Linear regression is the algorithm that, given a set of vectors ${\bf x}_i \in \mathbb{R}^p$, and a set of targets $y_i \in \mathbb{R}$, returns a vector ${\bf w} \in \mathbb{R}^p$ minimizing the summed prediction error $\sum_i ({\bf x}_i\cdot{\bf w} - y_i)^2$.
Ridge regression is the algorithm that instead returns a vector ${\bf w} \in \mathbb{R}^p$ minimizing the penalized error function $\lambda |{\bf w}|^2 + \sum_i ({\bf x}_i\cdot{\bf w} - y_i)^2$, where $\lambda \in \mathbb{R}$ is a parameter of the algorithm.
Intuitively, it seems that linear regression operates when ${\bf x}_i$ lives in a vector space, while ridge regression operates when ${\bf x}_i$ lives in a Euclidean space. More specifically, given an invertible matrix ${\bf M} \in \mathbb{R}^{p \times p}$, if we transform ${\bf x}_i \mapsto {\bf M x}_i$, then the linear regression solution transforms as ${\bf w} \mapsto {{\bf M}^{-1}}^T {\bf w}$. However for ridge regression, this only works if $\bf M$ is orthogonal.
How can one express this formally in the language of categories, functors, and natural transformations?
UPDATE: I think I have an answer. It is too long for this page, written up here.
