Given $A, B \in \mathbb{R}^{k \times d}$ with $d < k$, any solution $X \in \mathbb{R}^{k \times k}$ minimizing $$ \lVert X A - B \rVert_F^2$$ is of the form $$ X = B A^+ + (I - A^+ A) W $$ for some $W \in \mathbb{R}^{k \times k}$. Setting $W = 0$ gives the minimum norm solution.
Now, consider the same minimization problem with the constraint that $X$ is orthogonal (i.e. $X^T X = I$). Letting $\kappa: \mathbb{R}^{k \times k} \rightarrow \textrm{O}(N)$ denote the Procrustes projection (e.g. For $M$ with SVD $M = U \Sigma V^T,$ $\kappa(M) = U V^T$), the minimial solution is given by $$ X = \kappa(B A^T)$$ However, given that $d < k$, this solution is not unique because (I think, roughly; see this answer ) that $d - k$ orthogonal columns of $U$ and/or $V$ can be chosen arbitrarily. Thus, it seems there is really a space of solutions to the Procrustes problem when $d < k$.
My question is how to succinctly express this space of orthogonal solutions in terms of $A$ and $B$ (or their SVDs $A = U_A \Sigma_A V_A^T, B = U_B \Sigma_B V_B^T$, $U_A, U_B \in \mathbb{R}^{k \times d}$, $\Sigma_A, \Sigma_B, V_A, V_B \in \mathbb{R}^{d \times d}$, etc.) in a manner similar to the least squares problem above?
EDIT: I realize now the question basically boils down to the following (which is obvious in hindsight). Given a $k \times k$ real matrix $A$ with $\textrm{rank}(A) = d < k$, what is the space of all pairs of $k \times k$ orthogonal matrices $U, V$ such that $A$ can be expressed as $A = U \Sigma V^T$ where $\Sigma$ is the diagonal matrix of the singular values of $A$?
