Projection onto positive semidefinite (PSD) matrices with bounded rank

Question

Let $\bf A$ be a symmetric matrix and let $\Omega_k$ denote the set of symmetric positive semidefinite matrices (SPSD) with rank at most $k$. Consider the following optimization problem

$$ \arg \min_{{\bf X} \in \Omega_k} \frac{1}{2} \left\| {\bf X} − {\bf A} \right\|_{\text{F}}^{2} $$

where $ {\left\| \cdot \right\|}_{\text{F}} $ is the Frobenius norm.

I know that the minimizer can be computed using the spectral decomposition of $\bf A$. What I am not sure about is how to go about the proof. It would be great if someone could give a sketch of the proof (or suggest a reference).

Answer 1

I tried my best to be very explicit with the details here. Please let me know if any place needs more exposition.

---

Simple case: we're lucky and $A$ and $X$ commute.

Assume that $X$ commutes with $A$. Then, they are simultaneously diagonalizable as $X=UL_XU^T$ and $A=UL_AU^T$ for an orthogonal matrix $U$ and diagonal matrices $L_X$ and $L_A$ of the eigenvalues. The minimization problem in this case is: $$\min\limits_{X\in\Omega}\frac{1}{2}\|UL_XU^T-UL_AU^T\|_F^2$$ but, since $U$ is an orthogonal matrix and the Frobenius norm is invariant under orthogonal transforms, we can also write $$\|UL_XU^T-UL_AU^T\|_F^2 = \|L_XU^T-L_AU^T\|_F^2 = \|L_X-L_A\|_F^2$$ so that our problem is equivalent to $$\min\limits_{X\in\Omega}\frac{1}{2}\|L_X-L_A\|_F^2.$$ Taking into account that $L_X$ and $L_A$ are $n\times n$ diagonal matrices (and thus identified with vectors in $\mathbb{R}^n$), the constraint $X\in\Omega$ is equivalent to $L_X\in\mathbb{R}^n_+$ (this is the psd part of the constraint) with at most $k$ nonzero components (this is the rank-$k$ part of the constraint). But, this new optimization problem is actually easily solved: we just project the vector $L_A$ onto $\mathbb{R}^n_+$ and keep only the fist $k$ nonzero components (we assume the components are sorted according to magnitude, otherwise instead of keeping the first $k$ components we would keep the $k$ most positive (largest) components).

To summarize, letting $L_A = (\sigma_1, \sigma_2, \ldots, \sigma_n)$, we take $L_X = (R(\sigma_1),R(\sigma_2),\ldots,R(\sigma_k),0,\ldots,0)$ where $R(x)=\max(0,x)$ is the ReLU function and our rank-$k$ psd solution is then $UL_XU^T$.

---

General case: what if we're not lucky?

The question remains: if $X$ does not commute with $A$ what can we do? It turns out that considering the case where they commute is sufficient, due to a trick/inequality related to the Frobenius norm.

von-Neumann's trace inequality says, for $A$ real and symmetric and $X$ real psd with spectral decompositions $A=VL_AV^T$ and $X=UL_XU^T$ for $U$ and $V$ orthogonal matrices and $L_A$ and $L_X$ diagonal matrices as before,

$$\|X-A\|_F^2 \geq \|L_A-L_X\|_F^2$$

Now, assume that there is a minimizer $Y\in\Omega$ of our original problem which does not commute with $A$. The spectral decomposiiton of $Y$ will be written $WL_YW^T$ for some orthogonal matrix $W$. Then we have $$\|Y-A\|_F^2 \geq \|L_Y-L_A\|_F^2 = \|VL_YV^T - A\|_F^2$$ where the first inequality is due to the von-Neumann trace inequality and the second equality is the invariance of the Frobenius norm with respect to orthogonal transformations. The matrix $Z=VL_YV^T$ is in $\Omega$ since $Y\in\Omega$.

Thus, given some minimizer $Y$ which is not commuting with $A$, we can always construct the matrix $Z\in\Omega$ which does commute with $A$ and which is at least as good in terms of the objective function:

$$\|Y-A\|_F^2\geq \|Z-A\|_F^2$$

Answer 2

I would use the Projected Sub Gradient Method for this problems.
Yet, the set of Semidefinite Matrices with Bounded Rank isn't Convex.
Hence the method can't be guaranteed to converge to the best solution.

So defining $ \mathcal{ {\Omega} }_{k} = \left\{ X \in \mathbb{R}^{n \times n} \mid X \in \mathcal{S}_{+}^{n}, \, \operatorname{rank} \left( X \right) \leq k \right\} $ as the set of matrices with rank up to $ k $ and Symmetric Positive Semi Definite.

One can do the following steps to project into the set:

1. Project the Matrix into the Set of Symmetric Matrices.
2. Calculate the Matrix Eigen Values by the Eigen Decomposition of a Matrix.
3. Threshold the Eigen Values such that any non positive value is zeroed.
4. Keep the $ k $ largest eigen values.
5. Recompose the matrix by the eigen values and eigen vectors.

Regarding deeper analysis I found:

1. Semidefinite Projections, Regularization Algorithms and Polynomial Optimization.
2. Sublinear Time Low Rank Approximation of PSD Matrices.
3. Low Rank PSD Approximation in Input Sparsity Time.
4. Positive Semidefinite Rank and Nested Spectrahedra.
5. Positive Semidefinite Rank.
6. Rank Restricted Semidefinite Matrices and Image Closedness.