Recover primal solution from dual for matrix completion

Question

Consider the following primal/dual SDPs $$ \min\limits_X \; \lVert X \rVert_* : \mathcal{A}(X) = b \qquad \max\limits_z \; b^T z : \lVert \mathcal{A}^*(z) \rVert \leq 1 $$ where $\lVert X \rVert_* = \sum_{i=1}^{\text{rank}(X)} \sigma_i(X)$ is the nuclear norm (sum of singular values), $\lVert X \rVert = \sigma_1(X)$ is the operator norm, and $\mathcal{A} : \mathbb{R}^{m, n} \rightarrow \mathbb{R}^p$ is a linear operator. Note this problem setup comes from (2.7) of http://www.eecs.berkeley.edu/~brecht/papers/07.rfp.lowrank.pdf.

I want to make this very concrete for the problem of matrix completion. That is, we are given $\alpha \in \{1, ..., m\}^{p}$, $\beta \in \{1, ..., n\}^{p}$, and we set $\mathcal{A}(X)_k = X_{\alpha_k, \beta_k}$ for $k=1,...,p$. To compute the adjoint $\mathcal{A}^*(z)$, let $X \in \mathbb{R}^{m,n}$ and $z \in \mathbb{R}^{p}$. Then $$\langle \mathcal{A}(X), z \rangle = \sum_{k=1}^{p} X_{\alpha_k, \beta_k} z_k = \langle X, \mathcal{A}^*(z) \rangle = \sum_{i=1}^{m} \sum_{j=1}^{n} X_{i,j} (\mathcal{A}^*(z))_{i,j}$$ Therefore, to achieve equality it suffices to set $(\mathcal{A}^*(z))_{\alpha_k, \beta_k} = z_k$ for $k = 1,...,p$ and $0$ otherwise (and since adjoints are unique we know it has to be this).

Now for the question: how do I recover the optimal value $X_{opt}$ of the primal, from a solution $z_{opt}$ of the dual? For the purposes of this question let's assume that the solution to both problems is unique and the optimal value is finite (call it $opt$).

What I have done is this so far. I'm just going to assume strong duality holds everywhere (the Slater conditions are easy enough to check) and derive the dual to see what the relationship between the primal and dual variables is $$ \begin{align*} \min_{X : \mathcal{A}(X) = b} \lVert X \rVert_* &= \min_{X: \mathcal{A}(X) = b} \max_{Y:\lVert Y \rVert \leq 1} \langle X, Y \rangle = \max_{Y:\lVert Y \rVert \leq 1}\min_{X: \mathcal{A}(X) = b}\langle X, Y \rangle \end{align*} $$ Now to solve the inner minimization, I appealed to duality again, writing the Lagrangian as $$ \begin{align*} \mathcal{L}(X, \nu) = \langle X, Y \rangle + \langle \nu, (\mathcal{A}(X) - b) \rangle &= \langle X, Y \rangle + \langle \nu, \mathcal{A}(X) \rangle - \langle \nu , b \rangle \\ &= \langle X, Y \rangle + \langle X, \mathcal{A}^*(\nu) \rangle - \langle \nu , b \rangle \end{align*} $$ So solving for $\min\limits_X \mathcal{L}(X, \nu)$ $$ \min\limits_X \mathcal{L}(X, \nu) = \begin{cases} - \langle \nu, b \rangle &\text{if } Y + \mathcal{A}^*(\nu) = 0 \\ -\infty &\text{o.w.} \end{cases} $$ So plugging the dual of the inner minimization in $$ \max_{Y:\lVert Y \rVert \leq 1}\min_{X: \mathcal{A}(X) = b}\langle X, Y \rangle = \max_{Y:\lVert Y \rVert \leq 1}\max_{\nu:Y = -\mathcal{A}^*(\nu)} -\langle \nu, b \rangle $$ which yields the dual form given in the paper.

Now from this we see, solving the dual allows us to recover the $Y_{opt} = \mathcal{A}^*(z_{opt})$ which obtains the maximum. But I'm not sure how to recover $X_{opt}$ from $Y_{opt}$. We know that (a) $\langle X_{opt}, Y_{opt} \rangle = opt$ and (b) $\mathcal{A}(X_{opt}) = b$. But because of the way $\mathcal{A}^*(z)$ works, $\langle X_{opt}, \mathcal{A}^*(z_{opt}) \rangle = opt$ is an equation which only involves the $p$ variables of $X_{opt}$ which we already know about; the $mn - p$ unknowns drop out. Where did I go wrong?

Answer 1

Many iterative methods for solving the $z$ problem will actually recover $X$ as a natural consequence. For instance, if you were to convert these problems to semidefinite programs and solve them using a symmetric primal-dual method, then both problems are solved simultaneously, and the primal and dual solutions will typically converge to optimality at the same rate. A first-order method with Nesterov-style smoothing will produce $X$ as well, as a Lagrange multipler. However, in my experience, with first order methods the primal variable ($z$ in this case) will converge more quickly than the dual ($X$).

So in practice, you will not need to separately recover $X_\text{opt}$ from $z_\text{opt}$; it will just be there. But if a magic oracle dropped $z_\text{opt}$ into your lap, you can indeed recover $X_\text{opt}$ with an SVD and a linear system solve. The derivation below might be useful to help you "scrub" a nearly-optimal $z$ and $X$ obtained from one of these algorithms. I will explain how at the end.

To see why we can do this, note that Cauchy-Schwarz tells us that for any $X$, $Y$, $$\langle X, Y \rangle \leq \| X \|_* \| Y \|$$ When is equality satisfied? That is, when does $$\langle X, Y\rangle = \|X\|_* \|Y\| = \sigma_1(Y) \sum_i \sigma_i(X) \quad ?$$ My claim: this occurs when $X$ and $Y$ share singular directions, and the rank of $X$ is no greater than the multiplicity of the first singular vector of $Y$. That is, $X=U\Sigma_X V^H$ and $Y=U\Sigma_Y V^H$ for identical $U$ and $V$, and $$\sigma_i(X) = 0 \quad \forall i ~~\text{s.t.}~~ \sigma_i(Y) < \sigma_1(Y).$$ If you don't believe me, take some time to work it out. It may help your intuition to think of this as the matrix version of the vector inequality $x^Ty\leq \|x\|_1\|y\|_\infty$.

Now let's use this to solve the problem. First, some facts about our models:

• The dual problem is strictly feasible (e.g., $z=0$).
• If there is at least one solution to $\mathcal{A}(X)=b$, then the primal problem is strictly feasible as well, as its constraints are linear and the objective function does not have a restricted domain.
• Thus strong duality is achieved in any case; and if $\mathcal{A}(X)=b$ has a solution, we also have both primal and dual attainment: there exists at least one optimal primal/dual pair $(X_\text{opt},z_\text{opt})$ satisfying $\|X_\text{opt}\|_*=b^Tz_\text{opt}$.
• We can guarantee that $\|\mathcal{A}^*(z_\text{opt})\|=1$. If this were not the case---that is, if $\|\mathcal{A}^*(z_\text{opt})\|=\alpha<1$---then we could scale $z_\text{opt}$ by $\alpha^{-1}$, and it would remain feasible and achieve a larger objective value; a contradiction.

So, given an optimal dual value $z_\text{opt}$, let $\gamma=b^Tz_\text{opt}$ and $Y\triangleq\mathcal{A}^*(z_\text{opt})$. Let $Y=U\Sigma V$ be the SVD of $Y$, and $q$ be the multiplicity of $\sigma_1(Y)$; that is, if $q=\max\{i\,|\,\sigma_i(Y)=1\}$. As you rightly point out, we have $$\langle X_\text{opt},Y\rangle = \gamma = b^Tz_\text{opt} = \|X_\text{opt}\|_* = \|X_\text{opt}\|_*\|Y\|$$ In other words, $X_\text{opt}$ and $Y$ satisfy the Cauchy Schwarz inequality with equality. Therefore, $$X_\text{opt} = \sum_{i=1}^q \sigma_i u_i v_i^H$$ for some $\sigma_i$, $i=1,2,\dots q$. Substitution into the primal equality constraints gives us $$\sum_{i=1}^q \sigma_i \mathcal{A}(u_iv_i^H) = b, \quad \sum_{i=1}^q \sigma_i = \gamma$$ which is a system of $p+1$ equations and $q$ unknowns. Any solution of this linear system will give us the singular values $\sigma_i$, from which we can construct $X_\text{opt}$. A solution must exist, otherwise it would contradict the claim that $X_\text{opt}$ is optimal.

As I said, you can use this to "scrub" a slightly inaccurate solution $X$ obtained from a numerical method. Of course, you could just look at the singular values of $X$ and drop the ones that are "very" small. But how do you know how small is small? A better solution is to look at $Y$, and determine how many of its singular values are close to $1$---a more absolute measure. This numerical measure of multiplicity will tell you the likely rank of the true solution $X_\text{opt}$.