1

I am trying to find the dual function $g(\lambda, \nu)$ to this problem

$$\min\limits_{Ax = b} \|x\|$$

Step 1. Form the Lagrangian

$$L(x, \lambda, \nu) = \|x\| + \nu^T(Ax-b) = \|x\| + \nu^TAx - \nu^Tb$$

Step 2. Take the inf over all $x$ to get $g(\lambda, \nu)$

$g(\lambda, \nu)$ = $\inf\limits_x (\|x\| + \nu^TAx - \nu^Tb)$

Then by property of $\inf$, we have:

$\inf\limits_x (\|x\| + \nu^TAx - \nu^Tb) = \inf\limits_x \|x\| + \inf\limits_x \nu^TAx - \inf\limits_x \nu^Tb$

Notice:

  • $- \inf\limits_x \nu^Tb = -\nu^Tb$, not much to do here

  • $ \inf\limits_x \nu^TAx = \inf\limits_x (A^T\nu)^Tx$ looks like a dual norm i.e. $\|A^T\nu\|_{*} = \inf\limits_x\{ (A^T\nu)^Tx | \|x\|\leq 1\}$, but here we have a constraint $\|x\|\leq 1$

Then we have

$g(\lambda, \nu)$ = $\inf\limits_x (\|x\| + \nu^TAx - \nu^Tb) = \inf\limits_x \|x\| + \|A^T\nu\|_{*} - \nu^Tb$, $\|x\| \leq 1$

Does anyone know how to deal with the $\|x\| \leq 1$ constraint and proceed from above?

Fraïssé
  • 11,656

1 Answers1

4

Your're almost there...

Recall the definition of convex conjugates. Recall that the convex conjugate of a norm $\|.\|$ is the indicator function of the unit ball of the dual norm $\|.\|_*$. Now,

\begin{equation} \begin{split} g(\nu) &:= \underset{x}{\inf }L(x,\nu) = \underset{x}{\inf }\|x\| + \nu^T Ax - \nu^T b = \nu^Tb -\underset{x}{\sup }x^T(-A^T\nu) - \|x\|\\ &= \nu^Tb -\|.\|^*(-A^T\nu) = \begin{cases}\nu^Tb, &\mbox{ if }\|A^T\nu\|_* \le 1,\\-\infty, &\mbox{ else.}\end{cases} \end{split} \end{equation}

Thus the dual problem is: \begin{equation} \text{Maximize } \nu^Tb\text{ subject to }\|A^T\nu\|_* \le 1. \end{equation}

Example: For example, if $\|.\|$ is the $\ell_1$-norm, then your original problem is the well-known Basis Pursuit problem, and the dual we've obtained is a linear program (you're maximizing a linear function on a polytope).

Notes: Using the Fenchel-Rockafellar duality Theorem, you can obtain the sought-for dual formulation in exactly one line!

dohmatob
  • 9,753
  • Hi thank you for the answer! I have two questions, 1. Is there a simple way to show that the convex conjugate of $| \cdot |$ is the indicator function on the unit ball of dual norm $| \cdot |{}$. I don't think I have encountered this claim in my study. How did you go from $\nu^Tb - \sup\limits_x (x^T(-A^T\nu) - |x|)$ to $v^Tb - |\cdot|^(-A^T\nu)$? Specifically what is a $|\cdot|^*$? Lastly, how did the $|A^Tv|{*} \leq 1$ come about? --- (I don't think I know much about convex conjugates but I will look into it) – Fraïssé Nov 05 '15 at 15:54
  • $f^$ denotes the convex conjugate of $f$. Thus $|.|^$ denotes the convex conjugate of the norm $|.|$. About convex conjugates of norms, see my answer to this question http://math.stackexchange.com/a/1477721/168758. To get a big picture of all this, please read on convex conjugates https://en.wikipedia.org/wiki/Convex_conjugate. I think you have the necessary background to understand these things. – dohmatob Nov 05 '15 at 16:49
  • Hey Dohmatob I am revisiting this problem and I am convinced that there must be a easier method. Pg 5 https://web.stanford.edu/class/ee364a/lectures/duality.pdf shows an elegant relationship between the primal and the dual problem via the dual norm. Do you know if there is a simpler way to see why given a primal function under the primal norm, the dual problem is constrained by the dual norm? – Fraïssé Dec 10 '15 at 23:28
  • This is because the dual norm is defined as the support function of the unit ball of the primal norm. Any other 'reason' will invariable be a restatement of the this fact, using a possibly different wording :) – dohmatob Dec 11 '15 at 07:39
  • @dohmatob, Great Answer. +1. It would be great if you showed the explicit solution for $ {L}_{2} $ Norm. – Royi Apr 02 '18 at 14:18