Orthogonal Projection onto $ {L}_{1} $ Ball with Box Constraints

Question

How could one derive the Orthogonal Projection Operator of the $ {L}_{1} $ Ball with Box Inequality constraints?

The convex optimization problem is formulated by:

\begin{align*} \arg \min_{x} \quad & \frac{1}{2} {\left\| x - u \right\|}_{2}^{2} \\ \text{subject to} \quad & {\left\| x \right\|}_{1} \leq \lambda \\ \quad & -1 \leq {x}_{i} \leq 1 \quad \forall i = 1, 2, \ldots, n \end{align*}

Answer 1

I don't think you can find it analytically, but we can construct an algorithm to compute it efficiently using duality.

Taking only the last constraint inside the Lagrangian: $$ L(x, \lambda) = \frac{1}{2}\|x - u\|_2^2 + \lambda(\|x\|_1 - 1) = \sum_{i=1}^n \left[ \frac{1}{2}(x_i-u_i)^2 + \lambda|x_i| \right] - \lambda, $$ with $\lambda \geq 0$. Remembering the additional constraints $-1 \leq x_i \leq 1$, which can also be written as $\|x\|_{\infty} \leq 1$, a dual objective is: $$ q(\lambda) = \inf_{\|x\|_{\infty} \leq 1} L(x, \lambda) = \sum_{i=1}^n \inf_{|x_i| \leq 1} \left\{ \frac{1}{2}(x_i-u_i)^2 + \lambda |x_i| \right\} - \lambda. $$

We concentrate on solving problems of the form $$\phi(u, \lambda) \equiv\inf_{|z|\leq 1} \left \{ \tfrac{1}{2}(z - u)^2 + \lambda|z| \right \},$$ and the dual objective is $$ q(\lambda)=\sum_{i=1}^n \phi(u_i;\lambda) - \lambda. $$

The problem of computing $\phi$ has a strongly convex objective over a convex set, and here is a unique minimizer. To solve it, we can split the problem into two problems - one for non-negative $z$ while the other is for non-positive $z$. The minimum is attained at the solution of the problem having a smaller optimum. We will use $[x]_{[a,b]}$ to denote $x$ clamped to the interval $[a,b]$.

1. The case of non-negative $z$ is $$ \min_{0 \leq z \leq 1} \left \{ \tfrac{1}{2}(z - u)^2 + \lambda z \right \}. $$ The minimizer above is $z^* = [u - \lambda]_{[0,1]}$, and the optimal value is $$ \tfrac{1}{2}([u - \lambda]_{[0,1]} - u)^2 + \lambda [u - \lambda]_{[0,1]} $$
2. The case of non-positive $z$ is $$ \min_{-1 \leq z \leq 0} \left \{ \tfrac{1}{2}(z - u)^2 - \lambda z \right \}. $$ The minimizer is $z^* = [u+\lambda]_{[-1,0]}$, and the minimum is $$ \tfrac{1}{2}([u+\lambda]_{[-1,0]} - u)^2 - \lambda [u+\lambda]_{[-1,0]} $$ The value of $\phi(u, \lambda)$ is the smaller among the two cases, and the minimizer is the minimizer of the chosen case.

With the above, you can easily write code which evaluates $q(\lambda)$ for any $\lambda$, and now you can employ any one-dimensional maximization algorithm, such as golden section search, to solve the dual problem $\max_{\lambda \geq 0} q(\lambda)$. Then, the optimal $x_i$'s are exactly the minimizers $z^*$ in the definition of $\phi(u_i, \lambda)$.

Answer 2

Reformulating the problem with arbitrary boundaries (Preserving $ \lambda $ for the Lagrange Multiplier):

$$\begin{align*} \arg \min_{x} \quad & \frac{1}{2} {\left\| x - y \right\|}_{2}^{2} \\ \text{subject to} \quad & {\left\| x \right\|}_{1} \leq r \\ & {l}_{i} \leq {x}_{i} \leq {u}_{i} \; \forall i = 1, 2, \ldots, n \end{align*}$$

The Lagrangian is given by (While the box constraints are implied):

$$ L \left( x, \lambda \right) = \frac{1}{2} {\left\| x - y \right\|}_{2}^{2} + \lambda \left( {\left\| x \right\|}_{1} - r \right) = \sum_{i = 1}^{n} \left[ \frac{1}{2} {\left( {x}_{i} - {y}_{i} \right)}^{2} + \lambda \left| {x}_{i} \right| \right] - \lambda r $$

Namely the problem can be formulated component wise.

The dual function is given by:

$$ q \left( \lambda \right) = \inf_{ {l}_{i} \leq {x}_{i} \leq {u}_{i} } L \left( x, \lambda \right) = \inf_{ {l}_{i} \leq {x}_{i} \leq {u}_{i} } \left\{ \sum_{i = 1}^{n} \left[ \frac{1}{2} {\left( {x}_{i} - {y}_{i} \right)}^{2} + \lambda \left| {x}_{i} \right| \right] - \lambda r \right\} $$

Now, let's think about the solution a little bit.
The solution $ {x}_{i}^{\ast} $ would like to have the same sign of $ {y}_{i} $ as if it would have different sign being $ 0 $ would clearly be a better solution (Lower objective value both for the squared term and for the absolute value term).

This above is the basic intuition for the trick done by @gerw.

Now, imagine we know the sign of each element of the solution - $ {s}_{i} = \operatorname{sign} \left( {x}_{i} \right) $.

Then the problem is given as:

$$\begin{align*} \arg \min_{x} \quad & \frac{1}{2} {\left\| x - y \right\|}_{2}^{2} \\ \text{subject to} \quad & {s}^{T} x \leq r \\ & {l}_{i} \leq {x}_{i} \leq {u}_{i} \; \forall i = 1, 2, \ldots, n \end{align*}$$

Then the Lagangian is given by:

$$ L \left( x, \lambda \right) = \frac{1}{2} {\left\| x - y \right\|}_{2}^{2} + \lambda \left( {s}^{T} x - r \right) = \sum_{i = 1}^{n} \left[ \frac{1}{2} {\left( {x}_{i} - {y}_{i} \right)}^{2} + \lambda {s}_{i} {x}_{i} \right] - \lambda r $$

Namely the problem can be formulated component wise.

The dual function is given by:

$$ q \left( \lambda \right) = \inf_{ {l}_{i} \leq {x}_{i} \leq {u}_{i} } L \left( x, \lambda \right) = \inf_{ {l}_{i} \leq {x}_{i} \leq {u}_{i} } \left\{ \sum_{i = 1}^{n} \left[ \frac{1}{2} {\left( {x}_{i} - {y}_{i} \right)}^{2} + \lambda {s}_{i} {x}_{i} \right] - \lambda r \right\} $$

It is easy to show that the solution $ {x}^{\ast} = \operatorname{Proj}_{\left[ l, u \right]} \left( y - \lambda s \right) $ obeys all the KKT conditions of the problem.

One of the KKT conditions is given by:

$$ \lambda \left( {s}^{T} x - r \right) = 0 \Rightarrow \lambda \left( {s}^{T} \operatorname{Proj}_{\left[ l, u \right]} \left( y - \lambda s \right) - r \right) = 0 $$

From the above it is easy to derive the optimal $ \lambda $ and then by it to extract $ {x}^{\ast} $.

The only question is how to have $ s $ without having $ {x}^{\ast} $ first?
Keep in mind the objective function $ f \left( \lambda \right) = \lambda \left( {s}^{T} \operatorname{Proj}_{\left[ l, u \right]} \left( y - \lambda s \right) - r \right) $ which we need to find the root of.

So we can think about 3 exclusive cases:

• $ \operatorname{sign} \left( {l}_{i} \right) = \operatorname{sign} \left( {u}_{i} \right) \Rightarrow {s}_{i} = \operatorname{sign} \left( {l}_{i} \right) = \operatorname{sign} \left( {u}_{i} \right) $.
  This also suggest that for next cases $ {u}_{i} \geq 0 $ and $ {l}_{i} \leq 0 $ otherwise we're back into this case.
• $ \operatorname{sign} \left( {y}_{i} \right) = \operatorname{sign} \left( {l}_{i} \right) $ (Implicitly $ {l}_{i} \leq 0 $ and $ {y}_{i} \leq 0 $) hence $ {s}_{i} = \operatorname{sign} \left( {y}_{i} \right) $ hence we must update $ {u}_{i} = 0 $ as in the objective function we can't allow $ {x}_{i}^{\ast} > 0 $ as it means its sign will reverse.
• $ \operatorname{sign} \left( {y}_{i} \right) = \operatorname{sign} \left( {u}_{i} \right) $ (Implicitly $ {u}_{i} \geq 0 $ and $ {y}_{i} \geq 0 $) hence $ {s}_{i} = \operatorname{sign} \left( {y}_{i} \right) $ hence we must update $ {l}_{i} = 0 $ as in the objective function we can't allow $ {x}_{i}^{\ast} < 0 $ as it means its sign will reverse.

So we have set of rules to set $ s $ and update the boundaries (Component wise as this function has component wise property) to validate the objective function.

I wrote a MATLAB code which implements the solution at my Mathematics StackExchange Question 2824418 - GitHub Repository.
The code validates the result to a reference calculated by CVX.

Answer 3

I think this projection can be solved quite efficiently. First, we observe that the projection $c$ of $u$ has coefficient-wise the same sign as $u$. Then, $c$ is already the solution of $$\text{Minimize}\quad \frac12 \|x - u\|_2^2 \\\text{s.t.}\quad a \le x \le b \\\text{and}\quad s^\top x \le \lambda,$$ where $s_i = \operatorname{sign}(u_i)$, $a = \min(s,0)$, $b = \max(s,0)$.

By the introduction of a multiplier $\mu \ge 0$ for the linear constraint, we find that $(x,\mu)$ is the unique solution of $$ x = Proj_{[a,b]}( u - \mu \, s) \\ 0 \ge s^\top x -\lambda \perp \mu \ge 0.$$ Now, it is easy to check that $$\mu \mapsto s^\top Proj_{[a,b]}(u - \mu s) - \lambda$$ is monotone, thus a root can be found by a bisection algorithm.

I think that the above idea can be used to obtain an algorithm running in $O(n \log n)$, if you first sort the entries of $|u|$.

Answer 4

This is extension of @Alex answer to arbitrary box constraints with a little easier method to find the solution.

Reformulating the problem with arbitrary boundaries (Preserving $ \lambda $ for the Lagrange Multiplier):

$$\begin{align*} \arg \min_{x} \quad & \frac{1}{2} {\left\| x - y \right\|}_{2}^{2} \\ \text{subject to} \quad & {\left\| x \right\|}_{1} \leq r \\ & {l}_{i} \leq {x}_{i} \leq {u}_{i} \; \forall i = 1, 2, \ldots, n \end{align*}$$

The Lagrangian is given by (While the box constraints are implied):

$$ L \left( x, \lambda \right) = \frac{1}{2} {\left\| x - y \right\|}_{2}^{2} + \lambda \left( {\left\| x \right\|}_{1} - r \right) = \sum_{i = 1}^{n} \left[ \frac{1}{2} {\left( {x}_{i} - {y}_{i} \right)}^{2} + \lambda \left| {x}_{i} \right| \right] - \lambda r $$

Namely the problem can be formulated component wise.

The dual function is given by:

$$ q \left( \lambda \right) = \inf_{ {l}_{i} \leq {x}_{i} \leq {u}_{i} } L \left( x, \lambda \right) = \inf_{ {l}_{i} \leq {x}_{i} \leq {u}_{i} } \left\{ \sum_{i = 1}^{n} \left[ \frac{1}{2} {\left( {x}_{i} - {y}_{i} \right)}^{2} + \lambda \left| {x}_{i} \right| \right] - \lambda r \right\} $$

Now, taking advantage of the component wise form one could solve $ n $ scalar problem of the form:

$$ {x}^{\ast}_{i} = \arg \min_{ {l}_{i} \leq {x}_{i} \leq {u}_{i} } \frac{1}{2} {\left( {x}_{i} - {y}_{i} \right)}^{2} + \lambda \left| {x}_{i} \right| $$

The key to easily solve this problem is knowing the sign of optimal solution $ {x}^{\ast}_{i} $.
As in general the solution is given by (Just a derivative of the above):

$$ {x}^{\ast}_{i} = \operatorname{Proj}_{\left[ {l}^{\ast}_{i}, {u}^{\ast}_{i} \right]} \left( {y}_{i} - \lambda \operatorname{sign} \left( {x}^{\ast}_{i} \right) \right) $$

Pay attention for $ {l}^{\ast}_{i} $ and $ {u}^{\ast}_{i} $. We can't use $ l $ and $ u $ as the effective boundaries changes according to $ \operatorname{sign} \left( {x}^{\ast}_{i} \right) $.

In general, if the boundaries won't interfere with the sign of the optimal solution, the optimal solution will have the same sign as $ {y}_{i} $. It is easy to validate this as if the sign is different having $ {x}^{\ast}_{i} = 0 $ will have better objective value as it has lower value for the squared term and zero out the absolute value term. Yet there are cases the boundaries enforces different sign.

The intuition above yields 3 exclusive cases here:

• $ \operatorname{sign} \left( {l}_{i} \right) = \operatorname{sign} \left( {u}_{i} \right) \Rightarrow \operatorname{sign} \left( {x}_{i} \right) = \operatorname{sign} \left( {l}_{i} \right) = \operatorname{sign} \left( {u}_{i} \right) $ then $ {l}^{\ast}_{i} = {l}_{i} $ and $ {u}^{\ast}_{i} = {u}_{i} $. This also suggest that for next cases $ {u}_{i} \geq 0 $ and $ {l}_{i} \leq 0 $ otherwise we're back into this case.
• $ \operatorname{sign} \left( {y}_{i} \right) = \operatorname{sign} \left( {l}_{i} \right) $ (Implicitly $ {l}_{i} \leq 0 $ and $ {y}_{i} \leq 0 $) hence $ \operatorname{sign} \left( {x}_{i} \right) = \operatorname{sign} \left( {y}_{i} \right) $ hence we must update $ {u}^{\ast}_{i} = 0 $ (While $ {l}^{\ast}_{i} = {l}_{i} $) as in the objective function we can't allow $ {x}_{i}^{\ast} > 0 $ as it means its sign will reverse (Due to the value of $ \lambda $).
• $ \operatorname{sign} \left( {y}_{i} \right) = \operatorname{sign} \left( {u}_{i} \right) $ (Implicitly $ {u}_{i} \geq 0 $ and $ {y}_{i} \geq 0 $) hence $ \operatorname{sign} \left( {x}_{i} \right) = \operatorname{sign} \left( {y}_{i} \right) $ hence we must update $ {l}^{\ast}_{i} = 0 $ (While $ {u}^{\ast}_{i} = {u}_{i} $) as in the objective function we can't allow $ {x}_{i}^{\ast} < 0 $ as it means its sign will reverse (Due to the value of $ \lambda $).

So now we have the optimal solution for each case:

• $ \operatorname{sign} \left( {l}_{i} \right) = \operatorname{sign} \left( {u}_{i} \right) \Rightarrow {x}^{\ast}_{i} = \operatorname{Proj}_{\left[ {l}_{i}, {u}_{i} \right]} \left( {y}_{i} - \lambda \operatorname{sign} \left( {l}_{i} \right) \right) = \operatorname{Proj}_{\left[ {l}_{i}, {u}_{i} \right]} \left( {y}_{i} - \lambda \operatorname{sign} \left( {u}_{i} \right) \right) $.
• $ \operatorname{sign} \left( {y}_{i} \right) = \operatorname{sign} \left( {l}_{i} \right) \Rightarrow {x}^{\ast}_{i} = \operatorname{Proj}_{\left[ {l}_{i}, 0 \right]} \left( {y}_{i} - \lambda \operatorname{sign} \left( {y}_{i} \right) \right) $.
• $ \operatorname{sign} \left( {y}_{i} \right) = \operatorname{sign} \left( {u}_{i} \right) \Rightarrow {x}^{\ast}_{i} = \operatorname{Proj}_{\left[ 0, {u}_{i} \right]} \left( {y}_{i} - \lambda \operatorname{sign} \left( {y}_{i} \right) \right) $.

So now, for each value of $ \lambda $ we have a way to evaluate $ q \left( \lambda \right) $ efficiently by plugging in the optimal value of each $ {x}^{\ast}_{i} $.
Since the problem is convex the Dual Function is guaranteed to be concave hence its maximum can easily found by any one variable function maximization method.

Once we find the optimal value of $ \lambda $ it is easy to derive the optimal value of $ x $ from the above.

I wrote a MATLAB code which implements the solution at my Mathematics StackExchange Question 2824418 - GitHub Repository (See ProjectL1BallDual.m).
The code validates the result to a reference calculated by CVX.