Solve Linear Least Squares with $ {L}_{1} $ Norm Regularization with Linear Equality and Non Negativity Constraints

Question

I'm searching for a way to solve the following problem in MATLAB.

$$\begin{array}{ll} \text{minimize} & \|Ax-b\|_2 + \lambda \|Cx\|_1\\ \text{subject to} & D x = 0\\ & -x \leq 0\end{array}$$

In MATLAB, lsqlin doesn't take the $\ell_1$ regularization into account, and l1_ls is only considering the constraint $x \geq 0$. Is there a way to solve the problem in MATLAB?

You can solve this problem using primal-dual methods, i.e. chambolle-pock. — Jürgen Sukumaran, Nov 03 '20 at 10:33
I think that the verb "minimize" is lacking; besides, can you give some details about your matrices $A,C,D$ ? Sizes, being sparse or not... — Jean Marie, Nov 03 '20 at 10:49
@TSF Thanks for the very interesting reference to Chambolle-Pock (https://hal.archives-ouvertes.fr/hal-01346507/document) — Jean Marie, Nov 03 '20 at 10:52
I have taken the liberty to modify a little your problem setting. Do you agree ? — Jean Marie, Nov 03 '20 at 11:02
... in particular placing the $2$ and the $1$ as indices instead of exponents. But maybe, one has to add an exponent $2$ to $AX+B$ ? Please answer. — Jean Marie, Nov 03 '20 at 11:05
Why not considering the minimization of $|Ax-b|_2 + \lambda |Cx|_1+\mu |D x|_2$ ? — Jean Marie, Nov 03 '20 at 13:02
@Jean Marie, I think, it is somewhat different. But, similary, I tried to solve it using Lagrangian and dual function, but it is so complicated because I am not goot at optimization. — geun-young lee, Nov 04 '20 at 02:37
@geun-younglee The beauty of CVX is that you don't need to be good at optimization to be a healthy consumer of numerical optimization algorithms. — Rodrigo de Azevedo, Nov 04 '20 at 09:16

Royi · Answer 1 · 2023-11-24T21:27:04.497

The problem is given by:

$$\begin{aligned} \arg \min_{x} \quad & \frac{1}{2} {\left\| A x - b \right\|}_{2}^{2} + \lambda {\left\| C x \right\|}_{1} \\ \text{subject to} \quad & D x = 0 \\ & x \geq 0 \end{aligned}$$

Probably the fastest solver to this kind of a problem is by some adaptation of the ADMM algorithm.

Yet, a simple solver can be done by the Projected Sub Gradient Method.
In this case we need a projection onto the convex set $ \mathcal{D} = \left\{ x \mid D x = 0, x \geq 0 \right\} $. Since the set $ \left\{ x \mid x \geq 0 \right\} $ isn't a sub space we can't use the alternating projection method (See Projections onto Convex Sets). So we'll use the Dykstra's Projection Algorithm from Orthogonal Projection onto the Intersection of Convex Sets.

The projection onto each set are given by (You may have a look at Projection of $ z $ onto the Affine Half Space $ \left\{ x \mid A x = b, \; x \geq 0 \right\} $):

$ \operatorname{Proj}_{ \left\{ x \mid D x = 0 \right\} } \left( y \right) = y - {D}^{T} {\left( D {D}^{T} \right)}^{-1} \left( D y \right) $.
$ \operatorname{Proj}_{ \left\{ x \mid x \geq 0 \right\} } \left( y \right) = \max \left( y, 0 \right) $.

The Sub Gradient of the objective function is given by $ {A}^{T} \left( A x - b \right) + \lambda {C}^{T} \operatorname{sign} \left( D x \right) $.

Throw all of this into a simple iterative Sub Gradient and you have a solver.
The MATLAB code is available at my StackExchange Mathematics Q3892375 GitHub Repository.

P. S.

By generating all matrices randomly the solution is usually given by the zero vector.

Solve Linear Least Squares with $ {L}_{1} $ Norm Regularization with Linear Equality and Non Negativity Constraints

1 Answers1