2

UPDATED: I was forgetting this basic fact (that the sum of convex functions are themselves convex), and therefore I thought that I needed a global optimizer. I now see that that is not the case and my problem does indeed reduce to the classical lasso problem.

Original:

I have an objective function that looks like the follow:

$L(x) = ||Ax-y||^2_{L2}+ ||x-x_{ref}||_{L2}^2 + ||x||^2_{L1}$

where $A$ is a matrix, $y,x_{ref}$ are vectors. I ideally want to find the global minimum for this function, but I am unsure what methods are suitable for this kind of operation. Having searched this site and done some reading I am aware of ISTA methods and in particular FISTA.

As far as I can see FISTA or similar algorithms will not really be applicable in my case since they assume a smooth convex function in combination with a nonsmooth convex function, which is not exactly what I have since I have an extra term convex term. Besides, ISTA methods are local optimization algorithms and I believe I need a global optimization algorithm (though perhaps I could do it with an ensemble of local optimizations).

I am unsure what algorithms are suitable to solving such optimization, especially the $L_1$-norm is making me uncertain since I know quite a few algorithms are assuming differentiability. On the other hand I know there are also global optimization algorithms like evolution based ones that have no such assumptions, but I believe those shouldn't be necessary since I still have a relatively well behaving function.

If size is a consideration then the problem I am trying to solve should be relatively small in size:

$n \sim 200$

$m \sim 100$

$A$ - shape [n,m]

$x,x_{ref}$ - shape [m]

$y$ - shape [n]

Edit: updated the formula to reflect the intended $L2$ norm

tbolind
  • 45
  • 1
    MInimization of L1 norm is easy to express as a linear problem (LP). I'm not sure what you mean by $(Ax-y)^2$, ie. squaring a vector, but assuming you mean some norms like L2 then your problem is convex and it is a quadratic problem (QP) or second-order cone problem (SOCP). There is a rich universe of solvers and modelling tools for these problem types. – Michal Adamaszek Jul 25 '23 at 11:53
  • 1
    Here is an example in this direction https://www.cvxpy.org/examples/machine_learning/lasso_regression.html – Michal Adamaszek Jul 25 '23 at 11:54
  • @MichalAdamaszek I do indeed mean $L2$ norm for the other two terms in my code. So all three terms are by themselves convex, but the combination of them are not. So I doubt I can use standard convex solvers for such problems? – tbolind Jul 25 '23 at 12:57
  • 1
    "So all three terms are by themselves convex, but the combination of them are not" Erm... How is that possible? I was taught that the sum of convex functions is still convex. Also, a strictly convex function that tends to $+\infty$ at infinity has a unique local minimum, which is also global. So I see no reason not to invoke some standard convex minimization technique at least for the start. – fedja Jul 25 '23 at 23:07
  • @fedja you are quite right. I was forgetting this basic fact (that the sum of convex functions are themselves convex), and therefore I thought that I needed a global optimizer. I now see that that is not the case and my problem does indeed reduce to the classical lasso problem. Thank you both for you help! – tbolind Jul 26 '23 at 06:18

0 Answers0