0

How to solve such optimization problem \begin{align} \min_{x \in \mathbb{R}^n} \ f(x) + \sum_{i=1}^m g_i(x) \, \end{align} where $f(x)$ and $g_i(x)$ are closed proper convex functions. We can also assume that $f(x)$ and $g_i(x)$ are "proximable" functions [1][2].

please note that it is a large-scale optimization problem. So, what algorithms would you suggest to solve such a problem (besides consensus ADMM)?

Thank you so much in advance.

user550103
  • 2,773
  • could you tell more about $f$ and $g_i$ and the scale of the problem? – LinAlg Dec 28 '18 at 17:26
  • @LinAlg investigating various applications of such generic forms (within electrical engineering domain). So, for simplicity, we can assume that $f$ is an indicator function and the set can be a $\ell$2 norm ball, and $g_i$ can be quadratic forms (with positive semidefinite matrices). For instance, $m = 100$ and $n = 8096$ – user550103 Dec 29 '18 at 07:37

2 Answers2

0

Your problem is optimizing $$\min_x \left\{ \sum_{i=1}^m g_i(x) : ||Ax-b||_2^2 \leq 1 \right\}.$$ Since all $g_i$ are quadratic forms, their sum is also a quadratic form. You therefore optimize one quadratic form subject to one quadratic constraint.

The scale of your problem ($n=8096$) is small. The Hessian only takes 512 MB of memory. Newton-based methods (such as sequential quadratic programming or interior point methods) will have no problem with this formulation, and are probably much faster than ADMM. Give CPLEX or Gurobi a try.

LinAlg
  • 20,093
  • thank you. Actually I need to run such optimization problems simultaneously/parallel for 1024. So, it becomes huge problem per se (also building a test bed). I need to be really memory efficient as well. I agree that second-order methods would be much faster than first-order methods. – user550103 Dec 29 '18 at 15:20
  • Furthermore, I am trying to understand the various possibilities to solve such generic problems (besides consensus ADMM since I have investigated that algorithm already). – user550103 Dec 29 '18 at 15:24
  • @user550103 You should check L-BFGS for memory efficiency. Computing derivatives is just matrix multiplication here. I'd love to see a comparison with ADMM. – LinAlg Dec 29 '18 at 15:35
  • Thanks, I will check out LBFGS method (hoping such method can be deployed on the test beds)... – user550103 Dec 29 '18 at 15:42
0

Look at Quadratic form of the Hybrid Steepest Descent (See Quadratic Optimization of Fixed Points of Non Expensive Mappings in Hilbert Space).

There are some methods there with little memory requirements.

I used them in my answer to Orthogonal Projection onto the Intersection of Convex Sets as an alternative to ADMM Consensus Trick.

Royi
  • 10,050