0

I trying to minimize this objective function.

$$J(x) = \frac{1}{2}x^THx + c^Tx$$

First I thought I could use Newtown's Method, but later I found Gradient Descent, which is more suitable for this type of problem.

I first turn $J(x)$ into the deritivate.

$$\Delta J(x) = x^TH + c^T$$

And I want to find $x$ when $\Delta J(x)$ is small as possible. I could use gradient descent for that.

$$x_{n+1} = x_n - \gamma \Delta J(x_n)$$

Where $\gamma$ is a small number (tuning factor in an applied engineering world). Also $x_n$ is known.

But how about constraints? Can I juse an if-statement on $x_{n+1}$ to check if $x$ is reach its limits?

Example. Assume that $x \in \Re^2$. Then I can take two if statements.

If x(0) < 0, then x(0) = 0. If x(1) < 0, then x(0) = 0.

If x(0) > 10, then x(0) = 10. If x(1) > 10, then x(0) = 10.

Or what do you think about this?

Rodrigo de Azevedo
  • 23,223
euraad
  • 3,052
  • 4
  • 35
  • 79
  • Try projected gradient descent – ProAmateur Nov 18 '19 at 20:32
  • What is your constraint? – Duns Nov 18 '19 at 21:06
  • @Dunkel I don't know right now. But setting limits on vector $x$ would be a constraint. – euraad Nov 18 '19 at 21:47
  • @madnessweasley So I should have different "tuning factors"? – euraad Nov 18 '19 at 21:48
  • I cannot put a good answer but I can give you idea for trick that could work. You might want try borrowing idea from regularized regression (ridge or lasso). Instead of $L_1$ penalty you can try any form of function you want. This is rather applied way of approaching your problem, maybe experiment with function that grows exponentially as it reaches its constraint boundary. – Yohanes Alfredo Nov 19 '19 at 14:53
  • @YohanesAlfredo I got the advice from madnessweasley that I should use projected gratient descent. OK. But how to I minimize this $min|y - x|$ if vector $y$ is known and I want to find $x$ and $x$ is constrained? – euraad Nov 19 '19 at 17:04
  • The idea of projected gradient descent is simply, do one step of GD but save it as intermediate output $y$. if your $y$ is within feasible set, keep it (set $x_{t+1} = y$. Otherwise, you take the closest feasible point from your intermediate output based on some distance metric as your new updated point. Theoretically easy idea to do but quite difficult to compute since finding this projection is not that easy. https://ttic.uchicago.edu/~nati/Teaching/TTIC31070/2015/Lecture16.pdf – Yohanes Alfredo Nov 19 '19 at 18:10
  • @YohanesAlfredo Sorry. To much math. Can't I use if-statments instead? – euraad Nov 19 '19 at 18:19

0 Answers0