19

In Boyd's Convex Optimization, pp. 243,

for any optimization problem ... for which strong duality obtains, any pair of primal and dual optimal points must satisfy the KKT conditions

i.e. $\mathrm{strong ~ duality} \implies \mathrm{KKT ~ is ~ necessary ~ condition ~ for ~ optimal ~ solution}$

and in pp. 244,

(When the primal problem is convex) if $\tilde{x}, \tilde{\lambda}, \tilde{\mu}$ are any points that satisfy the KKT conditions, then $\tilde{x}$ and $(\tilde{\lambda}, \tilde{\mu})$ are primal and dual optimal, with zero duality gap.

If duality gap = 0, the problem satisfies strong duality, and in the 3rd paragraph:

If a convex optimization problem ... satisfies Slater’s condition, then the KKT conditions provide necessary and sufficient conditions for optimality

For me it means: (for any convex problems KKT is already sufficient for optimal)

$$\mathrm{KKT} \implies \mathrm{optimal ~ with ~ zero ~ duality ~ gap} \implies \mathrm{strong ~ duality} \implies \mathrm{KKT ~ is ~ also ~ necessary}$$

so KKT is necessary and sufficient for any convex problems? (Because Slater's condition can be automatically satisfied for the zero duality gap)

Alexander Zhang
  • 473
  • 1
  • 3
  • 9

2 Answers2

20

The KKT conditions are not necessary for optimality even for convex problems. Consider $$ \min x $$ subject to $$ x^2\le 0. $$ The constraint is convex. The only feasible point, thus the global minimum, is given by $x=0$. The gradient of the objective is $1$ at $x=0$, while the gradient of the constraint is zero. Thus, the KKT system cannot be satisfied.

daw
  • 54,637
  • 2
  • 44
  • 85
  • 2
    slater or any other constraint qualification does hold in your example – Red shoes Nov 19 '17 at 19:17
  • @Redshoes ? simply no. please check again. the constraint says $x^2$ LESS than zero. – daw Nov 20 '17 at 07:03
  • in fact, no constraint qualification holds: Please calculate the a KKT point – daw Nov 20 '17 at 07:04
  • Sorry I wanted to say ,, does NOT hold.. it was a typo. That's why KKT is not satisfied in minimum point – Red shoes Nov 21 '17 at 07:43
  • Isn't linear independence constraints qualification working? Gradient of x^2 is 2x is that not linear independence? – holala Jun 14 '22 at 05:11
  • @holala not if $x=0$ – daw Jun 14 '22 at 06:05
  • If I am following correctly $\nabla f_i ( x^\ast)$ s.t. $f_i (x^\ast) = 0$ is linearly dependent if $x^\ast = 0$, is this correct? – holala Jun 14 '22 at 17:18
  • 1
    This fails Boyd's qualification of the theorem, where he assumes strong duality holds. Strong duality does NOT hold in your example @daw – user3180 Mar 21 '23 at 22:58
  • 2
    The crux thing here is that the tangent cone at $0$ and the set of linearized feasible directions at $0$ is that they do not match. In particular, the former is ${0}$, while the latter is $\mathbb{R}$. – Blue Jan 25 '24 at 01:49
13

Boyd and Vandenberghe considers convex optimization problems of the form \begin{align} \text{minimize} &\quad f_0(x) \\ \text{subject to} & \quad f_i(x) \leq 0 \quad \text{for } i = 1,\ldots, m \\ &\quad a_i^T x = b_i \quad \text{for } i = 1,\ldots, p, \end{align} where $f_0,\ldots, f_m$ are convex functions. The optimization variable is $x \in \mathbb R^n$ and the domain of the optimization problem is $$ \mathcal D = \bigcap_{i=0}^m \textbf{dom} \,f_i. $$ (See equation (4.15), p. 136 in Boyd and Vandenberghe.)

Let $x \in \mathbb R^n$, $\lambda \in \mathbb R^m$, and $\nu \in \mathbb R^p$. Then the following two statements are equivalent:

  1. $x$ and $(\lambda,\nu)$ together satisfy the KKT conditions.
  2. $x$ and $(\lambda,\nu)$ are primal and dual optimal, and strong duality holds.

If Slater's condition is satisfied, then strong duality is guaranteed to hold, and so we can make a simpler and more useful statement. In this case, the following are equivalent:

  1. $x$ and $(\lambda,\nu)$ together satisfy the KKT conditions.
  2. $x$ and $(\lambda,\nu)$ are primal and dual optimal.

Warning: If strong duality does not hold, then it is possible for $x$ and $(\lambda,\nu)$ to be primal and dual optimal without satisfying the KKT conditions. An example where this occurs is given below.

By the way, if Slater's condition holds, then dual optimal variables $(\lambda,\nu)$ are guaranteed to exist. So if $x$ is primal optimal, then $x$ and $(\lambda,\nu)$ together satisfy the KKT conditions.

Example: [Problem 5.21 in Boyd and Vandenberghe] Consider the convex optimization problem \begin{align} \text{minimize} &\quad e^{-x} \\ \text{subject to} & \quad \frac{x^2}{y}\leq 0 \end{align} with variables $x$ and $y$ and domain $$ \mathcal D = \{ (x,y) \mid y > 0 \}. $$ The primal optimal value is $1$. The Lagrangian is $$ L(x,y,\lambda) = e^{-x} + \lambda \frac{x^2}{y}. $$ The dual function is $$ g(\lambda) = \inf_{(x,y) \in \mathcal D} L(x,y,\lambda) =\begin{cases} 0 & \quad \text{if } \lambda \geq 0, \\ -\infty & \quad \text{otherwise.} \end{cases} $$ The dual problem is \begin{align} \text{maximize} &\quad 0 \\ \text{subject to} &\quad \lambda \geq 0. \end{align} The optimization variable in the dual problem is $\lambda$. We can see that any $\lambda \geq 0$ is dual optimal.

So, any $(x,y)$ with $x = 0$ and $y > 0$ is primal optimal, and any $\lambda \geq 0$ is dual optimal. Although the primal and dual optimal values are both attained, strong duality does not hold.

littleO
  • 54,048
  • What's an example of a problem where strong duality holds but Slater's conditions do not hold? – Y. S. Jul 27 '18 at 02:23
  • @Y.S. I think an example is minimize $x$ subject to $x^2 \leq 0$. The Lagrangian is $L(x,z)= x+ z x^2$ and the dual function is $g(z) = -1/4z$ if $z \neq 0$ and $g(z) = -\infty$ otherwise. The dual optimal value is $0$, but the dual optimal value is not attained. Maybe you wanted an example where the dual optimal value is attained, I'll have to think about that. (Is that impossible?) – littleO Jul 27 '18 at 20:24
  • Cool example! But then is slater's condition satisfied? The primal constraint has no interior. – Y. S. Jul 28 '18 at 19:50
  • @Y.S. I think Slater's condition for an inequality constraint $h(x) \leq 0$ would say that there exists a point $\tilde x$ such that $h(\tilde x) < 0$. – littleO Jul 28 '18 at 20:04
  • but in this scenario there isn't, right? since there is no $\tilde x$ where $\tilde x^2 <0$? (But yeah, misunderstood Slater's!) (Although that kind of implies there is interior, no?) – Y. S. Jul 28 '18 at 20:35
  • Overall, I feel like it must be really hard to find a scenario where slater holds but strong duality doesn't! – Y. S. Jul 28 '18 at 20:45
  • @Y.S. Oh, I thought you were asking for a case where strong duality holds but Slater's condition doesn't hold. I think that if Slater's condition holds then strong duality is guaranteed to hold, so it should be impossible to find a scenario where Slater holds but strong duality doesn't. – littleO Jul 28 '18 at 21:15
  • 2
    Btw, I realized I can simplify the above example. The primal problem could be to minimize $0$ subject to $x^2 \leq 0$. Then the Lagrangian is $L(x,z) = a x^2$ and the dual function is $g(z) = 0$ if $z \geq 0$ and $g(z) = -\infty$ otherwise. The dual optimal value is $0$, and in this case the dual optimal value is attained (for example by $z = 0$). – littleO Jul 28 '18 at 21:17
  • Oh yes, I kind of lost track of the objective there! And yeah I like this second example! These corner cases are pretty interesting! – Y. S. Jul 28 '18 at 22:37
  • 1
    Hi, do you have an example of the warning "If strong duality does not hold, then it is possible for and (,) to be primal and dual optimal without satisfying the KKT conditions"? I googled a lot but can't find one such example. – soloice Jun 25 '21 at 14:56
  • If a smooth convex problem has at least one KKT point, then all minimizers satisfy the KKT conditions? It seems that all minimizers are qualified when at least one is qualified as well (with respect to Guignard constraint qualification). This is quite weird for me. – R. W. Prado Sep 13 '22 at 18:33
  • @soloice I just added an example. Does that look correct to you? – littleO Nov 09 '22 at 08:40