0

Considering a convex function $f:\mathbb{R}^n \rightarrow \mathbb{R}$, if it is a $\beta$-smooth function, namely $$\forall x,y\in \mathbb{R}^n, \quad f(y) \leq f(x) + \nabla f(x)^\mathrm{T}(y-x)+\frac{\beta}{2}\lVert y-x \rVert^2 $$

then we have $$\lVert \nabla f(x)\rVert^2 \leq \beta f(x) $$

This conclusion comes from a lecture note from Dimitris Papailiopoulos's course ECE 901 but without proof.

I just wanna figure out how to prove this conclusion? Thx.

Matics
  • 307
  • 2
  • 11
  • Seems that this property does not hold. A counterexample due to @Willie is any linear function $f(x)=Lx$ is convex and $\beta$-smooth for any $\beta \geq 0$. But $| \nabla f(0)|^2 = |L|^2 \not\leq \beta f(0) = 0$ whenever $L \neq 0$. – Matics May 09 '17 at 02:58
  • On further study I do think you're right about the second inequality. The lecture notes themselves say that linear & affine functions are $0$-smooth. But $|\nabla f(x)|^2$ is a constant in those cases. – Michael Grant May 09 '17 at 04:00
  • Now I know the inequality only holds for positive-valued smooth functions. – Matics May 09 '20 at 16:05

1 Answers1

1

You copied incorrectly the definition of $\beta$-smooth. In the lecture notes you linked to, $\beta$-smooth is defined by $$ \| \nabla f(x) - \nabla f(y) \| \leq \beta \| x - y\|. $$ (What you copied down is almost the definition of $\beta$-strong convex; you have the wrong inequality.)

That said, the claim you are asking about is false (at least without additional assumptions). Any linear function $f(x) = Lx$ is convex and $\beta$-smooth for any $\beta \geq 0$. But $\| \nabla f(0)\|^2 = \|L\|^2 \not\leq \beta f(0) = 0$ whenever $L \neq 0$.

Willie Wong
  • 75,276
  • Thank you for your counterexample, I think you are right, this claim should be false. However, the definition above seems equivalent to your definition, which means $\quad f(y) \leq f(x) + \nabla f(x)^\mathrm{T}(y-x)+\frac{\beta}{2}\lVert y-x \rVert^2$ is equivalent to $| \nabla f(x) - \nabla f(y) | \leq \beta | x - y|.$ from Hazan's book Hazan - Introduction to Online Convex Optimization in Page 22. But anyone who knows how to prove it? – Matics May 09 '17 at 02:52
  • 1
    I found the proof of the equivalence https://math.stackexchange.com/questions/2020979/show-that-lipschitz-nabla-fx-nabla-fy-leq-l-x-y-is-implied-b?rq=1 – Matics May 09 '17 at 05:27
  • 1
    I now realize that the correct form should be $| \nabla f(x)|^2 \leq 2\beta (f(x) - \inf_u f(u))$, whose proof can be found in Theorem 4.23 of this lecture note (https://arxiv.org/pdf/1912.13213). – Matics Aug 09 '24 at 04:35