10

A function $f$ is said to be strongly convex with respect to a norm $\|\cdot\|_p$ if for all $x,y$, $$f(x) \geq f(y) + \nabla f(y)^T(x-y) + \frac{1}{2}\|x-y\|^2_p.$$

There are a bunch of functions used in machine learning, statistics, etc. that are extremely well known to be strongly convex with respect to the $2$ or $1$ norm

Examples:

$\sum_{j = 1}^m x_j^2$ is 2-strongly convex with respect to $\|\cdot\|_2$

$\sum_{j = 1}^m x_j \log(x_j) $ is 1-strongly convex with respect to $\|\cdot\|_1$

Does there exist any strongly convex function with respect to a norm $\|\cdot\|_p, p>2$?

Norman
  • 1,246
  • Does $0 \geq \dfrac{1}{2} (\max_i |x_i - y_i| )^2$ hold for all $x,y \in \mathbb{R}$? – Norman Jul 23 '20 at 07:06
  • All norms are equivalent on $\mathbb R^m$, so a strongly convex function is strongly convex wrt any norm. – Gabriel Romon Jul 23 '20 at 07:11
  • 1
    @GabrielRomon I'm perplexed. A lot of efforts are put in to prove things are strongly convex wrt a certain norm. Such as the Pinsker's inequality. https://mltheory.github.io/CS7545/scribe/lec3.pdf – Norman Jul 23 '20 at 07:17
  • Do you care about the constant before the norm term ? If so, my argument is useless. – Gabriel Romon Jul 23 '20 at 07:19
  • 1
    @GabrielRomon If I don't care, then what you say is 100% true, and any strongly convex function in one norm is strongly convex in another with a slightly adjusted factor. If I do care, then I'm stuck – Norman Jul 23 '20 at 07:23
  • 1
    A lot of optimization algorithms will require you to know or be able to estimate the strong convexity constant to determine things like step size. You probably care. – Jürgen Sukumaran Jul 23 '20 at 07:26
  • It might be worth noting that sometimes the phrase "$\beta$-strongly convex" is used to refer to the inequality above with the $2$ norm fixed, and $\beta|x-y|_2^2/2$ instead of $|x-y|_p^2/2$. In those contexts, $\beta$ is called the strong convexity constant. Would you allow considering a function with a different constant for each $p$? i.e. a function $f$ such that, for every $p$, there exists $\beta_p$ such that the inequality above holds where $\beta_p|x-y|_p^2/2$ replaces $|x-y|_p^2/2$? – Zim Jul 23 '20 at 14:35

1 Answers1

3

It depends on what you mean by that. In finite dimensional spaces, as other people noted, all norms are equivalent. However, the constants relating them can scale with the dimension.

If you are interested in functions that would have a strong convexity constant that is independent of the dimension (while having dimension independent divergence on sets of dimension-independent diameters), then no such function exists for $p > 2.$ For proof, see Example 4.1 in https://arxiv.org/pdf/1301.0465.pdf.

user1085605
  • 31