Lipschitz Hessian implies Lipschitz Hessian diagonal?

Question

I am working with an optimization problem where the Hessian is assumed Lipschitz, i.e.,

$$\| \nabla^2 f(x) - \nabla^2 f(y)\| \leq L\|x-y \|\tag{1}$$ and positive semi-definite. i.e., $\nabla^2 f(x) \succeq 0$. Let $\text{diag}\:\nabla^2 f(x)$ be the diagonal matrix whose elements are the diagonal elements of the Hessian $\nabla^2 f(x)$. That is,

$$\nabla^2 f(x) = \widehat{\nabla^2 f(x)} + \text{diag}\:\nabla^2 f(x)\tag{2}.$$

My goal is to prove that $\text{diag}\:\nabla^2 f(x)$ is also Lipschitz continuous.

To do so I have started by this post to get

$$\begin{aligned}z^T \nabla^2 f z \leq L z^Tz &\Leftrightarrow\\ z^T\widehat{\nabla^2 f} z + z^T\text{diag}\:\nabla^2 f z \leq L z^Tz &\Leftrightarrow \\ z^T\widehat{\nabla^2 f} z \leq z^T (L\:I -\text{diag}\:\nabla^2 f)z& \end{aligned}\tag{3}$$ Then by this post and the $\nabla^2 f\succeq 0$, we have $\text{diag}\:\nabla^2 f \succeq 0$. Combining the above we have $$z^T\widehat{\nabla^2 f} z \leq z^T (L\:I -\text{diag}\:\nabla^2 f)z \leq L z^Tz\tag{4}$$ which by setting $z=x-y$ and using again this post, we get

$$\| \widehat{\nabla^2 f(x)} - \widehat{\nabla^2 f(y)}\| \leq L\|x-y \|\tag{5}$$

Finally, using triangle inequality in $(1)$ and by applying $(5)$ we get

$$\begin{aligned} -\|\widehat{\nabla^2 f(x)} -\widehat{\nabla^2 f(y)} \| + \|\text{diag}\:\nabla^2 f(x) - \text{diag}\:\nabla^2 f(y) \|\leq L \|x -y \| &\Leftrightarrow \\ \|\text{diag}\:\nabla^2 f(x) - \text{diag}\:\nabla^2 f(y) \| \leq L \|x-y \| + \|\widehat{\nabla^2 f(x)} -\widehat{\nabla^2 f(y)} \| & \Leftrightarrow \\ \|\text{diag}\:\nabla^2 f(x) - \text{diag}\:\nabla^2 f(y) \| \leq 2 L \|x-y \|. \end{aligned}\tag{6}$$

Could you please someone verify if this procedure is correct? If it is correct can we make it tighter? Generally, in this case are we interested for a smaller Lipschitz constant or something that maximizes the upper bound? Any help is highly appreciated.

Answer 1

I am looking you comments again after a year. Please let me concentrate them in this post hopping to understand better what you have written in you comments and make a question in the end.

Here are the facts. The first comment fact is

If $f:\Bbb R^n\to\Bbb R^m$ is Lipschitz continuous then $f_i:\Bbb R^n\to\Bbb R$, for $i=1,2,\dots, m$, is also Lipschitz continuous.$\tag{A}$

The second comment fact is

If $f_i:\Bbb R^n\to\Bbb R$, for $i=1,2,\dots, m$, is Lipschitz continuous then $f:\Bbb R^n\to\Bbb R^m$ is also Lipschitz continuous. $\tag{B}$

The third comment fact is

$F(x):=(\nabla^2f_{11}(x),\ldots,\nabla^2f_{1n}(x),\ldots,\nabla^2f_{i1}(x),\ldots,\nabla^2f_{in}(x),\ldots,\nabla^2f_{n1}(x),\ldots,\nabla^2f_{nn}(x))$ $\tag{C}$

The forth comment fact is

$M_n(\Bbb R)\cong\Bbb R^{n^2} \tag{D}$

this will take us from the matrix space to vector space and back.

The fifth fact is

$$\| \nabla^2 f(x) - \nabla^2 f(y)\| \leq L\|x-y \| \tag{E}$$

As far as I understand form your comments, using (E) and (D), we can state that $F(x)$ is Lipschitz continuous. Then, using (A), we can state that

$$||\nabla^2f_{ij}(x) - \nabla^2f_{ij}(y)|| \leq L_{ij}||x-y|| \tag{F}$$

Then, restricting into the case of $i=j$, we have

$$||\nabla^2f_{ii}(x) - \nabla^2f (y)|| \leq L_{ii}||x-y|| \tag{G}$$

and using (B), we can go to

$$||\text{diag} \nabla^2f(x) - \text{diag} \nabla^2f(y)|| \leq L'||x-y|| \tag{H}$$

where $\text{diag} \nabla^2f(x)$ is a $m \times m$ diagonal matrix whose elements are the diagonal elements of $\nabla^2f(x)$.

My question:

I gathered all the information together to enhance my understanding of the process, particularly focusing on point (D). As I comprehend it, (D) serves as a bridging element that facilitates the transition between matrix space and vector space, while preserving the Lipschitz continuity characteristic. I'm uncertain about how (D) ensures the transfer of the Lipschitz continuity property between these spaces. Could you provide further insight on how (D) maintains Lipschitz continuity during the transition?