6

Let $\mathcal{C}$ be a non-empty, closed and convex set. I am trying to prove that the projection operator defined as follows

$$ P(z) := \underset{x \in \mathcal{C}}{\operatorname{argmin}} \frac{1}{2}\|x-z\|^2_2 $$

is non-expansive. To show this, I proceed as:

$$ \| P(z_1) - P(z_2) \| = \| x_1 - x_2 \|$$

where $x_1, x_2 \in \mathcal{C}$. Now, I know that $\|x_1-x_2\| \leq \|z_1-z_2\|$. But, how to prove this last part? Can JL lemma be used someway?

Rodrigo de Azevedo
  • 23,223
CKM
  • 1,642
  • What is $C$? This is not true in general for arbitrary set $C$ but holds e.g. for $C$ convex. – air Sep 08 '15 at 07:38
  • Assume $C$ is convex. – CKM Sep 08 '15 at 08:23
  • The projection operator onto a convex set is more than just non-expansive, it is in fact firmly non-expansive, i.e $|P(z_1) - P(z_2)|^2 + |Q(z_1) - Q(z_2)|^2 \le |z_1 - z_2|^2$ $\forall (z_1, z_2) \in \mathcal{X}^2$, where $Q := Id - P$. – dohmatob Sep 08 '15 at 23:34
  • Thanks for clarifying. Do you have proof for this? – CKM Sep 09 '15 at 08:04
  • @dohmatob -- What you wrote is true when $P$ is linear, but for the general case, when $C$ is just convex and no a subspace, I am not sure it is true. Can you prove it? – uniquesolution Nov 11 '15 at 14:54
  • @uniquesolution: This is true for the proximal operator of any proper covex lower-semi continuous mapping. This is a classical result due to G.J Minty (Monotone nonlinear operators in Hilbert spaces --1962). In your case, $P = \text{prox}_{i_C}$, and it's an easy excercise to show that $i_C \in \Gamma_0$. – dohmatob Nov 11 '15 at 16:11
  • @dohmatob Oh, thanks. Looks like a nice result. I'm not familiar with the proximal operator of a convex lower-semi continuous mapping. Thankfully there is an easier proof that the projection onto a convex and closed set is non-expansive... – uniquesolution Nov 11 '15 at 16:32
  • I'm talking about firm non-expansivieness, which is much stronger than non-expensiveness. OK, long short, this is an invitation to study "proximal operator theory", which is very powerful and much easier than people usually imagine... – dohmatob Nov 11 '15 at 16:40
  • @chandresh I do not think that what you have is true in general. $\begin{equation} |P(z_1) - P(z_2)|| \le |x_1-x_2| \end{equation} $ is true for convex sets. but $\begin{equation} |P(z_1) - P(z_2)|| =|x_1-x_2| \end{equation} $ only if the set $\mathcal{C}$ is a subspace (which are convex but a special kind of convex. Right? – makansij Apr 21 '18 at 02:32
  • how does the answer change if the projection operator is in terms of a weighted norm i.e. $$ P(z) = \min_{x \in X} \frac{1}{2} | x - x |^2_W $$? – Charlie Parker May 09 '19 at 18:10

1 Answers1

13

As in your post, let $z_1$, $z_2$ be arbitrary points.

Recall the variational characterization of the projection operator onto nonempty, closed, convex sets:

$$ \langle z_1 - P(z_1), x- P(z_1) \rangle\leq 0 \; \forall \; x \in C $$

Now also notice that by definition $P(z_2) \in C$ thus we get:

$$ \langle z_1 - P(z_1), P(z_2)- P(z_1) \rangle\leq 0 $$

Similarly we also get:

$$ \begin{aligned} &\langle z_2 - P(z_2), P(z_1)- P(z_2) \rangle\leq 0 \\ \Rightarrow &\langle P(z_2) - z_2, P(z_2)- P(z_1) \rangle\leq 0 \end{aligned} $$

Adding these two inequalities, rearranging and finally applying the Cauchy-Schwarz inequality, we get:

$$ \begin{aligned} \langle P(z_2) - P(z_1), P(z_2)- P(z_1) \rangle &\leq \langle z_2 - z_1, P(z_2)- P(z_1) \rangle \\ & \leq \vert\vert z_2 - z_1 \vert\vert \; \vert\vert P(z_2) - P(z_1) \vert\vert \end{aligned} $$

Thus:

$$ \begin{aligned} &\vert\vert P(z_2) - P(z_1) \vert\vert^2 \leq \vert\vert z_2 - z_1 \vert\vert \; \vert\vert P(z_2) - P(z_1) \vert\vert \\ \Rightarrow &\vert\vert P(z_2) - P(z_1) \vert\vert \leq \vert\vert z_2 - z_1 \vert\vert \end{aligned} $$

air
  • 2,871
  • Thanks. That makes sense. One small thing. I wanted to read more about variational characterization of the projection operator. Is there any any source or link? – CKM Sep 08 '15 at 09:31
  • Sometimes it is called the Projection theorem, you can find it in many more theoretical convex analysis books (e.g. Proposition 1.1.9 in the book Convex Optimization Theory by Dimitri Bertsekas). He also has some slides from his lectures at MIT OCW, see the 5th slide (http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-253-convex-analysis-and-optimization-spring-2012/lecture-notes/MIT6_253S12_lec03.pdf) – air Sep 08 '15 at 14:53
  • Did the asker forget to accept @air's answer (it completely answers the question as it stands) ? – dohmatob Nov 12 '15 at 07:58
  • 2
    FWIW: That variational characterization is called the Bourbaki-Cheney-Goldstein inequality. – dohmatob Nov 12 '15 at 07:58
  • 1
    Can you explain how the adding/rearranging works? I just cannot see it, even after 2 hours. – mavavilj Feb 18 '19 at 02:46
  • how does your answer change if the projection operator is in terms of a weighted norm i.e. $$ P(z) = \min_{x \in X} \frac{1}{2} | x - x |^2_W $$? – Charlie Parker May 09 '19 at 18:09
  • It does not change; at least if by weighted norm you mean $||x||^2_W := \sum w_i x_i^2$, where $w_i > 0$. – air May 09 '19 at 18:18
  • @mavavilj Let $u:=z_2-z_1$ and $v:=P(z_2)-P(z_1)$. Adding the two inequalities gives $\langle v-u,v\rangle\le0,$ which by rearranging becomes $\langle v,v\rangle\le\langle u,v\rangle.$ – Anne Bauval Feb 24 '25 at 08:51