On expectation of maximum of gaussians

Question

Let $X_1,\ldots,X_n$ be i.i.d $\mathcal{N}(0,1)$ random variables. I am trying to prove that \begin{align} (a)\ \ \mathbb{E} \left[ \max_{i}X_i\right] & \asymp\mathbb{E} \left[ \max_{i}|X_i|\right] \asymp \sqrt{\log n},\\ (b) \ \ \mathbb{E} \left[ \max_{i}X_i\right] &= \sqrt{2 \log n}+o(1) \end{align} where $A \asymp B$ means there exists universal constants $m,M >0$ such that $mA \leq B \leq MA$.

For part (a), I was able to prove the upper bound that $\mathbb{E} \left[ \max_{i}X_i\right] \leq \sqrt{2 \log n}$ using Jensen's inequality. How do I prove the lower bound and the fact that the two expectations are equivalent? I've been given the following hint: $\mathbb{P}(\max_{i}X_i \geq t)=1-\mathbb{P}(X_1 \leq t)^n$.

Are you aware of the formula $E[X]=\int_0^\infty P(X \geq t) dt$ for nonnegative random variables $X$? This seems helpful, because if you ignore the (exponentially unlikely) possibility that the maximum is negative, you reduce to $E[X] \approx \int_0^\infty 1-\Phi(t)^n dt$ where $\Phi$ is the standard normal CDF. The error here is something like the expectation of (a Gaussian multiplied by $1_{X<0}$) times $2^{-n}$. — Ian, Mar 25 '16 at 16:10
@Ian: Yeah, but I am unable to gain anything from that formula. Am I missing something? — pikachuchameleon, Mar 25 '16 at 16:13

Ian · Accepted Answer · 2020-03-15T17:17:03.220

8

A mostly-worked-out answer to the lower bound in part a:

$$E[\max_i X_i]=E[\max_i X_i 1_{\max_i X_i \geq 0}]+E[\max_i X_i 1_{\max_i X_i<0}].$$

We want to throw out that negative piece. Intuitively, it is unlikely to happen at all and it has bounded expectation. More rigorously, it goes to zero in probability (the probability of it being nonzero is $2^{-n}$) and is pointwise decreasing in magnitude, so by dominated convergence

$$E[\max_i X_i] \geq E[\max_i X_i 1_{\max_i X_i \geq 0}] + o(1) \\ =\int_0^\infty 1-\Phi(t)^n dt + o(1).$$

using the hint and a standard fact from Lebesgue integration of nonnegative functions. Denote the first term by $I$.

by simply throwing out regions of positive area.

On $[0,1]$ we have the simple bound $1-\Phi(t)^n \geq 1-\Phi(1)^n$. On $[1,\sqrt{2 \log(n)}]$ we have the bound $\Phi(t) \leq 1-\frac{1}{\sqrt{2 \pi}} e^{-t^2/2}$. (Cf. https://mikespivey.wordpress.com/2011/10/21/normaltails/) Hence

$$I \geq 1-\Phi(1)^n + \int_1^{\sqrt{2 \log(n)}} 1-\left ( 1-\frac{1}{\sqrt{2 \pi}} e^{-t^2/2} \right )^n dt.$$

As for the remaining piece, we're integrating a decreasing function, so we get a lower bound by substituting in the upper limit:

$$I \geq 1-\Phi(1)^n+\int_1^{\sqrt{2 \log(n)}} 1-\left ( 1-\frac{1}{\sqrt{2 \pi}} n^{-1} \right )^n dt.$$

The sequence of numbers in the integrand converges to $1-e^{-\frac{1}{\sqrt{2 \pi}}}>0$, so it is bounded below by $1-e^{-\frac{1}{\sqrt{2 \pi}}}-\varepsilon=:C$ for large enough $n$ depending on $\varepsilon$. Then we get the bound

$$I \geq C(\sqrt{2 \log(n)}-1)+1-\Phi(1)^n.$$

Returning to the original problem we have

$$E[\max_i X_i] \geq C(\sqrt{2 \log(n)}-1)+1-\Phi(1)^n+o(1)$$

which gives the lower bound for part a for sufficiently large $n$. A finite collection of $n$ can always be handled (why?) so we are done.

To solve part b we would need to be able to repeat the derivation to get $C=1$, and I'm not really sure how to do that. One idea would be to change variables to $u=\Phi(t)$, which would give

$$\int_0^\infty 1-\Phi(t)^n dt = \int_{1/2}^1 (1-u^n)\frac{dt}{du} du.$$

where $\frac{dt}{du}$ is the reciprocal of the normal density, written as a function of the normal CDF itself. Perhaps it is possible to get an appropriate series expansion for this quantity to get the result.

edited Mar 15 '20 at 17:17

answered Mar 25 '16 at 17:07

Ian

104,572

I think in the very first line, the bound should be less than or equals. Can you check that please? – pikachuchameleon Mar 25 '16 at 17:09
@pikachuchameleon You are right; getting the lower bound will actually require estimating the error there. Let me work on that. – Ian Mar 25 '16 at 17:10
@pikachuchameleon It's fixed now. – Ian Mar 25 '16 at 17:14
Also, you wrote that expectation as an integral. There, instead of $\Phi$, it should be the cdf of $|X_i|$. Is that correct? – pikachuchameleon Mar 25 '16 at 17:14
@pikachuchameleon No, that part is actually correct. Intuitively, because I did the splitting based on the signs, if you were to change variables back to make $I$ an integral in the probability space variable $\omega$, it would no longer range over the whole probability space $\Omega$. Part of the probability space is confined to the negative term which I handled separately. – Ian Mar 25 '16 at 17:16
@pikachuchameleon Now I think I understand your question, because what I answered is not exactly what you asked. My answer proves that $E[\max_i X_i] \asymp \sqrt{ \log(n)}$. $E[\max_i |X_i|] \geq E[\max_i X_i]$, so for that you can take $m=1$. The analysis at the top with the $2^{-n}$ allows you to construct $M$. This gives the left relation. Now you can prove that $\asymp$ is an equivalence relation (irrespective of this context) to deduce the other relation. – Ian Mar 25 '16 at 17:30
I have the following doubts in your solution: 1) $E[\max_i X_i 1_{\max_i X_i<0}]=2^{-n} E [\max_i X_i | X_i \leq 0,\forall i ] $. How did you conclude that $E [\max_i X_i | X_i \leq 0,\forall i ] $ is a constant and decreases in magnitude with n? That way, $A$ should be dependent on n in the above solution. 2) $E[\max_i X_i 1_{\max_i X_i \geq 0}] =E[\max_i |X_i|]=\int_0^\infty 1- F(t)^n dt $, where $F$ is the cdf of $|X_1|$. That's why I expressed my earlier doubt. So in your integral instead of $\Phi$, it should be $2\Phi-1$. – pikachuchameleon Mar 25 '16 at 19:45
I didn't say that this conditional expectation is constant. I said that its magnitude is bounded below by a constant. That's true because it is an increasing sequence of negative numbers. 2. $E[\max_i X_i 1_{\max_i X_i \geq 0}]$ is not the same as $E[\max_i |X_i|]$. For instance consider that when $\max_i X_i<0$, we have that $\max_i X_i 1_{\max_i X_i \geq 0} = 0$ while $\max_i |X_i|>0$.

Ian

Mar 25 '16 at 21:33

Let us continue this discussion in chat. – pikachuchameleon Mar 25 '16 at 21:44

$\Phi(t) \leq 1-\frac{1}{\sqrt{2\pi}} e^{-t^2/2} \iff 1-P(Z > t) \leq 1-\frac{1}{\sqrt{2\pi}} e^{-t^2/2} \iff P(Z > t) \geq \frac{1}{\sqrt{2\pi}} e^{-t^2/2}$. But the biggest lower tail bound I know of is $P(Z > t) \geq \frac{1}{\sqrt{2\pi}} \frac{t}{t^2+1} e^{-t^2/2}$, which is not $\geq \frac{1}{\sqrt{2\pi}} e^{-t^2/2}$ over the region $[1,\sqrt{2\log n}$, as you said. This seems to be important in your proof. Am I mistunderstanding something? – Daniel Xiang Apr 28 '19 at 17:54

@DanielXiang I think you are right and there is a mistake here, in that I need to use a lower bound for the upper tail and I actually used the expression for an upper bound. Still, it suffices to show that $1-(1-\Phi(\sqrt{2\log(n)}))^n$ is bounded below away from zero, and I think the bound you just gave is easily sufficient for this purpose. – Ian Apr 28 '19 at 23:27

Maurizio Barbato · Answer 2 · 2024-05-21T15:43:37.830

For the sake of completeness, I give here the full proof of the existence of $M>0$ such that $\mathbb{E} \left[ \max_{i}|X_i|\right] \leq M \mathbb{E} \left[ \max_{i}X_i\right]$ for all $n \geq 2$ (note that for $n=1$ the inequality cannot be true). First of all, let us note that $\mathbb{E} \left[ \max \{X_1,X_2\} \right] > 0$. This easily follows by noting that, if $\phi$ is the density function of the standard normal, then \begin{equation} \int_{x_1 \leq 0, x_2 \leq 0} \max \{x_1,x_2\} \phi(x_1)\phi(x_2)dx_1 dx_2 = \\= \int_{x_1 \geq 0, x_2 \geq 0} -\min \{x_1,x_2\} \phi(x_1)\phi(x_2)dx_1 dx_2, \end{equation} so that \begin{equation} \left| \int_{x_1 \leq 0, x_2 \leq 0} \max \{x_1,x_2\} \phi(x_1)\phi(x_2)dx_1 dx_2 \right| \leq \\ \leq \int_{x_1 \geq 0, x_2 \geq 0} \max \{x_1,x_2\} \phi(x_1)\phi(x_2)dx_1 dx_2. \end{equation} Note also that $n \mapsto \mathbb{E} [\max \{X_1,\dots,X_n \} ]$ is increasing.

Now we have (check the first inequality by considering separately the case in which there are both positive and negative $X_i's$ from that one in which all $X_i$'s are positive or negative): \begin{equation} \max \{|X_1|,\dots,|X_n|\} \leq \max \{X_1,\dots,X_n \} \mathbb{1}_{\{\max_{i} X_i \geq 0 \}} + \\ + \max \{-X_1,\dots,-X_n\} \mathbb{1}_{\{\max_{i} -X_i \geq 0 \}} = \\= \max \{X_1,\dots,X_n \} - \max \{X_1,\dots,X_n \} \mathbb{1}_{\{\max_{i} X_i < 0 \}} + \\+ \max \{-X_1,\dots,-X_n \} - \max \{-X_1,\dots,-X_n \} \mathbb{1}_{\{\max_{i} -X_i < 0 \}}, \end{equation}

so, by noting that $-X_1,\dots,-X_n$ are i.i.d. standard normal variables, we get \begin{equation} \mathbb{E} \left[ \max_{i}|X_i|\right] \leq 2 \mathbb{E} \left[ \max_{i}X_i\right] + 2 C_n, \end{equation} where $C_n = - \mathbb{E} \left[ \max \{X_1,\dots,X_n \} \mathbb{1}_{ \{ \max_{i} X_i < 0 \}} \right]$. As noted by Ian in his answer, we have $C_n \rightarrow 0$, since \begin{equation} C_n= - \mathbb{E} \left[ \max \{X_1,\dots,X_n \} \mathbb{1}_{ \{ \max_{i} X_i < 0 \}} \right] = \\ = -2^{-n}\mathbb{E} \left[ \max \{X_1,\dots,X_n \} | \mathbb{1}_{ \{ \max_{i} X_i < 0 \}} \right], \end{equation} being $n \mapsto -\mathbb{E} \left[ \max \{X_1,\dots,X_n \} | \mathbb{1}_{ \{ \max_{i} X_i < 0 \}} \right]$ a positive decreasing sequence. So $C_n \leq C$ fo some $C > 0$. Choose then $M$ such that \begin{equation} M \geq \frac{2C}{\mathbb{E} \left[ \max \{X_1,X_2 \}\right]} + 2, \end{equation} and you will get for $n \geq 2$: \begin{equation} \mathbb{E} \left[ \max_{i} |X_i| \right] \leq 2 \mathbb{E} \left[ \max_{i}X_i\right] + 2 C_n \leq 2 \mathbb{E} \left[ \max_{i}X_i\right] + 2 C \leq M \mathbb{E} \left[ \max_{i}X_i\right]. \end{equation} I want also add here that a proof of the equation (b) of this post was given by van der wolf in his answer to the post Expectation of the maximum of gaussian random variables.

On expectation of maximum of gaussians

2 Answers2

Linked