8

The assumed regression model is $E(Y_i|x)=\gamma+\beta(x-\bar{x})$ and $Var(Y|x)=\sigma^2$.

So I have: $E(RSS)= E(\sum\limits_{i=1}^n (y_i-\hat{y_i})^2)= E(\sum\limits_{i=1}^n (y_i-\hat{\gamma}-\hat{\beta}(x_i-\bar{x}))^2)= E(\sum\limits_{i=1}^n (y_i^2-2y_i\hat{\gamma} + \hat{\gamma}^2 - 2\hat{\beta}(x_i-\bar{x})y_i + 2\hat{\beta}(x_i-\bar{x})\hat{\gamma} + \hat{\beta}^2(x_i-\bar{x})^2)= \sum\limits_{i=1}^n (E(y_i^2) - 2E(y_i\hat{\gamma})+ E(\hat{\gamma}^2) - 2(x_i-\bar{x})E(y_i\hat{\beta})+ 2(x_i-\bar{x})E(\hat{\gamma}\hat{\beta})+ (x_i-\bar{x})^2E(\hat{\beta}^2)).$

But here I'm stuck. For $E(y_i^2), E(\hat{\gamma})$ and $E(\hat{\beta}^2)$ I can use the definition $E(X^2)= Var(X)+(E(X))^2$. But what should I use for the others? Clearly, $y_i$ and $\hat{\gamma}$ are dependent, then do I need to find their covariance to take the expectation? $\hat{\gamma}$ and $\hat{\beta}$ are uncorrelated by definition so I would know what to with them. However, I am not sure about $y_i$ and $\hat{\beta}$, are they correlated?

Keksainis
  • 101
  • 1
  • 2
  • 5

1 Answers1

6

From Rice - Mathematical Statistics and Data Analysis, using matrix notation.

Consider the sum of squared residuals for the general linear regression problem $||\mathbf{Y-HY}||^2$, where $\mathbf{H=X(X^TX)^{-1}X}$, then:

$$ \mathbb{E}||\mathbf{Y-HY}||^2 = \mathbb{E}(\mathbf{Y^T(I-H)Y}) = [\mathbb{E}(\mathbf{Y})]^T(\mathbf{I-H})[\mathbb{E}(Y)] + \sigma^2 tr(\mathbf{I-H}). $$

As $\mathbb{E}(\mathbf{Y})=\mathbf{X\beta}$, we get $(\mathbf{I-H})[\mathbb{E}(Y)] = 0$. Furthermore, $tr(\mathbf{I-H})= tr(\mathbf{I}) - tr(\mathbf{H})= n-p$.

Thus, $$\mathbb{E}||\mathbf{Y-HY}||^2 = (n-p) \sigma^2$$

For your two parameter model, we then have:

$$ \mathbb{E}(RSS) = (n-2)\sigma^2 \> \square $$

Padster
  • 93
  • 5
    How do you see that $E(Y^T(I-H)Y) = E(Y)^T(I-H)E(Y) + \sigma^2 \text{Tr}(I-H)$? Is there a name for the rule you're using there? – littleO Dec 19 '19 at 07:16
  • 1
    Your definition of the hat matrix is missing a transpose. It should be: $\mathbf{H=X(X^TX)^{-1}X^T}$ – James Sep 05 '21 at 19:39
  • @littleO - see https://math.stackexchange.com/questions/2228398/trace-trick-for-expectations-of-quadratic-forms – James Sep 05 '21 at 20:00
  • @littleO I think you can see a more complete explanation of this result here: https://en.wikipedia.org/wiki/Quadratic_form_(statistics)#Expectation – Daniel Kessler Jun 23 '22 at 19:08