Regression models questions

Question

I was studying regression models and found the formula:

$$V_Y^2 = V_\hat{Y}^2 + V_\varepsilon^2$$

• $Y = a+bX+\varepsilon$, which is the dependent variable.

• $\hat{Y}=a+bX$, which is the predictive variable (therefore $Y = \hat{Y} + \varepsilon$)

• $\varepsilon$ is the prediction error

From this equation we can divide each term by $V_Y^2$ and get:

$$\frac{V_Y^2}{V_Y^2} = \frac{V_\hat{Y}^2}{V_Y^2} + \frac{V_\varepsilon^2}{V_Y^2}$$

$$1 = R^2 + \frac{V_\varepsilon^2}{V_Y^2}$$

$$R^2 = 1 - \frac{V_\varepsilon^2}{V_Y^2}$$

• $R^2$ is the coefficient of determination

My first question is why we define:

$$R^2 = \frac{V_\hat{Y}^2}{V_Y^2}$$

---

Moreover, I found that the error variance is:

$$V_\varepsilon^2=\frac{\sum_{i=1}^{N}(y_i-\hat{y}_i)^2}{N}$$

This is because we assume $\bar{\varepsilon}=0$, where $\bar{\varepsilon}$ represents the average of the error $\varepsilon$, from the equation:

$$V_\varepsilon^2=\frac{\sum_{i=1}^{N}(\varepsilon_i-\bar{\varepsilon})^2}{N}$$

Furthermore, $\varepsilon_i=y_i-\hat{y}_i$, so we get the original equation.

My second question is why we make the assumption that $\bar{\varepsilon}=0$.

Answer 1

Question 1: Why do we define $R^2$ in such way?

This might be due to the fact that as $V_Y^2$ contains the error $\varepsilon$ then we want to minimize this value so that $R^2$ is maximized. For $\varepsilon=0$ we get ${V_\hat{Y}^2} = {V_Y^2}$, therefore $R^2=1$.

---

Question 2: Why do we assume $\bar{\varepsilon}=0$?

First, we will assume lineal regression.

This might be due to the fact that we know:

$$\bar{\varepsilon}=\sum_{i=1}^{N}\frac{\varepsilon_i}{N}$$

We also know that:

$$RSS=\sum_{i=1}^{N}\varepsilon_i^2=\sum_{i=1}^{N}(y_i-\hat{y}_i)^2$$

where $RSS$ is the sum of squared residuals.

From $\hat{y}_i=\hat{\beta}_0+\hat{\beta}_1x_1+\dots+\hat{\beta}_Nx_i$, we can rewrite it as:

$$RSS=\sum_{i=1}^{N}(y_i-(\hat{\beta}_0+\hat{\beta}_1x_1+\dots+\hat{\beta}_Nx_i))^2$$

We must take the partial derivatives of $RSS$ with respect to $\hat{\beta}_k, \forall k\in[1,N]$, where $k$ is an integer, and set them to zero to minimize the RSS.

For $\hat{\beta_0}$:

$$\frac{\partial RSS}{\partial \hat{\beta_0}}=0$$

Therefore:

$$\frac{\partial RSS}{\partial \hat{\beta_0}}=-2\sum_{i=1}^{N}(y_i-(\hat{\beta}_0+\hat{\beta}_1x_1+\dots+\hat{\beta}_Nx_i))=0$$

$$\sum_{i=1}^{N}(y_i-(\hat{\beta}_0+\hat{\beta}_1x_1+\dots+\hat{\beta}_Nx_i)=0$$

$$\sum_{i=1}^{N}\varepsilon_i=0$$

Generalizing for $k$:

$$\frac{\partial RSS}{\partial \hat{\beta_k}} = 0$$

$$\frac{\partial RSS}{\partial \hat{\beta_k}} = -2\sum_{i=1}^{N} x_k \cdot(y_i-(\hat{\beta}_0+\hat{\beta}_1x_1+\dots+\hat{\beta}_kx_i))=0$$

$$\sum_{i=1}^{N} x_k \cdot(y_i-(\hat{\beta}_0+\hat{\beta}_1x_1+\dots+\hat{\beta}_kx_i))=0$$

We will assume $x_k$ is always a constant.

$$\sum_{i=1}^{N} x_k \cdot \varepsilon_i=0$$

$$x_k \cdot \sum_{i=1}^{N} \varepsilon_i=0$$

$$\sum_{i=1}^{N}\varepsilon_i=0$$

We obtain mathematically that the residuals must sum to zero, so:

$$\bar{\varepsilon}=\sum_{i=1}^{N}\frac{\varepsilon_i}{N}=\frac{1}{N}\sum_{i=1}^{N}\varepsilon_i=\frac{1}{N} \cdot 0 = 0$$