Consider the multiple linear regression model $Y_i = \beta_1 x_{i1} + \dots + \beta_p x_{ip} + \epsilon_i$, with $E(\epsilon_i) = 0$, $Cov(\epsilon_i, \epsilon_k) = 0$, if $i \ne k$ and $Var(\epsilon_i) = \sigma^2$, for $i, k = 1, \dots, n$. If $\hat{y}_i$ is the least squares fit of $y_i$ and $\hat{\epsilon}_i$ is the corresponding estimated residual for $i = 1, \dots, n$, then which of the following statements are always correct?
To determine which of the given statements are always correct for the residuals and fitted values in a multiple linear regression model, we need to understand the properties of least squares estimates. Given the model:
\(Y_i = \beta_1 x_{i1} + \dots + \beta_p x_{ip} + \epsilon_i\), where \(\epsilon_i\) are the random errors satisfying \(E(\epsilon_i) = 0\) and \(Cov(\epsilon_i, \epsilon_k) = 0\) for \(i \ne k\) with \(Var(\epsilon_i) = \sigma^2\).
We examine the properties of the estimated residuals \(\hat{\epsilon}_i = y_i - \hat{y}_i\) and the least squares fit \(\hat{y}_i\):
- The sum of residuals is zero: \(\sum_i \hat{\epsilon}_i = 0\). This is a fundamental property of least squares estimation which ensures that the model fits the data's mean value.
- The residuals are orthogonal to the fitted values: \(\sum_i \hat{y}_i \hat{\epsilon}_i = 0\). This means that the residuals have no linear relationship with the predicted values, which ensures that the best possible linear fit is achieved.
- The residuals are orthogonal to the predictors: \(\sum_i x_{ij} \hat{\epsilon}_i = 0\) for all \(j = 1, \dots, p\). This ensures that the predictions are unbiased and that each predictor contributes independently to explaining variability in the response variable.
Based on these properties, we can conclude the following statements are always correct:
- \(\sum_i x_{ij} \hat{\epsilon}_i = 0; \text{ for all } j = 1, \dots, p\). This statement is correct because the residuals have zero correlation with each predictor, ensuring an unbiased fit of the model.
- \(\sum_i \hat{y}_i \hat{\epsilon}_i = 0\). This statement is correct based on the orthogonality between the fitted values and residuals.
Thus, in conclusion, the correct options are:
\(\sum_i x_{ij} \hat{\epsilon}_i = 0; \text{ for all } j = 1, \dots, p\).
\(\sum_i \hat{y}_i \hat{\epsilon}_i = 0\).
Important Questions from Gauss-Markov Models
- Let $X_1, X_2,..., X_n$ ($n > 5$) be independent random variables such that
$X_t = \alpha + \alpha^2 t + \epsilon_t$, for $t = 1,..., n$,
where, $\epsilon_1, \epsilon_2,..., \epsilon_n$ are independent and identically distributed $N(0, \sigma^2)$ random variables. Here $\alpha \in R$ and $\sigma > 0$ are unknown parameters. Which of the following statements are true? Consider the linear statistical model
$y_{ij} = \mu + \tau_i + \varepsilon_{ij}; \quad i = 1, 2, \cdots, a; \quad j = 1, 2, \cdots, n$
where $\mu$ is unknown, $\tau_i$ are independently and identically distributed as $N(0, \sigma_\tau^2)$, $\varepsilon_{ij}$ are independently and identically distributed as $N(0, \sigma^2)$; $\tau_i$ and $\varepsilon_{ij}$ are independent for all $i$ and $j$. Note that $\tau_i$ is the $i^{th}$ treatment effect. Suppose $SS_{total}, SS_{treatment}, SS_{error}$ are total sum of squares, total treatment sum of squares and error sum of squares, respectively. To test :$H_0 : \sigma_\tau^2 = 0$ vs $H_A : \sigma_\tau^2 > 0$ which of the following statements is not true?
Consider the model $Y = X\beta + \epsilon$, where $Y = \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix}, X = \left( (x_{ij}) \right)_{n \times p}$ and $\beta = \begin{bmatrix} \beta_1 \\ \vdots \\ \beta_p \end{bmatrix}, \epsilon = \begin{bmatrix} \epsilon_1 \\ \epsilon_2 \\ \vdots \\ \epsilon_n \end{bmatrix}$. $E(\epsilon) = 0$ and $D(\epsilon) = \sigma^2 I_n, p < n$. Let $\hat{\beta}$ be the solution of $X^T X \beta = X^T Y$. Which of the following are true?
Let $X_1, X_2, \ldots$ be a random sample from uniform $(0, 3\theta), \theta > 0$. Define
$T_n = \frac{1}{3} \max\{X_1, X_2, \ldots, X_n\}$. Which of the following is NOT true?
- Consider the following regression problem $y_i = \beta_1 e^i + \beta_2 e^{-i} + \epsilon_i; \ i = 1, \ldots, n$. Here $\epsilon_1, \ldots, \epsilon_n$ are i.i.d. $N(0, \sigma^2)$ random variables. If $\hat{\beta}_1$ and $\hat{\beta}_2$ are the least square estimators of $\beta_1$ and $\beta_2$, respectively, then which of the following statements are correct?
