Consider the linear model $Y \sim N_n(X\beta, \sigma^2 I)$, where $X$ is a $n \times (k + 1)$ matrix of rank $k + 1 < n$. Let $\hat{\beta}$ and $\hat{\sigma}^2$ be the maximum likelihood estimators of $\beta$ and $\sigma^2$ respectively. Then which of the following statements are true?
Analysis of Maximum Likelihood Estimator Properties
The question concerns the properties of Maximum Likelihood Estimators (MLEs) \(\hat{\beta}\) and \(\hat{\sigma}^2\) for the linear model \(Y \sim N_n(X\beta, \sigma^2 I)\), where \(X\) is an \(n \times (k+1)\) matrix of rank \(k+1\).
Option A: Covariance of \(\hat{\beta}\)
The MLE for \(\beta\) is \(\hat{\beta} = (X'X)^{-1}X'Y\).
The correct covariance matrix is \(\text{Cov}(\hat{\beta}) = \sigma^2 (X'X)^{-1}\).
Option A states \(\text{Cov}(\hat{\beta}) = \sigma^2 X'X\), which is incorrect.
Option B: Independence of \(\hat{\beta}\) and \(\hat{\sigma}^2\)
A standard result for the normal linear model is that the MLE \(\hat{\beta}\) is statistically independent of the Residual Sum of Squares \(SSE = (Y-X\hat{\beta})'(Y-X\hat{\beta})\).
Since the MLE \(\hat{\sigma}^2 = \frac{1}{n} SSE\) is derived from SSE, \(\hat{\beta}\) and \(\hat{\sigma}^2\) are independently distributed.
Therefore, this statement is true.
Option C: Sufficiency of \(\hat{\sigma}^2\)
The pair of estimators \((\hat{\beta}, \hat{\sigma}^2)\) is jointly sufficient for the parameter pair \((\beta, \sigma^2)\).
However, \(\hat{\sigma}^2\) alone is not sufficient for \(\sigma^2\) when \(\beta\) is an unknown nuisance parameter.
Therefore, this statement is false.
Option D: \(\hat{\sigma}^2\) as a Quadratic Form
The MLE for \(\sigma^2\) is \(\hat{\sigma}^2 = \frac{1}{n} (Y - X\hat{\beta})'(Y - X\hat{\beta})\).
Define the projection matrix \(P_X = X(X'X)^{-1}X'\). Then \(Y - X\hat{\beta} = (I - P_X)Y\).
Substituting this, \(\hat{\sigma}^2 = \frac{1}{n} [(I - P_X)Y]' [(I - P_X)Y] = \frac{1}{n} Y'(I - P_X)'(I - P_X)Y\).
As \(I - P_X\) is symmetric and idempotent, \(\hat{\sigma}^2 = Y'AY\) where \(A = \frac{1}{n} (I - P_X)\).
The rank of \(A\) is \(\text{rank}(I - P_X) = n - \text{rank}(P_X)\).
Since \(\text{rank}(X) = k+1\), \(\text{rank}(P_X) = k+1\).
Consequently, the rank of \(A\) is \(n - (k+1) = n - k - 1\).
Therefore, this statement is true.
Conclusion
The analysis confirms that Options B and D are the true statements regarding the MLEs \(\hat{\beta}\) and \(\hat{\sigma}^2\).
Important Questions from Best Linear Unbiased Estimators
Consider the simple linear regression model $Y_i = \beta x_i + \epsilon_i, i = 1,2,..., n$, where $\beta > 0$, $\sum_{i=1}^n x_i^2 > 0$, and the uncorrelated errors $\epsilon_i, i = 1,2,..., n$, have zero mean and finite variance $\sigma^2 (> 0)$. Let $\tilde{\beta}_1 = \sum_{i=1}^n a_i^*Y_i$, where $a_i$'s minimize $E(\sum_{i=1}^n a_i Y_i - \beta)^2$ with respect to scalars $a_1, a_2,..., a_n$. Let $\tilde{\beta}_2$ be the ordinary least squares estimator of $\beta$. Which of the following statements are true?
Consider the model $y_i = i\beta + \varepsilon_i, i = 1, 2, 3$ where $\varepsilon_1, \varepsilon_2, \varepsilon_3$ are independent with mean $0$ and variance $\sigma^2, 2\sigma^2, 3\sigma^2$ respectively. Which of the following is the best linear unbiased estimate of $\beta$?
- Let $X_1, X_2, \ldots, X_n$ be a random sample from $f_\theta(x)$, a probability density function or a probability mass function. Define $s_n^2 = \frac{1}{n-1} \sum_{i=1}^n (X_i - \bar{X}_n)^2$, where $\bar{X}_n = \frac{1}{n} \sum_{i=1}^n X_i$. Then $s_n^2$ is unbiased for $\theta$ if
Let $\{X_n\}$ be a sequence of independent random variables where the distribution of $X_n$ is normal with mean $\mu$ and variance $n$ for $n = 1, 2, \ldots$ Define
$\bar{X}_n = \frac{\sum_{i=1}^n X_i}{n}$ and $S_n = \frac{\sum_{i=1}^n \frac{X_i}{i}}{\sum_{i=1}^n \frac{1}{i}}$.
Which of the following are true?
- Let $Y_1, Y_2, Y_3$ be uncorrelated observations with common variance $\sigma^2$ and expectations given by $\mathbb{E}(Y_1) = \beta_1, \mathbb{E}(Y_2) = \beta_2$ and $\mathbb{E}(Y_3) = \beta_1 + \beta_2,$ where $\beta_1, \beta_2$ are unknown parameters. The best linear unbiased estimator of $\beta_1 + \beta_2$ is
