Consider the simple linear regression model $Y_i = \beta x_i + \epsilon_i, i = 1,2,..., n$, where $\beta > 0$, $\sum_{i=1}^n x_i^2 > 0$, and the uncorrelated errors $\epsilon_i, i = 1,2,..., n$, have zero mean and finite variance $\sigma^2 (> 0)$. Let $\tilde{\beta}_1 = \sum_{i=1}^n a_i^*Y_i$, where $a_i$'s minimize $E(\sum_{i=1}^n a_i Y_i - \beta)^2$ with respect to scalars $a_1, a_2,..., a_n$. Let $\tilde{\beta}_2$ be the ordinary least squares estimator of $\beta$. Which of the following statements are true?
$Var(\tilde{\beta}_1) < Var(\tilde{\beta}_2), E(\tilde{\beta}_1 - \beta)^2 < E(\tilde{\beta}_2 - \beta)^2$
OLS Estimator Properties
The Ordinary Least Squares (OLS) estimator $\tilde{\beta}_2$ for the simple linear regression model $Y_i = \beta x_i + \epsilon_i$ (regression through the origin) is given by:
$ \tilde{\beta}_2 = \frac{\sum_{i=1}^n x_i Y_i}{\sum_{i=1}^n x_i^2} $
Key properties of $\tilde{\beta}_2$ under the given assumptions ($\epsilon_i$ uncorrelated, $E(\epsilon_i)=0$, $Var(\epsilon_i)=\sigma^2$, $\sum x_i^2 > 0$) are:
- Unbiasedness: $E(\tilde{\beta}_2) = \beta$.
- Variance: Let $S_x = \sum_{i=1}^n x_i^2$. Then, $Var(\tilde{\beta}_2) = \frac{\sigma^2}{S_x}$.
- Mean Squared Error (MSE): Since $\tilde{\beta}_2$ is unbiased, $MSE(\tilde{\beta}_2) = Var(\tilde{\beta}_2) = \frac{\sigma^2}{S_x}$.
BLIMMSE Estimator Characteristics
The estimator $\tilde{\beta}_1 = \sum_{i=1}^n a_i^* Y_i$ is defined such that the coefficients $a_i^*$ minimize the mean squared error $E(\sum_{i=1}^n a_i Y_i - \beta)^2$. This means $\tilde{\beta}_1$ is the Best Linear Minimum Mean Square Error (BLIMMSE) estimator of the form $\sum a_i Y_i$.
The minimum MSE achievable by estimators of this form is:
$ MSE(\tilde{\beta}_1) = E(\tilde{\beta}_1 - \beta)^2 = \frac{\beta^2 \sigma^2}{\sigma^2 + \beta^2 S_x} $
The variance of $\tilde{\beta}_1$ is:
$ Var(\tilde{\beta}_1) = \frac{\beta^4 \sigma^2 S_x}{(\sigma^2 + \beta^2 S_x)^2} $
Note that $\tilde{\beta}_1$ is generally a biased estimator.
Variance Comparison: $\tilde{\beta}_1$ vs $\tilde{\beta}_2$
We compare the variances:
- $Var(\tilde{\beta}_1) = \frac{\beta^4 \sigma^2 S_x}{(\sigma^2 + \beta^2 S_x)^2}$
- $Var(\tilde{\beta}_2) = \frac{\sigma^2}{S_x}$
The ratio of variances is:
$ \frac{Var(\tilde{\beta}_1)}{Var(\tilde{\beta}_2)} = \frac{\beta^4 \sigma^2 S_x}{(\sigma^2 + \beta^2 S_x)^2} \cdot \frac{S_x}{\sigma^2} = \frac{(\beta^2 S_x)^2}{(\sigma^2 + \beta^2 S_x)^2} = \left(\frac{\beta^2 S_x}{\sigma^2 + \beta^2 S_x}\right)^2 $
Since $\sigma^2 > 0$, we have $\beta^2 S_x < \sigma^2 + \beta^2 S_x$. Thus, $\frac{\beta^2 S_x}{\sigma^2 + \beta^2 S_x} < 1$, which implies $\left(\frac{\beta^2 S_x}{\sigma^2 + \beta^2 S_x}\right)^2 < 1$. Therefore:
$ Var(\tilde{\beta}_1) < Var(\tilde{\beta}_2) $
MSE Comparison: $\tilde{\beta}_1$ vs $\tilde{\beta}_2$
We compare the mean squared errors:
- $MSE(\tilde{\beta}_1) = \frac{\beta^2 \sigma^2}{\sigma^2 + \beta^2 S_x}$
- $MSE(\tilde{\beta}_2) = \frac{\sigma^2}{S_x}$
The ratio of MSEs is:
$ \frac{MSE(\tilde{\beta}_1)}{MSE(\tilde{\beta}_2)} = \frac{\beta^2 \sigma^2}{\sigma^2 + \beta^2 S_x} \cdot \frac{S_x}{\sigma^2} = \frac{\beta^2 S_x}{\sigma^2 + \beta^2 S_x} $
Since $\sigma^2 > 0$, we have $\beta^2 S_x < \sigma^2 + \beta^2 S_x$. Thus, $\frac{\beta^2 S_x}{\sigma^2 + \beta^2 S_x} < 1$. Therefore:
$ MSE(\tilde{\beta}_1) < MSE(\tilde{\beta}_2) $
This translates to $E(\tilde{\beta}_1 - \beta)^2 < E(\tilde{\beta}_2 - \beta)^2$.
Final Conclusion
Both comparisons show:
- $Var(\tilde{\beta}_1) < Var(\tilde{\beta}_2)$
- $E(\tilde{\beta}_1 - \beta)^2 < E(\tilde{\beta}_2 - \beta)^2$
These findings match the conditions stated in Option 2.
Important Questions from Best Linear Unbiased Estimators
Consider the model $y_i = i\beta + \varepsilon_i, i = 1, 2, 3$ where $\varepsilon_1, \varepsilon_2, \varepsilon_3$ are independent with mean $0$ and variance $\sigma^2, 2\sigma^2, 3\sigma^2$ respectively. Which of the following is the best linear unbiased estimate of $\beta$?
Consider the linear model $Y \sim N_n(X\beta, \sigma^2 I)$, where $X$ is a $n \times (k + 1)$ matrix of rank $k + 1 < n$. Let $\hat{\beta}$ and $\hat{\sigma}^2$ be the maximum likelihood estimators of $\beta$ and $\sigma^2$ respectively. Then which of the following statements are true?
- Let $X_1, X_2, \ldots, X_n$ be a random sample from $f_\theta(x)$, a probability density function or a probability mass function. Define $s_n^2 = \frac{1}{n-1} \sum_{i=1}^n (X_i - \bar{X}_n)^2$, where $\bar{X}_n = \frac{1}{n} \sum_{i=1}^n X_i$. Then $s_n^2$ is unbiased for $\theta$ if
Let $\{X_n\}$ be a sequence of independent random variables where the distribution of $X_n$ is normal with mean $\mu$ and variance $n$ for $n = 1, 2, \ldots$ Define
$\bar{X}_n = \frac{\sum_{i=1}^n X_i}{n}$ and $S_n = \frac{\sum_{i=1}^n \frac{X_i}{i}}{\sum_{i=1}^n \frac{1}{i}}$.
Which of the following are true?
- Let $Y_1, Y_2, Y_3$ be uncorrelated observations with common variance $\sigma^2$ and expectations given by $\mathbb{E}(Y_1) = \beta_1, \mathbb{E}(Y_2) = \beta_2$ and $\mathbb{E}(Y_3) = \beta_1 + \beta_2,$ where $\beta_1, \beta_2$ are unknown parameters. The best linear unbiased estimator of $\beta_1 + \beta_2$ is
