Regression, \{(Y_i, X_i) : i = 1, \ldots, n\} iid: Y_i = X_i^\top \beta + U_i, \qquad \underset{k\times1}{X_i} = \begin{pmatrix} X_{i,1} \\ \vdots \\ X_{i,k} \end{pmatrix}

OLS from n observations, \operatorname{rank}\left(X\right) = k (Lecture 1); substitute Y_i = X_i^\top \beta + U_i: \begin{aligned} \hat\beta_n &= (X^\top X)^{-1} X^\top Y \\ &= \Big(\underbrace{\sum_{i=1}^n X_i X_i^\top}_{k\times k}\Big)^{-1} \sum_{i=1}^n X_i Y_i \\ &= \Big(\sum_{i=1}^n X_i X_i^\top\Big)^{-1} \sum_{i=1}^n X_i \big(X_i^\top \beta + U_i\big) \\ &= \Big(\sum_{i=1}^n X_i X_i^\top\Big)^{-1} \Big(\sum_{i=1}^n X_i X_i^\top\Big) \beta + \Big(\sum_{i=1}^n X_i X_i^\top\Big)^{-1} \sum_{i=1}^n X_i U_i \\ &= \beta + \Big(\sum_{i=1}^n X_i X_i^\top\Big)^{-1} \sum_{i=1}^n X_i U_i \\ &= \beta + \Big({\color{red}\frac{1}{n}} \sum_{i=1}^n X_i X_i^\top\Big)^{-1} {\color{red}\frac{1}{n}} \sum_{i=1}^n X_i U_i \end{aligned}

Squares and cross-products: \underset{k\times k}{X_i X_i^\top} = \begin{pmatrix} X_{i,1}^2 & X_{i,1} X_{i,2} & \cdots & X_{i,1} X_{i,k} \\ X_{i,2} X_{i,1} & X_{i,2}^2 & \cdots & X_{i,2} X_{i,k} \\ \vdots & \vdots & \ddots & \vdots \\ X_{i,k} X_{i,1} & X_{i,k} X_{i,2} & \cdots & X_{i,k}^2 \end{pmatrix}

Element (r, s): \Big[\sum_{i=1}^n X_i X_i^\top\Big]_{rs} = \sum_{i=1}^n X_{i,r} X_{i,s}

X_1 U_1, \ldots, X_n U_n: iid; weak LLN, element by element: \frac{1}{n} \sum_{i=1}^n X_i U_i \to_p \mathrm{E}\left[X_i U_i\right]

Assume: \mathrm{E}\left[X_i U_i\right] = 0

Weak LLN for element j requires: \mathrm{E}\left[|X_{i,j} U_i|\right] < \infty, \qquad j = 1, \ldots, k

Cauchy-Schwarz inequality: \mathrm{E}\left[|X_{i,j} U_i|\right] \le \sqrt{\mathrm{E}\left[X_{i,j}^2\right]\, \mathrm{E}\left[U_i^2\right]}

Assume: \mathrm{E}\left[U_i^2\right] < \infty, \qquad \mathrm{E}\left[X_{i,j}^2\right] < \infty, \quad j = 1, \ldots, k

Element (r, s) of \mathrm{E}\left[X_i X_i^\top\right] well defined, Cauchy-Schwarz: \mathrm{E}\left[|X_{i,r} X_{i,s}|\right] \le \sqrt{\mathrm{E}\left[X_{i,r}^2\right]\, \mathrm{E}\left[X_{i,s}^2\right]} < \infty

Weak LLN, element by element: \frac{1}{n} \sum_{i=1}^n X_i X_i^\top \to_p \mathrm{E}\left[X_i X_i^\top\right]

\mathrm{E}\left[X_i X_i^\top\right]: symmetric, p.s.d.

Assume: \mathrm{E}\left[X_i X_i^\top\right] \ \text{ p.d.}

Slutsky, h(\cdot) = (\cdot)^{-1} continuous at a p.d. matrix (Lecture 8): \Big(\frac{1}{n} \sum_{i=1}^n X_i X_i^\top\Big)^{-1} \to_p \big(\mathrm{E}\left[X_i X_i^\top\right]\big)^{-1}

Assumptions for consistency: \begin{aligned} & \{(Y_i, X_i) : i = 1, \ldots, n\} \ \text{ iid} \\ & Y_i = X_i^\top \beta + U_i \\ & \mathrm{E}\left[X_i U_i\right] = 0 \\ & \mathrm{E}\left[U_i^2\right] < \infty \\ & \mathrm{E}\left[X_{i,j}^2\right] < \infty, \quad j = 1, \ldots, k \\ & \mathrm{E}\left[X_i X_i^\top\right] \ \text{ p.d. (no perfect multicollinearity)} \end{aligned}

U_i = Y_i - X_i^\top \beta: \{(Y_i, X_i)\} \ \text{ iid} \quad \Longrightarrow \quad \{(U_i, X_i)\} \ \text{ iid}

(Y_1, X_1) independent of (Y_2, X_2); Y_1 and X_1 need not be independent.

Weaker than \mathrm{E}\left[U_i \mid X_i\right] = 0: \mathrm{E}\left[X_i U_i\right] = 0 \quad \nRightarrow \quad \mathrm{E}\left[Y_i \mid X_i\right] = X_i^\top \beta \quad \text{(in general)}

Definition. \hat\beta_n is a consistent estimator of \beta if \hat\beta_n \to_p \beta.

Element by element, by definition (Lecture 8): \hat\beta_n \to_p \beta \quad \Longleftrightarrow \quad \hat\beta_{n,j} \to_p \beta_j, \quad j = 1, \ldots, k

Theorem. Under the assumptions for consistency, \hat\beta_n \to_p \beta.

The two limits, then Slutsky for the product: \begin{aligned} \hat\beta_n &= \beta + \Big(\underbrace{\frac{1}{n} \sum_{i=1}^n X_i X_i^\top}_{\to_p\, \mathrm{E}\left[X_i X_i^\top\right]}\Big)^{-1} \underbrace{\frac{1}{n} \sum_{i=1}^n X_i U_i}_{\to_p\, 0} \\ &\to_p \beta + \big(\mathrm{E}\left[X_i X_i^\top\right]\big)^{-1} \cdot 0 \\ &= \beta \end{aligned}

Therefore: \boxed{\ \hat\beta_n \to_p \beta\ }

\hat\beta_n - \beta \to_p 0. Approximate the distribution of \hat\beta_n - \beta: scale by \sqrt{n} (Lecture 8).

Assumption I. \begin{aligned} & \{(Y_i, X_i) : i = 1, \ldots, n\} \ \text{ iid} \\ & Y_i = X_i^\top \beta + U_i, \qquad \beta \in \mathbb{R}^k \end{aligned}

OLS: \begin{aligned} \hat\beta_n &= \Big(\sum_{i=1}^n X_i X_i^\top\Big)^{-1} \sum_{i=1}^n X_i Y_i \\ &= \beta + \Big(\sum_{i=1}^n X_i X_i^\top\Big)^{-1} \sum_{i=1}^n X_i U_i \\ \Longrightarrow\quad \hat\beta_n - \beta &= \Big(\sum_{i=1}^n X_i X_i^\top\Big)^{-1} \sum_{i=1}^n X_i U_i \end{aligned}

Assumption II. \underset{k\times k}{Q} = \mathrm{E}\left[X_i X_i^\top\right] \quad \text{p.d. and finite}

Weak LLN, then Slutsky: \begin{aligned} \frac{1}{n} \sum_{i=1}^n X_i X_i^\top &\to_p \mathrm{E}\left[X_i X_i^\top\right] = Q \\ \Big(\frac{1}{n} \sum_{i=1}^n X_i X_i^\top\Big)^{-1} &\to_p Q^{-1} = \big(\mathrm{E}\left[X_i X_i^\top\right]\big)^{-1} \end{aligned}

Assumption III. \begin{aligned} & \mathrm{E}\left[X_i U_i\right] = 0 \quad \text{($X$'s and $U$'s uncorrelated)} \\ & \underset{k\times k}{\Omega} = \mathrm{Var}\left(X_i U_i\right) \quad \text{finite and p.d.} \end{aligned}

Variance of X_i U_i, with \mathrm{E}\left[X_i U_i\right] = 0: \begin{aligned} \Omega &= \mathrm{Var}\left(X_i U_i\right) \\ &= \mathrm{E}\left[X_i U_i (X_i U_i)^\top\right] \\ &= \mathrm{E}\left[X_i \underset{1\times 1}{U_i}\, \underset{1\times 1}{U_i} X_i^\top\right] \\ &= \mathrm{E}\left[U_i^2 X_i X_i^\top\right] \end{aligned}

X_1 U_1, \ldots, X_n U_n: iid k \times 1 (Assumption I), mean 0, variance \Omega finite and p.d. (Assumption III).

Multivariate CLT (Lecture 8): \frac{1}{\sqrt{n}} \sum_{i=1}^n X_i U_i \to_d N\big(0,\ \underbrace{\mathrm{E}\left[U_i^2 X_i X_i^\top\right]}_{\Omega}\big)

We will use, without proof (Cramér convergence theorem): A_n \to_p A, A non-random, and Z_n \to_d Z: A_n Z_n \to_d A Z

The two limits: {\color{red}\sqrt{n}}(\hat\beta_n - \beta) = \underbrace{\Big({\color{red}\frac{1}{n}} \sum_{i=1}^n X_i X_i^\top\Big)^{-1}}_{\to_p\, Q^{-1}}\ \underbrace{{\color{red}\frac{1}{\sqrt{n}}} \sum_{i=1}^n X_i U_i}_{\to_d\, N(0,\, \Omega)} \to_d Q^{-1} N(0, \Omega)

Linear map of a normal vector (Lecture 4): W \sim N(\mu, \Sigma) \quad \Longrightarrow \quad \Gamma W \sim N(\Gamma \mu,\ \Gamma \Sigma \Gamma^\top)

\Gamma = Q^{-1}; Q = \mathrm{E}\left[X_i X_i^\top\right] = Q^\top, so (Q^{-1})^\top = Q^{-1}: \begin{aligned} Q^{-1} N(0, \Omega) &\overset{d}{=} N\big(Q^{-1} \cdot 0,\ Q^{-1} \Omega (Q^{-1})^\top\big) \\ &= N\big(0,\ Q^{-1} \Omega Q^{-1}\big) \end{aligned}

Theorem. Under Assumptions I to III: \begin{aligned} & \boxed{\, \sqrt{n}(\hat\beta_n - \beta) \to_d N\Big(0,\, \underbrace{\big(\mathrm{E}\left[X_i X_i^\top\right]\big)^{-1} \mathrm{E}\left[U_i^2 X_i X_i^\top\right] \big(\mathrm{E}\left[X_i X_i^\top\right]\big)^{-1}}_{\mathrm{AsyVar}(\hat\beta_n)}\Big)\, } \\ & \mathrm{AsyVar}(\hat\beta_n) = Q^{-1} \Omega Q^{-1} \quad \text{(sandwich)} \end{aligned}

CDF form, for all a \ne 0 and t \in \mathbb{R}: \mathrm{P}\big(a^\top \sqrt{n}(\hat\beta_n - \beta) \le t\big) \to \Phi\Big(\frac{t}{\sqrt{a^\top Q^{-1} \Omega Q^{-1} a}}\Big)

a = e_j, element j: \sqrt{n}(\hat\beta_{n,j} - \beta_j) \to_d N\big(0,\ [Q^{-1} \Omega Q^{-1}]_{jj}\big)

Suppose: \mathrm{E}\left[U_i^2 \mid X_i\right] = \sigma^2 \quad \text{(homoskedasticity)}

Law of iterated expectations: \begin{aligned} \Omega &= \mathrm{E}\left[U_i^2 X_i X_i^\top\right] \\ &= \mathrm{E}\left[\mathrm{E}\left[U_i^2 X_i X_i^\top \mid X_i\right]\right] \\ &= \mathrm{E}\left[\underbrace{\mathrm{E}\left[U_i^2 \mid X_i\right]}_{=\,\sigma^2} X_i X_i^\top\right] \\ &= \sigma^2 \mathrm{E}\left[X_i X_i^\top\right] \\ &= \sigma^2 Q \end{aligned}

The sandwich collapses: \begin{aligned} \mathrm{AsyVar}(\hat\beta_n) &= Q^{-1} \Omega Q^{-1} \\ &= Q^{-1} (\sigma^2 Q) Q^{-1} \\ &= \sigma^2 Q^{-1} \end{aligned}

Therefore: \boxed{\ \sqrt{n}(\hat\beta_n - \beta) \to_d N\Big(0,\ \sigma^2 \big(\mathrm{E}\left[X_i X_i^\top\right]\big)^{-1}\Big)\ }

Normal errors (Lecture 4, Assumptions 1 to 5), exact for every n (requires stronger assumptions), a \ne 0: \begin{aligned} & Y = X\beta + U, \qquad U \mid X \sim N(0, \sigma^2 I_n) \\ \Longrightarrow\quad & \hat\beta_n \mid X \sim N\big(\beta,\ \sigma^2 (X^\top X)^{-1}\big) \\ \Longrightarrow\quad & \mathrm{P}\big(a^\top (\hat\beta_n - \beta) \le t \mid X\big) = \Phi\Big(\frac{t}{\sqrt{\sigma^2 a^\top (X^\top X)^{-1} a}}\Big) \end{aligned}

Homoskedastic errors, large sample, a \ne 0: \begin{aligned} & \text{Assumptions I and II}, \quad \mathrm{E}\left[X_i U_i\right] = 0, \quad \mathrm{E}\left[U_i^2 \mid X_i\right] = \sigma^2 > 0 \\ \Longrightarrow\quad & \mathrm{P}\big(a^\top \sqrt{n}(\hat\beta_n - \beta) \le t\big) \to \Phi\Big(\frac{t}{\sqrt{\sigma^2 a^\top \big(\mathrm{E}\left[X_i X_i^\top\right]\big)^{-1} a}}\Big) \end{aligned}

General, large sample, a \ne 0: \begin{aligned} & \text{Assumptions I to III}; \quad Q = \mathrm{E}\left[X_i X_i^\top\right], \quad \Omega = \mathrm{E}\left[U_i^2 X_i X_i^\top\right] \\ \Longrightarrow\quad & \mathrm{P}\big(a^\top \sqrt{n}(\hat\beta_n - \beta) \le t\big) \to \Phi\Big(\frac{t}{\sqrt{a^\top Q^{-1} \Omega Q^{-1} a}}\Big) \end{aligned}

Unknown: \begin{aligned} V &= \mathrm{AsyVar}(\hat\beta_n) = Q^{-1} \Omega Q^{-1} \\ Q &= \mathrm{E}\left[X_i X_i^\top\right], \qquad \Omega = \mathrm{E}\left[U_i^2 X_i X_i^\top\right] \end{aligned}

Estimator, sample averages in place of expectations: \hat V_n = \hat Q_n^{-1} \hat\Omega_n \hat Q_n^{-1}

\hat Q_n: weak LLN, then Slutsky (Assumption II): \begin{aligned} \hat Q_n &= \frac{1}{n} \sum_{i=1}^n X_i X_i^\top \to_p \mathrm{E}\left[X_i X_i^\top\right] = Q \\ \hat Q_n^{-1} &\to_p Q^{-1} \end{aligned}

\hat\Omega_n, residuals \hat U_i in place of U_i: \hat\Omega_n = \frac{1}{n} \sum_{i=1}^n \hat U_i^2 X_i X_i^\top

Residual and error, i fixed (continuous mapping theorem, without proof): \begin{aligned} \hat U_i &= Y_i - X_i^\top \hat\beta_n \\ &= X_i^\top \beta + U_i - X_i^\top \hat\beta_n \\ &= U_i + X_i^\top \underbrace{(\beta - \hat\beta_n)}_{\to_p\, 0} \\ \Longrightarrow\quad \hat U_i &\to_p U_i \\ \Longrightarrow\quad \hat U_1^2 + \hat U_2^2 &\to_p U_1^2 + U_2^2 \end{aligned}

Definition. Small o_p: Z_n = o_p(1) \quad \Longleftrightarrow \quad Z_n \to_p 0

One o_p(1) term for each i, \hat U_i = U_i + o_p^i(1): \begin{aligned} \hat\Omega_n &= \frac{1}{n} \sum_{i=1}^n \hat U_i^2 X_i X_i^\top \\ &= \frac{1}{n} \sum_{i=1}^n \big(U_i + o_p^i(1)\big)^2 X_i X_i^\top \\ &\overset{?}{=} \frac{1}{n} \sum_{i=1}^n U_i^2 X_i X_i^\top + o_p(1) \end{aligned}

\sum_{i=1}^n o_p^i(1) can be large.

Weak LLN? Summands not iid: \hat U_i = U_i - X_i^\top \underbrace{(\hat\beta_n - \beta)}_{\text{random}}

\hat U_1 and \hat U_2: correlated via \hat\beta_n.

To show: \hat\Omega_n = \frac{1}{n} \sum_{i=1}^n \hat U_i^2 X_i X_i^\top \to_p \mathrm{E}\left[U_i^2 X_i X_i^\top\right] = \Omega, \qquad \underset{k\times k}{\Omega}, \ \ \underset{k\times k}{X_i X_i^\top}

Frobenius norm, \underset{k\times k}{A} (Euclidean norm of the k^2 elements): \begin{aligned} \|A\| &= \sqrt{\sum_{r=1}^k \sum_{s=1}^k A_{rs}^2} \\ &= \sqrt{\operatorname{tr}(A^\top A)} \end{aligned}

Element by element \Longleftrightarrow in norm (Lecture 8); to show: \|\underbrace{\hat\Omega_n - \Omega}_{k\times k}\| \to_p 0

Substitute \hat U_i = U_i - X_i^\top(\hat\beta_n - \beta); square, (a - b)^2 = a^2 + b^2 - 2ab: \begin{aligned} \hat\Omega_n &= \frac{1}{n} \sum_{i=1}^n \hat U_i^2 X_i X_i^\top \\ &= \frac{1}{n} \sum_{i=1}^n \big(\underbrace{Y_i - X_i^\top \hat\beta_n}_{\hat U_i}\big)^2 X_i X_i^\top \\ &= \frac{1}{n} \sum_{i=1}^n \big(U_i - \underbrace{X_i^\top (\hat\beta_n - \beta)}_{1\times1}\big)^2 X_i X_i^\top \\ &= \frac{1}{n} \sum_{i=1}^n \Big(U_i^2 + \big(X_i^\top (\hat\beta_n - \beta)\big)^2 - 2\, U_i X_i^\top (\hat\beta_n - \beta)\Big) X_i X_i^\top \\ &= \frac{1}{n} \sum_{i=1}^n U_i^2 X_i X_i^\top + \frac{1}{n} \sum_{i=1}^n \big(X_i^\top (\hat\beta_n - \beta)\big)^2 X_i X_i^\top \\ &\qquad - 2\, \frac{1}{n} \sum_{i=1}^n \big(U_i X_i^\top (\hat\beta_n - \beta)\big) X_i X_i^\top \end{aligned}

First term, weak LLN, \Omega finite (Assumption III): \frac{1}{n} \sum_{i=1}^n U_i^2 X_i X_i^\top \to_p \mathrm{E}\left[U_i^2 X_i X_i^\top\right] = \Omega

To show: the other two terms \to_p 0.

Cauchy-Schwarz: \big|X_i^\top (\hat\beta_n - \beta)\big| \le \|X_i\|\, \|\hat\beta_n - \beta\|

\underset{k\times k}{A} = X_i X_i^\top = A^\top; \operatorname{tr}(BC) = \operatorname{tr}(CB): \begin{aligned} \|X_i X_i^\top\| &= \sqrt{\operatorname{tr}\big(\underbrace{X_i X_i^\top}_{A^\top}\ \underbrace{X_i X_i^\top}_{A}\big)} \\ &= \sqrt{\operatorname{tr}\big(X_i \underbrace{X_i^\top X_i}_{=\,\|X_i\|^2} X_i^\top\big)} \\ &= \sqrt{\operatorname{tr}\big(X_i \underbrace{\|X_i\|^2}_{1\times1} X_i^\top\big)} \\ &= \sqrt{\|X_i\|^2 \operatorname{tr}\big(X_i X_i^\top\big)} \\ &= \|X_i\| \sqrt{\operatorname{tr}\big(X_i X_i^\top\big)} \\ &= \|X_i\| \sqrt{\operatorname{tr}\big(X_i^\top X_i\big)} \\ &= \|X_i\| \sqrt{\|X_i\|^2} \\ &= \|X_i\|^2 \end{aligned}

Cauchy-Schwarz and \|X_i X_i^\top\| = \|X_i\|^2: \begin{aligned} \underbrace{\Big\|\frac{1}{n} \sum_{i=1}^n \big(U_i X_i^\top (\hat\beta_n - \beta)\big) X_i X_i^\top\Big\|}_{1\times1} &\le \frac{1}{n} \sum_{i=1}^n |U_i|\, \big|X_i^\top (\hat\beta_n - \beta)\big|\, \|X_i X_i^\top\| \\ &\le \frac{1}{n} \sum_{i=1}^n \underset{1\times1}{|U_i|}\, \underset{1\times1}{\|X_i\|}\, \underset{1\times1}{\|\hat\beta_n - \beta\|}\, \underset{1\times1}{\|X_i\|^2} \\ &= \Big(\underbrace{\frac{1}{n} \sum_{i=1}^n |U_i|\, \|X_i\|^3}_{\to_p\, \mathrm{E}\left[|U_i|\, \|X_i\|^3\right]}\Big) \cdot \underbrace{\|\hat\beta_n - \beta\|}_{\to_p\, 0} \end{aligned}

Weak LLN, provided \mathrm{E}\left[|U_i|\, \|X_i\|^3\right] < \infty. Cauchy-Schwarz twice: \begin{aligned} \mathrm{E}\left[|U_i|\, \|X_i\|^3\right] &= \mathrm{E}\left[|U_i|\, \|X_i\| \cdot \|X_i\|^2\right] \\ &\le \sqrt{\mathrm{E}\left[U_i^2 \|X_i\|^2\right]\, \mathrm{E}\left[\|X_i\|^4\right]} \\ &\le \sqrt{\sqrt{\mathrm{E}\left[U_i^4\right]\, \mathrm{E}\left[\|X_i\|^4\right]} \cdot \mathrm{E}\left[\|X_i\|^4\right]} \end{aligned}

Assume: \mathrm{E}\left[U_i^4\right] < \infty, \qquad \mathrm{E}\left[\|X_i\|^4\right] < \infty

Slutsky, bound \to_p 0; non-negative norm: \Big\|\frac{1}{n} \sum_{i=1}^n \big(U_i X_i^\top (\hat\beta_n - \beta)\big) X_i X_i^\top\Big\| \to_p 0

Triangle inequality and \|cA\| = |c|\, \|A\|, then Cauchy-Schwarz and \|X_i X_i^\top\| = \|X_i\|^2: \begin{aligned} \Big\|\frac{1}{n} \sum_{i=1}^n \big(\underbrace{X_i^\top (\hat\beta_n - \beta)}_{1\times1}\big)^2 X_i X_i^\top\Big\| &\le \frac{1}{n} \sum_{i=1}^n \big(X_i^\top (\hat\beta_n - \beta)\big)^2\, \|X_i X_i^\top\| \\ &\le \frac{1}{n} \sum_{i=1}^n \|X_i\|^2\, \|\hat\beta_n - \beta\|^2\, \|X_i\|^2 \\ &= \Big(\underbrace{\frac{1}{n} \sum_{i=1}^n \|X_i\|^4}_{\to_p\, \mathrm{E}\left[\|X_i\|^4\right] < \infty}\Big) \cdot \underbrace{\|\hat\beta_n - \beta\|^2}_{\to_p\, 0} \end{aligned}

Slutsky, bound \to_p 0; non-negative norm: \Big\|\frac{1}{n} \sum_{i=1}^n \big(X_i^\top (\hat\beta_n - \beta)\big)^2 X_i X_i^\top\Big\| \to_p 0

Three terms: \begin{aligned} \hat\Omega_n &= \frac{1}{n} \sum_{i=1}^n \hat U_i^2 X_i X_i^\top \\ &= \frac{1}{n} \sum_{i=1}^n U_i^2 X_i X_i^\top + \underbrace{\frac{1}{n} \sum_{i=1}^n \big(X_i^\top (\hat\beta_n - \beta)\big)^2 X_i X_i^\top}_{{\color{red}\to_p\, 0}} \\ &\qquad - 2\, \underbrace{\frac{1}{n} \sum_{i=1}^n \big(U_i X_i^\top (\hat\beta_n - \beta)\big) X_i X_i^\top}_{{\color{red}\to_p\, 0}} \\ &= \underbrace{\frac{1}{n} \sum_{i=1}^n U_i^2 X_i X_i^\top}_{\to_p\, \mathrm{E}\left[U_i^2 X_i X_i^\top\right] = \Omega} {}+ o_p(1) \\ &\to_p \Omega \end{aligned}

Theorem. Under Assumptions I to III, \mathrm{E}\left[U_i^4\right] < \infty and \mathrm{E}\left[\|X_i\|^4\right] < \infty, with or without homoskedasticity: \boxed{\ \hat V_n = \hat Q_n^{-1} \hat\Omega_n \hat Q_n^{-1} \to_p Q^{-1} \Omega Q^{-1} = V\ } \quad \text{(Slutsky, $\hat Q_n^{-1} \to_p Q^{-1}$)}

\overset{a}{\sim}: approximately distributed, for large n: \sqrt{n}(\hat\beta_n - \beta) \overset{a}{\sim} N(0, V) \quad \Longrightarrow \quad \hat\beta_{n,j} \overset{a}{\sim} N\Big(\beta_j,\ \frac{[V]_{jj}}{n}\Big)

Definition. Robust (heteroskedasticity-consistent, White) standard error; {\hat Q_n = \frac{1}{n} X^\top X}: \begin{aligned} & \text{std. err}_{\text{rob}}\big(\hat\beta_{n,j}\big) \\ &= \sqrt{[\hat V_n]_{jj} / n} \\ &= \sqrt{\Big[\frac{1}{n} \Big(\frac{1}{n} X^\top X\Big)^{-1} \Big(\frac{1}{n} \sum_{i=1}^n \hat U_i^2 X_i X_i^\top\Big) \Big(\frac{1}{n} X^\top X\Big)^{-1}\Big]_{jj}} \\ &= \sqrt{\Big[\frac{1}{n} \cdot n (X^\top X)^{-1} \cdot \frac{1}{n} \sum_{i=1}^n \hat U_i^2 X_i X_i^\top \cdot n (X^\top X)^{-1}\Big]_{jj}} \\ &= \sqrt{\Big[(X^\top X)^{-1} \Big(\sum_{i=1}^n \hat U_i^2 X_i X_i^\top\Big) (X^\top X)^{-1}\Big]_{jj}} \end{aligned}

Lecture 2, heteroskedastic errors, \mathrm{Var}\left(U_i \mid X\right) = \sigma_i^2; \hat U_i^2 in place of \sigma_i^2: \mathrm{Var}\left(\hat\beta \mid X\right) = (X^\top X)^{-1} \Big(\sum_{i=1}^n \sigma_i^2 X_i X_i^\top\Big) (X^\top X)^{-1}

We will use, without proof (Cramér convergence theorem): Z_n \to_d Z, {c_n \to_p c \ne 0}, c non-random: \frac{Z_n}{c_n} \to_d \frac{Z}{c}

Slutsky, \sqrt{[\hat V_n]_{jj}} \to_p \sqrt{[V]_{jj}} > 0 (V p.d.); \sqrt{n}(\hat\beta_{n,j} - \beta_j) \to_d N(0, [V]_{jj}): \begin{aligned} \frac{\hat\beta_{n,j} - \beta_j}{\text{std. err}_{\text{rob}}\big(\hat\beta_{n,j}\big)} &= \frac{\hat\beta_{n,j} - \beta_j}{\sqrt{[\hat V_n]_{jj} / n}} \\ &= \frac{\sqrt{n}(\hat\beta_{n,j} - \beta_j)}{\sqrt{[\hat V_n]_{jj}}} \\ &\to_d \frac{N\big(0, [V]_{jj}\big)}{\sqrt{[V]_{jj}}} \\ &\overset{d}{=} N(0, 1) \end{aligned}

Asymptotic confidence interval: \begin{aligned} CI_{1-\alpha} = \Big[ & \hat\beta_{n,j} - z_{1-\alpha/2}\, \text{std. err}_{\text{rob}}\big(\hat\beta_{n,j}\big), \\ & \hat\beta_{n,j} + z_{1-\alpha/2}\, \text{std. err}_{\text{rob}}\big(\hat\beta_{n,j}\big)\Big] \end{aligned}

Coverage, t-ratio \to_d Z \sim N(0, 1): \begin{aligned} \mathrm{P}\big(\beta_j \in CI_{1-\alpha}\big) &= \mathrm{P}\Big(\Big|\frac{\hat\beta_{n,j} - \beta_j}{\text{std. err}_{\text{rob}}\big(\hat\beta_{n,j}\big)}\Big| \le z_{1-\alpha/2}\Big) \\ &\to \mathrm{P}\big(|Z| \le z_{1-\alpha/2}\big) \\ &= 1 - \alpha \end{aligned}

Lecture 3: \text{std. err}\big(\hat\beta_{n,j}\big) = \sqrt{s^2 \big[(X^\top X)^{-1}\big]_{jj}}, \qquad s^2 = \frac{1}{n - k} \sum_{i=1}^n \hat U_i^2

In terms of \hat Q_n = \frac{1}{n} X^\top X: \begin{aligned} s^2 (X^\top X)^{-1} &= \frac{1}{n}\, s^2 \Big(\frac{1}{n} X^\top X\Big)^{-1} \\ &= \frac{1}{n}\, s^2 \hat Q_n^{-1} \end{aligned}

Usual: s^2 \hat Q_n^{-1}, built for \sigma^2 Q^{-1} (homoskedasticity); robust: \hat V_n, for V: \text{std. err}\big(\hat\beta_{n,j}\big) = \sqrt{[s^2 \hat Q_n^{-1}]_{jj} / n}, \qquad \text{std. err}_{\text{rob}}\big(\hat\beta_{n,j}\big) = \sqrt{[\hat V_n]_{jj} / n}

\hat U_i = U_i - X_i^\top(\hat\beta_n - \beta); U_i X_i^\top (\hat\beta_n - \beta) = (\hat\beta_n - \beta)^\top X_i U_i; {\mathrm{E}\left[U_i^2\right] \le \sqrt{\mathrm{E}\left[U_i^4\right]} < \infty}; weak LLN, consistency, Slutsky: \begin{aligned} \frac{1}{n} \sum_{i=1}^n \hat U_i^2 &= \frac{1}{n} \sum_{i=1}^n \big(U_i - X_i^\top (\hat\beta_n - \beta)\big)^2 \\ &= \frac{1}{n} \sum_{i=1}^n U_i^2 - 2\, \frac{1}{n} \sum_{i=1}^n U_i X_i^\top (\hat\beta_n - \beta) \\ &\qquad + \frac{1}{n} \sum_{i=1}^n \big(X_i^\top (\hat\beta_n - \beta)\big)^2 \\ &= \frac{1}{n} \sum_{i=1}^n U_i^2 - 2\, (\hat\beta_n - \beta)^\top \frac{1}{n} \sum_{i=1}^n X_i U_i \\ &\qquad + (\hat\beta_n - \beta)^\top \Big(\frac{1}{n} \sum_{i=1}^n X_i X_i^\top\Big) (\hat\beta_n - \beta) \\ &\to_p \mathrm{E}\left[U_i^2\right] - 2 \cdot 0^\top 0 + 0^\top Q\, 0 \\ &= \mathrm{E}\left[U_i^2\right] \end{aligned}

\frac{n}{n - k} \to 1: \begin{aligned} s^2 &= \frac{n}{n - k} \cdot \frac{1}{n} \sum_{i=1}^n \hat U_i^2 \\ &\to_p 1 \cdot \mathrm{E}\left[U_i^2\right] = \mathrm{E}\left[U_i^2\right] \end{aligned}

Homoskedasticity, law of iterated expectations: \mathrm{E}\left[U_i^2\right] = \mathrm{E}\left[\mathrm{E}\left[U_i^2 \mid X_i\right]\right] = \sigma^2

Both estimators of \mathrm{AsyVar}(\hat\beta_n) = \sigma^2 Q^{-1} consistent: \begin{aligned} \hat V_n &\to_p Q^{-1} \Omega Q^{-1} = \sigma^2 Q^{-1} \\ s^2 \hat Q_n^{-1} &\to_p \sigma^2 Q^{-1} \end{aligned}

Ratio of the two standard errors, Slutsky, \sigma^2 [Q^{-1}]_{jj} = [V]_{jj} > 0: \begin{aligned} \frac{\text{std. err}_{\text{rob}}\big(\hat\beta_{n,j}\big)}{\text{std. err}\big(\hat\beta_{n,j}\big)} &= \sqrt{\frac{[\hat V_n]_{jj} / n}{[s^2 \hat Q_n^{-1}]_{jj} / n}} \\ &= \sqrt{\frac{[\hat V_n]_{jj}}{[s^2 \hat Q_n^{-1}]_{jj}}} \\ &\to_p \sqrt{\frac{\sigma^2 [Q^{-1}]_{jj}}{\sigma^2 [Q^{-1}]_{jj}}} \\ &= 1 \end{aligned}

Therefore: \boxed{\ \frac{\text{std. err}_{\text{rob}}\big(\hat\beta_{n,j}\big)}{\text{std. err}\big(\hat\beta_{n,j}\big)} \to_p 1\ }

Usual t-ratio, A_n Z_n \to_d A Z: \begin{aligned} \frac{\hat\beta_{n,j} - \beta_j}{\text{std. err}\big(\hat\beta_{n,j}\big)} &= \underbrace{\frac{\text{std. err}_{\text{rob}}\big(\hat\beta_{n,j}\big)}{\text{std. err}\big(\hat\beta_{n,j}\big)}}_{\to_p\, 1} \cdot \underbrace{\frac{\hat\beta_{n,j} - \beta_j}{\text{std. err}_{\text{rob}}\big(\hat\beta_{n,j}\big)}}_{\to_d\, N(0, 1)} \\ &\to_d N(0, 1) \end{aligned}

Without homoskedasticity: s^2 \hat Q_n^{-1} \to_p \mathrm{E}\left[U_i^2\right]\, Q^{-1} \ne Q^{-1} \Omega Q^{-1} = V \quad \text{(in general)}

Example, k = 1, no intercept: Q = \mathrm{E}\left[X_i^2\right], \Omega = \mathrm{E}\left[U_i^2 X_i^2\right]: \begin{aligned} V &= \frac{\mathrm{E}\left[U_i^2 X_i^2\right]}{\big(\mathrm{E}\left[X_i^2\right]\big)^2}, \qquad \text{usual: } \frac{\mathrm{E}\left[U_i^2\right]}{\mathrm{E}\left[X_i^2\right]} \\ V - \frac{\mathrm{E}\left[U_i^2\right]}{\mathrm{E}\left[X_i^2\right]} &= \frac{\mathrm{E}\left[U_i^2 X_i^2\right] - \mathrm{E}\left[U_i^2\right]\, \mathrm{E}\left[X_i^2\right]}{\big(\mathrm{E}\left[X_i^2\right]\big)^2} \\ &= \frac{\mathrm{Cov}\left(U_i^2, X_i^2\right)}{\big(\mathrm{E}\left[X_i^2\right]\big)^2} \end{aligned}

\mathrm{Cov}\left(U_i^2, X_i^2\right) > 0, errors more variable where |X_i| is large: usual standard error too small.

Robust: \hat V_n \to_p V with or without homoskedasticity.

Normal errors (Lecture 4, Assumptions 1 to 5): \begin{aligned} & U \mid X \sim N(0, \sigma^2 I_n) \\ \Longrightarrow\quad & \hat\beta_n \mid X \sim N\big(\beta,\ \sigma^2 (X^\top X)^{-1}\big) \quad \text{for every } n \end{aligned}

Variance of \sqrt{n}(\hat\beta_n - \beta) given X, and its limit under Assumptions I and II: \begin{aligned} \mathrm{Var}\left(\sqrt{n}(\hat\beta_n - \beta) \mid X\right) &= n\, \mathrm{Var}\left(\hat\beta_n \mid X\right) \\ &= n\, \sigma^2 (X^\top X)^{-1} \\ &= \sigma^2 \Big(\frac{1}{n} X^\top X\Big)^{-1} \\ &= \sigma^2 \hat Q_n^{-1} \\ &\to_p \sigma^2 Q^{-1} = \mathrm{AsyVar}(\hat\beta_n) \end{aligned}

\sigma^2 estimated by s^2 (typed notes, Lecture 4): \begin{aligned} & \frac{\hat\beta_{n,j} - \beta_j}{\text{std. err}\big(\hat\beta_{n,j}\big)} \,\Big|\, X \sim t_{n-k} \quad \text{for every } n \\ & t_{n-k,\,1-\alpha/2} \to z_{1-\alpha/2} \end{aligned}

Normal errors are homoskedastic, so robust and usual agree in large samples (\mathrm{E}\left[U_i^4\right] = 3\sigma^4, \mathrm{E}\left[\|X_i\|^4\right] < \infty): \begin{aligned} \mathrm{E}\left[U_i^2 \mid X\right] &= \big[\sigma^2 I_n\big]_{ii} = \sigma^2 \\ \Longrightarrow\quad \mathrm{E}\left[U_i^2 \mid X_i\right] &= \mathrm{E}\left[\mathrm{E}\left[U_i^2 \mid X\right] \mid X_i\right] = \sigma^2 \\ \Longrightarrow\quad \frac{\text{std. err}_{\text{rob}}\big(\hat\beta_{n,j}\big)}{\text{std. err}\big(\hat\beta_{n,j}\big)} &\to_p 1 \end{aligned}

Validity of the t-ratio (large n: conditions for \hat V_n \to_p V):

Rows 1 and 2, t_{n-k,\,1-\alpha/2} \to z_{1-\alpha/2} and \text{std. err}_{\text{rob}} / \text{std. err} \to_p 1; the three intervals agree in large samples: \begin{aligned} \text{usual, exact (row 1):}\quad & \hat\beta_{n,j} \pm t_{n-k,\,1-\alpha/2}\, \text{std. err}\big(\hat\beta_{n,j}\big) \\ \text{usual, large } n\text{:}\quad & \hat\beta_{n,j} \pm z_{1-\alpha/2}\, \text{std. err}\big(\hat\beta_{n,j}\big) \\ \text{robust, large } n\text{:}\quad & \hat\beta_{n,j} \pm z_{1-\alpha/2}\, \text{std. err}_{\text{rob}}\big(\hat\beta_{n,j}\big) \end{aligned}

\hat\beta_n = \beta + \big(\frac{1}{n} \sum X_i X_i^\top\big)^{-1} \frac{1}{n} \sum X_i U_i; iid, \mathrm{E}\left[X_i U_i\right] = 0, \mathrm{E}\left[U_i^2\right] < \infty, \mathrm{E}\left[X_{i,j}^2\right] < \infty, \mathrm{E}\left[X_i X_i^\top\right] p.d. \Longrightarrow \hat\beta_n \to_p \beta.

\sqrt{n}(\hat\beta_n - \beta) = \big(\frac{1}{n} \sum X_i X_i^\top\big)^{-1} \frac{1}{\sqrt{n}} \sum X_i U_i.

Assumptions I to III \Longrightarrow \sqrt{n}(\hat\beta_n - \beta) \to_d N(0, Q^{-1} \Omega Q^{-1}), Q = \mathrm{E}\left[X_i X_i^\top\right], \Omega = \mathrm{E}\left[U_i^2 X_i X_i^\top\right].

\mathrm{E}\left[U_i^2 \mid X_i\right] = \sigma^2 \Longrightarrow \Omega = \sigma^2 Q, \mathrm{AsyVar}(\hat\beta_n) = \sigma^2 Q^{-1}.

\hat V_n = \hat Q_n^{-1} \hat\Omega_n \hat Q_n^{-1}, \hat Q_n = \frac{1}{n} \sum X_i X_i^\top, \hat\Omega_n = \frac{1}{n} \sum \hat U_i^2 X_i X_i^\top.

\hat\Omega_n = \frac{1}{n} \sum U_i^2 X_i X_i^\top + o_p(1) under Assumptions I to III, \mathrm{E}\left[U_i^4\right] < \infty, \mathrm{E}\left[\|X_i\|^4\right] < \infty; \hat V_n \to_p V.

\text{std. err}_{\text{rob}}(\hat\beta_{n,j}) = \sqrt{\big[(X^\top X)^{-1} \big(\sum \hat U_i^2 X_i X_i^\top\big) (X^\top X)^{-1}\big]_{jj}}; t-ratio \to_d N(0, 1); \hat\beta_{n,j} \pm z_{1-\alpha/2}\, \text{std. err}_{\text{rob}}: coverage \to 1 - \alpha.

\text{std. err}(\hat\beta_{n,j}) = \sqrt{[s^2 \hat Q_n^{-1}]_{jj} / n}, s^2 \to_p \mathrm{E}\left[U_i^2\right]; homoskedasticity: \text{std. err}_{\text{rob}} / \text{std. err} \to_p 1.

Heteroskedasticity: s^2 \hat Q_n^{-1} \to_p \mathrm{E}\left[U_i^2\right]\, Q^{-1} \ne V in general; only the robust standard error is always valid.

Normality: \mathrm{Var}\left(\sqrt{n}(\hat\beta_n - \beta) \mid X\right) = \sigma^2 \hat Q_n^{-1} \to_p \sigma^2 Q^{-1}; usual t-ratio exactly t_{n-k}, t_{n-k,\,1-\alpha/2} \to z_{1-\alpha/2}; homoskedastic, so {\text{std. err}_{\text{rob}} / \text{std. err} \to_p 1}.