Model and OLS estimator: Y_i = X_i^\top \beta + U_i, \qquad \hat\beta = \Big(\sum_{i=1}^n X_i X_i^\top\Big)^{-1} \sum_{i=1}^n X_i Y_i

Normal errors, exact distribution given the regressors: \begin{aligned} & U_i \mid X_i \sim N(0, \sigma^2) \\ \Longrightarrow\quad & \hat\beta \mid X_1, \ldots, X_n \sim N\Big(\beta,\ \sigma^2 \Big(\sum_{i=1}^n X_i X_i^\top\Big)^{-1}\Big) \end{aligned}

Without normality, write OLS in terms of sample averages: \hat\beta = \Big({\color{red}\frac{1}{n}} \sum_{i=1}^n X_i X_i^\top\Big)^{-1} {\color{red}\frac{1}{n}} \sum_{i=1}^n X_i Y_i

Use approximations; all limits below are as n \to \infty.

Non-random sequence: \{\underset{1\times1}{a_n} \in \mathbb{R} : n \ge 1\}

Definition. a_n converges to a, a_n \to a, a = \lim a_n, if for all \varepsilon > 0 there is N_\varepsilon \in \mathbb{N}: |a_n - a| < \varepsilon \quad \text{for all } n \ge N_\varepsilon

a: good approximation of a_n for all n \ge N_\varepsilon.

Sequence a_n with bands a \pm \varepsilon and a \pm \varepsilon', \varepsilon' < \varepsilon:

\hat\theta_n: estimator of \theta from n observations; random sequence: \{\underset{1\times1}{\hat\theta_n} : n \ge 1\}, \qquad \hat\theta_n = \hat\theta_n(\omega), \quad \omega \in \Omega

OLS from n observations: \hat\beta_n = \hat\beta_n(\omega), random.

Non-random sequences from \hat\theta_n; apply the definition of a limit to them: \begin{aligned} \text{Moments, } r > 0 &: \quad a_n = \mathrm{E}\left[|\hat\theta_n - \theta|^r\right] \\ \text{Probabilities, } \delta > 0 &: \quad a_n = \mathrm{P}\big(|\hat\theta_n - \theta| > \delta\big) \end{aligned}

Definition. \hat\theta_n converges to \theta in r-th mean, r > 0, if: \mathrm{E}\left[|\hat\theta_n - \theta|^r\right] \to 0

a_n = \mathrm{E}\left[|\hat\theta_n - \theta|^r\right] \to 0: for all \varepsilon > 0 there is N_\varepsilon \in \mathbb{N}: \mathrm{E}\left[|\hat\theta_n - \theta|^r\right] < \varepsilon \quad \text{for all } n \ge N_\varepsilon

r = 2, mean squared error (MSE): \mathrm{E}\left[(\hat\theta_n - \theta)^2\right]

Non-random sequence, \delta > 0 fixed: a_n = \mathrm{P}\big(|\hat\theta_n - \theta| > \delta\big), \qquad n \ge 1

Definition. \hat\theta_n converges in probability to \theta, \hat\theta_n \to_p \theta, if for all \delta > 0: \mathrm{P}\big(|\hat\theta_n - \theta| > \delta\big) \to 0

Equivalently, for all \delta > 0: \begin{aligned} \mathrm{P}\big(|\hat\theta_n - \theta| \le \delta\big) &= 1 - \underbrace{\mathrm{P}\big(|\hat\theta_n - \theta| > \delta\big)}_{\to\,0} \\ &\to 1 \end{aligned}

Example: densities for \hat\theta_n \sim N(\theta, 1/n).

Shaded: \mathrm{P}(|\hat\theta_n - \theta| > \delta).

\hat\theta_n \to_p \theta: distribution of \hat\theta_n more concentrated around \theta.

Markov’s inequality connects tail probabilities to expectations; the core device for establishing convergence in probability.

Theorem. X: random variable, M > 0: \mathrm{P}\big(|X| \ge M\big) \le \frac{\mathrm{E}\left[|X|\right]}{M}

Expectation of an indicator, X with density f_X, split at -M and M: \begin{aligned} \mathrm{E}\left[\mathbf{1}(|X| \ge M)\right] &= \int_{-\infty}^{\infty} \mathbf{1}(|x| \ge M)\, f_X(x)\, dx \\ &= \int_{-\infty}^{-M} \underbrace{\mathbf{1}(|x| \ge M)}_{=\,1} f_X(x)\, dx \\ &\qquad + \overbrace{\int_{-M}^{M} \underbrace{\mathbf{1}(|x| \ge M)}_{=\,0 \text{ for } |x| < M} f_X(x)\, dx}^{=\,0} \\ &\qquad + \int_{M}^{\infty} \underbrace{\mathbf{1}(|x| \ge M)}_{=\,1} f_X(x)\, dx \\ &= \int_{-\infty}^{-M} f_X(x)\, dx + \int_{M}^{\infty} f_X(x)\, dx \\ &= \mathrm{P}(X \le -M) + \mathrm{P}(X \ge M) \\ &= \mathrm{P}\big(|X| \ge M\big) \end{aligned}

Split 1 into two indicators: 1 = \mathbf{1}(|X| \ge M) + \mathbf{1}(|X| < M)

Expectation of |X|: \begin{aligned} \mathrm{E}\left[|X|\right] &= \mathrm{E}\left[|X| \cdot 1\right] \\ &= \mathrm{E}\left[|X|\, \big(\mathbf{1}(|X| \ge M) + \mathbf{1}(|X| < M)\big)\right] \\ &= \mathrm{E}\left[|X|\, \mathbf{1}(|X| \ge M)\right] + \underbrace{\mathrm{E}\left[|X|\, \mathbf{1}(|X| < M)\right]}_{\ge\,0} \\ &\ge \mathrm{E}\left[|X|\, \mathbf{1}(|X| \ge M)\right] \\ &\ge \mathrm{E}\left[M\, \mathbf{1}(|X| \ge M)\right] \quad \substack{|X| \ge M \\ \text{where } \mathbf{1}(|X| \ge M) = 1} \\ &= M\, \mathrm{E}\left[\mathbf{1}(|X| \ge M)\right] \\ &= M\, \mathrm{P}\big(|X| \ge M\big) \end{aligned}

Divide by M: \boxed{\ \mathrm{P}\big(|X| \ge M\big) \le \frac{\mathrm{E}\left[|X|\right]}{M}\ }

Claim. \mathrm{E}\left[|\hat\theta_n - \theta|^r\right] \to 0 \ \text{ for some } r > 0 \quad \Longrightarrow \quad \hat\theta_n \to_p \theta

Markov’s inequality, for all \delta > 0: \begin{aligned} \mathrm{P}\big(|\hat\theta_n - \theta| > \delta\big) &= \mathrm{P}\big(|\hat\theta_n - \theta|^r > \delta^r\big) \\ &\le \mathrm{P}\big(\underbrace{|\hat\theta_n - \theta|^r}_{X} \ge \underbrace{\delta^r}_{M}\big) \\ &\le \frac{1}{\delta^r}\, \underbrace{\mathrm{E}\left[|\hat\theta_n - \theta|^r\right]}_{\to\,0} \\ &\to 0 \end{aligned}

Therefore: \boxed{\ \hat\theta_n \to_p \theta\ }

Claim. For an estimator \hat\theta_n of \theta: \begin{aligned} & \text{1. } \mathrm{E}\left[\hat\theta_n\right] \to \theta \\ & \text{2. } \mathrm{Var}\left(\hat\theta_n\right) \to 0 \\ \Longrightarrow\quad & \hat\theta_n \to_p \theta \end{aligned}

Recall: \underbrace{\mathrm{E}\left[(\hat\theta_n - \theta)^2\right]}_{\text{MSE}} = \mathrm{Var}\left(\hat\theta_n\right) + \underbrace{\big(\mathrm{E}\left[\hat\theta_n\right] - \theta\big)^2}_{\text{Bias}^2}

Markov with r = 2, for all \delta > 0: \begin{aligned} \mathrm{P}\big(|\hat\theta_n - \theta| > \delta\big) &\le \frac{1}{\delta^2}\, \mathrm{E}\left[(\hat\theta_n - \theta)^2\right] \\ &= \frac{1}{\delta^2} \Big(\underbrace{\mathrm{Var}\left(\hat\theta_n\right)}_{\to\,0 \text{ by 2.}} + \big(\underbrace{\mathrm{E}\left[\hat\theta_n\right] - \theta}_{\to\,0 \text{ by 1.}}\big)^2\Big) \\ &\to 0 \end{aligned}

Therefore: \boxed{\ \hat\theta_n \to_p \theta\ }

\mathrm{E}\left[\bar X_n\right] = \mu, so bias is zero: \begin{aligned} \mathrm{E}\left[\bar X_n\right] &= \frac{1}{n} \sum_{i=1}^n \mathrm{E}\left[X_i\right] \\ &= \mu \end{aligned}

Variance of \bar X_n: \begin{aligned} \mathrm{Var}\left(\bar X_n\right) &= \mathrm{Var}\left(\frac{1}{n} \sum_{i=1}^n X_i\right) \\ &= \frac{1}{n^2}\, \mathrm{Var}\left(\sum_{i=1}^n X_i\right) \\ &= \frac{1}{n^2} \sum_{i=1}^n \sum_{j=1}^n \mathrm{Cov}\left(X_i, X_j\right) \\ &= \frac{1}{n^2} \Big(\sum_{i=1}^n \mathrm{Var}\left(X_i\right) + \underbrace{\sum_{i=1}^n \sum_{j \ne i} \mathrm{Cov}\left(X_i, X_j\right)}_{=\,0}\Big) \\ &= \frac{1}{n^2} \sum_{i=1}^n \underbrace{\mathrm{Var}\left(X_i\right)}_{=\,\sigma^2} \\ &= \frac{1}{n^2}\, (\sigma^2 \cdot n) \\ &= \frac{\sigma^2}{n} \\ &\to 0 \end{aligned}

Chebyshev with \mathrm{E}\left[\bar X_n\right] = \mu, for all \delta > 0: \begin{aligned} \mathrm{P}\big(|\bar X_n - \mu| > \delta\big) &\le \frac{1}{\delta^2}\, \mathrm{Var}\left(\bar X_n - \mu\right) \\ &= \frac{1}{\delta^2}\, \mathrm{Var}\left(\bar X_n\right) \\ &= \frac{1}{\delta^2}\, \frac{\sigma^2}{n} \\ &\to 0 \end{aligned}

Therefore: \boxed{\ \bar X_n \to_p \mu = \mathrm{E}\left[X_i\right]\ }

\mathrm{E}\left[X_i\right] = \mu_i, not constant; \mathrm{Var}\left(X_i\right) = \sigma^2 < \infty; \mathrm{Cov}\left(X_i, X_j\right) = 0 for i \ne j.

Centre each X_i at its mean: \begin{aligned} & Y_i = X_i - \mathrm{E}\left[X_i\right] \\ \Longrightarrow\quad & \mathrm{E}\left[Y_i\right] = 0, \qquad \mathrm{Var}\left(Y_i\right) = \sigma^2, \qquad \mathrm{Cov}\left(Y_i, Y_j\right) = 0 \ \text{ for } i \ne j \end{aligned}

LLN for Y_1, \ldots, Y_n: \begin{aligned} \frac{1}{n} \sum_{i=1}^n X_i - \frac{1}{n} \sum_{i=1}^n \mathrm{E}\left[X_i\right] &= \frac{1}{n} \sum_{i=1}^n \big(X_i - \mathrm{E}\left[X_i\right]\big) \\ &= \frac{1}{n} \sum_{i=1}^n Y_i \\ &\to_p 0 \end{aligned}

Y_1, \ldots, Y_n: random variables; for all i and all j \ne i: \begin{aligned} & \mathrm{E}\left[Y_i\right] = 0, \qquad \mathrm{Var}\left(Y_i\right) = \mathrm{E}\left[Y_i^2\right] = \sigma_i^2 \ \text{ not constant} \\ & \mathrm{Cov}\left(Y_i, Y_j\right) = 0 \end{aligned}

Bounded variances, \bar\sigma^2 not depending on n: \sigma_i^2 \le \max_{j = 1, \ldots, n} \sigma_j^2 \le \bar\sigma^2

Variance of the average \to 0 \Longrightarrow \frac{1}{n} \sum_{i=1}^n Y_i \to_p 0, by Chebyshev: \begin{aligned} \mathrm{Var}\left(\frac{1}{n} \sum_{i=1}^n Y_i\right) &= \frac{1}{n^2}\, \mathrm{Var}\left(\sum_{i=1}^n Y_i\right) \\ &= \frac{1}{n^2} \sum_{i=1}^n \mathrm{Var}\left(Y_i\right) \\ &= \frac{1}{n^2} \sum_{i=1}^n \sigma_i^2 \\ &\le \frac{1}{n^2}\, n \bar\sigma^2 \\ &= \frac{\bar\sigma^2}{n} \\ &\to 0 \end{aligned}

Variance of the average, non-zero covariances: \begin{aligned} \mathrm{Var}\left(\frac{1}{n} \sum_{i=1}^n Y_i\right) &= \frac{1}{n^2} \sum_{i=1}^n \sum_{j=1}^n \mathrm{Cov}\left(Y_i, Y_j\right) \\ &= \frac{1}{n^2} \sum_{i=1}^n \mathrm{Var}\left(Y_i\right) + \frac{1}{n^2} \sum_{i=1}^n \sum_{j \ne i} \mathrm{Cov}\left(Y_i, Y_j\right) \\ &= \frac{1}{n^2} \sum_{i=1}^n \mathrm{Var}\left(Y_i\right) + 2\, \frac{1}{n^2} \sum_{i=1}^{n-1} \sum_{j=i+1}^n \mathrm{Cov}\left(Y_i, Y_j\right) \end{aligned}

Number of distinct covariances, order n^2: \frac{n^2 - n}{2} = \frac{n(n-1)}{2}

Triangle inequality, |a + b| \le |a| + |b|: \begin{aligned} \Big|\frac{1}{n^2} \sum_{i=1}^n \sum_{j \ne i} \mathrm{Cov}\left(Y_i, Y_j\right)\Big| &\le \frac{1}{n^2} \underbrace{\sum_{i=1}^n \sum_{j \ne i} \big|\mathrm{Cov}\left(Y_i, Y_j\right)\big|}_{\text{if of smaller order than } n^2} \\ &\to 0 \end{aligned}

Double sum smaller order than n^2: for example, each observation correlated with only a fixed number of others, with bounded covariances.

\varepsilon_1, \varepsilon_2, \ldots: iid (independent and identically distributed), 0 < \mathrm{Var}\left(\varepsilon_i\right) < \infty.

Neighbours share one \varepsilon: \begin{aligned} Y_i &= \varepsilon_i + \varepsilon_{i+1} \\ Y_{i+1} &= \varepsilon_{i+1} + \varepsilon_{i+2} \\ Y_{i+2} &= \varepsilon_{i+2} + \varepsilon_{i+3} \end{aligned}

Covariances of Y_i and Y_{i+h}: \mathrm{Cov}\left(Y_i, Y_{i+1}\right) \ne 0, \qquad \mathrm{Cov}\left(Y_i, Y_{i+h}\right) = 0 \quad \text{for } h \ge 2

Independence instead of uncorrelatedness: finite variance not required.

Theorem. X_1, \ldots, X_n: iid random variables: \begin{aligned} & \mathrm{E}\left[|X_1|\right] < \infty \\ \Longrightarrow\quad & \bar X_n = \frac{1}{n} \sum_{i=1}^n X_i \to_p \mathrm{E}\left[X_1\right] \\ & \mathrm{E}\left[X_1\right] = \int_{-\infty}^{\infty} x\, f_X(x)\, dx \quad \text{with density } f_X \end{aligned}

Stated without proof; this is the LLN we will use.

Estimator \hat\theta_n and parameter \theta, k \times 1: \hat\theta_n = \begin{pmatrix} \hat\theta_{n,1} \\ \vdots \\ \hat\theta_{n,k} \end{pmatrix}, \qquad \theta = \begin{pmatrix} \theta_1 \\ \vdots \\ \theta_k \end{pmatrix}

Definition. \hat\theta_n converges in probability to \theta, \hat\theta_n \to_p \theta, two versions: \begin{aligned} \text{1. Element by element:} \quad & \hat\theta_{n,j} \to_p \theta_j \quad \text{for all } j = 1, \ldots, k \\ \text{2. In norm:} \quad & \|\hat\theta_n - \theta\| \to_p 0, \qquad \|x\| = \sqrt{x_1^2 + \cdots + x_k^2} \end{aligned}

Claim. 1 \Longleftrightarrow 2. \|\hat\theta_n - \theta\|^2 = (\hat\theta_{n,1} - \theta_1)^2 + \cdots + (\hat\theta_{n,k} - \theta_k)^2

1 \Longrightarrow 2: each element converges to zero, so the finite sum converges to zero.

2 \Longrightarrow 1: non-negative summands; the sum converges to zero only if each summand does.

Claim. For 1 \times 1 sequences \hat\theta_{n,1} and \hat\theta_{n,2}: \begin{aligned} & \hat\theta_{n,1} \to_p \theta_1, \quad \hat\theta_{n,2} \to_p \theta_2 \\ \Longrightarrow\quad & \hat\theta_{n,1} + \hat\theta_{n,2} \to_p \theta_1 + \theta_2 \end{aligned}

Triangle inequality, then a union bound, for all \delta > 0: \begin{aligned} & \mathrm{P}\big(|\hat\theta_{n,1} + \hat\theta_{n,2} - (\theta_1 + \theta_2)| > \delta\big) \\ &\le \mathrm{P}\big(|\hat\theta_{n,1} - \theta_1| + |\hat\theta_{n,2} - \theta_2| > \delta\big) \\ &\le \mathrm{P}\Big(|\hat\theta_{n,1} - \theta_1| > \frac{\delta}{2} \ \text{ or } \ |\hat\theta_{n,2} - \theta_2| > \frac{\delta}{2}\Big) \\ &\le \mathrm{P}\Big(|\hat\theta_{n,1} - \theta_1| > \frac{\delta}{2}\Big) + \mathrm{P}\Big(|\hat\theta_{n,2} - \theta_2| > \frac{\delta}{2}\Big) \\ &\to 0 \end{aligned}

Therefore: \boxed{\ \hat\theta_{n,1} + \hat\theta_{n,2} \to_p \theta_1 + \theta_2\ }

More general than sums: convergence is preserved by continuous functions.

Theorem. \hat\theta_n: k \times 1; \theta \in \mathbb{R}^k non-random; h : \mathbb{R}^k \to \mathbb{R}^m continuous at \theta; \to_p in norm: \hat\theta_n \to_p \theta \quad \Longrightarrow \quad h(\hat\theta_n) \to_p h(\theta)

h continuous at \theta: for all \delta > 0 there is \eta_\delta > 0: \|\hat\theta_n - \theta\| \le \eta_\delta \quad \Longrightarrow \quad \|h(\hat\theta_n) - h(\theta)\| \le \delta

Inclusion of events, then probabilities, for all \delta > 0: \begin{aligned} & \big\{\|\hat\theta_n - \theta\| \le \eta_\delta\big\} \subset \big\{\|h(\hat\theta_n) - h(\theta)\| \le \delta\big\} \\ \Longrightarrow\quad & \underbrace{\mathrm{P}\big(\|\hat\theta_n - \theta\| \le \eta_\delta\big)}_{\to\,1 \text{ (in norm)}} \le \mathrm{P}\big(\|h(\hat\theta_n) - h(\theta)\| \le \delta\big) \le 1 \\ \Longrightarrow\quad & \mathrm{P}\big(\|h(\hat\theta_n) - h(\theta)\| \le \delta\big) \to 1 \\ \Longrightarrow\quad & \boxed{\ h(\hat\theta_n) \to_p h(\theta)\ } \end{aligned}

\hat\theta_{n,1} \to_p \theta_1, \hat\theta_{n,2} \to_p \theta_2; element by element \Longrightarrow in norm; Slutsky for product and ratio: \begin{aligned} \hat\theta_{n,1}\, \hat\theta_{n,2} &\to_p \theta_1\, \theta_2 \\ \frac{\hat\theta_{n,1}}{\hat\theta_{n,2}} &\to_p \frac{\theta_1}{\theta_2} \quad \text{if } \theta_2 \ne 0 \end{aligned}

h(\theta_1, \theta_2) = \theta_1 / \theta_2: not continuous at \theta_2 = 0.

X_1, \ldots, X_n iid, \mathrm{E}\left[|X_1|\right] < \infty; weak LLN, then Slutsky for h(u) = u^2 and h(u) = 1/u: \begin{aligned} (\bar X_n)^2 &\to_p \big(\mathrm{E}\left[X_1\right]\big)^2 \\ \frac{1}{\bar X_n} &\to_p \frac{1}{\mathrm{E}\left[X_1\right]} \quad \text{if } \mathrm{E}\left[X_1\right] \ne 0 \end{aligned}

X_i: k \times 1, iid, \mathrm{E}\left[X_i X_i^\top\right] finite; weak LLN, element by element: \frac{1}{n} \sum_{i=1}^n X_i X_i^\top \to_p \mathrm{E}\left[X_i X_i^\top\right]

\mathrm{E}\left[X_i X_i^\top\right]: symmetric, p.s.d.

Assume \mathrm{E}\left[X_i X_i^\top\right] p.d.; h(\cdot) = (\cdot)^{-1}: continuous at a p.d. matrix.

Element by element \Longrightarrow in norm, then Slutsky on the k^2 elements: \Big(\frac{1}{n} \sum_{i=1}^n X_i X_i^\top\Big)^{-1} \to_p \big(\mathrm{E}\left[X_i X_i^\top\right]\big)^{-1}

Standard normal CDF: \Phi(t) = \mathrm{P}(Z \le t), \qquad Z \sim N(0, 1)

For X \sim N(\mu, \sigma^2), \sigma > 0: \begin{aligned} \mathrm{P}(X \le t) &= \mathrm{P}(X - \mu \le t - \mu) \\ &= \mathrm{P}\Big(\underbrace{\frac{X - \mu}{\sigma}}_{=\,Z \sim N(0,1)} \le \frac{t - \mu}{\sigma}\Big) \\ &= \Phi\Big(\frac{t - \mu}{\sigma}\Big) \end{aligned}

For any X, \mu \in \mathbb{R} and \sigma > 0: \mathrm{P}(X \le t) = \Phi\Big(\frac{t - \mu}{\sigma}\Big) \ \text{ for all } t \in \mathbb{R} \quad \Longleftrightarrow \quad X \sim N(\mu, \sigma^2)

\hat\theta_n, \theta: 1 \times 1.

Definition. \hat\theta_n asymptotically normal: for some \omega > 0: \begin{aligned} \mathrm{P}\big(\sqrt{n}(\hat\theta_n - \theta) \le t\big) &= {\color{red}\mathrm{P}\Big(\frac{\sqrt{n}(\hat\theta_n - \theta)}{\omega} \le \frac{t}{\omega}\Big)} \\ &\to {\color{red}\Phi\Big(\frac{t}{\omega}\Big)} \quad \text{for all } t \in \mathbb{R} \\ \Longleftrightarrow\quad \frac{\sqrt{n}(\hat\theta_n - \theta)}{\omega} &\to_d N(0, 1) \\ \Longleftrightarrow\quad \sqrt{n}(\hat\theta_n - \theta) &\to_d N(0, \omega^2) \end{aligned}

Convergence in distribution concerns distribution functions, not convergence of the random variables themselves.

Approximate using the normal CDF; valid as n \to \infty: \mathrm{P}\big(\sqrt{n}(\hat\theta_n - \theta) \le t\big) \approx \Phi\Big(\frac{t}{\omega}\Big)

X_1, \ldots, X_n iid, mean \theta, variance 0 < \sigma^2 < \infty; \hat\theta_n = \bar X_n: \hat\theta_n - \theta \to_p 0 \quad \text{(LLN)}

Variance of \hat\theta_n - \theta (proof of the LLN, step 1): \begin{aligned} \mathrm{Var}\left(\hat\theta_n - \theta\right) &= \mathrm{Var}\left(\bar X_n\right) \\ &= \frac{\sigma^2}{n} \\ &\to 0 \end{aligned}

Scaled by \sqrt{n}: \begin{aligned} \mathrm{Var}\left(\sqrt{n}(\hat\theta_n - \theta)\right) &= n\, \mathrm{Var}\left(\hat\theta_n - \theta\right) \\ &= n \cdot \frac{\sigma^2}{n} \\ &= \sigma^2 \end{aligned}

Scaled by n^\alpha: \begin{aligned} \mathrm{Var}\left(n^\alpha(\hat\theta_n - \theta)\right) &= \frac{n^{2\alpha}}{n}\, \sigma^2 \\ &= n^{2\alpha - 1} \sigma^2 \\ &\to \begin{cases} \infty & \text{if } \alpha > \tfrac12 \\ \sigma^2 & \text{if } \alpha = \tfrac12 \\ 0 & \text{if } \alpha < \tfrac12 \end{cases} \end{aligned}

Theorem. X_1, \ldots, X_n: iid random variables: \begin{aligned} & \mathrm{E}\left[X_i\right] = 0, \qquad \mathrm{Var}\left(X_i\right) = \mathrm{E}\left[X_i^2\right] = \sigma^2, \quad 0 < \sigma^2 < \infty \\ \Longrightarrow\quad & \frac{1}{\sqrt{n}} \sum_{i=1}^n X_i \to_d N(0, \sigma^2) \end{aligned}

CDF form, for all t \in \mathbb{R}: \mathrm{P}\Big(\frac{1}{\sqrt{n}} \sum_{i=1}^n X_i \le t\Big) \to \Phi\Big(\frac{t}{\sigma}\Big)

In terms of \bar X_n: \sqrt{n}\, \bar X_n = \frac{1}{\sqrt{n}} \sum_{i=1}^n X_i \to_d N(0, \sigma^2)

X_1, \ldots, X_n iid, \mathrm{E}\left[X_i\right] = \mu \ne 0, \mathrm{Var}\left(X_i\right) = \sigma^2, 0 < \sigma^2 < \infty: \frac{1}{\sqrt{n}} \sum_{i=1}^n X_i \nrightarrow_d N(\mu, \sigma^2)

Add and subtract \mu: \begin{aligned} \frac{1}{\sqrt{n}} \sum_{i=1}^n X_i &= \frac{1}{\sqrt{n}} \sum_{i=1}^n (X_i - \mu) + \frac{1}{\sqrt{n}}\, n\mu \\ &= \underbrace{\frac{1}{\sqrt{n}} \sum_{i=1}^n \underbrace{(X_i - \mu)}_{\mathrm{E}\left[X_i - \mu\right] \,=\, 0}}_{\to_d\, N(0, \sigma^2)} + \underbrace{\sqrt{n}\, \mu}_{\to\, \pm\infty} \end{aligned}

CLT for X_i - \mu, for all t \in \mathbb{R}: \begin{aligned} \mathrm{P}\big(\sqrt{n}(\underbrace{\bar X_n - \mu}_{\to_p\,0}) \le t\big) &= \mathrm{P}\Big(\sqrt{n}\, \frac{1}{n} \sum_{i=1}^n (X_i - \mu) \le t\Big) \\ &= \mathrm{P}\Big(\frac{1}{\sqrt{n}} \sum_{i=1}^n (X_i - \mu) \le t\Big) \\ &\to \Phi\Big(\frac{t}{\sigma}\Big) \end{aligned}

Therefore: \boxed{\ \sqrt{n}(\bar X_n - \mu) \to_d N(0, \sigma^2)\ }

MGF of X, assumed finite for t near 0: M(t) = \mathrm{E}\left[e^{tX}\right] = \int_{-\infty}^{\infty} e^{tx} f_X(x)\, dx

Normal MGF: X \sim N(\mu, \sigma^2) \quad \Longleftrightarrow \quad \mathrm{E}\left[e^{tX}\right] = e^{\mu t + \sigma^2 t^2 / 2} \ \text{ for all } t \in \mathbb{R}

One can show: MGFs converge for all t \Longrightarrow convergence in distribution.

Need to show, for all t \in \mathbb{R}: \mathrm{E}\left[e^{t \frac{1}{\sqrt{n}} \sum_{i=1}^n X_i}\right] \to e^{\sigma^2 t^2 / 2}

Definition. Small o: a_n \to 0 \ \Longleftrightarrow\ a_n = o(1), \qquad o\Big(\frac{t^2}{n}\Big) = \underbrace{\frac{t^2}{n}}_{\to\,0}\, o(1)

M and its derivatives at 0: \begin{aligned} M(0) &= \mathrm{E}\left[e^{tX_i}\right] \Big|_{t=0} = 1 \\ M^{(1)}(t) &= \frac{d}{dt} \mathrm{E}\left[e^{tX_i}\right] \\ &= \mathrm{E}\left[\frac{d}{dt} e^{tX_i}\right] \\ &= \mathrm{E}\left[X_i e^{tX_i}\right] \\ M^{(1)}(0) &= \mathrm{E}\left[X_i\right] = 0 \\ M^{(2)}(t) &= \mathrm{E}\left[\frac{d^2}{dt^2} e^{tX_i}\right] \\ &= \mathrm{E}\left[X_i^2 e^{tX_i}\right] \\ M^{(2)}(0) &= \mathrm{E}\left[X_i^2\right] = \sigma^2 \end{aligned}

Second-order expansion around 0, then substitute the derivatives: \begin{aligned} M\Big(\frac{t}{\sqrt{n}}\Big) &= M(0) + M^{(1)}(0)\, \frac{t}{\sqrt{n}} + \frac{M^{(2)}(0)}{2}\, \frac{t^2}{n} + o\Big(\frac{t^2}{n}\Big) \\ &= 1 + 0 + \frac{\sigma^2}{2}\, \frac{t^2}{n} + \frac{t^2}{n}\, o(1) \\ &= 1 + \frac{c_n}{n}, \qquad c_n = \frac{\sigma^2 t^2}{2} + t^2\, o(1) \to \frac{\sigma^2 t^2}{2} \end{aligned}

We will use, for a_n \to \infty: \lim \Big(1 + \frac{1}{a_n}\Big)^{a_n} = e

MGF of \frac{1}{\sqrt{n}} \sum_{i=1}^n X_i, from steps 2 and 3, t \ne 0, a_n = n / c_n \to \infty: \begin{aligned} \mathrm{E}\left[e^{t \frac{1}{\sqrt{n}} \sum_{i=1}^n X_i}\right] &= \Big(M\Big(\frac{t}{\sqrt{n}}\Big)\Big)^n \\ &= \Big(1 + \frac{c_n}{n}\Big)^n \\ &= \Big(1 + \frac{1}{n / c_n}\Big)^n \\ &= \Big[\underbrace{\Big(1 + \frac{1}{n / c_n}\Big)^{n / c_n}}_{\to\, e}\Big]^{c_n} \\ &\to e^{\sigma^2 t^2 / 2} \quad (c_n \to \sigma^2 t^2 / 2,\ \text{step 3}) \end{aligned}

t = 0: both sides equal 1.

e^{\sigma^2 t^2 / 2}: MGF of N(0, \sigma^2) (step 1).

Therefore: \boxed{\ \frac{1}{\sqrt{n}} \sum_{i=1}^n X_i \to_d N(0, \sigma^2)\ }

l’Hôpital’s rule in 1/n: \begin{aligned} \lim \frac{\ln(1 + 1/n)}{1/n} &= \lim \frac{1/(1 + 1/n)}{1} \\ &= 1 \end{aligned}

Take logarithms: \begin{aligned} \ln\Big[\Big(1 + \frac{1}{n}\Big)^n\Big] &= n \ln\Big(1 + \frac{1}{n}\Big) \\ &= \frac{\ln(1 + 1/n)}{1/n} \\ &\to 1 \\ \Longrightarrow\quad \Big(1 + \frac{1}{n}\Big)^n &= e^{\ln[(1 + 1/n)^n]} \\ &\to e^1 = e \end{aligned}

Same steps, 1/a_n \to 0 in place of 1/n: (1 + 1/a_n)^{a_n} \to e.

Theorem. X_1, \ldots, X_n: iid random k \times 1 vectors: \begin{aligned} & \mathrm{E}\left[X_i\right] = 0, \qquad \underset{k\times k}{\Sigma} = \mathrm{E}\left[X_i X_i^\top\right] \ \text{ finite and p.d.} \\ \Longrightarrow\quad & \frac{1}{\sqrt{n}} \sum_{i=1}^n X_i \to_d N(0, \Sigma) \end{aligned}

Recall: linear map of a normal vector; a \in \mathbb{R}^k, a \ne 0: \underset{k\times1}{W} \sim N(0, \Sigma) \quad \Longrightarrow \quad \underset{1\times1}{a^\top W} \sim N(0, \underset{1\times1}{a^\top \Sigma a})

Equivalent statements (one can show): \begin{aligned} & \frac{1}{\sqrt{n}} \sum_{i=1}^n X_i \to_d N(0, \Sigma) \\ \Longleftrightarrow\quad & a^\top \Big(\frac{1}{\sqrt{n}} \sum_{i=1}^n X_i\Big) \to_d N(0, a^\top \Sigma a) \quad \text{for all } a \ne 0 \\ \Longleftrightarrow\quad & \mathrm{P}\Big(a^\top \Big(\frac{1}{\sqrt{n}} \sum_{i=1}^n X_i\Big) \le t\Big) \to \Phi\Big(\frac{t}{\sqrt{a^\top \Sigma a}}\Big) \quad \text{for all } a \ne 0,\ t \in \mathbb{R} \end{aligned}

\Sigma p.d.: a^\top \Sigma a > 0 \quad \text{for all } a \ne 0

Linear combination with a: a^\top \Big(\frac{1}{\sqrt{n}} \sum_{i=1}^n X_i\Big) = \frac{1}{\sqrt{n}} \sum_{i=1}^n \underbrace{a^\top X_i}_{1\times1}

a^\top X_1, \ldots, a^\top X_n: iid.

Recall mean and variance with \Gamma = a^\top, \mathrm{Var}\left(X_i\right) = \mathrm{E}\left[X_i X_i^\top\right] = \Sigma: \begin{aligned} \mathrm{E}\left[a^\top X_i\right] &= a^\top \mathrm{E}\left[X_i\right] = 0 \\ \mathrm{Var}\left(a^\top X_i\right) &= a^\top \mathrm{Var}\left(X_i\right)\, a = a^\top \Sigma a \end{aligned}

Univariate CLT, 0 < a^\top \Sigma a < \infty: \begin{aligned} & \frac{1}{\sqrt{n}} \sum_{i=1}^n a^\top X_i \to_d N(0, a^\top \Sigma a) \quad \text{for all } a \ne 0 \\ \Longrightarrow\quad & \boxed{\ \frac{1}{\sqrt{n}} \sum_{i=1}^n X_i \to_d N(0, \Sigma)\ } \quad \text{(equivalent statements)} \end{aligned}

a_n \to a: for all \varepsilon > 0, some N_\varepsilon with |a_n - a| < \varepsilon for all n \ge N_\varepsilon.

r-th mean, r > 0: \mathrm{E}\left[|\hat\theta_n - \theta|^r\right] \to 0; r = 2: MSE.

\hat\theta_n \to_p \theta: \mathrm{P}(|\hat\theta_n - \theta| > \delta) \to 0 for all \delta > 0.

Markov, M > 0: \mathrm{P}(|X| \ge M) \le \mathrm{E}\left[|X|\right]/M; Chebyshev, \delta > 0: \mathrm{P}(|Y - \mathrm{E}\left[Y\right]| > \delta) \le \mathrm{Var}\left(Y\right)/\delta^2.

r-th mean \Longrightarrow \to_p; \mathrm{E}\left[\hat\theta_n\right] \to \theta, \mathrm{Var}\left(\hat\theta_n\right) \to 0 \Longrightarrow \hat\theta_n \to_p \theta.

LLN: uncorrelated, \mathrm{E}\left[X_i\right] = \mu, \mathrm{Var}\left(X_i\right) = \sigma^2 < \infty \Longrightarrow \mathrm{Var}\left(\bar X_n\right) = \sigma^2/n, \bar X_n \to_p \mu.

Y_i = X_i - \mathrm{E}\left[X_i\right], \mathrm{Var}\left(Y_i\right) \le \bar\sigma^2 not depending on n, \sum_i \sum_{j \ne i} |\mathrm{Cov}\left(Y_i, Y_j\right)| of smaller order than n^2 \Longrightarrow \frac{1}{n} \sum Y_i \to_p 0.

Weak LLN: iid, \mathrm{E}\left[|X_1|\right] < \infty \Longrightarrow \bar X_n \to_p \mathrm{E}\left[X_1\right].

Vectors: \hat\theta_{n,j} \to_p \theta_j for all j \Longleftrightarrow \|\hat\theta_n - \theta\| \to_p 0.

Slutsky: \hat\theta_n \to_p \theta, h continuous at \theta \Longrightarrow h(\hat\theta_n) \to_p h(\theta).

X_i iid, \mathrm{E}\left[X_i X_i^\top\right] finite and p.d. \Longrightarrow \Big(\frac{1}{n} \sum X_i X_i^\top\Big)^{-1} \to_p \big(\mathrm{E}\left[X_i X_i^\top\right]\big)^{-1}.

\sqrt{n}(\hat\theta_n - \theta) \to_d N(0, \omega^2), \omega > 0: \mathrm{P}(\sqrt{n}(\hat\theta_n - \theta) \le t) \to \Phi(t/\omega) for all t.

X_i iid, mean \theta, variance 0 < \sigma^2 < \infty \Longrightarrow \mathrm{Var}\left(n^\alpha(\bar X_n - \theta)\right) = n^{2\alpha - 1} \sigma^2; \alpha = \tfrac12: variance \sigma^2.

CLT: iid, \mathrm{E}\left[X_i\right] = 0, \mathrm{Var}\left(X_i\right) = \sigma^2, 0 < \sigma^2 < \infty \Longrightarrow \frac{1}{\sqrt{n}} \sum X_i \to_d N(0, \sigma^2).

\mathrm{E}\left[X_i\right] = \mu, other conditions the same \Longrightarrow \sqrt{n}(\bar X_n - \mu) \to_d N(0, \sigma^2).

Proof of the CLT, \mathrm{E}\left[X_i\right] = 0, MGF M(t) = \mathrm{E}\left[e^{tX_i}\right] finite near 0: \big(M(t/\sqrt{n})\big)^n = \big(1 + c_n/n\big)^n \to e^{\sigma^2 t^2/2}, c_n \to \sigma^2 t^2/2.

Multivariate CLT: iid k \times 1, \mathrm{E}\left[X_i\right] = 0, \Sigma = \mathrm{E}\left[X_i X_i^\top\right] finite and p.d. \Longrightarrow \frac{1}{\sqrt{n}} \sum X_i \to_d N(0, \Sigma).

Equivalent statements: \frac{1}{\sqrt{n}} \sum X_i \to_d N(0, \Sigma) \Longleftrightarrow \frac{1}{\sqrt{n}} \sum a^\top X_i \to_d N(0, a^\top \Sigma a) for all a \ne 0.