Central Limit Theorem.

We saw before that for any sequence of independent identically distributed random variables X_{1}, ⋯, X_{n}, ⋯ the sum S_{n} = X_{1} + ⋯ + X_{n} has the property that

lim_{n→∞} \frac{S_{n}}{n} = 0

in probability provided the expectation exists and equals 0. If we assume that the Variance of the random variables is finite and equals σ^{2} > 0, then we have

Theorem 3.17. The distribution of \frac{S_{n}}{\sqrt{n}} converges as n → ∞ to the normal distribution with density

p(x) = \frac{1}{\sqrt{2π}σ} exp[−\frac{x^{2}}{2σ^{2}} ].  (3.1)

Proof. If we denote by ϕ(t) the characteristic function of any X_{i} then the characteristic function of \frac{S_{n}}{\sqrt{n}} is given by

ψ_{n}(t) = [ϕ(\frac{t}{\sqrt{n}})]^{n}

We can use the expansion

ϕ(t) = 1 − \frac{σ^{2}t^{2}}{2} + o (t^{2})

to conclude that

ϕ(\frac{t}{\sqrt{n}}) = 1 − \frac{σ^{2}t^{2}}{2n} + o (\frac{1}{n})

and it then follows that

lim_{n→∞} ψ_{n}(t) = ψ(t) = exp[−\frac{σ^{2}t^{2}}{2} ].

Since ψ(t) is the characteristic function of the normal distribution with density p(x) given by equation 3.1, we are done.

Exercise 3.14. A more direct proof is possible in some special cases. For instance if each X_{i} = ±1 with probability \frac{1}{2}, S_{n} can take the values n − 2k with 0 ≤ k ≤ n,

P [S_{n} = 2k − n] = \frac{1}{2^{n}} \binom{n}{k}

and

P [a ≤ \frac{S_{n}}{\sqrt{n}} ≤ b] = \frac{1}{2^{n}} ∑_{k:a\sqrt{n}≤2k−n≤b\sqrt{n}} \binom{n}{k}.

Use Stirling’s approximation to prove directly that

lim_{n→∞} P [a ≤ \frac{S_{n}}{\sqrt{n}} ≤ b] = ∫_{a}^{b} \frac{1}{\sqrt{2}π} exp[−\frac{x^{2}}{2} ] dx

Actually for the proof of the central limit theorem we do not need the random variables {X_{j}} to have identical distributions. Let us suppose that they all have zero means and that the variance of X_{j} is σ_{j}^{2}. Define s_{n}^{2} = σ_{1}^{2} + ⋯ + σ_{n}^{2}. Assume s_{n}^{2} → ∞ as n → ∞. Then Y_{n} = \frac{S_{n}}{\sqrt{s}_{n}} has zero mean and unit variance. It is not unreasonable to expect that

lim_{n→∞} P [Y_{n} ≤ a] = ∫_{−∞}^{a} \frac{1}{\sqrt{2}π} exp[−\frac{x^{2}}{2} ] dx

under certain mild conditions.

Theorem 3.18. (Lindeberg). If we denote by α_{i} the distribution of X_{i}, the condition

lim_{n→∞} \frac{1}{s_{n}^{2}} ∑_{i=1}^{n} ∫_{|x|≥ϵs_{n}} x^{2}dα_{i} = 0

for each ϵ > 0 is sufficient for the central limit theorem to hold.

Proof. The first step in proving this limit theorem as well as other limit theorems that we will prove is to rewrite

Y_{n} = X_{n,1} + X_{n,2} + ⋯ + X_{n,k_{n}} + A_{n}

wher X_{n,j} are k_{n} mutually independent random variables and A_{n} is a constant. In our case k_{n} = n, A_{n} = 0, and X_{n,j} = \frac{X_{j}}{s_{n}} for 1 ≤ j ≤ n. We denote by

ϕ_{n,j}(t) = E[e^{i t X_{n,j}}] = ∫ e^{i t x}dα_{n,j} = ∫ e^{i t \frac{x}{sn}} dα_{j} = ϕ_{j}(\frac{t}{s_{n}})

where α_{n,j} is the distribution of X_{n,j}. The functions ϕ_{j} and ϕ_{n,j} are the characteristic functions of α_{j} and α_{n,j} respectively. If we denote by μ_{n} the distribution of Y_{n}, its characteristic function μ̂_{n}(t) is given by

μ̂_{n}(t) = ∏_{j=1}^{n} ϕ_{n,j}(t)

and our goal is to show that

lim_{n→∞} μ̂_{n}(t) = exp[−\frac{t^{2}}{2} ]

This will be carried out in several steps. First, we define

ψ_{n,j}(t) = exp[ϕ_{n,j}(t) − 1]

and

ψ_{n}(t) = ∏_{j=1}^{n} ψ_{n,j}(t).

We show that for each finite T,

lim_{n→∞} sup_{|t|≤T} sup_{1≤j≤n} |ϕ_{n,j}(t) − 1| = 0

and

sup_{n} sup_{|t|≤T} ∑_{j=1}^{n} |ϕ_{n,j}(t) − 1| < ∞

This would imply that

lim_{n→∞} sup_{|t|≤T} | log μ̂_{n}(t) − log ψ_{n}(t)|
≤  lim_{n→∞} sup_{|t|≤T} ∑_{j=1}^{n} | log ϕ_{n,j}(t) − [ϕ_{n,j}(t) − 1]|

≤  lim_{n→∞} sup_{|t|≤T} C ∑_{j=1}^{n} |ϕ_{n,j}(t) − 1|^{2}
≤  C lim_{n→∞} { sup_{|t|≤T} sup_{1≤j≤n} |ϕ_{n,j}(t) − 1|}{ sup_{|t|≤T} ∑_{j=1}^{n} |ϕ_{n,j}(t) − 1|}
=  0

by the expansion

log r = log(1 + (r − 1)) = r − 1 + O(r − 1)^{2}.

The proof can then be completed by showing

lim_{n→∞} sup_{|t|≤T} | log ψ_{n}(t) + \frac{t^{2}}{2} | = lim_{n→∞} sup_{|t|≤T} |[ ∑_{j=1}^{n}(ϕ_{n,j}(t) − 1)] + \frac{t^{2}}{2} | = 0

We see that

sup_{|t|≤T} |ϕ_{n,j}(t) − 1| = sup_{|t|≤T} | ∫ [ exp[i t x] − 1]dα_{n,j}|
= sup_{|t|≤T} | ∫ [ exp[i t \frac{x}{s_{n}} ] − 1]dα_{j}|
= sup_{|t|≤T} | ∫ [ exp[i t \frac{x}{s_{n}} ] − 1 − i t \frac{x}{s_{n}}]dα_{j}| (3.2)
≤ C_{T} ∫ \frac{x^{2}}{s_{n}^{2}} dα_{j}  (3.3)
= C_{T} ∫_{|x|<ϵs_{n}} \frac{x^{2}}{s_{n}^{2}} dα_{j} + C_{T} ∫_{|x|≥ϵs_{n}} \frac{x^{2}}{s_{n}^{2}} dα_{j}
≤ C_{T} ϵ^{2} + C_{T} \frac{1}{s_{n}^{2}} ∫_{|x|≥ϵs_{n}} x^{2}dα_{j}  (3.4)

We have used the mean zero condition in deriving equation 3.2 and the estimate |e^{ix} − 1 − ix| ≤ cx^{2} to get to the equation 3.3. If we let n → ∞ by Lindeberg’s condition, the second term of equation 3.4 goes to 0. Therefore

lim sup_{n→∞} sup_{1≤j≤k_{n}} sup_{|t|≤T} |ϕ_{n,j}(t) − 1| ≤ ϵ^{2}C_{T}

Since, ϵ > 0 is arbitrary, we have

lim_{n→∞} sup_{|t|≤T} |ϕ_{n,j}(t) − 1| = 0

Next

sup_{|t|≤T} ∑_{j=1}^{n} |ϕ_{n,j}(t) − 1| ≤ C_{T} ∑_{j=1}^{n} ∫ \frac{x^{2}}{s_{n}^{2}} dα_{j} ≤ C_{T} \frac{1}{s_{n}^{2}} ∑_{j=1}^{n} σ_{j}^{2} = C_{T}

uniformly in n. Finally for each ϵ > 0,

lim_{n→∞} sup_{|t|≤T} |[ ∑_{j=1}^{n}(ϕ_{n,j}(t) − 1)] + \frac{t^{2}}{2} |

≤ lim_{n→∞} sup_{|t|≤T} ∑_{j=1}^{n} |ϕ_{n,j}(t) − 1 + \frac{σ_{j}^{2}t^{2}}{2s_{n}^{2}} |
= lim_{n→∞} sup_{|t|≤T} ∑_{j=1}^{n} | ∫ [ exp[i t \frac{x}{s_{n}} ] − 1 − i t \frac{x}{s_{n}} + \frac{t^{2}x^{2}}{2s_{n}^{2}} ] dα_{j}|
≤ lim_{n→∞} sup_{|t|≤T} ∑_{j=1}^{n} | ∫_{|x|<ϵs_{n}} [ exp[i t \frac{x}{s_{n}} ] − 1 − i t \frac{x}{s_{n}} + \frac{t^{2}x^{2}}{2s_{n}^{2}} ] dα_{j}|

+ lim_{n→∞} sup_{|t|≤T} ∑_{j=1}^{n} | ∫_{|x|≥ϵs_{n}} [ exp[i t \frac{x}{s_{n}} ] − 1 − i t \frac{x}{s_{n}} + \frac{t^{2}x^{2}}{2s_{n}^{2}} ] dα_{j}|

≤ lim_{n→∞} C_{T} ∑_{j=1}^{n} ∫_{|x|<ϵs_{n}} \frac{|x|^{3}}{s_{n}^{3}} dα_{j}

+ lim_{n→∞} C_{T} ∑_{j=1}^{n} ∫_{|x|≥ϵs_{n}} \frac{x^{2}}{s_{n}^{2}} dα_{j}

≤ ϵC_{T} lim sup_{n→∞} ∑_{j=1}^{n} ∫ \frac{x^{2}}{s_{n}^{2}} dα_{j}

+ lim_{n→∞} C_{T} ∑_{j=1}^{n} ∫_{|x|≥ϵs_{n}} \frac{x^{2}}{s_{n}^{2}} dα_{j}

= ϵC_{T}

by Lindeberg’s condition. Since ϵ > 0 is arbitrary the result is proven.

Exercise 3.15. Lyapunov’s condition is the following: for some δ > 0

lim_{n→∞} \frac{1}{s_{n}^{2+δ}} ∑_{j=1}^{n} ∫ |x|^{2+δ} dα_{j} = 0

Prove that Lyapunov’s condition implies Lindeberg’s condition.

Exercise 3.16. Consider the case of mutually independent random variables \{X_{j}\}, where X_{j} = ±a_{j} with probability \frac{1}{2}. What do Lyapunov’s and Lindeberg’s conditions demand of {a_{j}}? Can you find a sequence {a_{j}} that does not satisfy Lyapunov’s condition for any δ > 0 but satisfies Lindeberg’s condition? Try to find a sequence {a_{j}} such that the central limit theorem is not valid.

Accompanying Laws.

As we stated in the previous section, we want to study the behavior of the sum of a large number of independent random variables. We have k_{n} independent random variables {X_{n,j} : 1 ≤ j ≤ k_{n}} with respective distributions {α_{n,j}}. We are interested in the distribution μ_{n} of Z_{n} = ∑_{j=1}^{k_{n}} X_{n,j}. One important assumption that we will make on the random variables {X_{n,j}} is that no single one is significant. More precisely for every δ > 0,

lim_{n→∞} sup_{1≤j≤k_{n}} P [ |X_{n,j}| ≥ δ] = lim_{n→∞} sup_{1≤j≤k_{n}} α_{n,j}[ |x| ≥ δ] = 0

The condition is referred to as uniform infinitesimality.

The following construction will play a major role. If α is a probability distribution on the line and α̂(t) is its characteristic function, for any nonnegative real number a > 0, ψ_{a}(t) = exp[a(ϕ(t) − 1)] is again a characteristic distribution. In fact, if we denote by α^{k} the k-fold convolution of α with itself, ψ_{a} is seen to be the characteristic function of the proabaility distribution

e^{−a} ∑_{j=0}^{∞} \frac{a^{j}}{j!} α^{k}

which is a convex combination α^{k} with weights e^{−a} \frac{a^{j}}{j!} . We use the construction mostly with a = 1. If we denote the probability distribution with characteristic function ψ_{a}(t) by e_{a}(α) one checks easily that e_{a+b}(α) = e_{a}(α) ∗ e_{b}(α). In particular e_{a}(α) = e\frac{a}{n} (α)^{n}. Probability distributions β that can be written for each n ≥ 1 as the n-fold convolution β_{n}^{n} of some probability distribution β_{n} are called infinitely divisible. In particular for every a ≥ 0 and α, e_{a}(α) is an infinitely divisible probability distribution. These are called compound Poisson distributions. A special case when α = δ_{1} the degenerate distribution at 1, we get for e_{a}(δ_{1}) the usual Poisson distribution with parameter a. We can interpret e_{a}(α) as the distribution of the sum of a random number of independent random variables withcommon distrribution α. The random n has a distribution which is Poisson with parameter a and is independent of the random variables involved in the sum.

In order to study the distribution μ_{n} of Z_{n} it will be more convenient to replace α_{n,j} by an infinitely divisible distribution β_{n,j}. This is done as follows. We define

a_{n,j} = ∫_{|x|≤1} x dα_{n,j},

α′_{n,j} as the translate of α_{n,j} by −a_{n,j}, i.e.

α′_{n,j} = α_{n,j} ∗ δ−a_{n,j},
β′_{n,j} = e_{1}(α_{n,j}),
β_{n,j} = β′_{n,j} ∗ a_{n,j}

and finally

λ_{n} = ∏_{j=1}^{k_{n}} β_{n,j}

A main tool in this subject is the following theorem. We assume always that the uniform infinitesimality condition holds.

Theorem 3.19. (Accompanying Laws.) In order that, for some constants A_{n}, the distribution μ_{n} ∗ δ_{A_{n}} of Z_{n} + A_{n} may converge to the limit μ it is necessary and sufficient that, for the same constants A_{n}, the distribution λ_{n} ∗ δ_{A_{n}} converges to the same limit μ.

Proof. First we note that, for any δ > 0,

lim sup_{n→∞} sup_{1≤j≤k_{n}} |a_{n,j}| = lim sup_{n→∞} sup_{1≤j≤k_{n}} | ∫_{|x|≤1} x dα_{n,j}|
≤ lim sup_{n→∞} sup_{1≤j≤k_{n}} | ∫_{|x|≤δ} x dα_{n,j}|

+ lim sup_{n→∞} sup_{1≤j≤k_{n}} | ∫_{δ<|x|≤1} x dα_{n,j}|

≤ δ + lim sup_{n→∞} sup_{1≤j≤k_{n}} α_{n,j}[ |x| ≥ δ]
= δ

Therefore

lim_{n→∞} sup_{1≤j≤k_{n}} |a_{n,j}| = 0

This means that α′_{n,j} are uniformly infinitesimal just as α_{n,j} were. Let us suppose that n is so large that sup_{1≤j≤k_{n}} |a_{n,j}| ≤ \frac{1}{4}. The advantage in going from α_{n,j} to α′_{n,j} is that we can calculate

a′_{n,j} = ∫_{|x|≤1} x dα′_{n,j}
= ∫_{|x−a_{n,j}|≤1}(x − a_{n,j}) dα_{n,j}
= ∫_{|x−a_{n,j}|≤1} x dα_{n,j} − a_{n,j} α_{n,j}[ |x − a_{n,j}| ≤ 1 ]
= ∫_{|x−a_{n,j}|≤1} x dα_{n,j} − a_{n,j} + α_{n,j}[ |x − a_{n,j}| > 1 ]

and estimate |a′_{n,j}| by

|a′_{n,j}| ≤ Cα_{n,j}[ |x| ≥ \frac{3}{4} ] ≤ Cα′_{n,j}[ |x| ≥ \frac{1}{2} ]

In other words we may assume with out loss of generality that α_{n,j} satisfy the bound

|a_{n,j}| ≤ Cα_{n,j}[ |x| ≥ \frac{1}{2} ]

and forget all about the change from α_{n,j} to α′_{n,j}.

Then, just as in the proof of the Lindeberg theorem, we proceed to estimate

lim_{n→∞} sup_{|t|≤T} | log λ̂_{n}(t) − log μ̂_{n}(t)|

≤ lim_{n→∞} sup_{|t|≤T} | ∑_{j=1}^{k_{n}} [ log α̂_{n,j}(t) − (α̂_{n,j}(t) − 1)]|

≤ lim_{n→∞} sup_{|t|≤T} ∑_{j=1}^{k_{n}} | log α̂_{n,j}(t) − (α̂_{n,j}(t) − 1)|

≤ lim_{n→∞} sup_{|t|≤T} C ∑_{j=1}^{k_{n}} |α̂_{n,j}(t) − 1|^{2}

= 0

provided we prove that if either λ_{n} or μ_{n} has a limit after translation by some constants A_{n}, then

sup_{n} sup_{|t|≤T} ∑_{j=1}^{k_{n}} |α̂_{n,j}(t) − 1| ≤ C < ∞  (3.5)

Let us first suppose that λ_{n} has a weak limit as n → ∞ after translation by A_{n}. The characteristic functions

exp [ ∑_{j=1}^{k_{n}}(α̂_{n,j}(t) − 1)) + itA_{n}] = exp[f_{n}(t)]

have a limit, which is again a characteristic function. Since the limiting characteristic function is continuous and equals 1 at t = 0, and the convergence is uniform near 0, on some small interval |t| ≤ T_{0} we have the bound

sup_{n} sup_{|t|≤T_{0}} [1 − Realf_{n}(t) ] ≤ C

or equivalently

sup_{n} sup_{|t|≤T_{0}} ∑_{j=1}^{k_{n}} ∫ (1 − cos t x) dα_{n,j} ≤ C

and from the subadditivity property (1 − cos 2 t x) ≤ 4(1 − cos t x) this bound extends to arbitrary interval |t| ≤ T,

sup_{n} sup_{|t|≤T} ∑_{j=1}^{k_{n}} ∫ (1 − cos t x) dα_{n,j} ≤ C_{T}

If we integrate the inequality with respect to t over the interval [−T, T] and divide by 2T, we get

sup_{n} ∑_{j=1}^{k_{n}} ∫ (1 − \frac{sin T x}{T x}) dα_{n,j} ≤ C_{T}

from which we can conclude that

sup_{n} ∑_{j=1}^{k_{n}} α_{n,j}[ |x| ≥ δ] ≤ C_{δ} < ∞

for every δ > 0 by choosing T = \frac{2}{δ}. Moreover using the inequality (1 − cos x) ≥ c x^{2} valid near 0 for a suitable choice of c we get the estimate

sup_{n} ∑_{j=1}^{k_{n}} ∫_{|x|≤1} x^{2} dα_{n,j} ≤ C < ∞

Now it is straight forward to estimate, for t ∈ [−T, T],

|α̂_{n,j}(t) − 1| = | ∫ [exp(i t x) − 1] dα_{n,j}|
= | ∫_{|x|≤1}[exp(i t x) − 1] dα_{n,j}|
+| ∫_{|x|>1}[exp(i t x) − 1] dα_{n,j}|
≤ | ∫_{|x|≤1}[exp(i t x) − 1 − i t x] dα_{n,j}|
+| ∫_{|x|>1}[exp(i t x) − 1] dα_{n,j}| + T |a_{n,j}|
≤ C_{1} ∫_{|x|≤1} x^{2} dα_{n,j} + C_{2} α_{n,j} [x : |x| ≥ \frac{1}{2} ]

which proves the bound of equation 3.5.

Now we need to establish the same bound under the assumption that μ_{n} has a limit after suitable translations. For any probability measure α we define barα by ᾱ(A) = α(−A) for all Borel sets. The distribution α ∗ ᾱ is denoted by |α|^{2}. The characteristic functions of ᾱ and |α|^{2} are respectively ¯ α̂(t) and |α̂(t)|^{2} where α̂(t) is the characteristic function of α. An elementary but important fact is |α∗A|^{2} = |α|^{2} for any translate A. If μ_{n} has a limit so does |μ_{n}|^{2}. We conclude that the functions

lim_{n→∞} |μ̂_{n}(t)|^{2} = lim_{n→∞} ∏_{j=1}^{k_{n}} |α̂_{n,j}(t)|^{2}

exists and defines a characteristic function which is continuous at 0 with a value of 1. Moreover because of uniform infinitesimality,

lim_{n→∞} inf_{|t|≤T} |α_{n,j}(t)| = 1

It is easy to conclude that there is a T_{0} > 0 such that, for |t| ≤ T_{0},

sup_{n} sup_{|t|≤T_{0}} ∑_{j=1}^{k_{n}} [1 − |α̂_{n,j}(t)|^{2}] ≤ C_{0} < ∞

and by subadditivity for any finite T,

sup_{n} sup_{|t|≤T} ∑_{j=1}^{k_{n}} [1 − |α̂_{n,j}(t)|^{2}] ≤ C_{T} < ∞

providing us with the estimates

(2)  sup_{n} ∑_{j=1}^{k_{n}} |α_{n,j}|^{2} [ |x| ≥ δ] ≤ C_{δ} < ∞

for any δ > 0, and

(3)  sup_{n} ∑_{j=1}^{k_{n}} ∫ ∫_{|x−y|≤2}(x − y)^{2}dα_{n,j}(x) dα_{n,j}(y) ≤ C < ∞

We now show that (2) and (3) imply (1).

|α_{n,j}|^{2}[ x : |x| ≥ \frac{δ}{2} ]

≥ ∫_{|x|≤ \frac{δ}{2}} α_{n,j} [x : |x − y| ≥ \frac{δ}{2} ] dα_{n,j}(y)
≥ α_{n,j} [ x : |x| ≥ δ] α_{n,j} [ x : |x| ≤ \frac{δ}{2} ]
≥ \frac{1}{2} α_{n,j} [ x : |x| ≥ δ]

by uniform infinitesimality. Therfore (2) implies

(2′)  sup_{n} ∑_{j=1}^{k_{n}} α_{n,j} [ x : |x| ≥ δ] ≤ C_{δ} < ∞

We now turn to

∫ ∫_{|x−y|≤2}(x − y)^{2}dα_{n,j}(x) dα_{n,j}(y)
≥ {α_{n,j} [y : |y| ≤ 1 ]}{ inf_{|y|≤1} ∫_{|x|≤1}(x − y)^{2}dα_{n,j}(x)}

The first term on the right can be assumed to be atleast \frac{1}{2} by uniform infinitesimality. The second term

∫_{|x|≤1}(x − y)^{2}dα_{n,j}(x) ≥ ∫_{|x|≤1} x^{2}dα_{n,j}(x) − 2y ∫_{|x|≤1} x dα_{n,j}(x)
≥ ∫_{|x|≤1} x^{2}dα_{n,j}(x) − 2| ∫_{|x|≤1} x dα_{n,j}(x)|
≥ ∫_{|x|≤1} x^{2}dα_{n,j}(x) − Cα_{n,j} [x : |x| ≥ \frac{1}{2} ]

The last step is a consequence of what we assumed in the beginning with out loss of generality i.e.

∫_{|x|≤1} x dα_{n,j}(x) ≤ Cα_{n,j} [x : |x| ≥ \frac{1}{2} ]

Because of (2′) we can now assert

(3′)  sup_{n} ∑_{j=1}^{k_{n}} ∫_{|x|≤1} x^{2} dα_{n,j} ≤ C < ∞

One can now derive (3) from (2′) and (3′) as in the earlier part. Exercise 3.17. Let k_{n} = n^{2} and α_{n,j} = δ_{\frac{1}{n}} for 1 ≤ j ≤ n^{2}. μ_{n} = δ_{n} and show that without centering λ_{n} ∗ δ_{−n} converges to a different limit.