Chapter 3

Independent Sums

Independence and Convolution

One of the central ideas in probabilty is the notion of independence. In intuitive terms two events are independent if they have no influence on each other. The formal definition is

Definition 3.1. Two events A and B are independent if P [A ∩ B] = P [A].P [B]

Exercise 3.1. If A and B are independent prove that so are A^{c} and B.

Definition 3.2. Two random variables X and Y are independent if the events X ∈ A and Y ∈ B are independent for any two Borel sets A and B on the line i.e.

P [X ∈ A, Y ∈ B] = P [X ∈ A].P [Y ∈ B]

for all Borel sets A and B.

There is a natural extension to a finite or even an infinite collection of random variables.

Definition 3.3. A finite collection collection {X_{j} : 1 ≤ j ≤ n} of random variables are said to be independent if for any n Borel sets A_{1}, …, A_{n} on the line

P [ ∩_{1≤j≤n} [X_{j} ∈ A_{j}]] = Π_{1≤j≤n}P [X_{j} ∈ A_{j}]

Definition 3.4. An infinite collection of random variables is said to independent if every finite subcollection is independent.

Lemma 3.1. Two random variables X, Y defined on (Ω, Σ, P) are independent if and only if the measure induced on R^{2} by (X, Y), is the product measure α × β where α and β are the distributions on R induced by X and Y respectively.

Proof: Left as an exercise.

The important thing to note is that if X and Y are independent and one knows their distributions α and β, then their joint distribution is automatically determined as the product measure.

If X and Y are random variables having α and β for their distributions, the distribution of the sum Z = X + Y is determined as follows. First we construct the product measure α × β on R × R and then consider the induced distribution of the function f(x, y) = x + y. This distribution, called the convolution of α and β, is denoted by α ∗ β. An elementary calculation using Fubini’s theorem provides the following identities.

(α ∗ β)(A) = ∫ α(A − x) dβ = ∫ β(A − x) dα  (3.1)

In terms of characteristic function, we can express the characteristic function of the convolution as

∫ exp[ i t x]d(α ∗ β) = ∫ ∫ exp[ i t (x + y) ] d α d β
= ∫ exp[ i t x] d α ∫ exp[ i t x] d β

or equivalently

ϕ_{α∗β}(t) = ϕ_{α}(t)ϕ_{β}(t)  (3.2)

which provides a direct way of calculating the distributions of sums of indepenedent random variables by the use of characteristic functions.

Exercise 3.2. If X and Y are independent show that for any two measurable functions f and g, f(x) and g(Y) are independent.

Exercise 3.3. Use Fubini’s theorem to show that if X and Y are independent and if f and g are measurable functions with both E[|f(X)|] and E[|f(Y)|] finite then

E[f(X)g(Y)] = E[f(X)]E[g(Y)]

Exercise 3.4. If X and Y are any two random variables then E(X + Y) = E(X) + E(Y), and if X and Y are independent then

Var(X + Y) = Var(X) + Var(Y)

where

Var(X) = E[[X − E[X]]^{2}] = E[X^{2}] − [E[X]]^{2}

If X_{1}, X_{2}, ⋯, X_{n} are n independent random variables, then the distribution of their sum S_{n} = X_{1} + X_{2} + ⋯ + X_{n} can be computed in terms of the distributions of the summands. If α_{j} is the distribution of X_{j}, then the distribution of μ_{n} of S_{n} is given by the convolution μ_{n} = α_{1} ∗ α_{2} ∗ ⋯ ∗ α_{n} that can be calculated inductively by μ_{j+1} = μ_{j} ∗ α_{j+1}. In terms of their characteristic functions ψ_{n}(t) = ϕ_{1}(t)ϕ_{2}(t) ⋯ ϕ_{n}(t). The first two moments of S_{n} are computed easily.

E(S_{n}) = E(X_{1}) + E(X_{2}) + ⋯ E(X_{n})

and

Var(S_{n}) = E[S_{n} − E(S_{n})]^{2}
= ∑_{j} E[X_{j} − E(X_{j})]^{2}

+2 ∑_{1≤i<j≤n} E[X_{i} − E(X_{i})][X_{j} − E(X_{j})]

For i ≠ j, because of independence

E[X_{i} − E(X_{i})][X_{j} − E(X_{j})] = E[X_{i} − E(X_{i})]E[X_{j} − E(X_{j})] = 0

and we get the formula

Var(S_{n}) = Var(X_{1}) + Var(X_{2}) + ⋯ + Var(X_{n})  (3.3)

Weak Law of Large Numbers

Let us look at the distribution of the number of succesees in n independent trials, with the probability of success in a single trial being equal to p.

P \{S_{n} = r\} = \binom{n}{r}p^{r}(1 − p)^{n−r}

and

P \{|S_{n} − np| ≥ nδ\} = ∑_{|r−np|≥nδ} \binom{n}{r}p^{r}(1 − p)^{n−r}

≤ \frac{1}{n^{2}δ^{2}} ∑_{|r−np|≥nδ}(r − np)^{2}\binom{n}{r}p^{r}(1 − p)^{n−r}

≤ \frac{1}{n^{2}δ^{2}} ∑_{1≤r≤n}(r − np)^{2}\binom{n}{r}p^{r}(1 − p)^{n−r}
= \frac{1}{n^{2}δ^{2}} E[S_{n} − np]^{2} = \frac{1}{n^{2}δ^{2}} Var(S_{n})
= \frac{1}{n^{2}δ^{2}} np(1 − p)
= \frac{p(1 − p)}{nδ^{2}}

We have used the fact that S_{n} = X_{1} + X_{2} + ⋯ + X_{n} where the X_{i} are independent and have the simple distribution P {X_{i} = 1} = p and P {X_{i} = 0} = 1 − p. Therefore E(S_{n}) = np and Var(S_{n}) = nVar(X_{1}) = np(1 − p)

It follows now that

lim_{n→∞} P \{|S_{n} − np| ≥ nδ\} = lim_{n→∞} P \{|\frac{S_{n}}{n} − p| ≥ δ\} = 0

or the average \frac{S_{n}}{n} converges to p in probability. This is seen easily to be equivalent to the statement that the distribution of \frac{S_{n}}{n} converges to the distribution degenerate at p.

The above argument works for any sequence of independent and identically distributed random variables. If we assume that E(X_{i}) = m and Var(X_{i}) = σ^{2} < ∞, then E(\frac{S_{n}}{n}) = m and Var(\frac{S_{n}}{n}) = \frac{σ^{2}}{n}. Chebychev’s inequality states that for any random variable X

P {|X − E[X]| ≥ δ} = ∫_{|X−E[X]|≥δ} dP
≤ \frac{1}{δ^{2}} ∫_{|X−E[X]|≥δ}[X − E[X]]^{2}dP
= \frac{1}{δ^{2}} ∫ [X − E[X]]^{2}dP
= \frac{1}{δ^{2}} Var(X)  (3.4)

This can be used to prove the weak law of large numbers for the general case of independent identically distributed random variables with finite second moments.

Theorem 3.2. If X_{1}, X_{2} …, X_{n}, … is a sequence of independent identically distributed random variables with E[X_{j}] ≡ m and VarX_{j} ≡ σ^{2} then for

S_{n} = X_{1} + X_{2} + ⋯ + X_{n}

we have

lim_{n→∞} P [|\frac{S_{n}}{n} − m| ≥ δ = 0

for any δ > 0.

Proof: Use Chebychev’s inequality to estimate

P [|\frac{S_{n}}{n} − m| ≥ δ] ≤ \frac{1}{δ^{2}} Var(\frac{S_{n}}{n}) = \frac{σ^{2}}{nδ^{2}}

Actually it is enough to assume that E|X_{i}| < ∞ and the existence of the second moment is not needed. We will provide two proofs of the statement

Theorem 3.3. If X_{1}, X_{2}, ⋯ X_{n} are independent and identically distributed with a finite first moment and E(X_{i}) = m, then \frac{X_{1}+X_{2}+···+X_{n}}{n} converges to m in probability as n → ∞.

Proof 1. Let C be a large constant and let us define X_{i}^{C} as the truncated random variable X_{i}^{C} = X_{i} if |X_{i}| ≤ C and X_{i}^{C} = 0 otherwise.

Let Y_{i}^{C} = X_{i} − X_{i}^{C} so that X_{i} = X_{i}^{C} + Y_{i}^{C}. Then

\frac{1}{n} ∑_{1≤i≤n} X_{i} = \frac{1}{n} ∑_{1≤i≤n} X_{i}^{C} + \frac{1}{n} ∑_{1≤i≤n} Y_{i}^{C}
= ξ_{n}^{C} + η_{n}^{C}

If we denote by a_{C} = E(X_{i}^{C}) and b_{C} = E(Y_{i}^{C}) we always have m = a_{C} + b_{C}. Consider the quantity

δ_{n} = E[|\frac{1}{n} ∑_{1≤i≤n} X_{i} − m|]
= E[|ξ_{n}^{C} + η_{n}^{C} − m|]
≤ E[|ξ_{n}^{C} − a_{C}|] + E[|η_{n}^{C} − b_{C}|]
≤ [E[|ξ_{n}^{C} − a_{C}|^{2}]]^{\frac{1}{2}} + 2E[|Y_{i}^{C}|]

Taking the limsup as n → ∞, by the earlier argument for each fixed cutoff level C the first term goes to zero. Therefore

lim sup_{n→∞} δ_{n} ≤ 2E[|Y_{i}^{C}|]

If we now let the cutoff level C to go to infinity, by the integrability of X_{i}, E|Y_{i}^{C}| → 0 as C → ∞ and we are done. The final step of eastablishing that for any sequence Y_{n} of random variables E[|Y_{n}|] → 0 implies that Y_{n} → 0 in probability, is left as an exercise.

Proof 2. We can use characteristic functions. If we denote the characteristic function of X_{i} by ϕ(t), then the characteristic function of \frac{1}{n} ∑_{1≤i≤n} X_{i} is given by ψ_{n}(t) = [ϕ(\frac{t}{n})]^{n}. The existence of the first moment assures us that ϕ(t) is differentiable at t = 0 with a derivative equal to im where m = E(X_{i}). Therefore

ϕ(\frac{t}{n}) = 1 + \frac{i m t}{n} + o(\frac{1}{n})

and this implies

lim_{n→∞} ψ_{n}(t) = exp[ i m t]

and therefore the distribution of \frac{S_{n}}{n} tends to the degenerate distribution at the point m. The weak law of large numbers is thereby established.

Exercise 3.5. If the underlying distribution is a Cauchy distribution with density \frac{1}{π(1+x^{2})} and characteristic function ϕ(t) = e^{−|t|}, prove that the weak law does not hold.

Exercise 3.6. The weak law may hold sometimes even if the mean does not exist. If we dampen the tails of the Cauchy ever so slightly with a density f(x) = \frac{c}{(1+x^{2}) log(1+x^{2})}, show that the weak law of large numbers holds.

Exercise 3.7. In the case of the Binomial distribution with p = \frac{1}{2}, use Stirling’s formula

n! ≃ \sqrt{2π} e^{−n} n^{n+ \frac{1}{2}}

to estimate the probability

∑_{r≥nx} \binom{n}{r}\frac{1}{2^{n}}

and show that it decays geometrically in n. Can you calculate the geometric ratio

ρ(x) = lim_{n→∞} [ ∑_{r≥nx} \binom{n}{r}\frac{1}{2^{n}} ]^{\frac{1}{n}}

explicitly as a function of x for x > \frac{1}{2}?

Strong Limit Theorems

The weak law of large numbers is really a result concerning the behavior of

\frac{S_{n}}{n} = \frac{X_{1} + X_{2} + ⋯ + X_{n}}{n}

where X_{1}, X_{2}, ⋯, X_{n}, … is a sequence of independent and identically distributed random variables on some probability space (Ω, B, P). Under the assumption that X_{i} are integrable with an integral equal to m, the weak law asserts that as n → ∞ \frac{S_{n}}{n} → m in Probability. Since almost everywhere convergence is generally stronger than convergence in Probabilty one may ask if

P [ω : lim_{n→∞} \frac{S_{n}(ω)}{n} = m] = 1

This is called the Strong Law of Large Numbers. Strong laws are statements that hold for almost all ω.

Let us look at functions of the form f_{n} = χ_{An}. It is easy to verify that f_{n} → 0 in probability if and only if P (A_{n}) → 0. On the other hand

Lemma 3.4. (Borel-Cantelli). If

∑_{n} P (A_{n}) < ∞

then

P [ω : lim_{n→∞} χ_{An}(ω) = 0] = 1.

If the events A_{n} are mutually independent the converse is also true.

Remark: Note that the complementary event

[ω : lim sup_{n→∞} χ_{An}(ω) = 1]

is the same as ∩_{n=1}^{∞} ∪_{j=n}^{∞} A_{j}, or the event that infinitely many of the events {A_{j}} occcur.

Proof. By monotone convergence theorem if ∑_{n} P (A_{n}) < ∞, then ∑_{n} χ_{An}(ω) = S(ω) is finite almost everywhere and

E(S(ω)) = ∑_{n} P (A_{n}) < ∞.

If an infinite series has a finite sum then the n-th term must go to 0, thereby proving the direct part. To prove the converse we need to show that if ∑_{n} P (A_{n}) = ∞, then lim_{m→∞} P (∪_{n=m}^{∞}A_{n}) > 0. We can use independence and the continuity of probability under monotone limits, to calculate for every m,

P (∪_{n=m}^{∞}A_{n}) = 1 − P (∩_{n=m}^{∞}A_{n}^{c})
= 1 − ∏_{n=m}^{∞}(1 − P (A_{n})) = 1 − 0 = 1

and we are done.

Another digression that we want to make into measure theory at this point is to discuss Kolmogorov consistency theorem. How do we know that there are probability spaces that admit a sequence of independent identically distributed random variables with specified distributions? By the construction of product measures that we outlined earlier we can construct a measure on R^{n} for every n which is the joint distribution of the first n random variables. Let us denote by P_{n} this probabilty measure on R^{n}. They are consistent in the sense that if we project in the natural way from R^{n+1} → R^{n}, P_{n+1} projects to P_{n}. Such a family is called a consistent family of finite dimensional distributions. We look at the space Ω = R^{∞} consisting of all real sequences ω = {x_{n} : n ≥ 1} with a natural σ-field Σ generated by the field F of finite dimensional cylinder sets of the form B = {ω : (x_{1}, ⋯, x_{n}) ∈ A} where A varies over Borel sets in R^{n} and varies over positive integers.

Theorem 3.5. (Kolmogorov Connsistency Theorem). Given a consistent family of finite dimensional distributions P_{n}, there exists a unique P on (Ω, Σ) such that for every n, under the natural projection π_{n}(ω) = (x_{1}, ⋯, x_{n}), the induced measure P π_{n}^{−1} = P_{n} on R^{n}.

Proof. The consistency is just what is required to be able to define P on F by

P (B) = P_{n}(A).

Once we have P defined on the field F, we have to prove the countable additivity of P on F. The rest is then routine. Let B_{n} ∈ F and B_{n} ↓ Φ, the empty set. If possible let P (B_{n}) ≥ δ for all n and for some δ > 0. Then B_{n} = π_{k_{n}}^{−1}A_{k_{n}} for some k_{n} and without loss of generality we assume that k_{n} = n, so that B_{n} = π_{n}^{−1}A_{n} for some Borel set A_{n} ⊂ R^{n}. According to Exercise 3.8 below, we can find a closed bounded subset K_{n} ⊂ A_{n} such that

P_{n}(A_{n} − K_{n}) ≤ \frac{δ}{2^{n+1}}

and define C_{n} = π_{n}^{−1}K_{n} and D_{n} = ∩_{j=1}^{n}C_{j} = π_{n}^{−1}F_{n} for some closed bounded set F_{n} ⊂ K_{n} ⊂ R^{n}. Then

P (D_{n}) ≥ δ − ∑_{j=1}^{n} \frac{δ}{2^{j+1}} ≥ \frac{δ}{2}.

D_{n} ⊂ B_{n}, D_{n} ↓ Φ and each D_{n} is nonempty. If we take ω^{(n)} = {x_{j}^{n} : j ≥ 1} to be an arbitrary point from D_{n}, by our construction (x_{1}^{n}, ⋯ x_{m}^{n}) ∈ F_{m} for n ≥ m. We can definitely choose a subsequence (diagonlization) such that x_{j}^{n_{k}} converges for each j producing a limit ω = (x_{1}, ⋯, x_{m}, ⋯) and, for every m, we will have (x_{1}, ⋯, x_{m}) ∈ F_{m}. This implies that ω ∈ D_{m} for every m, contradicting D_{n} ↓ Φ. We are done.

Exercise 3.8. We have used the fact that given any borel set A ⊂ R^{n}, and a probability measure α on R^{n}, for any ϵ > o, there exists a closed bounded subset K_{ϵ} ⊂ A such that α(A − K_{ϵ}) ≤ ϵ. Prove it by showing that the class of sets A with the above property is a monotone class that contains finite disjoint unioms of measurable rectangles and therefore contains the Borel σ-field. To prove the last fact, establish it first for n = 1. To handle n = 1, repeat the same argument starting from finite disjoint unions of right-closed left-open intevals. Use the countable additivity to verify this directly.

The following is a strong version of the Law of Large Numbers.

Theorem 3.6. If X_{1}, ⋯, X_{n} ⋯ is a sequence of independent identically distributed random variables with E|X_{i}|^{4} = C < ∞, then

lim_{n→∞} \frac{S_{n}}{n} = lim_{n→∞} \frac{X_{1} + ⋯ + X_{n}}{n} = E(X_{1})

with probability 1.

Proof. We can assume without loss of generality that E(X_{i}) = 0 . Just take Y_{i} = X_{i} − E(X_{i}). A simple calculation shows

E[(S_{n})^{4}] = nE(X_{1}^{4}) + 3n(n − 1)E[(X_{1}^{2})]^{2} ≤ nC + 3n^{2}σ^{4}

and by applying a Chebychev type inequality using fourth moments,

P [|\frac{S_{n}}{n} | ≥ δ] = P [ |S_{n}| ≥ nδ] ≤ \frac{nC + 3n^{2}σ^{4}}{n^{4}δ^{4}} .

Because

∑_{n=1}^{∞} P [ |\frac{S_{n}}{n} | ≥ δ] < ∞

we can apply the Borel-Cantelli Lemma and we are done.

Series of Independent Random variables

We wish to investigate conditions under which an infinite series with independent summands

S = ∑_{j=1}^{∞} X_{j}

converges with probability 1. The basic steps are the following inequalities due to Kolomogorov and Lévy that control the behaviour of sums of independent random variables. They both deal with the problem of estimating

T_{n}(ω) = sup_{1≤k≤n} |S_{k}(ω)| = sup_{1≤k≤n} | ∑_{j=1}^{k} X_{j}(ω)|

where X_{1}, ⋯, X_{n} are n independent random variables.

Lemma 3.7. (Kolmogorov’s Inequality). Assume that EX_{i} = 0 and Var(X_{i}) = σ_{i}^{2} < ∞ and let s_{n}^{2} = ∑_{j=1}^{n} σ_{j}^{2}. Then

P \{T_{n}(ω) ≥ ℓ\} ≤ \frac{s_{n}^{2}}{ℓ^{2}} .

Proof. The important point here is that the estimate depends only on s_{n}^{2} and not on the number of summands. In fact the Chebychev bound on S_{n} is

P \{|S_{n}| ≥ ℓ\} ≤ \frac{s_{n}^{2}}{ℓ^{2}}

and the supremum does not cost anything.

Let us define the events E_{k} = {|S_{1}| < ℓ, ⋯, |S_{k−1}| < ℓ, |S_{k}| ≥ ℓ} and then {T_{n} ≥ ℓ} = ∪_{k=1}^{n}E_{k} is a disjoint union of E_{k}. If we use the independence of S_{n} − S_{k} and S_{k}χ_{E_{k}} that only depends on X_{1} ⋯, X_{k}

P \{E_{k}\} ≤ \frac{1}{ℓ^{2}} ∫_{E_{k}} S_{k}^{2} dP
≤ \frac{1}{ℓ^{2}} ∫_{E_{k}} [S_{k}^{2} + (S_{n} − S_{k})^{2}] dP

= \frac{1}{ℓ^{2}} ∫_{E_{k}} [S_{k}^{2} + 2S_{k}(S_{n} − S_{k}) + (S_{n} − S_{k})^{2}] dP
= \frac{1}{ℓ^{2}} ∫_{E_{k}} S_{n}^{2} dP

Summing over k from 1 to n

P \{T_{n} ≥ ℓ\} ≤ \frac{1}{ℓ^{2}} ∫_{T_{n}≥ℓ} S_{n}^{2} dP ≤ \frac{s_{n}^{2}}{ℓ^{2}} .

Lemma 3.8. (Lévy’s Inequality). Assume that

P \{|X_{i} + ⋯ + X_{n}| ≥ \frac{ℓ}{2}\} ≤ δ

for all 1 ≤ i ≤ n. Then

P \{T_{n} ≥ ℓ\} ≤ \frac{δ}{1 − δ}

Proof. Let E_{k} be as in the previous lemma.

P \{(T_{n} ≥ ℓ) ∩ |S_{n}| ≤ \frac{ℓ}{2}\} = ∑_{k=1}^{n} P \{E_{k} ∩ |S_{n}| ≤ \frac{ℓ}{2}\}
≤ ∑_{k=1}^{n} P \{E_{k} ∩ |S_{n} − S_{k}| ≥ \frac{ℓ}{2}\}
= ∑_{k=1}^{n} P \{|S_{n} − S_{k}| ≥ \frac{ℓ}{2}\} P (E_{k})
≤ δ ∑_{k=1}^{n} P (E_{k})
= δP {T_{n} ≥ ℓ}

On the other hand,

P \{(T_{n} ≥ ℓ) ∩ |S_{n}| > \frac{ℓ}{2}\} ≤ P \{|S_{n}| > \frac{ℓ}{2}\} ≤ δ

Adding the two,

P {T_{n} ≥ ℓ} ≤ δP {T_{n} ≥ ℓ} + δ

or

P \{T_{n} ≥ ℓ\} ≤ \frac{δ}{1 − δ}

We are now ready to prove

Theorem 3.9. (Lévy’s Theorem). If X_{1}, X_{2}, …, X_{n}, … are independent, then the following are equivalent.

(i) The distribution α_{n} of S_{n} = X_{1} + ⋯ + X_{n} converges weakly to a probability distribution α on R.

(ii) The random variable S_{n} = X_{1} + ⋯ + X_{n} converges in probability to a limit S(ω).

(iii) The random variable S_{n} = X_{1} + ⋯ + X_{n} converges with probability 1 to a limit S(ω).

Proof. Clearly (iii) ⇒ (ii) ⇒ (i) are trivial. We will establish (i) ⇒ (ii) ⇒ (iii).

(i) ⇒ (ii). The characteristic functions ϕ_{j}(t) of X_{j} are such that

ϕ(t) = ∏_{i=1}^{∞} ϕ_{j}(t)

is a convergent infinite product. Since the limit ϕ(t) is continuous at t = 0 and ϕ(0) = 1 it is nonzero in some interval |t| ≤ T around 0. Therefore for |t| ≤ T,

lim_{{n→∞ \atop m→∞}} ∏_{m+1}^{n} ϕ_{j}(t) = 1

By Exercise 3.9 below, this implies that for all t,

lim_{{n→∞ \atop m→∞}} ∏_{m+1}^{n} ϕ_{j}(t) = 1

and consequently, the distribution of S_{n} − S_{m} converges to the distribution degenerate at 0. This implies the convergence in probability to 0 of S_{n} − S_{m} as m, n → ∞. Therefore for each δ > 0,

lim_{{n→∞ \atop m→∞}} P \{|S_{n} − S_{m}| ≥ δ\} = 0

establishing (ii).

(ii) ⇒ (iii). To establish (iii), because of Exercise 3.10 below, we need only show that for every δ > 0

lim_{{n→∞ \atop m→∞}} P \{ sup_{m<k≤n} |S_{k} − S_{m}| ≥ δ\} = 0

and this follows from (ii) and Lévy’s inequality.

Exercise 3.9. Prove the inequality 1 − cos 2t ≤ 4(1 − cos t) for all real t. Deduce the inequality 1 − Real ϕ(2t) ≤ 4[1 − Real ϕ(t)], valid for any characteristic function. Conclude that if a sequence of characteristic functions converges to 1 in an interval around 0, then it converges to 1 for all real t.

Exercise 3.10. Prove that if a sequence S_{n} of random variables is a Cauchy sequence in Probaility, i.e. for each δ > 0,

lim_{{n→∞ \atop m→∞}} P \{|S_{n} − S_{m}| ≥ δ\} = 0

then there is a random variable S such that S_{n} → S in probability, i.e for each δ > 0,

lim_{n→∞} P {|S_{n} − S| ≥ δ} = 0.

Exercise 3.11. Prove that if a sequence S_{n} of random variables satisfies

lim_{{n→∞ \atop m→∞}} P \{ sup_{m<k≤n} |S_{k} − S_{m}| ≥ δ\} = 0

for every δ > 0 then there is a limiting random variable S(ω) such that

P { lim_{n→∞} S_{n}(ω) = S(ω)} = 1

Exercise 3.12. Prove that whenever X_{n} → X in probability the distribution α_{n} of X_{n} converges weakly to the distribution α of X.

Now it is straightforward to find sufficient conditions for the convergence of an infinite series of independent random variables.

Theorem 3.10. (Kolmogorov’s one series Theorem). If E(X_{i}) = 0 and ∑_{i=1}^{∞} Var(X_{i}) < ∞, then

S(ω) = ∑_{i=1}^{∞} X_{i}(ω)

converges with probability 1.

Proof. By a direct application of Kolmogorov’s inequality

lim_{{n→∞ \atop m→∞}} P \{ sup_{m<k≤n} |S_{n} − S_{m}| ≥ δ\} ≤ lim_{{n→∞ \atop m→∞}} ∑_{j=m+1}^{n} E(X_{i}^{2})
= lim_{{n→∞ \atop m→∞}} ∑_{j=m+1}^{n} Var(X_{i}) = 0

lim_{{n→∞ \atop m→∞}} P \{|S_{n} − S_{m}| ≥ δ\} ≤ 0

or we can first prove convergence in probability

lim_{{n→∞ \atop m→∞}} P \{|S_{n} − S_{m}| ≥ δ\} = 0

by a simple application of Chebychev’s inequality and then apply Lévy’s Theorem.

Theorem 3.11. (Kolomogorov’s two series theorem). If a_{i} = E(X_{i}) and σ_{i}^{2} = Var(X_{i}) are such that ∑_{i} a_{i} and ∑_{i} σ_{i}^{2} converge, then the series ∑_{i} X_{i} converges with probability 1.

Proof. Define Y_{i} = X_{i} − a_{i} and apply the previous theorem to Y_{i}. Of

course in general random variables need not have finite expectations or variances. If {X_{i}} is any sequence of random variables we can take a cut off value C and define Y_{i} = X_{i} if |X_{i}| ≤ C, and Y_{i} = 0 otherwise. The Y_{i} are then independent and bounded in absolute value by C. The theorem can be applied to Y_{i} and if we impose the additional condition that

∑_{i} P {X_{i} ≠ Y_{i}} = ∑_{i} P {|X_{i}| > C} < ∞

by an application of Borel-Cantelli Lemma, with Probabilty 1, X_{i} = Y_{i} for all sufficiently large i. The convergence of ∑_{i} X_{i} and ∑_{i} Y_{i} are therefore equivalent. We get then Theorem 3.12. (Kolmogorov’s three series theorem). For the convergence of an infinite series of independent random variables ∑_{i} X_{i} it is sufficient that the the following three infinite series converge.

(i) For some cut off value C > 0, ∑_{i} P {|X_{i}| > C} converges.

(ii) If Y_{i} is defined to equal X_{i} if |X_{i}| ≤ C, and 0 otherwise, ∑_{i} E(Y_{i}) converges.

(iii) With Y_{i} as in (ii), ∑_{i} Var(Y_{i}) converges.

The converse is also true. If ∑_{i} X_{i} converges for a sequence of independent random variables, we must necessarily have |X_{n}| ≤ C eventually with probability 1. By Borel-Cantelli Lemma the first series must converge. This means that in order to prove the necessity we can assume with out loss of generality that |X_{i}| are all bounded say by 1. we may also assume that E(X_{i}) = 0 for each i. Otherwise let us take independent random variables X′_{i} that have the same distribution as X_{i}. Then ∑_{i} X_{i} as well as ∑_{i} X′_{i} converge with probability 1 and therefore so does ∑_{i}(X_{i} − X′_{i}). The random variables Z_{i} = X_{i} − X′_{i} are independent and bounded by 2. They have mean 0. If we can show ∑ Var(Z_{i}) is convergent, since Var(Z_{i}) = 2Var(X_{i}) we would have proved the convergence of the the third series. Now it is elementary to conclude that since both ∑_{i} X_{i} as well as ∑_{i}(X_{i} − E(X_{i})) converge, the series ∑_{i} E(X_{i}) must be convergent as well. So all we need is the following

Lemma 3.13. If ∑_{i} X_{i} is convergent for a series of independent random variables with mean 0 that are individually bounded by C, then ∑_{i} Var(X_{i}) is convergent.

Proof. Let F_{n} = {ω : |S_{1}| ≤ ℓ, |S_{2}| ≤ ℓ, ⋯, |S_{n}| ≤ ℓ} where S_{k} = X_{1} + ⋯ + X_{k}. If the series converges with probablity 1, we must have, for some ℓ and δ > 0, P (F_{n}) ≥ δ for all n. We have

∫_{F_{n−1}} S_{n}^{2} dP = ∫_{F_{n−1}}[S_{n−1} + X_{n}]^{2} dP
= ∫_{F_{n−1}}[S_{n−1}^{2} + 2S_{n−1}X_{n} + X_{n}^{2}] dP

= ∫_{F_{n−1}} S_{n−1}^{2} dP + σ_{n}^{2}P (F_{n−1})
≥ ∫_{F_{n−1}} S_{n−1}^{2} dP + δσ_{n}^{2}

and on the other hand,

∫_{F_{n−1}} S_{n}^{2} dP = ∫_{F_{n}} S_{n}^{2} dP + ∫_{F_{n−1}∩F_{n}^{c}} S_{n}^{2} dP
≤ ∫_{F_{n}} S_{n}^{2} dP + P (F_{n−1} ∩ F_{n}^{c}) (ℓ + C)^{2}

providing us with the estimate

δσ_{n}^{2} ≤ ∫_{F_{n}} S_{n}^{2} dP − ∫_{F_{n−1}} S_{n−1}^{2} dP + P (F_{n−1} ∩ F_{n}^{c}) (ℓ + C)^{2}

Since F_{n−1} ∩ F_{n}^{c} are disjoint and |S_{n}| ≤ ℓ on F_{n} ,

∑_{j=1}^{∞} σ_{j}^{2} ≤ \frac{1}{δ}[ℓ^{2} + (ℓ + C)^{2}]

This concludes the proof.

Strong Law of Large Numbers

We will now prove the theorem that if X_{1} ⋯, X_{n}, ⋯ is a sequence of i.i.d. random variables with E|X_{i}| < ∞ and E(X_{i}) = 0, then Theorem 3.14. With probability 1,

lim_{n→∞} \frac{S_{n}}{n} = 0

where S_{n} = X_{1} + ⋯ + X_{n}. Proof. We define

\{ \begin{matrix}Y_{n} = X_{n} if |X_{n}| ≤ n \\ Y_{n} = 0 otherwise \end{matrix}
a_{n} = P [X_{n} ≠ Y_{n}], b_{n} = E[Y_{n}] and c_{n} = Var(Y_{n}).

First we note that

∑_{n} a_{n} = ∑_{n} P [|X_{1}| > n] ≤ E|X_{1}| < ∞

lim_{n→∞} b_{n} = 0

and

∑_{n} \frac{c_{n}}{n^{2}} ≤ ∑_{n} \frac{E[Y_{n}^{2}]}{n^{2}} = ∑_{n} ∫_{|x|≤n} \frac{x^{2}}{n^{2}} dα
= ∫ x^{2}( ∑_{n≥x} \frac{1}{n^{2}} ) dα ≤ C ∫ |x| dα < ∞

where α is the common distribution of X_{i}. From the three series theorem and the Borel-Cantelli Lemma, we conclude that ∑_{n} \frac{Y_{n}−b_{n}}{n} as well as ∑_{n} \frac{X_{n}−b_{n}}{n} converge almost surely. It is elementary to verify that for any series ∑_{n} \frac{x_{n}}{n} that converges, \frac{x_{1}+···+x_{n}}{n} → 0 as n → ∞. We therefore conclude that

P \{ lim_{n→∞} [\frac{X_{1} + ⋯ + X_{n}}{n} − \frac{b_{1} + ⋯ + b_{n}}{n}] = 0\} = 1

Since b_{n} → 0 as n → ∞, the theorem is proved.

Exercise 3.13. Show that for a sequence of i.i.d. random variables X_{1}, ⋯, X_{n}, ⋯, if the strong law of large numbers holds with some limit, i.e.

P [ lim_{n→∞} \frac{S_{n}}{n} = ξ] = 1

for some random variable ξ, which may or may not be a constant with probability 1, then necessarily E|X_{i}| < ∞. Consequently ξ = E(X_{i}) with probabilty 1.

One may ask why the limit cannot be a proper random variable. There is a general theorem that forbids it called Kolmogorov’s Zero-One law. Let us look at the space Ω of real sequences {x_{n} : n ≥ 1}. We have the σ-field B, the product σ-field on Ω. In addition we have the sub σ-fields B^{n} generated by {x_{j} : j ≥ n}. B^{n} are ↓ with n and B^{∞} = ∩_{n} B^{n} which is also a σ-field is called the tail σ-field. The typical set in B^{∞} is set depending on the tail behavior of the sequence. For example the sets {ω : x_{n} is bounded }, {ω : lim sup_{n} x_{n} = 1} are in B^{∞} where as {ω : sup_{n} |x_{n}| = 1} is not.

Theorem 3.15. (Kolmogorov’s Zero-One Law). If A ∈ B^{∞} and P is any product measure (not necessarily with identical components) P (A) = 0 or 1.

Proof. The proof depends on showing that A is independent of itself so that P (A) = P (A ∩ A) = P (A)P (A) = [P (A)]^{2} and therefore equals 0 or 1. The proof is elementary. Since A ∈ B^{∞} ⊂ B^{n+1} and P is a product measure, A is independent of B_{n} = σ-field generated by {x_{j} : 1 ≤ j ≤ n}. It is therefore independent of sets in the field F = ∪_{n}B_{n}. The class of sets A that are independent of A is a monotone class. Since it contains the field F it contains the σ-field B generated by F. In particular since A ∈ B, A is independent of itself.

Corollary 3.16. Any random variable measurable with respect to the tail σ-field B^{∞} is equal with probaility 1 to a constant relative to any given product measure.

Warning. For different product measures the constants can be different.

Exercise 3.14. How can that happen?