Chapter 2

Weak Convergence

Characteristic Functions

If we have a probability distribution α on the line then its characteristic function is defined by

ϕ(t) = ∫ exp[ i t x] dα  (2.1)

The above definition makes sense. We write the integrand e^{itx} as cos tx + i sin tx and integrate each part to see that

|ϕ(t)| ≤ 1

for all real t.

Exercise 2.1. Calculate the characteristic functions for the following distributions:

1. α is the degenerate distribution δ_{a} with probability one at the point a.

2. α is the binomial distribution masses

p_{k} = P rob[X = k] = \binom{n}{k}p^{k}(1 − p)^{n−k}

for 0 ≤ k ≤ n.

Theorem 2.1. The characteristic function ϕ(t) of any probability distribution is a uniformly continuous function of t that is positive definite i.e. for any n complex numbers ξ_{1}, ⋯, ξ_{n} and real nubers t_{1}, ⋯, t_{n}

∑_{i,j=1}^{n} ϕ(t_{i} − t_{j}) ξ_{i} ξ̄_{j} ≥ 0

Proof: Let us note that

∑_{i,j=1}^{n} ϕ(t_{i} − t_{j}) ξ_{i} ξ̄_{j} = ∑_{i,j=1}^{n} ξ_{i} ξ̄_{j} ∫ exp[ i(t_{i} − t_{j})x] dα
= ∫ | ∑_{j=1}^{n} ξ_{j} exp[i t_{j} x]|^{2} dα ≥ 0

To prove uniform continuity we see that

|ϕ(t) − ϕ(s)| ≤ ∫ | exp[ i(t − s) x] − 1| dP

which tends to 0 by the bounded convergence theorem if |t − s| → 0.

The characteristic function of course carries some information about the distribution α. In particular if ∫ |x| dα < ∞, then ϕ(·) is continuously diffrentiable and ϕ′(0) = i ∫ x dα.

Exercise 2.2. Prove it!

Warning: The converse need not be true. ϕ(·) can be continuously differentiable but ∫ |x| dP could be ∞.

Exercise 2.3. Construct a counterexample along the followimg lines. Take a discrete distribution, symmetric around 0 with

α\{n\} = α\{−n\} = p(n) ≃ \frac{1}{n^{2} log n}

Then show that ∑_{n} \frac{(1−cos nt)}{n^{2} log n} is a continuously differentiable function of t.

Exercise 2.4. The story with higher moments m_{r} = ∫ x^{r} dα is similar. If any of them, say m_{r} exists, then ϕ(·) is r times continuously differentiable and ϕ^{(r)}(0) = i^{r}m_{r}. The converse is false for odd r, but true for even r.

The next question is how to recover the distribution function F (x) from ϕ(t). If we go back to the Fourier inversion formula, we can ‘guess’ that

F′(x) = \frac{1}{2π} ∫_{−∞}^{∞} exp[−i t x] ϕ(t) dt

and therefore

F (b) − F (a) = \frac{1}{2π} ∫_{a}^{b} dx ∫_{−∞}^{∞} exp[−i t x] ϕ(t) dt
= \frac{1}{2π} ∫_{−∞}^{∞} ϕ(t) dt ∫_{a}^{b} exp[−i t x] dx
= \frac{1}{2π} ∫_{−∞}^{∞} ϕ(t)\frac{exp[− i t b]−exp[− i t a]}{− i t} dt
= lim_{T →∞} \frac{1}{2π} ∫_{−T}^{T} ϕ(t)\frac{exp[− i t b]−exp[− i t a]}{− i t}

We will in fact prove it if a and b are points of continuity of F. We compute the right hand side as

lim_{T →∞} \frac{1}{2π} ∫_{−T}^{T} \frac{exp[− i t b] − exp[− i t a]}{− i t} dt ∫ exp[ i t x] dα

= lim_{T →∞} \frac{1}{2π} ∫ dα ∫_{−T}^{T} \frac{exp[i t (x−b) ]−exp[i t (x−a) ]}{− i t} dt
= lim_{T →∞} \frac{1}{2π} ∫ dα ∫_{−T}^{T} \frac{sin t (x−a)−sin t (x−b)}{t} dt
= \frac{1}{2} ∫ [sign(x − a) − sign(x − b)] dα
= F (b) − F (a)

provided a and b are continuity points. We have applied the bounded convergence theorem to take the limit as T → ∞.

As a consequence we conclude that the distribution function and hence α is determined uniquely by the characteristic function.

Weak Convergence

One of the basic ideas in Limit Theorems is the notion of weak Convergence of a sequence of probability distributions on the line R. Since the role of a probability measure is to assign probabilities to sets, we should expect that if two probability measures are to be close then they should assign for a given set, probabilities that are nearly equal. This suggets the definition

d(P_{1}, P_{2}) = sup_{A∈B} |P_{1}(A) − P_{2}(A)|

as the distance between two probability measures P_{1} and P_{2} on a measurable space (Ω, B). This is too strong. If we take P_{1} and P_{2} to be degenerate distributions with probability 1 concentrated at two points x_{1} and x_{2} on the line one can see that, as soon as x_{1} ≠ x_{2}, d(P_{1}, P_{2}) = 1, and the above metric is not sensitive to how close the two points x_{1} and x_{2} are. It only cares that they are unequal. The problem is not because of the supremum. We can take A to be an interval [a, b] that includes x_{1} but omits x_{2} and |P_{1}(A)−P_{2}(A)| = 1. On the other hand if the end points of the interval are kept away from x_{1} or x_{2} the situation is not that bad. This leads to the following definition.

Definition 2.1. A sequence α_{n} of probabiity distributions on R is said to converge weakly to a probability distribution α if,

lim_{n→∞} α_{n}[I] = α[I]

for any interval I = [a, b] such that the single point sets a and b have probability 0 under α. Remark. One says that a sequence X_{n} of random variables converges in law or in distribution to X if the distributions α_{n} of X_{n} converges weakly to α the distribution of X.

One can state this more succinctly in terms of the distribution functions F_{n}(x) and F (x) corrresponding to the measures α_{n} and α respectively. Definition 2.2. A sequence α_{n} of probability measures on the real line R with distribution functions F_{n}(x) is said to converge weakly to a limiting probability measure α weakly (in symbols α_{n} ⇒ α or F_{n} ⇒ F) if

lim_{n→∞} F_{n}(x) = F (x)

for every x that is a continuity point of F.

There are equivalent formulations interms of expectations and characteristic functions.

Theorem 2.2. The following are equivalent.

1. α_{n} ⇒ α or F_{n} ⇒ F

2. For every bounded continuous function f(x) on R

lim_{n→∞} ∫_{R} f(x) dα_{n} = ∫_{R} f(x) dα

3. If ϕ_{n}(t) and ϕ(t) are respectively the characteristic functions of α_{n} and α, for every real t,

lim_{n→∞} ϕ_{n}(t) = ϕ(t)

Proof: We first prove (a ⇒ b). Let ϵ > 0 be arbitrary. Find continuity points a and b of F such that a < b, F (a) ≤ ϵ and 1 − F (b) ≤ ϵ. Since F_{n}(a) and F_{n}(b) converge to F (a) and F (b), for n large enough, F_{n}(a) ≤ 2ϵ and 1 − F_{n}(b) ≤ 2ϵ. Divide the interval [a, b] into a finite number N = N_{δ} of small subintervals I_{j} = (a_{j}, a_{j+1}], 1 ≤ j ≤ N with a = a_{1} < a_{2} < ⋯ < a_{N+1} = b such that all the end points {a_{j}} are points of continuity of F and the oscillation of the continuous function f in each I_{j} is less than a preassigned number δ. Since the continuous function f is uniformly continuous in the closed bounded interval [a, b] this is always possible for any given δ > 0. Let h(x) = ∑_{j=1}^{N} χ_{I_{j}}f(a_{j}) be the simple function equal to f(a_{j}) on I_{j} and 0 outside ∪_{j}I_{j} = (a, b]. We have |f(x) − h(x)| ≤ δ on (a, b]. If f(x) is bounded by M, then

| ∫ f(x) dα_{n} − ∑_{j=1}^{N} f(a_{j})[F_{n}(a_{j+1}) − F_{n}(a_{j})]| ≤ δ + 4Mϵ

and

| ∫ f(x) dα − ∑_{j=1}^{N} f(a_{j})[F (a_{j+1}) − F (a_{j})]| ≤ δ + 2Mϵ

Since lim_{n→∞} F_{n}(a_{j}) = F (a_{j}) for every 1 ≤ j ≤ N, we conclude that

lim sup_{n→∞} | ∫ f(x) dα_{n} − ∫ f(x) dα| ≤ 2δ + 6Mϵ

Since ϵ and δ are arbitrary small numbers we are done. (b ⇒ c) is

trivial. We can make the choice of f(x) = exp[i t x] = cos tx + i sin tx which is a bounded continuous function. (c ⇒ a) is the hardest. It is

carried out in several steps. Actually we will prove a stronger version as a separate theorem. Theorem 2.3. For each n ≥ 1, let ϕ_{n}(t) be the characteristic function of a probability distribution α_{n}. Assume that lim_{n→∞} ϕ_{n}(t) = ϕ(t) exists for each t and ϕ(t) is continuous at t = 0. Then ϕ(t) is the characteristic function of some probability distribution α and α_{n} ⇒ α.

Proof: Step 1. Let r_{1}, r_{2}, ⋯ be an enumeration of the rational numbers. For each j consider the sequence {F_{n}(r_{j}) : n ≥ 1} where F_{n} is the distribution function corresponding to ϕ_{n}(·). It is a sequence bounded by 1 and we can extract a subsequence that converges. By the diagonalization process we can choose a subseqence G_{k} = F_{n_{k}} such that

lim_{n→∞} G_{k}(r) = b_{r}

exists for every rational number r. From the monotonicity of F_{n} in x we conclude that if r_{1} < r_{2}, then b_{r_{1}} ≤ b_{r_{2}}.

Step 2. From the skeleton b_{r} we reconstruct a right continuous monotone function G(x). We defiine

G(x) = inf_{r>x} b_{r}

Clearly G is nondecreasing. If x_{n} ↓ x, any r > x satisfies r > x_{n} for sufficiently large n. This allows us to conclude that G(x) = inf_{n} G(x_{n}) for any sequence x_{n} ↓ x, proving that G(x) is right continuous.

Step 3. Next we show that at any continuity point x of G

lim_{n→∞} G_{n}(x) = G(x)

Let r > x be a rational number. Then G_{n}(x) ≤ G_{n}(r) and G_{n}(r) → b_{r} as n → ∞. Hence

lim sup_{n→∞} G_{n}(x) ≤ b_{r}

This is true for every rational r > x, and therefore

lim sup_{n→∞} G_{n}(x) ≤ G(x)

Suppose now that we have y < x. Find a rational r such that y < r < x.

lim inf_{n→∞} G_{n}(x) ≥ lim inf_{n→∞} G_{n}(r) = b_{r} ≥ G(y)

As this is true for every y < x,

lim inf_{n→∞} G_{n}(x) ≥ sup_{y<x} G(y) = G(x − 0) = G(x)

the last step being a consequence of the assumption that x is a point of continuity of G i.e. G(x − 0) = G(x).

Warning. This does not mean that G is necessarily a distribution function. Consider F_{n}(x) = 0 for x < n and 1 for x ≥ n. In this case lim_{n→∞} F_{n}(x) = G(x) exists and G(x) ≡ 0, which is not a distribution function.

Step 4. We will use the continuity at t = 0, of ϕ(t), to show that G is a distribution function. If ϕ(t) is the characteristic function of α

\frac{1}{2T} ∫_{−T}^{T} ϕ(t) dt = ∫ [\frac{1}{2T} ∫ exp[i t x] dt] dα
= ∫ \frac{sin T x}{T x} dα
≤ ∫ |\frac{sin T x}{T x} | dα
= ∫_{|x|<ℓ} |\frac{sin T x}{T x} | dα + ∫_{|x|≥ℓ} |\frac{sin T x}{T x} | dα
≤ α[|x| < ℓ] + \frac{1}{T ℓ} α[|x| ≥ ℓ]

We have used Fubini’s theorem in the first line and the bounds | sin x| ≤ |x| and | sin x| ≤ 1 in the last line. We can rewrite this as

1 − \frac{1}{2T} ∫_{−T}^{T} ϕ(t) dt ≥ 1 − α[|x| < ℓ] − \frac{1}{T ℓ} α[|x| ≥ ℓ]
= (1 − \frac{1}{T ℓ})α[|x| ≥ ℓ]
≥ (1 − \frac{1}{T ℓ})[1 − F (ℓ) + F (−ℓ)]

Finally, if we pick ℓ = \frac{2}{T},

[1 − F (\frac{2}{T}) + F (−\frac{2}{T})] ≤ 2[1 − \frac{1}{2T} ∫_{−T}^{T} ϕ(t) dt]

Since this inequality is valid for any distribution function F and its characteristic function ϕ, we conclude that, for every n ≥ 1,

[1 − F_{n_{k}}(\frac{2}{T}) + F_{n_{k}}(−\frac{2}{T})] ≤ 2[1 − \frac{1}{2T} ∫_{−T}^{T} ϕ_{n_{k}}(t) dt].

We can pick T such that ±\frac{2}{T} are continuity points of G. If we now pass to the limit and use the bounded convergence theorem, we obtain

[1 − G(\frac{2}{T}) + G(−\frac{2}{T})] ≤ 2[1 − \frac{1}{2T} ∫_{−T}^{T} ϕ(t) dt]

Since ϕ(0) = 1 and ϕ is continuous at t = 0, by letting T → 0 in such a way that ±\frac{2}{T} are continuity points of G, we conclude that

1 − G(∞) + G(−∞) = 0

or G is indeed a distribution function

Step 4. We now complete the rest of the proof, i.e. show that α_{n} ⇒ α. We have G_{k} = F_{n_{k}} ⇒ G as well as ψ_{k} = ϕ_{n_{k}} → ϕ. Therefore G must equal F which has ϕ for its characteristic function. Since the argument works for any subsequence of F_{n}, every subsequence of F_{n} will have a further subsequence that converges weakly to the same limit F uniquely determined as the distribution function whose characteristic function is ϕ(·). Consequently F_{n} ⇒ F or α_{n} ⇒ α.

Definition 2.3. A subset A of probability distributions on R is said to be totally bounded if, given any sequence α_{n} from A, there is subsequence that converges weakly to some limiting probability distribution α.

Theorem 2.4. In order that a family A of probability distributions to be totally bounded it is necessary and sufficient that either of the following equivalent conditions hold. a)

lim_{ℓ→∞} sup_{α∈A} α[x : |x| ≥ ℓ] = 0

b)

lim_{h→0} sup_{α∈A} sup_{|t|≤h} |1 − ϕ_{α}(t)| = 0

Here ϕ_{α}(t) is the characteristic function of α.

Condition a) is often called the uniform tightness property.

Proof. The proof is already contained in the details of the proof of the earlier theorem. We can always choose a subsequence such that the distribution functions converge at rationals and try to reconstruct the limiting distribution function from the limits at rationals. The crucial step is to prove that the limit is a distribution function. Either of the two conditions a) or b) will guarantee this. If a) is violated it is straight forward to pick a sequence from A for which the distribution functions have a limit which is not a distribution function. Then A cannot be totally bounded. a) is therefore necessary. That a)⇒ b), is a consequence of the estimate

|1 − ϕ(t)| ≤ ∫ | exp[ i t x] − 1| dα
= ∫_{|x|≤ℓ} | exp[ i t x] − 1| dα + ∫_{|x|>ℓ} | exp[ i t x] − 1| dα
≤ |t|ℓ + 2α[x : |x| > ℓ]

The regularity of ϕ(t) at t = 0 is related to the decay rate of the tail probabilities.

Exercise 2.5. Compute ∫ |x|^{p} dα in terms of the characteristic function ϕ(t) for p in the range 0 < p < 2.

Hint: Look at the formula

∫_{−∞}^{∞} \frac{1 − cos tx}{|t|^{p+1}}dt = C_{q}|x|^{p}

and use Fubini’s theorem.

Bochner’s Theorem

We are now ready to prove the converse of Theorem 2.1 due to Bochner.

Theorem 2.5. (Bochner). If ϕ(t) is a positive definite function which is continuous at t = 0 and is normalized so that ϕ(0) = 1, then ϕ is the characteristic function of some probability ditribution on R.

Proof: The proof depends on constructing approximations ϕ_{n}(t) which are in fact characteristic functions and satisfy ϕ_{n}(t) → ϕ(t) as n → ∞. Then we can apply the preceeding theorem and the probability measures corresponding to ϕ_{n} will have a weak limit which will have ϕ for its characteristic function.

Step 1. Let us establish a few elementary properties of positive definite functions.

1) If ϕ(t) is a positive definite function so is ϕ(t)exp[ i t a] for any real a. The proof is elementary and requires just direct verification.

2) If ϕ_{j}(t) are positive definite for each j then so is any linear combination ϕ(t) = ∑_{j} w_{j}ϕ_{j}(t) with nonnegative weights w_{j}. If each ϕ_{j}(t) is normalized with ϕ_{j}(0) = 1 and ∑_{j} w_{j} = 1, then of of course ϕ(0) = 1 as well.

3) If ϕ is positive definite then ϕ satisfies ϕ(0) ≥ 0, ϕ(−t) = \overline{ϕ(t)} and |ϕ(t)| ≤ ϕ(0) for all t.

We use the fact that the matrix {ϕ(t_{i} − t_{j}) : 1 ≤ i, j ≤ n} is Hermitian positive definite for any n real numbers t_{1}, ⋯, t_{n}. The first assertion follows from the the positivity of ϕ(0)|z|^{2}, the second is a consequence of the Hermitian property and if we take n = 2 with t_{1} = t and t_{2} = 0 as a consequence of the positive definiteness of the 2 × 2 matrix we get |ϕ(t)|^{2} ≤ |ϕ(0)|^{2}

4) For any s, t we have |ϕ(t) − ϕ(s)|^{2} ≤ 4ϕ(0)|ϕ(0) − ϕ(t − s)| We use the positive definiteness of the 3 × 3 matrix

\begin{matrix}1 & ϕ(t − s) & ϕ(t) \\ \overline{ϕ(t − s)} & 1 & ϕ(s) \\ \overline{ϕ(t)} & \overline{ϕ(s)} & 1 \end{matrix}

which is {ϕ(t_{i} − t_{j})} with t_{1} = t, t_{2} = s and t_{3} = 0. In particular the determinant has to be nonnegative.

0 ≤ 1 + ϕ(s)ϕ(t − s)\overline{ϕ(t)} + \overline{ϕ(s)}\overline{ϕ(t − s)}ϕ(t) − |ϕ(s)|^{2}
−|ϕ(t)|^{2} − |ϕ(t − s)|^{2}
= 1 − |ϕ(s) − ϕ(t)|^{2} − |ϕ(t − s)|^{2} − ϕ(t)\overline{ϕ(s)}(1 − ϕ(t − s))
−\overline{ϕ(t)}ϕ(s)(1 − \overline{ϕ(t − s)})
≤ 1 − |ϕ(s) − ϕ(t)|^{2} − |ϕ(t − s)|^{2} + 2|1 − ϕ(t − s)|

Or

\begin{aligned}|ϕ(s) − ϕ(t)|^{2} & ≤ & 1 − |ϕ(s − t)|^{2} + 2|1 − ϕ(t − s)| \\ & ≤ & 4|1 − ϕ(s − t)| \end{aligned}

5) It now follows from 4) that if a positive definite function is continuous at t = 0, it is continuous everywhere.

Step 2. First we show that if ϕ(t) is a positive definite function which is continuous on R and is absolutely integrable, then

f(x) = \frac{1}{2π} ∫_{−∞}^{∞} exp[− i t x]ϕ(t) dt ≥ 0

is a continuous function and

∫_{−∞}^{∞} f(x)dx = 1.

Moreover the function

F (x) = ∫_{−∞}^{x} f(y) dy

defines a distribution function with characteristic function

ϕ(t) = ∫_{−∞}^{∞} exp[i t x]f(x) dx  (2.2)

If ϕ is integrable on (−∞, ∞), then f(x) is clearly bounded and continuous. To see that it is nonnegative we write

f(x) = lim_{T →∞} \frac{1}{2π} ∫_{−T}^{T} (1 − \frac{|t|}{T})e^{− i t x}ϕ(t) dt  (2.3)

= lim_{T →∞} \frac{1}{2πT} ∫_{0}^{T} ∫_{0}^{T} e^{− i (t−s) x}ϕ(t − s) dt ds  (2.4)
= lim_{T →∞} \frac{1}{2πT} ∫_{0}^{T} ∫_{0}^{T} e^{− i t x} e^{i s x}ϕ(t − s) dt ds  (2.5)
≥ 0

We can use the dominated convergence theorem to prove 2.3, a change of variables to show 2.4 and finally a Riemann sum approximation to the integral and the positive definiteness of ϕ to show that 2.5 is nonnegative. It reamains to show the relation 2.2. Let us define

f_{σ}(x) = f(x) exp[ −\frac{σ^{2}x^{2}}{2} ]

and calculate

∫_{−∞}^{∞} e^{i t x}f_{σ}(x) dx = ∫_{−∞}^{∞} e^{i t x}f(x) exp[ −\frac{σ^{2}x^{2}}{2} ] dx
= \frac{1}{2π} ∫_{−∞}^{∞} ∫_{−∞}^{∞} e^{i t x}ϕ(s)e^{i s x} exp[ −\frac{σ^{2}x^{2}}{2} ] ds dx
= ∫_{−∞}^{∞} ϕ(s)\frac{1}{\sqrt{2π}σ} exp[ −\frac{(t−s)^{2}}{2σ^{2}} ] ds

by Fubini’s thorem. It is now clear that

∫_{−∞}^{∞} f_{σ}(x) dx = ∫_{−∞}^{∞} ϕ(s)\frac{1}{\sqrt{2π}σ} exp[ −\frac{s^{2}}{2σ^{2}} ] ds ≤ 1

and as a consequence of the monotone convergence theorem, by letting σ → 0, we have established our claim.

Step 3. If ϕ(t) is a positive definite function which is continuous, so is ϕ(t) exp[ i t y] for every y and for σ > 0, as well as the convex combination

ϕ_{σ}(t) = ∫_{−∞}^{∞} ϕ(t) exp[ i t y]\frac{1}{\sqrt{2π}σ} exp[ −\frac{y^{2}}{2σ^{2}} ] dy
= ϕ(t) exp[ −\frac{σ^{2}t^{2}}{2} ]

The previous step is applicable to ϕ_{σ}(t) which is clearly integrable on R and by letting σ → 0 we conclude by Theorem 2.3. that ϕ is a characteristic function as well.

Theorem 2.6. Let α_{n} ⇒ α on R and C be a closed subset of R. Then

lim sup_{n→∞} α_{n}(C) ≤ α(C).

If A ⊂ R is a continuity set of α i.e. α(∂A) = α(Ā − A^{o}) = 0, then

lim_{n→∞} α_{n}(A) = α(A)

Proof: The function d(x, C) = inf_{y∈C} |x−y| is a continuous and equals 0 precisely on C.

f(x) = \frac{1}{1 + d(x, C)}

is a continuous function bounded by 1, that is equal to 1 precisely on C and

f_{k}(x) = [f(x)]^{k} ↓ χ_{C}(x)

as k → ∞. From

lim_{n→∞} ∫ f_{k}(x) dα_{n} = ∫ f_{k}(x) dα

for each k ≥ 1, we deduce that

lim sup_{n→∞} α_{n}(C) ≤ α(C)

Taking complements we conclude that for any open set G ⊂ R

lim inf_{n→∞} α_{n}(G) ≥ α(G)

Combining the two parts, if A ⊂ R is a continuity set of α i.e. α(∂A) = α(Ā − A^{o}) = 0, then

lim_{n→∞} α_{n}(A) = α(A)

Exercise 2.6. If α_{n} ⇒ α it is not always true that ∫ x dα_{n} → ∫ x dα because while x is a continuous function it is not bounded. Construct a simple counterexample. On the positive side, let f(x) be a continuous function that is not necessarily bounded. Assume that there exists a positive continuous function g(x) satisfying

lim_{|x|→∞} \frac{|f(x)|}{g(x)} = 0

and

sup_{n} ∫ g(x) dα_{n} ≤ C < ∞

Then show that

lim_{n→∞} ∫ f(x) dα_{n} = ∫ f(x) dα

In particular if ∫ |x|^{k} dα_{n} remains bounded, then ∫ |x|^{j} dα_{n} → ∫ |x|^{j} dα for 1 ≤ j ≤ k − 1.