Probability Theory

S.R.S.Varadhan

Courant Institute of Mathematical Sciences New York University

Fall 1999

Chapter 1

Measure Theory

Introduction

The evolution of probability theory was based more on intuition rather than mathematical axioms during its early development. In the 1930’s A. N. Kolmogorov provided an axiomatic basis for probability theory and it is now the universally accepted model. There are certain ‘non commutative’ versions that have their origins in quantum mechanics that are generalizations of the Kolmogorov Model. We shall however use exclusively Kolmogorov’s framework.

The basic intuition in probability theory is the notion of randomness. There are experiments whose results are not predicatable and can be determined only after performing it and then observing the outcome. The simplest familiar examples are, the tossing of a fair coin, or the throwing of a balanced die. In the first experiment the result could be either a head or a tail and the throwing of a die could result in a score of any integer from 1 through 6. These are experiments with only a finite number of alternate outcomes. It is not difficult to imagine experiments that have countably or even uncountably many alternatives as possible outcomes.

Abstractly then, there is a space Ω of all possible outcomes and each individual outcome is represented as a point ω in that space Ω. Subsets of Ω are called events and they correspond to a collection of outcomes. If the outcome ω is in the subset A, then the event A is said to have occurred. For example in the case of a die the set A = {1, 3, 5} ⊂ Ω corresponds to the event ‘an odd number shows up’. With this terminology it is clear that union of sets corresponds to ‘or’, intersection to ‘and’, and complementation to ‘negation’.

One would expect that probabilities should be associated with each outcome and there should be a ‘Probability Function’ f(ω) which is the probabilty that ω occurs. In the case of coin tossing we may expect Ω = {H, T } and

f(T) = f(H) = \frac{1}{2}.

Or in the case of a die

f(1) = f(2) = ⋯ = f(6) = \frac{1}{6}

Since ‘Probaility’ is normalized so that certainty corresponds to a Probability of 1, one expects

∑_{ω∈Ω} f(ω) = 1  (1.1)

If Ω is uncountable this is a mess. There is no reasonable way of adding up an uncountable set of numbers each of which is 0. This suggests that it may not be possible to start with probabilities associated with individual outcomes and build a meaningful theory. The next best thing is to start with the notion that probabilities are already defined for events. In such a case, P (A) is defined for a class B of subsets A ⊂ Ω. The question that arises naturally is what should B be and what properties should P (·) defined on B have? It is natural to demand that the class B of sets for which probabilities are to be defined satisfy the following properties:

The sets Ω and Φ are in B. For any two sets A and B in B, the sets A ∪ B and A ∩ B are again in B. If A ∈ B, then the complement A^{c} is again in B. Any class of sets satisfying these properties is called a field.

Definition 1.1. A ‘probability’ or more precisely ‘a finitely additive probability measure’ is a nonnegative set function defined for sets in B that satisfies the following properties:

P (A) ≥ 0 for all A ∈ B  (1.2)

P (Ω) = 1 and P (Φ) = 0  (1.3)

If A ∈ B and B ∈ B are disjoint then

P (A ∪ B) = P (A) + P (B)  (1.4)

In particular

P (A^{c}) = 1 − P (A)  (1.5)

for all A ∈ B

A condition which is some what more technical, but important from a mathematical viewpoint is that of countable additivity. The class B, in addition to being a field is assumed to be closed under countable union (or equivalently, intersection); i.e. if A_{n} ∈ B for every n, then A = ∪_{n}A_{n} ∈ B . Such a class is called a σ-field. The ‘probability’ itself is presumed to be defined on a σ-field B.

A set function P defined on a σ-field is called a ‘countably additive probability measure’ if in addition to satsfying equations (1.2), (1.3) and (1.4) it satisfies the following countable additivity property: for any sequence of pairwise disjoint sets A_{n} with A = ∪_{n}A_{n}

P (A) = ∑_{n} P (A_{n})  (1.6)

Exercise 1.1. The limit of an increasing (or decreasing) sequence A_{n} of sets is defined as its union ∪_{n}A_{n} (or the intersection ∩_{n}A_{n}). A monotone class is defined as a class that is closed under monotone limits of an increasing or decreasing sequence of sets. Show that a field B is a σ-field if and only if it is a monotone class.

Exercise 1.2. Show that a finitely additive probability measure P (·) defined on a σ-field B, is countably additive, i.e. satisfies equation (1.6), if and only if either of the following two conditions is satsfied.

If A_{n} is any nonincreasing sequence of sets in B and A = lim_{n} A_{n} = ∩_{n}A_{n} then

P (A) = lim_{n} P (A_{n}).

If A_{n} is any nondecreasing sequence of sets in B and A = lim_{n} A_{n} = ∪_{n}A_{n} then

P (A) = lim_{n} P (A_{n}).

Exercise 1.3. If A, B ∈ B, and P is a finitely additive probability measure show that P (A ∪ B) = P (A) + P (B) − P (A ∩ B). How does this generalize to P (∪_{j=1}^{n}A_{j})?

Exercise 1.4. If P is a countably additive probability measure, show that for any sequence A_{n} ∈ B, P (∪_{n=1}^{∞}A_{n}) ≤ ∑_{n=1}^{∞} P (A_{n}).

Although we would like our ‘probability’ to be a countably additive probability measure, on a σ- field B of subsets of a space Ω it is not clear that are plenty of such things. As a first small step show

Exercise 1.5. If for n ≥ 1, ω_{n} ∈ Ω are distinct points and p_{n} ≥ 0 with ∑_{n} p_{n} = 1 then

P (A) = ∑_{n:ω_{n}∈A} p_{n}

defines a countably additive probability measure on Ω. ( This is still cheating because the measure P lives on a countable set.)

Definition 1.2. A probability measure P on a field F is said to be countably additive on F if for any sequence A_{n} ∈ F with A_{n} ↓ Φ, we have P (A_{n}) ↓ 0.

Exercise 1.6. Given any class F of subsets of Ω there is a unique σ-field B such that it is the smallest σ-field that contains F.

Definition 1.3. The σ-field in the above exercise is called the σ-field generated by F.

Construction of Measures

The following theorem is important for the construction of countably additive probability measures. A detailed proof can be found in any text on real variables. In an effort to be complete we will sketch the standard proof.

Theorem 1.1. (Caratheodory Extension Theorem). Any countably additive probabilty measure P on a field F extends uniquely as a countably additive probability measure to the σ-field B generated by F.

Proof: The proof proceeds along the following steps:

Step 1. Define an object P^{∗} called the outer measure for all sets A.

P^{∗}(A) = inf_{∪_{j}A_{j}⊃A} ∑_{j} P (A_{j})  (1.7)

where the infimum is taken over all countable collections {A_{j}} of sets from F that cover A. Without loss of generality we can assume that {A_{j}} are disjoint. (Replace A_{j} by (∩_{i=1}^{j−1}A_{i}^{c}) ∩ A_{j}). Step 2. Show that P^{∗} has the following properties:

1. P^{∗} is countably sub-additive,i.e.

P^{∗}(∪_{j}A_{j}) ≤ ∑_{j} P^{∗}(A_{j})

2. For A ∈ F, P^{∗}(A) ≤ P (A). (Trivial)

3. For A ∈ F, P^{∗}(A) ≥ P (A). (Need to use the countable additivity of P on F)

Step 3. Define a set E to be measurable if

P^{∗}(A) ≥ P^{∗}(A ∩ E) + P^{∗}(A ∩ E^{c})

holds for all sets A, and establish the following properties for the class M of measurable sets. The class of measurable sets M is a σ-field and P^{∗} is a countably additive measure on it. Step 4. Finally show that M ⊃ F. This implies that M ⊃ B and P^{∗} is an extension of P from F to B.

Uniqueness is quite simple. Let P_{1} and P_{2} be two countably additive probability measures on a σ-field B that agree on a field F ⊂ B. Let us define A = {A : P_{1}(A) = P_{2}(A)}. Then A is a monotone class i.e., if A_{n} ∈ A is increasing ( decreasing) then ∪_{n}A_{n} (∩_{n}A_{n}) ∈ A. We will leave the following elementary fact as an Exercise 1.7. The smallest monotone class generated by a field is the same as the σ-field generated by the field.

It now follows that A must contain the σ-field generated by F and that proves uniqueness.

The extension Theorem does not quite solve the problem of constructing countably additive probability measures. It reduces it to constructing them on fields. The following theorem is important in the theory of Lebesgue integrals and is very useful for the construction of countably additive probability measures on the real line. The proof will again be only sketched. The natural σ-field on which to define a probability measure on the line is the Borel σ-field. This is defined as the smallest σ-field containing all intervals and includes in particular all open sets.

Let us consider the class of subsets of the real numbers, I = {I_{a,b} : −∞ ≤ a < b ≤ ∞} where I_{a,b} = {x : a < x ≤ b} if b < ∞, and I_{a,∞} = {x : a < x < ∞}. In other words I is the collection of intervals that are left-open and right-closed. The class of sets that are finite disjoint unions of members of I is a field F, if the empty set is added to the class. If we are given a function F (x) on the real line which is nondecreasing, continuous from the right and satisfies

lim_{x→−∞} F (x) = 0 and lim_{x→∞} F (x) = 1

we can define a finitely additive probability measure P by first defining

P (I_{a,b}) = F (b) − F (a)

for intervals and then extending it to F by defining it as the sum for disjoint unions from I. Let us note that the Borel σ-field B on the real line is the σ-field generated by F.

Theorem 1.2. (Lebesgue). P is countably additive on F if and only if F (x) is a right continuous function of x. Therefore for each right continuous nondecreasing function F (x) with F (−∞) = 0 and F (∞) = 1 there is a unique probability measure P on the Borel subsets of the line, such that F (x) = P (I_{−∞,x}). Conversely every countably additive probability measure P on the Borel subsets of the line comes from some F. The correspondence between P and F is one-to-one.

Proof: The onlly difficult part is to establish the countable additivity of P on F from the right continuity of F (·). Let A_{j} ∈ F and A_{j} ↓ Φ, the empty set. Let us assume, P (A_{j}) ≥ δ > 0, for all j and then establish a contradiction.

Step 1. Take a large interval [−ℓ, ℓ] and replace A_{j} by B_{j} = A_{j}∩[−ℓ, ℓ]. In other words one assumes without loss of generality that A_{j} ⊂ [−ℓ, ℓ] for some ℓ < ∞. Step 2. If

A_{j} = ∪_{i=1}^{k_{j}}I_{a_{j,i},b_{j,i}}

use the right continuity of F to replace A_{j} by B_{j} which is a union of left open right closed intervals with the same right end points, but with left end points moved ever so slightly to the right. Achieve this in such a way that

P (A_{j} − B_{j}) ≤ \frac{δ}{10.2^{j}}

for all j. Step 3. Define C_{j} to be the closure of B_{j} by adding the left end points of the intervals that make up B_{j}. Let E_{j} = ∩_{i=1}^{j}B_{i} and D_{j} = ∩_{i=1}^{j}C_{i}. Then, (i) D_{j} are closed bounded sets, (ii) since A_{j} ⊃ D_{j} , D_{j} ↓ Φ, (iii) because D_{j} ⊃ E_{j} and P (E_{j}) ≥ δ − ∑_{j} P (A_{j} − B_{j}) ≥ \frac{9}{10}δ , each D_{j} is nonempty and this violates the finite intersection property that every decreasing sequence of bounded nonempty closed sets on the real line has a nonempty intersection, i.e. has atleast one common point.

The rest of the proof is left as an exercise.

The function F is called the distribution function corresponding to the probability measure P. Examples.

1. Suppose x_{1}, x_{2}, ⋯, x_{n}, ⋯ is a sequence of points and we have probabilities p_{n} at these ponts then for the discrete measure

P (A) = ∑_{n:x_{n}∈A} p_{n}

we have the distribution function

F (x) = ∑_{n:x_{n}≤ x} p_{n}

that only increases by jumps, the jump at x_{n} being p_{n}. The points {x_{n}} themselves can be discrete like integers or dense like the rationals.

2. If f(x) is a nonnegative integrable function with ∫_{−∞}^{∞} f(y)dy = 1 then F (x) = ∫_{−∞}^{x} f(y)dy is a distribution function which is continuous. In this case f is the density of the measure P and can be calculated as f(x) = F′(x).

3. There are (messy) examples of F that are continuous, but do not come from any density. More on this later.

Exercise 1.8. Let us try to construct the Lebesgue measure on the rationals Q ⊂ [0, 1]. We would like to have

P [I_{a,b}] = b − a

for all rational 0 ≤ a ≤ b ≤ 1. Show that it is impossible by showing that P [{q}] = 0 for the set {q} containing the single rational q while P [Q] = P [∪_{q∈Q}{q}] = 1. Where does the earlier proof break down? Once we have a countably additive probability measure P on a space (Ω, Σ) we will call the triple (Ω, Σ, P) a probabilty space.

Integration

An important notion is that of a random variable or a measurable function. Definition 1.4. A random variable or measurable function is map from Ω → R i.e. a real valued function on Ω such that for every Borel set B ⊂ R, f^{−1}(B) = {ω : f(ω) ∈ B} is a measurable subset of Ω i.e f^{−1}(B) ∈ Σ. Exercise 1.9. It is enough to check the requirement for sets B ⊂ R that are intervals or even just sets of the form (−∞, x] for −∞ < x < ∞.

A function that is measurable and satisfies |f(ω)| ≤ M all ω ∈ Ω for for some finite M is called a bounded measurable function. The following statements are the essential steps in developing an integration theory. Details can be found in any book on real variables.

1. If A ∈ Σ , the indicator function 1_{A}(x) defined to be 1 on A and 0 on A^{c}, is bounded and measurable.

2. Sums, products, limits, compositions and reasonable elementary operations like min and max performed on measurable functions lead to measurable functions.

3. If {A_{j} : 1 ≤ j ≤ n} is a disjoint partition of Ω into measurable sets, the function f(ω) = ∑_{j} c_{j}1_{A_{j}}(ω) is a measurable function and is referred to as a ‘simple’ function.

4. Any bounded measurable function f is a uniform limit of simple functions. To see this, if f is bounded by M, divide [−M, M] into n subintervals I_{j} of length \frac{2M}{n} with midpoints c_{j}. Let A_{j} = f^{−1}(I_{j}) = {ω : f(ω) ∈ I_{j}} and f_{n} = ∑_{j=1}^{n} c_{j}1_{A_{j}}. Clearly f_{n} is simple, sup_{ω} |f_{n}(ω) − f(ω)| ≤ \frac{M}{n}, and we are done.

5. For simple functions f = ∑ c_{j}1_{A_{j}} the integral ∫ f(ω)dP is defined to be ∑_{j} c_{j}P (A_{j}). It enjoys the following properties:

(a) If f and g are simple so is any linear combination af + bg for real constants a and b and

∫ (af + bg)dP = a ∫ fdP + b ∫ gdP.

(b) If f is simple so is |f| and | ∫ fdP | ≤ ∫ |f|dP ≤ sup_{ω} |f(ω)|.

6. If f_{n} is a sequence of simple functions converging to f uniformly, a_{n} = ∫ f_{n}dP is a Cauchy sequence of real numbers and therefore has a limit a as n → ∞. The integral ∫ fdP of f is defined to be this limit a. One can verify that a depends only on f and not on the sequence f_{n} chosen to approximate f.

7. Now the integral is defined for all bounded measurable functions and enjoys the following properties.

(a) If f and g are bounded measurable functions and a, b are real constants then for the linear combination af + bg which is a bounded measurable function

∫ (af + bg)dP = a ∫ fdP + b ∫ gdP.

(b) If f is a bounded measurable function so is |f| and | ∫ fdP | ≤ ∫ |f|dP ≤ sup_{ω} |f(ω)|.

(c) In fact a slightly stronger inequality is true. For any bounded measurable f,

∫ |f|dP ≤ P ({ω : |f(ω)| > 0}) sup_{ω} |f(ω)|

(d) If f is a bounded measurable function and A is a measurable set one defines

∫_{A} f(ω)dP = ∫ 1_{A}(ω)f(ω)dP

and we can write for any measurable set A,

∫ fdP = ∫_{A} fdP + ∫_{A^{c}} fdP

In addition to uniform convergence there are other weaker notions of convergence. Definition 1.5. A sequence f_{n} functions is said to converge to a function f everywhere or poinwise if

lim_{n→∞} f_{n}(ω) = f(ω)

for every ω ∈ Ω.

In dealing with sequences of functions on a space that has a measure defined on it, often it does not matter if the sequence fails to converge on a set of points that is insignificant. For example if we are dealing with the Lebesgue measure on the interval [0, 1] and f_{n}(x) = x^{n} then f_{n}(x) → 0 for all x except x = 1. A single point, being an interval of length 0 should be insignificant for the Lebesgue measure. Definition 1.6. A sequence f_{n} of measurable functions is said to converge to a measurable function f almost everywhere (usually abbreviated as a.e.) if there exists a measurable set N with P (N) = 0 such that

lim_{n→∞} f_{n}(ω) = f(ω)

for every ω ∈ N^{c}.

Note that almost everywhere convergence is always relative to a probability measure.

Another notion of convergence is the following: Definition 1.7. A sequence f_{n} of measurable functions is said to converge to a measurable function f in measure or in probability if

lim_{n→∞} P [ω : |f_{n}(ω) − f(ω)| ≥ ϵ] = 0

for every ϵ > 0.

Let us examine these notions in the context of indicator functions of sets f_{(}x) = 1_{A_{n}}(x). As soon as A ≠ B, sup_{ω} |1_{A}(ω) − 1_{B}(ω)| = 1, so that uniform convergence never really takes place. On the other hand one can verify that 1_{A_{n}}(ω) → 1_{A}(ω) for every ω if and only if the two sets

lim sup_{n} A_{n} = ∩_{n} ∪_{m≥n} A_{m}

and

lim inf_{n} A_{n} = ∪_{n} ∩_{m≥n} A_{m}

both coincide with A. Finally 1_{A_{n}}(ω) → 1_{A}(ω) in measure if and only if

lim_{n→∞} P (A_{n}ΔA) = 0

where for any two sets A and B the symmetric difference AΔB is defined as AΔB = (A ∩ B^{c}) ∪ (A^{c} ∩ B) = A ∪ B − A ∩ B. It is the set of points that belong to either set but not to both. For instance 1_{A_{n}} → 0 in measure if and only if P (A_{n}) → 0. Exercise 1.10. There is a difference between almost every where convergence and convergence in measure. The first is really stronger. Consider the interval [0, 1] and devide it successively into 2, 3, 4 ⋯ parts and enumerate the intervals in succession. That is, I_{1} = [0, \frac{1}{2}], I_{2} = [\frac{1}{2}, 1], I_{3} = [0, \frac{1}{3}], I_{4} = [\frac{1}{3}, \frac{2}{3}], I_{5} = [\frac{2}{3}, 1], and so on. If f_{n}(x) = 1_{I_{n}}(x) it easy to check that f_{n} tends to 0 in measure but not almost everywhere. Exercise 1.11. But the following statement is true. If f_{n} → f as n → ∞ in measure, then there is a subsequence f_{n_{j}} such that f_{n_{j}} → f almost everywhere as j → ∞.

Lemma 1.3If f_{n} → f almost everywhere then f_{n} → f in measure.

Proof: f_{n} → f outside N is equivalent to

∩_{n} ∪_{m≥n} [ω : |f_{m}(ω) − f(ω)| ≥ ϵ] ⊂ N

for every ϵ > 0. In particular by countable additivity

P [ω : |f_{n}(ω) − f(ω)| ≥ ϵ] ≤ P [∪_{m≥n}[ω : |f_{m}(ω) − f(ω)| ≥ ϵ] → 0

as n → ∞ and we are done.

Theorem 1.4. (Bounded Convergence Theorem). If f_{n} is uniformly bounded and f_{n} → f in measure then ∫ f_{n}dP → ∫ fdP.

Proof. Since

| ∫ f_{n}dP − ∫ fdP | = | ∫ (f_{n} − f)dP | ≤ ∫ |f_{n} − f|dP

we need only prove that if f_{n} → 0 in measure and |f_{n}| ≤ M then ∫ |f_{n}|dP → 0. To see this

∫ |f_{n}|dP = ∫_{|f_{n}|≤ϵ} |f_{n}|dP + ∫_{|f_{n}|>ϵ} |f_{n}|dP ≤ ϵ + MP [ω : |f_{n}(ω)| > ϵ]

and taking limits

lim sup_{n→∞} ∫ |f_{n}|dP ≤ ϵ

and since ϵ > 0 is arbitrary we are done.

The bounded convergence theorem is the essence of countable additivity. Let us look at the example of f_{n}(x) = x^{n} on 0 ≤ x ≤ 1 with Lebesgue measure. Clearly f_{n}(x) → 0 a.e. and therefore in meausre. While the convergence is not uniform, 0 ≤ x^{n} ≤ 1 for all n and x and so the bounded convergence theorem applies. In fact

∫_{0}^{1} x^{n}dx = \frac{1}{n + 1} → 0.

However if we replace x^{n} by nx^{n}, f_{n}(x) still goes to 0 a.e., but the sequence is no longer uniformly bounded and the integral does not go to 0. We now proceed to define integrals of nonnegative measurable functions. Definition 1.8. If f is a nonnegative measurable function we define

∫ fdP = {sup ∫ gdP : 0 ≤ g ≤ f, g bounded }

An important result is

Theorem 1.5(Fatou’s Lemma). If f_{n} ≥ 0 are measurable and f_{n} → f in measure then

∫ fdP ≤ lim inf_{n→∞} ∫ f_{n}dP.

Proof: Suppose 0 ≤ g ≤ f is bounded. Then the sequence h_{n} = f_{n} ∧ g is uniformly bounded and h_{n} → h = f ∧ g = g. Therefore ∫ gdP = lim_{n→∞} ∫ h_{n}dP and ∫ h_{n}dP ≤ ∫ f_{n}dP for every n. Hence ∫ gdP ≤ lim inf_{n→∞} ∫ f_{n}dP. Since this is true for every bounded measurable g ≤ f we are done.

Corollary 1.6. (Monotone Convergence Theorem). If f_{n} ↑ f monotonically then

∫ f_{n}dP → ∫ fdP as n → ∞.

Proof: Obviously ∫ f_{n}dP ≤ ∫ fdP and the rest is Fatou.

Now we try to define integrals of arbitrary measurable functions. A nonnegative measurable function is said to be integrable if ∫ fdP < ∞. A function f is said to be integrable if |f| is integrable and we define ∫ fdP = ∫ f^{+}dP − ∫ f^{−}dP where f^{+} = f ∨ 0 and f^{−} = −f ∧ 0 are the positive and negative parts of f. The integral has the following properties.

1. It is linear. If f and g are integrable so is af + bg for any two real constants and ∫ (af + bg)dP = a ∫ fdP + b ∫ gdP.

2. . | ∫ fdP | ≤ ∫ |f|dP for every integrable f.

3. If f = 0 except on a set N of measure 0, then f is integrable and ∫ fdP = 0. In particular if f = g almost everywhere then ∫ fdP = ∫ gdP.

Another important theorem is

Theorem 1.7. (The Dominated Convergence Theorem). If a sequence of measurable functions f_{n} → f in measure and |f_{n}| ≤ g for all n and ω for some integrable function g, then ∫ f_{n}dP → ∫ fdP as n → ∞.

Proof: g + f_{n} and g − f_{n} are nonnegative and converge in measure to g + f and g − f respectively. By Fatou’s lemma

lim inf_{n→∞} ∫ (g + f_{n})dP ≥ ∫ (g + f)dP

Since ∫ gdP is finite we can subtract it from both sides and get

lim inf_{n→∞} ∫ f_{n}dP ≥ ∫ fdP.

Working the same way with g − f_{n} yields

lim sup_{n→∞} ∫ f_{n}dP ≤ ∫ fdP

and we are done. Exercise 1.12. Take the unit interval with the Lebesgue measure and define f_{n}(x) = n^{a}1_{[0, \frac{1}{n}]}. Clearly for x ≠ 0 f_{n}(x) → 0 while ∫ f_{n}(x)dx = n^{α−1} → 0 if and only if α < 1. What is g(x) = sup_{n} f_{n}(x) and when is g integrable?

If F (ω) = f(ω) + ig(ω) is a complex valued measurable function with real and imaginary parts f(ω) and g(ω) that are integrable we define

∫ F (ω)dP = ∫ f(ω)dP + i ∫ g(ω)dP

Exercise 1.13. Show that for any complex function F (ω) = f(ω) + ig(ω) with measurable f and g, |F (ω)| is integrable, if and only if |f| and |g| are integrable and we then have

|∫ F (ω) dP | ≤ ∫ |F (ω)| dP

A measurable space (Ω, B) is a set Ω together with a σ-field B of subsets of Ω.

Transformations

Definition 1.9. Given two measurable spaces (Ω_{1}, B_{1}) and (Ω_{2}, B_{2}), a mapping or a transformation from F : Ω_{1} → Ω_{2}, i.e. a function ω_{2} = F (ω_{1}) that assigns for each point ω_{1} ∈ Ω_{1} a point ω_{2} = F (ω_{1}) ∈ Ω_{2}, is said to be measurable if for every measurable set A ∈ B_{2}, the inverse image

F^{−1}(A) = {ω_{1} : F (ω_{1}) ∈ A} ∈ B_{1}.

Exercise 1.14. Show that, in the above definition, it is enough to verify the property for A ∈ A where A is any class of sets that generates the σ-field B_{2}.

If F is a measurable map from (Ω_{1}, B_{1}) into (Ω_{2}, B_{2}) and P is a probability measure on (Ω_{1}, B_{1}), the induced probability measure Q on (Ω_{2}, B_{2}) is defined by

Q(A) = P (F^{−1}(A)) for A ∈ B_{2}  (1.8)

Exercise 1.15. Verify that Q indeed does define a probability measure on (Ω_{2}, B_{2}).

Q is called the induced measure and is denoted by Q = P F^{−1}.

Theorem 1.8. If f : Ω_{2} → R is a real valued measurable function on Ω_{2}, then g(ω_{1}) = f(F (ω_{1})) is a measurable real valued function on (Ω_{1}, B_{1}). Moreover g is integrable with respect to P if and only if f is integrable with respect to Q, and

∫_{Ω_{2}} f(ω_{2}) dQ = ∫_{Ω_{1}} g(ω_{1}) dP  (1.9)

Proof: If f(ω_{2}) = 1_{A}(ω_{2}) is the indicator function of a set A ∈ B_{2}, the claim in equation 1.9 is in fact the definition of measurability and the induced measure. We see that by linearity the claim extends easily from indicator functions to simple functions. By uniform limits, the claim can now be extended to bounded measurable functions. Monotone limits then extend it to nonnegative functions. By considering the positive and negative parts separately we are done.

A measurable function is just the generalization of the concept of a random variable introduced in section 2. We can either think of a random variable as special case of a measurable transformation where the target space is the real line. Or we can think of a measurable transformation as a random variable with values in an arbitrary target space. The induced measure Q = P F^{−1} is called the distribution of the random variable F under P. In particular, if F takes real values, Q is a probability distribution on R.

Exercise 1.16. When F is real valued show that

∫ F (ω)dP = ∫ x dQ

When F = (f_{1}, f_{2}, ⋯ f_{n}) takes values in R^{n}, the induced distribution Q on R^{n} is called the joint distribution of the n random variables f_{1}, f_{2} ⋯, f_{n}.

Exercise 1.17. If T_{1} is a measurable map from (Ω_{1}, B_{1}) into (Ω_{1}, B_{1}) and T_{2} is a measurable map from (Ω_{2}, B_{2}) into (Ω_{3}, B_{3}), then show that T = T_{2} ∘ T_{1} is a measurable map from (Ω_{1}, B_{1}) into (Ω_{3}, B_{3}). If P is a probability measure on (Ω_{1}, B_{1}), then on (Ω_{3}, B_{3}), the two measures P T^{−1} and (P T_{1}^{−1})T_{2}^{−1} are identical.

Product Spaces

Given two sets Ω_{1} and Ω_{2} the Cartesian product Ω = Ω_{1} × Ω_{2} is the set of pairs (ω_{1}, ω_{2}) with ω_{1} ∈ Ω_{1} and ω_{2} ∈ Ω_{2}. If Ω_{1} and Ω_{2} come with σ-fields B_{1} and B_{2} respectively, we can define a natural σ-field B on Ω as the σ-field generated by sets (measurable rectangles) of the form A_{1} × A_{2} with A_{1} ∈ B_{1} and A_{2} ∈ B_{2}. This σ-field will be called the product σ-field. Exercise 1.18. Show that sets that are finite disjoint unions of measurable rectangles constitute a field.

Given two probability measures P_{1} and P_{2} on (Ω_{1}, B_{1}) and (Ω_{2}, B_{2}) respectively we try to define on the product space (Ω, B) a probability measure P by defining for a measurable rectangle A = A_{1} × A_{2}

P (A_{1} × A_{2}) = P_{1}(A_{1}) × P_{2}(A_{2})

and extending it to the field F of finite disjoint unions of measurable rectangles as the obvious sum. Exercise 1.19. If E ∈ F has two representations as disjoint finite unions of measurable rectangles

E = ∪_{i} (A_{1}^{i} × A_{2}^{i}) = ∪_{j} (B_{1}^{j} × B_{2}^{j})

then

∑_{i} P_{1}(A_{1}^{i}) × P_{2}(A_{2}^{i}) = ∑_{j} P_{1}(B_{1}^{j}) × P_{2}(B_{2}^{j})

so that P (E) is well defined. P is a finitely additive probability measure on F.

Lemma 1.9. The measure P is countably additive on the field F. Proof: For any set E ∈ F let us define the section E_{ω_{2}} as

E_{ω_{2}} = {ω_{1} : (ω_{1}, ω_{2}) ∈ E}  (1.10)

Then P_{1}(E_{ω_{2}}) is a measurable function of ω_{2} (is in fact a simple function) and

P (E) = ∫_{Ω_{2}} P_{1}(E_{ω_{2}}) dP_{2}  (1.11)

Now let E_{n} ∈ F ↓ Φ, the empty set. Then it is easy to verify that E_{n,ω_{2}} defined by

E_{n,ω_{2}} = {ω_{1} : (ω_{1}, ω_{2}) ∈ E_{n}}

satisfies E_{n,ω_{2}} ↓ Φ for each ω_{2} ∈ Ω_{2}. From the countable additivity of P_{1} we conclude that P_{1}(E_{n,ω_{2}}) → 0 for each ω_{2} ∈ Ω_{2} and since, 0 ≤ P_{1}(E_{n,ω_{2}}) ≤ 1 for n ≥ 1, it follows from equation 1.11 and the bounded convergence theorem that

P (E_{n}) = ∫_{Ω_{2}} P_{1}(E_{n,ω_{2}}) dP_{2} → 0

establishing the countable additivity of P on F. We conclude that P extends uniquely as a countably additive measure to the σ-field B generated by F. We will call this the Product Measure P.

Corollary 1.10. For any A ∈ B if we denote by A_{ω_{1}} and A_{ω_{2}} the respective sections

A_{ω_{1}} = {ω_{2} : (ω_{1}, ω_{2}) ∈ A}

and

A_{ω_{2}} = {ω_{1} : (ω_{1}, ω_{2}) ∈ A}

then the functions P_{1}(A_{ω_{2}}) and P_{2}(A_{ω_{1}}) are measurable and

P (A) = ∫ P_{1}(A_{ω_{2}})dP_{2} = ∫ P_{2}(A_{ω_{1}}) dP_{1}.

In particular for a measurable set A, P (A) = 0 if and only if for almost all ω_{1} with respect to P_{1}, the sections A_{ω_{1}} have measure 0 or equivalently for almost all ω_{2} with respect to P_{2}, the sections A_{ω_{2}} have measure 0.

Proof: The assertion is clearly valid if A is rectangle of the form A_{1}×A_{2} with A_{1} ∈ B_{1} and A_{2} ∈ B_{2}. If A ∈ F, then it is a finite disjoint union of such rectangles and the assertion is extended to such a set by simple addition. Clearly, the class of sets for which the assertion is valid is a monotone class and since it contains the field F it also contains the σ-field B generated by the field F.

Warning. It is possible that a set A may not be measurable with respect to the product σ-field, but nevertheless the sections A_{ω_{1}} and A_{ω_{2}} are all measurable, P_{2}(A_{ω_{1}}) and P_{1}(A_{ω_{2}}) are measurable functions, but

∫ P_{1}(A_{ω_{2}})dP_{2} ≠ ∫ P_{2}(A_{ω_{1}}) dP_{1}.

In fact there is a rather nasty example where P_{1}(A_{ω_{2}}) ia identically 1 whereas P_{2}(A_{ω_{1}}) is identically 0. The next result concerns the equality

of the double integral, (i.e. the integral with respect to the product measure) and the repeated integrals in any order.

Theorem 1.11. (Fubini’s Theorem). Let f(ω) = f(ω_{1}, ω_{2}) be a measurable function of ω on (Ω, B). Then f can be considered as a function of ω_{2} for each fixed ω_{1} or the other way around. The functions g_{ω_{1}}(·) and h_{ω_{2}}(·) defined respectively on Ω_{2} and Ω_{1} by

g_{ω_{1}}(ω_{2}) = h_{ω_{2}}(ω_{1}) = f(ω_{1}, ω_{2})

are measurable for each ω_{1} and ω_{2}. If f is integrable then the functions g_{ω_{1}}(ω_{2}) and h_{ω_{2}}(ω_{1}) are integrable for almost all ω_{1} and ω_{2} respectively. Their integrals

G(ω_{1}) = ∫_{Ω_{2}} g_{ω_{1}}(ω_{2}) dP_{2}

and

H(ω_{2}) = ∫_{Ω_{1}} h_{ω_{2}}(ω_{1}) dP_{1}

are measurable, finite almost everywhere and integrable with repect to P_{1} and P_{2} respectively. Finally

∫ f(ω_{1}, ω_{2}) dP = ∫ G(ω_{1})dP_{1} = ∫ H(ω_{2})dP_{2}

Conversely for a nonnegative measurable function f if either G are H, which are always measurable, has a finite integral so does the other and f is integrable with its integral being equal to either of the repeated integrals, namely integrals of G and H.

Proof: The proof follows the standard pattern. It is a restatement of the earlier corollary if f is the indicator function of a measurable set A. By linearity it is true for simple functions and by passing to uniform limits, it is true for bounded measurable functions f. By monotone limits it is true for nonnegative functions and finally by taking the positive and negative parts seperately it is true for any arbitrary integrable function f.

Warning. The following could happen. f is a measurable function that takes both positive and negative values that is not integrable. Both the repeated integrals exist and are unequal. The example is not hard. Exercise 1.20. Construct a measurable function f(x, y), on the product space [0, 1] × [0, 1] of two copies of the unit interval with Lebesgue measure, which is not integrable but the repeated integrals make sense and or unequal, i.e.

∫_{0}^{1} dx ∫_{0}^{1} f(x, y) dy ≠ ∫_{0}^{1} dy ∫_{0}^{1} f(x, y) dx

Distributions and Expectations

Let us recall that a triplet (Ω, B, P) is a Probability Space if Ω is a set, B is a σ-field of subsets of Ω and P is a (countably additive) probability measure on B. A random variable X is a real valued measurable function on (Ω, B). Given such a function X it induces a probability distribution α on the Borel subsets of the line α = P X^{−1}. The distribution function F (x) corresponding to α is obviously

F (x) = α((−∞, x]) = P [ ω : X(ω) ≤ x]

The measure α is called the distibution of α and F (x) is called the distribution function of X. If g is a measurable function of the real variable x, then Y (ω) = g(X(ω)) is again a random variable and its distribution β = P Y^{−1} can be obtained as β = α g^{−1} from α. The Expectation of a random variable is defined if it is integrable and

E[X] = E^{P} [X] = ∫ X(ω) dP

By the change of variables formula (Exercise 3.3) it can be obtained directly from α as

E[X] = ∫ x dα

Here we are taking advantage of the fact that on the real line x is a very special real valued function. The value of the integral in this context is referred to as the expectation or mean of α. Of course it exists if and only if

∫ |x| dα < ∞

and

|∫ x dα| ≤ ∫ |x| dα

Similarly

E(g(X)) = ∫ g(X(ω)) dP = ∫ g(x) dα

and anything concerning X can be calculated from α. The statement X is a random variable with distribution α has to be interpreted in the sense that somewhere in the background there is a Probability Space and a random variable X on it, which has α for its distribution. Usually only α matters and the underlying (Ω, B, P) never emerges from the background and in a pinch we can always say Ω is the real line, B are the Borel sets , P is nothing but α and the random variable X(x) = x.

Some other related quantities are

V ar(X) = σ^{2}(X) = E[X^{2}] − [E[X]]^{2}.  (1.12)

V ar(X) is called the variance of X.

Exercise 1.21. Show that if it is defined V ar(X) is always nonnegative. Moreover V ar(X) = 0 if and only if for some value a, which is necessarily equal to E[X], P [X = a] = 1.

Some what more generally we can consider a measurable mapping X = (X_{1}, ⋯, X_{n}) of a probability space (Ω, B, P) into R^{n} as a vector of n random variables X_{1}(ø), X_{2}(ø), ⋯, X_{n}(ø). These are caled random vectors or vector valued random variables and the induced distribution α = P X^{−1} on R^{n} is called the distribution of X or the joint distribution of (X_{1}, ⋯, X_{n}). If we denote by π_{i} the coordinate maps (x_{1}, ⋯, x_{n}) → x_{i} from R^{n} → R, then

α_{i} = απ_{i}^{−1} = P X_{i}^{−1}

are called the marginals of α.

The Covariance between two random variables X and Y is defined as

Cov(X, Y) = E[(X −E(X))(Y −E(Y))] = E[XY]−E[X]E[Y] (1.13)

Exercise 1.22. If X_{1}, ⋯, X_{n} are n random variables the matrix

C_{i,j} = Cov(X_{i}, X_{j})

is called the covariance matrix. Show that it is a symmetric positive semi-definite matrix. Is every positive semi-definite the covariance matrix of some random vector?