Markov Chains (Continued)

Let us now consider an irreducible Markov Chain with one step transition probability π(x, y) that is periodic with period d > 1. Let us choose and fix a reference point x_{0} ∈ X. For each x ∈ X let D_{x_{0},x} = {n : π^{(n)}(x_{0}, x) > 0}.

Lemma 3.10. If n_{1}, n_{2} ∈ D_{x_{0},x} then d divides n_{1} − n_{2}.

Proof: Since the chain is irreducible there is an m such that π^{(m)}(x, x_{0}) > 0. By the Chapman-Kolmogorov equations π^{(m+n_{i})}(x_{0}, x_{0}) > 0 for i = 1, 2. Therefore m + n_{i} ∈ D_{x_{0}} = D_{x_{0},x_{0}} for i = 1, 2. This implies that d divides both m + n_{1} as well as m + n_{2}. Thus d divides n_{1} − n_{2}.

The residue modulo d of all the integers in D_{x_{0},x} are the same and equal some number r(x), satisfying 0 ≤ r(x) ≤ d − 1. By definition r(x_{0}) = 0. Let us define X_{j} = {x : r(x) = j}. Then {X_{j} : 0 ≤ j ≤ d−1} is a partition of X into disjoint sets with x_{0} ∈ X_{0}.

Lemma 3.11. If x ∈ X, then π^{(n)}(x, y) = 0 unless r(x) + n = r(y) modulo d.

Proof: Suppose that x ∈ X and π(x, y) > 0. Then if m ∈ D_{x_{0},x} then (m+1) ∈ D_{x_{0},y}. Therefore r(x)+1 = r(y) modulo d. The proof can be completed by induction. The chain marches through {X_{j}} in a cyclical way from a state in X_{j} to one in X_{j+1}

Theorem 3.12. Let X be irreducible and positive recurrent with period d. Then

lim_{{n→∞ \atop n+r(x)=r(y) modulo d}} π^{(n)}(x, y) = \frac{d}{m(y)}

Of course

π^{(n)}(x, y) = 0

unless n + r(x) = r(y) modulo d.

Proof: If we replace π by π̃ where π̃(x, y) = π^{(d)}(x, y), then π̃(x, y) = 0 unless both x and y are in the same X_{j}. The restriction of π̃ to each X_{j} defines an irreducible aperiodic Markov chain. Since each time step under π̃ is actually d units of time we can apply the earlier results and we will get for x, y ∈ X_{j} for some j,

lim_{k→∞} π^{(k d)}(x, y) = \frac{d}{m(y)}

We note that

π^{(n)}(x, y) = ∑_{1≤m≤n} f_{m}(x, y) π^{(n−m)}(y, y)
f_{m}(x, y) = P_{x}{τ_{y} = m} = 0 unless r(x) + m = r(y) modulo d
π^{(n−m)}(y, y) = 0 unless n − m = 0 modulo d
∑_{m} f_{m}(x, y) = 1

The theorem now follows.

Suppose now we have a chain that is not irreducible. Let us collect all the transient states and call the set X_{tr}. The complement consists of all the recurrent states and will be denoted by X_{re}.

Lemma 3.13. If x ∈ X_{re} and y ∈ X_{tr}, then π(x, y) = 0.

Proof: If x is a recuurrent state, and π(x, y) > 0, the chain will return to x infinitely often and each time there is a positive probability of visiting y. By the renewal property these are independent events and so y will be recurrent too.

The set of recurrent states X_{re} can be divided into one or more equivalence classes accoeding to the following procedure. Two recurrent states x and y are in the same equivalence class if f(x, y) = P_{x}{τ_{y} < ∞}, the probability of ever visiting y starting from x is positive. Because of recurrence if f(x, y) > 0 then f(x, y) = f(y, x) = 1. The restriction of the chain to a single equivalence class is irreducible and possibly periodic. Different equivalence classes could have different periods, some could be positive recurrent and others null recurrent. We can combine all our observations into the following theorem.

Theorem 3.14. If y is transient then ∑_{n} π^{(n)}(x, y) < ∞ for all x. If y is null recurrent (belongs to an equivalence class that is null recurrent) then π^{(n)}(x, y) → 0 for all x, but ∑_{n} π^{(n)}(x, y) = ∞ if x is in the same equivalence class or x ∈ X_{tr} with f(x, y) > 0. In all other cases π^{(n)}(x, y) = 0 for all n ≥ 1. If y is positive recurrent and belongs to an equivalence class with period d with m(y) = E^{P_{y}}{τ_{y}}, then for a nontransient x, π^{(n)}(x, y) = 0 unless x is in the same equivalence class and r(x) + n = r(y) modulo d. In such a case,

lim_{{n→∞ \atop r(x)+n=r(y) modulo d}} π^{(n)}(x, y) = \frac{d}{m(y)}

If x is transient then

lim_{{n→∞ \atop n=r modulo d}} π^{(n)}(x, y) = f(r, x, y)\frac{d}{m(y)}

where

f(r, x, y) = P_{x}{X_{kd+r} = y for some k ≥ 0}

Proof: The only statement that needs an explanation is the last one. The chain starting from a transient state x may at some time get into a positive recurrent equivalence class X_{j} with period d. If it does, it never leaves that class and so gets absorbed in that class. The probability of this is f(x, y) where y can be any state in X_{j}. However if the period d is greater than 1, there will be cyclical subclasses C_{1}, ⋯, C_{d} of X_{j}. Depending on which subclass the chain enters and when, the phase of its future is determined. There are d such possible phases. For instance, if the subclasses are ordered in the correct way, getting into C_{1} at time n is the same as getting into C_{2} at time n + 1 and so on. f(r, x, y) is the probability of getting into the equivalence class in a phase that visits the cyclical subclass containing y at times n that are equal to r modulo d.

Example (Random Walk): If X = Z^{d}, the integral lattice in R^{d}, a random walk is a Markov chain with transition probability π(x, y) = p(y − x) where {p(z)} specifies the probability distribution of a single step. We will assume for simplicity that p(z) = 0 except when z ∈ F where F consists of the 2 d neighbors of 0 and p(z) = \frac{1}{2d} for each z ∈ F. For ξ ∈ R^{d} the characteristic function of p̂(ξ) of p(·) is given by \frac{1}{d}(cos ξ_{1} + cos ξ_{2} + ⋯ + cos ξ_{d}). The chain is easily seen to irreducible, but periodic of period 2. Return to the starting point is possible only after an even number of steps.

π^{(2n)}(0, 0) = (\frac{1}{2π})^{d} ∫_{T^{d}}[p̂(ξ)]^{2n}dξ
≃ \frac{C}{n^{\frac{d}{2}}}

To see this asymptotic behavior let us first note that the integration can be restricted to the set where |p̂(ξ)| ≥ 1 − δ or near the 2 points (0, 0, ⋯, 0) and (π, π, ⋯, π) where |p̂(ξ)| = 1. Since the behaviour is similar at both points let us concentrate near the origin.

\frac{1}{d} ∑_{j=1}^{d} cos ξ_{j} ≤ 1 − c ∑_{j} ξ_{j}^{2} ≤ exp[−c ∑_{j} ξ_{j}^{2}]

for some c > 0 and

[\frac{1}{d} ∑_{j=1}^{d} cos ξ_{j}]^{2n} ≤ exp[−2 n c ∑_{j} ξ_{j}^{2}]

and with a change of variables the upper bound is clear. We have a similar lower bound as well. The random walk is recurrent if d = 1 or 2 but transient if d ≥ 3.

Exercise 3.11. If the distribution p(·) is arbitrary, determine when the chain is irreducible and when it is irreducible and aperiodic.

Exercise 3.12. If ∑_{z} zp(z) = m ≠ 0 conclude that the chain is transient by an application of the strong law of large numbers.

Exercise 3.13. If ∑_{z} zp(z) = m = 0, and if the covariance matrix, ∑_{z} z_{i}z_{j}p(z) = σ_{i,j}, is nondegenerate show that the transience or recurrence is determined by the dimension as in the case of the nearest neighbor random walk.

Exercise 3.14. Can you make sense of the formal calculation

∑_{n} π^{(n)}(0, 0) = ∑_{n}(\frac{1}{2π})^{d} ∫_{T^{d}}[p̂(ξ)]^{n}dξ

= (\frac{1}{2π})^{d} ∫_{T^{d}} \frac{1}{(1 − p̂(ξ))}dξ
= (\frac{1}{2π})^{d} ∫_{T^{d}} Real Part[\frac{1}{1 − p̂(ξ)}]dξ

to conclude that a necessary and sufficient condition for transience or recurrece is the convergence or divergence of the integral

∫_{T^{d}} Real Part[\frac{1}{1 − p̂(ξ)}]dξ

with an integrand

Real Part[\frac{1}{1 − p̂(ξ)}]

that is seen to be nonnegative ?

Hint: Consider instead the sum

∑_{n=0}^{∞} ρ^{n}π^{(n)}(0, 0) = ∑_{n}(\frac{1}{2π})^{d} ∫_{T^{d}} ρ^{n}[p̂(ξ)]^{n}dξ
= (\frac{1}{2π})^{d} ∫_{T^{d}} \frac{1}{(1 − ρp̂(ξ))}dξ
= (\frac{1}{2π})^{d} ∫_{T^{d}} Real Part[\frac{1}{1 − ρp̂(ξ)}]dξ

for 0 < ρ < 1 and let ρ → 1.

Example: The Queue Problem: In the example of customers arriving, except in the trivial cases of p_{0} = 0 or p_{0} + p_{1} = 1 the chain is irreducible and aperiodoc. Since the service rate is at most 1 if the arrival rate m = ∑_{j} j p_{j} > 1, then the queue will get longer and by an application of law of large numbers it is seen that the queue length will become infinite as time progresses. This is the transient behavior of the queue. If m < 1, one can expect the situation to be stable and there should be an asymptotic distribution for the queue length. If m = 1, it is the borderline case and one should probably expect this to be the null recurent case. The actual proofs are not hard. In time n the actual number of customers served is at most n because the queue may sometomes be empty. If {ξ_{i} : i ≥ 1} are the number of new customers arriving at time i and X_{0} is the initial number in the queue, then the number X_{n} in the queue at time n satisfies X_{n} ≥ X_{0} + (∑_{i=1}^{n} ξ_{i}) − n and if m > 1, it follows from the law of large numbers that lim_{n→∞} X_{n} = +∞, thereby establishing transience. To prove positive recurrence when m < 1 it is sufficient to prove that the equations

∑_{x} q(x)π(x, y) = q(y)

has a nontrivial nonnegative solution such that ∑_{x} q(x) < ∞. We shall proceed to show that this is indeed the case. Since the equation is linear we can alaways normalize tha solution so that ∑_{x} q(x) = 1. By iteration

∑_{x} q(x)π^{(n)}(x, y) = q(y)

for every n. If lim_{n→∞} π^{(n)}(x, y) = 0 for every x and y, because ∑_{x} q(x) = 1 < ∞, by the bounded convergence theorem the right hand side tends to 0 as n → ∞. therefore q ≡ 0 and is trivial. This rules out the transient and the null recurrent case. In our case π(0, y) = p_{y} and π(x, y) = p_{y−x+1} if y ≥ x − 1 and x ≥ 1. In all other cases π(x, y) = 0. The equations for {q_{x} = q(x)} are then

q_{0}p_{y} + ∑_{x=1}^{y+1} q_{x}p_{y−x+1} = q_{y} for y ≥ 0  (3.6)

Multiplying equation 3.6 by z^{n} and summimg from 1 to ∞, we get

q_{0} P (z) + \frac{1}{z} P (z) [Q(z) − q_{0}] = Q(z)

where P (z) and Q(z) are the generating functions

P (z) = ∑_{x=0}^{∞} p_{x}z^{x}
Q(z) = ∑_{x=0}^{∞} q_{x}z^{x}

We can solve for Q to get

\frac{Q(z)}{q_{0}} = P (z)[1 − \frac{P (z) − 1}{z − 1}]^{−1} = P (z) ∑_{k=0}^{∞} [\frac{P (z) − 1}{z − 1}]^{k}
= P (z) ∑_{k=0}^{∞} [ ∑_{j=1}^{∞} p_{j}(1 + z + ⋯ + z^{j−1})]^{k}

is a power series in z with nonnegative coefficients. If m < 1, we can let z → 1 to get

\frac{Q(1)}{q_{0}} = ∑_{k=0}^{∞} [ ∑_{j=1}^{∞} j p_{j}]^{k} = ∑_{k=0}^{∞} m^{k} = \frac{1}{1 − m} < ∞

proving positive recurrence.

The case m = 1 is a little bit harder. The calculations carried out earlier are still valid and we know in this case that there exists q(x) ≥ 0 such that each q(x) < ∞ for each x, ∑_{x} q(x) = ∞, and

∑_{x} q(x) π(x, y) = q(y)

In other words the chain admits an infinite invariant measure. Such a chain cannot be positive recurrent. To see this we note

q(y) = ∑_{x} π^{(n)}(x, y)q(x)

and if the chain were positive recurrent

lim_{n→∞} π^{(n)}(x, y) = q̃(y)

would exist and ∑_{y} q̃(y) = 1. By Fatou’s lemma

q(y) ≥ ∑_{x} q̃(y)q(x) = ∞

giving us a contradiction. To decide between transience and null recurrence a more detiled investigation is needed. We will outline a general procedure.

Suppose we have a state x_{0} that is fixed and would like to calculate F_{x_{0}}(ℓ) = P_{x_{0}}{τ_{x_{0}} ≤ ℓ}. If we can do this, then we can answer questions about transience, recurrence etc. If lim_{ℓ→∞} F_{x_{0}}(ℓ) < 1 then the chain is transient and otherwise recurrent. In the recurrent case the convergence or divergence of

E^{P_{x_{0}}}{τ_{x_{0}}} = ∑_{ℓ}[1 − F_{x_{0}}(ℓ)]

determines if it is positive or null recurrent. If we can determine

F_{y}(ℓ) = P_{y}{τ_{x_{0}} ≤ ℓ}

for y ≠ x_{0}, then for ℓ ≥ 1

F_{x_{0}}(ℓ) = π(x_{0}, x_{0}) + ∑_{y≠x_{0}} π(x_{0}, y)F_{y}(ℓ − 1)

We shall outline a procedure for determining for λ > 0,

U(λ, y) = E_{y}[ exp[−λτ_{x_{0}}]]

Clearly U(x) = U(λ, x) satisfies

U(x) = e^{−λ} ∑_{y} π(x, y)U(y) for x ≠ x_{0}  (3.7)

and U(x_{0}) = 1. One would hope that if we solve for these equations then we have our U. This requires uniqueness. Since our U is bounded in fact by 1, it is sufficient to prove uniqueness within the class of bounded solutions of equation 3.7. We will now establish that any bounded solution U of equation 3.7 with U(x_{0}) = 1, is given by

U(y) = U(λ, y) = E_{y}[ exp[−λτ_{x_{0}}]]

Let us define E_{n} = {X_{1} ≠ x_{0}, X_{2} ≠ x_{0}, ⋯, X_{n−1} ≠ x_{0}, X_{n} = x_{0}}. Then we will prove, by induction, that for any solution U of equation (3.7), with U(λ, x_{0}) = 1,

U(y) = ∑_{j=1}^{n} e^{−λ j} P_{y}{E_{j}} + e^{−λ n} ∫_{τ_{x_{0}}>n} U(X_{n}) dP_{y}  (3.8)

By letting n → ∞ we would obtain

U(y) = ∑_{j=1}^{∞} e^{−λ j} P_{y}{E_{j}} = E^{P_{y}}{e^{−λτ_{x_{0}}}}

because U is bounded and λ > 0.

∫_{τ_{x_{0}}>n} U(X_{n}) dP_{y}

= e^{−λ} ∫_{τ_{x_{0}}>n}[∑_{y} π(X_{n} , y) U(y)] dP_{y}

= e^{−λ}P_{y} {E_{n+1}} + e^{−λ} ∫_{τ_{x_{0}}>n}[ ∑_{y≠x_{0}} π(X_{n} , y) U(y)] dP_{y}

= e^{−λ}P_{y} {E_{n+1}} + e^{−λ} ∫_{τ_{x_{0}}>n+1} U(X_{n+1}) dP_{y}

completing the induction argument. In our case, if we take x_{0} = 0 and

try U_{σ}(x) = e^{−σ x} with σ > 0, for x ≥ 1

∑_{y} π(x, y)U_{σ}(y) = ∑_{y≥x−1} e^{−σ y}p_{y−x+1}
= ∑_{y≥0} e^{−σ (x+y−1)}p_{y}
= e^{−σ x}e^{σ} ∑_{y≥0} e^{−σ y}p_{y} = ψ(σ)U_{σ}(x)

where

ψ(σ) = e^{σ} ∑_{y≥0} e^{−σ y}p_{y}

Let us solve e^{λ} = ψ(σ) for σ which is the same as solving log ψ(σ) = λ for λ > 0 to get a solution σ = σ(λ) > 0. Then

U(λ, x) = e^{−σ(λ) x} = E^{P_{x}}{e^{−λτ_{0}}}

We see now that recurrence is equivalent to σ(λ) → 0 as λ ↓ 0 and positive recurrence to σ(λ) being differentiable at λ = 0. The function log ψ(σ) is convex and its slope at the origin is 1 − m. If m > 1 it dips below 0 initially for σ > 0 and then comes back up to 0 for some positive σ_{0} before turning positive for good. In that situation lim_{λ↓0} σ(λ) = σ_{0} > 0 and that is transience. If m < 1 then log ψ(σ) has a positive slope at the origin and σ′(0) = \frac{1}{ψ′(0)} = \frac{1}{1−m} < ∞. If m = 1, then log ψ has zero slope at the origin and σ′(0) = ∞. This concludes the discussion of this problem.

Exercise 3.15. The Urn Problem. We now turn to a discussion of the urn problem.

π(p, q; p + 1, q) = \frac{p}{p + q} and π(p, q; p, q + 1) = \frac{q}{p + q}

and π is zero otherwise. In this case the equation

F (p, q) = \frac{p}{p + q} F (p + 1, q) + \frac{q}{p + q} F (p, q + 1) for all p, q

which will play a role later, has lots of solutions. In particular, F (p, q) = \frac{p}{p+q} is one and for any 0 < x < 1

F_{x}(p, q) = \frac{1}{β(p, q)}x^{p−1}(1 − x)^{q−1}

where

β(p, q) = \frac{Γ(p)Γ(q)}{Γ(p + q)}

is a solution as well. The former is defined on p + q > 0 where as the latter is defined only on p > 0, q > 0. Actually if p or q is initially 0 it remains so for ever and there is nothing to study in that case. If f is a continuous function on [0, 1] then

F_{f}(p, q) = ∫_{0}^{1} F_{x}(p, q)f(x) dx

is a solution and if we want we can extend F_{f} by making it f(1) on q = 0 and f(0) on p = 0. It is a simple exercise to verify

lim_{{p,q→∞ \atop \frac{p}{q} →x}} F_{f}(p, q) = f(x)

for any continuous f on [0, 1]. We will show that the ratio ξ_{n} = \frac{p_{n}}{p_{n}+q_{n}} which is random, stabilizes asymptotically (i.e. has a limit) to a random variable ξ and if we start from p, q the distribution of ξ is the Beta distribution on [0, 1] with density

F_{x}(p, q) = \frac{1}{β(p, q)}x^{p−1}(1 − x)^{q−1}

Suppose we have a Markov Chain on some state space X with transition probability π(x, y) and U(x) is a bounded function on X that solves

U(x) = ∑_{y} π(x, y) U(y)

Such functions are called (bounded) Harmonic functions for the Chain. Consider the random variables ξ_{n} = U(X_{n}) for such an harmonic function. ξ_{n} are uniformly bounded by the bound for U. If we denote by η_{n} = ξ_{n} − ξ_{n−1} an elementary calculation reveals

E^{P_{x}}{η_{n+1}} = E^{P_{x}}{U(X_{n+1}) − U(X_{n})}
= E^{P_{x}}{E^{P_{x}}{U(X_{n+1}) − U(X_{n})}|F_{n}}}

where F_{n} is the σ-field generated by X_{0}, ⋯, X_{n}. But

E^{P_{x}}{U(X_{n+1}) − U(X_{n})}|F_{n}} = ∑_{y} π(X_{n}, y)[U(y) − U(X_{n})] = 0

A similar calculation shows that

E^{P_{x}}{η_{n}η_{m}} = 0

for m ≠ n. If we write

U(X_{n}) = U(X_{0}) + η_{1} + η_{2} + ⋯ + η_{n}

this is an orthogonal sum in L_{2}[P_{x}] and because U is bounded

E^{P_{x}}{|U(X_{n})|^{2}} = |U(x)|^{2} + ∑_{i=1}^{n} E^{P_{x}}{|η_{i}|^{2}} ≤ C

is bounded in n. Therefore lim_{n→∞} U(X_{n}) = ξ exists in L_{2}[P_{x}] and E^{P_{x}}{ξ} = U(x). Actually the limit exists almost surely and we will show it when we discuss martingales later. In our example if we take U(p, q) = \frac{p}{p+q}, as remarked earlier, this is a harmonic function bounded by 1 and therefore

lim_{n→∞} \frac{p_{n}}{p_{n} + q_{n}} = ξ

exists in L_{2}[P_{x}]. Moreover if we take U(p, q) = F_{f}(p, q) for some continuous f on [0, 1], because F_{f}(p, q) → f(x) as p, q → ∞ and \frac{p}{q} → x, U(p_{n}, q_{n}) has a limit as n → ∞ and this limit has to be f(ξ). On the other hand

E^{P_{p,q}}{U(p_{n}, q_{n})} = U(p_{0}, q_{0}) = F_{f}(p_{0}, q_{0})
= \frac{1}{β(p_{0}, q_{0})} ∫_{0}^{1} f(x)x^{p_{0}−1}(1 − x)^{q_{0}−1} dx

giving us

E^{P_{p,q}}\{f(ξ)\} = \frac{1}{β(p, q)} ∫_{0}^{1} f(x) x^{p−1}(1 − x)^{q−1} dx

thereby identifying the distribution of ξ under P_{p,q} as the Beta distribution with the right parameters.

Example: Let X be the set of integers. Assume that transitions from x are possible only to x − 1, x, and x + 1. The transition matrix π(x, y) appears as a tridiagonal matrix with π(x, y) = 0 unless |x − y| ≤ 1. For simplicity let us assume that π(x, x), π(x, x − 1) and π(x, x + 1) are positive for all x. The chain is then irreducible and aperiodic. Let us try to solve for

U(x) = P_{x}{τ_{0} = ∞}

that satisfies the equation

U(x) = π(x, x − 1) U(x − 1) + π(x, x) U(x) + π(x, x + 1) U(x + 1)

for x ≠ 0 with U(0) = 0. The equations decouple into a set for x > 0 and a set for x < 0. If we denote by V (x) = U(x + 1) − U(x) for x ≥ 0, then we always have

U(x) = π(x, x − 1) U(x) + π(x, x) U(x) + π(x, x + 1) U(x)

so that

π(x, x − 1) V (x − 1) − π(x, x + 1) V (x) = 0

or

\frac{V (x)}{V (x − 1)} = \frac{π(x, x − 1)}{π(x, x + 1)}

and therefore

V (x) = V (0) ∏_{i=1}^{x} \frac{π(i, i − 1)}{π(i, i + 1)}

and

U(x) = V (0)[1 + ∑_{y=1}^{x−1} ∏_{i=1}^{y} \frac{π(i, i − 1)}{π(i, i + 1)}]

If the chain is to be transient we must have for some choice of V (0), 0 ≤ U(x) ≤ 1 for all x > 0 and this will be possible only if

∑_{y=1}^{∞} ∏_{i=1}^{y} \frac{π(i, i − 1)}{π(i, i + 1)} < ∞

which then becomes a necessary condition for

P_{x}{τ_{0} = ∞} > 0

for x > 0. There is a similar condition on the negative side

∑_{y=1}^{∞} ∏_{i=1}^{y} \frac{π(−i, −i + 1)}{π(−i, −i − 1)} < ∞

Transience needs at least one of the two series to converge. Actually the converse is also true. If, for instance the series on the positive side converges then we get a function U(x) with 0 ≤ U(x) ≤ 1 and U(0) = 0 that satisfies

U(x) = π(x, x − 1) U(x − 1) + π(x, x) U(x) + π(x, x + 1) U(x + 1)

and by iteration one can prove that for each n,

U(x) = ∫_{τ_{0}>n} U(X_{n}) dP_{x} ≤ P {τ_{0} > n}

so the existence of a nontrivial U implies transience.

Exercise 3.16. Determine the conditions for positive recurrence.

Exercise 3.17. If we replace the set of integers by the set of nonnegative integers, [π(0, y) is nonzero only for y = 0 or 1 ] and such processes are called birth and death processes. Work out the conditions in that case.

Exercise 3.18. In the special case of a birth and death process with π(0, 1) = π(0, 0) = \frac{1}{2}, and for x ≥ 1, π(x, x) = \frac{1}{3}, π(x, x − 1) = \frac{1}{3} + a_{x}, π(x, x+1) = \frac{1}{3} −a_{x} with a_{x} = \frac{λ}{x^{α}} for large x, find conditions on positive α and real λ for the chain to be transient, null recurrent and positive recurrent.

Exercise 3.19. The notion of a Markov Chain makes sense for a finite chain X_{0}, ⋯, X_{n}. Formulate it precisely. Show that if the chain {X_{j} : 0 ≤ j ≤ n} is Markov so is the reversed chain {Y_{j} : 0 ≤ j ≤ n} where Y_{j} = X_{n−j} for 0 ≤ j ≤ n. Can the transition probabilities of the reversed chain be determined by the transition probabilities of the forward chain? If the forward chain has stationary transition proabilities does the same hold true for the reversed chain? What if we assume that the chain has a finte invariant probability distribution and we initialize the chain to start with an initial distribution which is the invariant distribution?

Exercise 3.20. Consider the simple chain on nonnegative integers with the following transition probailities. π(0, x) = p_{x} for x ≥ 0 with ∑_{x=0}^{∞} p_{x} = 1. For x > 0, π(x, x − 1) = 1 and π(x, y) = 0 for all other y. Determine conditions on {p_{x}} in order that the chain may be transient, null recurrent or positive recurrent. Determine the invariant probability measure in the positive recurrent case.

Exercise 3.21. Show that any null recurrent equivalence class must necessarily contain an infinite number of states. In patricular any Markov Chain with a finite state space has only transient and positive recurrent states and moreover the set of positive recurrent states must be non empty.