Probability for Data Science
eBook  ›  Chapter 4 · Continuous Random Variables
Section 4.2

Expectation, Moment, and Variance

4.2.1Definition and properties

As with discrete random variables, we can define expectation for continuous random variables. The definition is analogous: Just replace the summation with integration.

Definition 4.3

The expectation of a continuous random variable \(X\) is

$$\E[X] = \int_{\Omega} x \, f_X(x) \;dx.$$
Example 4.8

(Uniform random variable) Let \(X\) be a continuous random variable with PDF \(f_X(x) = \frac{1}{b-a}\) for \(a \le x \le b\), and \(0\) otherwise. The expectation is

$$\begin{aligned} \E[X] = \int_{\Omega} x f_X(x) \;dx = \int_a^b x \cdot \frac{1}{b-a} \;dx &= \frac{1}{b-a} \;\; \underset{\textcolor{black}{=\frac{x^2}{2}\big|_{a}^b}}{\underbrace{\int_a^b x \;dx}}\\ &= \frac{1}{b-a} \cdot \frac{b^2-a^2}{2} = \frac{a+b}{2}. \end{aligned}$$
Example 4.9

(Exponential random variable) Let \(X\) be a continuous random variable with PDF \(f_X(x) = \lambda e^{-\lambda x}\), for \(x \ge 0\). The expectation is

$$\begin{aligned} \E[X] &= \int_0^{\infty} x \; \lambda e^{-\lambda x} \;dx \\ &= -\int_0^{\infty} x \; \textcolor{red}{de^{-\lambda x}} \\ &= \underset{\textcolor{black}{=0}}{\underbrace{-xe^{-\lambda x}\bigg|_{0}^{\infty}}} + \int_0^{\infty} e^{-\lambda x} \;dx\\ &= -\frac{1}{\lambda} \;\; \underset{\textcolor{black}{=-1}}{\underbrace{e^{-\lambda x}\bigg|_{0}^{\infty}}} = \frac{1}{\lambda}, \end{aligned}$$

where the colored step is due to integration by parts.

If a function \(g\) is applied to the random variable \(X\), the expectation can be found using the following theorem.

Theorem 4.1

Let \(g: \Omega \rightarrow \R\) be a function and \(X\) be a continuous random variable. Then

$$\E[g(X)] = \int_{\Omega} g(x) \, f_X(x) \;dx.$$
Example 4.10

(Uniform random variable) Let \(X\) be a continuous random variable with \(f_X(x) = \frac{1}{b-a}\) for \(a \le x \le b\), and \(0\) otherwise. If \(g(\cdot) = (\cdot)^2\), then

$$\begin{aligned} \E[g(X)] = \E[X^2] &= \int_{\Omega} x^2 f_X(x) \;dx \\ &= \frac{1}{b-a} \cdot \underset{=\frac{b^3-a^3}{3}}{\underbrace{\int_{a}^{b} x^2 \;dx }} = \frac{a^2+ab+b^2}{3}. \end{aligned}$$
Practice Exercise 4.4

Let \(\Theta\) be a continuous random variable with PDF \(f_{\Theta}(\theta) = \frac{1}{2\pi}\) for \(0 \le \theta \le 2\pi\) and is 0 otherwise. Let \(Y = \cos (\omega t + \Theta)\). Find \(\E[Y]\).

Solution

Referring to Eq. (4.4), the function \(g\) is $$g(\theta) = \cos (\omega t + \theta).$$ Therefore, the expectation \(\E[Y]\) is

$$\begin{aligned} \E[Y] &= \int_{0}^{2\pi} \cos (\omega t + \theta)\; f_{\Theta}(\theta)\; d \theta \\ &= \frac{1}{2\pi}\int_{0}^{2\pi} \cos (\omega t + \theta)\; d \theta = 0, \end{aligned}$$

where the last equality holds because the integral of a sinusoid over one period is 0.

Practice Exercise 4.5

Let \(A \subseteq \Omega\). Let \(\mathbb{I}_{A}(X)\) be an indicator function such that

$$\mathbb{I}_{A}(X)= \begin{cases} 1, \quad\mbox{if}\; X \in A,\\ 0, \quad\mbox{if}\; X \not\in A. \end{cases}$$

Find \(\E[\mathbb{I}_{A}(X)]\).

Solution

The expectation is

$$\begin{aligned} \E[\mathbb{I}_{A}(X)] &= \int_{\Omega} \mathbb{I}_{A}(x) f_X(x) \;dx \\ &= \int_{x \in A} f_X(x) \;dx \\ &= \Pb[X \in A]. \end{aligned}$$

So the probability of \(\{X \in A\}\) can be equivalently represented in terms of expectation.

Practice Exercise 4.6

Is it true that \(\E[1/X] = 1/\E[X]\)?

Solution

No. This is because

$$\begin{aligned} \E\left[\frac{1}{X}\right] &= \int_{\Omega} \frac{1}{x} f_X(x)\;dx \\ &\not= \frac{1}{\int_{\Omega} x f_X(x)\;dx} \\ &= \frac{1}{\E[X]}. \end{aligned}$$

All the properties of expectation we learned in the discrete case can be translated to the continuous case. Specifically, we have that

Practice Exercise 4.7

Prove the above three statements.

Solution

The third statement is just the sum of the first two statements, so we just need to show the first two:

$$\begin{aligned} \E[aX] &= \int_{\Omega} ax f_X(x) \;dx = a \int_{\Omega} x f_X(x) \;dx = a\E[X],\\ \E[X+a]&= \int_{\Omega} (x+a) f_X(x) \;dx = \int_{\Omega} x f_X(x) \;dx + a = \E[X] + a. \end{aligned}$$

4.2.2Existence of expectation

As we discussed in the discrete case, not all random variables have an expectation.

Definition 4.4

A random variable \(X\) has an expectation if it is absolutely integrable, i.e.,

$$\E[|X|] = \int_{\Omega} |x| f_X(x) \;dx < \infty.$$

Being absolutely integrable implies that \(\E[|X|]\) is an upper bound of \(|\E[X]|\).

Theorem 4.2

For any random variable \(X\),

$$|\E[X]| \le \E[|X|].$$

Proof. Note that \(f_X(x) \ge 0\). Therefore,

$$\begin{aligned} - |x|\,f_X(x) \le x \; f_X(x) \le |x|\,f_X(x), \qquad \forall x. \end{aligned}$$

Thus, integrating all three terms yields

$$\begin{aligned} -\int_\Omega |x| f_X(x) \;dx \le \int_\Omega x \; f_X(x) \;dx \le \int_\Omega |x| f_X(x) \;dx, \end{aligned}$$

which is equivalent to \(-\E[|X|] \le \E[X] \le \E[|X|]\).

■
Example 4.11

Here is a random variable whose expectation is undefined. Let \(X\) be a random variable with PDF

$$f_X(x) = \frac{1}{\pi(1+x^2)}, \qquad x \in \R.$$

This random variable is called the Cauchy random variable. We can show that

$$\begin{aligned} \E[X] &= \int_{-\infty}^{\infty} x \cdot \frac{1}{\pi (1+x^2)} \;dx = \frac{1}{\pi} \int_{0}^{\infty} \frac{x}{(1+x^2)} \;dx + \frac{1}{\pi} \int_{-\infty}^{0} \frac{x}{(1+x^2)} \;dx. \end{aligned}$$

The first integral gives

$$\begin{aligned} \int_{0}^{\infty} \frac{x}{(1+x^2)} \;dx = \frac{1}{2} \log(1+x^2)\Big|_{0}^{\infty} = \infty, \end{aligned}$$

and the second integral gives \(-\infty\). Since neither integral is finite, the expectation is undefined. We can also check the absolute integrability criterion:

$$\begin{aligned} \E[|X|] &= \int_{-\infty}^{\infty} |x| \cdot \frac{1}{\pi (1+x^2)} \;dx\\ &\overset{(a)}{=} 2 \int_{0}^{\infty} \frac{x}{\pi (1+x^2)} \;dx \ge 2 \int_{1}^{\infty} \frac{x}{\pi (1+x^2)} \;dx\\ &\overset{(b)}{\ge} 2 \int_{1}^{\infty} \frac{x}{\pi (x^2 + x^2)} \;dx = \frac{1}{\pi}\log(x)\Big|_{1}^{\infty} = \infty, \end{aligned}$$

where in (a) we use the fact that the function being integrated is even, and in (b) we lower-bound \(\frac{1}{1+x^2} \ge \frac{1}{x^2+x^2}\) if \(x > 1\).

4.2.3Moment and variance

The moment and variance of a continuous random variable can be defined analogously to the moment and variance of a discrete random variable, replacing the summations with integrations.

Definition 4.5

The \(k\)th moment of a continuous random variable \(X\) is

$$\E[X^k] = \int_{\Omega} x^k \, f_X(x) \;dx.$$
Definition 4.6

The variance of a continuous random variable \(X\) is

$$\Var[X] = \E[(X-\mu)^2] = \int_{\Omega} (x-\mu)^2 f_X(x) \;dx,$$

where \(\mu \bydef \E[X]\).

It is not difficult to show that the variance can also be expressed as

$$\begin{aligned} \Var[X] = \E[X^2] - \mu^2, \end{aligned}$$

because

$$\begin{aligned} \Var[X] &= \E[(X-\mu)^2]\\ &= \E[X^2] - 2\E[X]\mu + \mu^2\\ &= \E[X^2] - \mu^2. \end{aligned}$$
Practice Exercise 4.8

(Uniform random variable) Let \(X\) be a continuous random variable with PDF \(f_X(x) = \frac{1}{b-a}\) for \(a \le x \le b\), and \(0\) otherwise. Find \(\Var[X]\).

Solution

We have shown that \(\E[X] = \frac{a+b}{2}\) and \(\E[X^2] = \frac{a^2+ab+b^2}{3}\). Therefore, the variance is

$$\begin{aligned} \Var[X] &= \E[X^2] - \E[X]^2 \\ &= \frac{a^2+ab+b^2}{3} - \left(\frac{a+b}{2}\right)^2 \\ &= \frac{(b-a)^2}{12}. \end{aligned}$$
Practice Exercise 4.9

(Exponential random variable) Let \(X\) be a continuous random variable with PDF \(f_X(x) = \lambda e^{-\lambda x}\) for \(x \ge 0\), and \(0\) otherwise. Find \(\Var[X]\).

Solution

We have shown that \(\E[X] = \frac{1}{\lambda}\). The second moment is

$$\begin{aligned} \E[X^2] &= \int_{0}^{\infty} x^2 \; \lambda e^{-\lambda x} \;dx \\ &= \left[-x^2 e^{-\lambda x}\right]_{0}^{\infty} + \int_0^{\infty} 2x e^{-\lambda x} \;dx \\ &= \frac{2}{\lambda}\int_0^{\infty} x \lambda e^{-\lambda x} \;dx \\ &= \frac{2}{\lambda} \cdot \frac{1}{\lambda} \\ &= \frac{2}{\lambda^2}. \end{aligned}$$

Therefore,

$$\begin{aligned} \Var[X] &= \E[X^2] - \E[X]^2 \\ &= \frac{2}{\lambda^2} - \frac{1}{\lambda^2} \\ &= \frac{1}{\lambda^2}. \end{aligned}$$