eBook › Chapter 10 · Random Processes
The Einstein-Wiener-Khinchin theorem
The Einstein-Wiener-Khinchin theorem is a fundamental result. It states that for any wide-sense stationary process, the power spectral density \(S_X(\omega)\) is the Fourier transform of the autocorrelation function.
Theorem 10.10 (The Einstein-Wiener-Khinchin theorem) For a WSS random process \(X(t)\),
$$S_X(\omega)=\mathcal{F}\left\{R_X(\tau)\right\},$$
whenever the Fourier transform of \(R_X(\tau)\) exists.
Proof. First, let's recall the definition of \(S_X(\omega)\):
$$S_X(\omega) \bydef \lim_{T\rightarrow\infty}\frac{1}{2T}{\E\left[|\widetilde{X}_T(\omega)|^2\right]}.$$
By expanding the expectation, we have
$$\begin{aligned}
\E[|\widetilde{X}_T(\omega)|^2] &= \E\left[
\left(\int_{-T}^{T} X(t)e^{-j \omega t}\;dt\right)
\left(\int_{-T}^{T} X(\theta)e^{-j \omega \theta}\;d\theta\right)^*\right]\\
&=\int_{-T}^{T}\int_{-T}^{T} \E\left[X(t)X(\theta)\right]e^{-j \omega (t-\theta)} \;dt\;d\theta \\
&=\int_{-T}^{T}\int_{-T}^{T} R_X(t-\theta)e^{-j \omega (t-\theta)} \;dt\;d\theta.
\end{aligned}$$
Our next step is to analyze \(R_X(t-\theta)\). Define
$$Q_X(v)=\mathcal{F} \left\{R_X(\tau)\right\}.$$
Then, by inverse Fourier transform
$$\begin{aligned}
R_X(\tau) = \frac{1}{2\pi} \int_{-\infty}^{\infty} Q_X(v) e^{j v \tau}\;dv,
\end{aligned}$$
and therefore
$$R_X(t-\theta)=\frac{1}{2\pi} \int_{-\infty}^{\infty} Q_X(v) e^{j v (t-\theta)}\;dv.$$
Substituting this into Equation (eq:2) yields
$$\begin{aligned}
\E\left[|\widetilde{X}_T(\omega)|^2\right]
&= \int_{-T}^{T}\int_{-T}^{T} \left(\frac{1}{2\pi} \int_{-\infty}^{\infty} Q_X(v) e^{j v (t-\theta)}\;dv\right)e^{-j \omega (t-\theta)} \;dt\;d\theta \\
&= \frac{1}{2\pi} \int_{-\infty}^{\infty} Q_X(v) \left(\int_{-T}^{T} e^{j t (v-\omega)} \;dt\right)\left(\int_{-T}^{T} e^{j \theta (\omega-v)} \;d\theta\right)\;dv.
\end{aligned}$$
We now need to simplify the two inner integrals. Recall by Fourier pair that
$$\textrm{rect} \left(\frac{t}{T}\right) \quad \underset{\longleftrightarrow}{\mathcal{F}} \quad T \textrm{sinc}\left(\frac{\omega T}{2}\right).$$
This implies that
$$\begin{aligned}
\int_{-T}^{T} e^{j t (v-\omega)}\;dt &= \int_{-T}^{T} e^{-j(\omega-v)t}\;dt \\
&=\int_{-\infty}^{\infty} \textrm{rect}\!\left(\frac{t}{2T}\right)e^{-j(\omega-v)t}\;dt
=2T \; \textrm{sinc}((\omega-v)T) \\
&=2T \; \frac{\sin((\omega-v)T)}{(\omega-v)T}.
\end{aligned}$$
Hence, we have
$$\E\left[|\widetilde{X}_T(\omega)|^2\right] = \frac{1}{2\pi} \int_{-\infty}^{\infty} Q_X(v) \left(2T \; \frac{\sin((\omega-v)T)}{(\omega-v)T} \right)^2 \;dv.$$
and so
$$\begin{aligned}
\frac{1}{2T}\E\left[|\widetilde{X}_T(\omega)|^2\right]
&= \frac{2T}{2\pi} \int_{-\infty}^{\infty} Q_X(v) \left(\frac{\sin((\omega-v)T)}{(\omega-v)T} \right)^2 \;dv.
\end{aligned}$$
As \(T \rightarrow \infty\) (see Lemma lemma: 10.5 below), we have
$$2T \left(\frac{\sin((\omega-v)T)}{(\omega-v)T}\right)^2 \;\; \longrightarrow \;\; 2\pi \delta(\omega-v).$$
Therefore,
$$\begin{aligned}
\lim_{T\rightarrow\infty} \frac{1}{2T}\E \left[|\widetilde{X}_T(\omega)|^2\right]
&= \frac{1}{2\pi} \int_{-\infty}^{\infty} Q_X(v) \left[\lim_{T\rightarrow\infty} 2T \left(\frac{\sin((\omega-v)T)}{(\omega-v)T}\right)^2 \right] \;dv \\
&= \int_{-\infty}^{\infty} Q_X(v) \delta(\omega-v)\;dv
= Q_X(\omega).
\end{aligned}$$
Since \(Q_X(\omega)=\mathcal{F}[R_X(\tau)]\), we conclude that
$$\begin{aligned}
S_X(\omega) &= \lim_{T\rightarrow\infty} \frac{1}{2T}\E[|\widetilde{X}_T(\omega)|^2]
= Q_X(\omega)
= \mathcal{F}[R_X(\tau)].
\end{aligned}$$
■ Lemma 10.5
$$\begin{aligned}
\lim_{T\rightarrow\infty}\frac{1}{2\pi} \int_{-\infty}^{\infty} Q_X(v) 2T\left( \frac{\sin((\omega-v)T)}{(\omega-v)T}\right)^2 \;dv
&= Q_X(\omega).
\end{aligned}$$
To prove this lemma, we first define \(\delta_T(\omega)=2T(\frac{\sin(\omega T)}{\omega T})^2\). It is sufficient to show that
$$\begin{aligned}
&\bigg|\lim_{T\rightarrow\infty}\frac{1}{2\pi} \int_{-\infty}^{\infty} Q_X(v) 2T\left( \frac{\sin((\omega-v)T)}{(\omega-v)T}\right)^2 \;dv \\
&\qquad\qquad - Q_X(\omega)\bigg|\rightarrow 0 \quad \textrm{ as }\quad T\rightarrow\infty.
\end{aligned}$$
We will proceed by demonstrating the following three facts about \(\delta_T(\omega)\):
$$\frac{1}{2\pi} \int_{-\infty}^{\infty} \delta_T(\omega) \; d\omega=1$$
For any \(\triangle>0\), $$\int_{\left\{\omega: |\omega|>\triangle\right\}} \delta_T(\omega) \; d\omega \rightarrow 0 \quad \textrm{ as } \quad T\rightarrow \infty$$
For any \(|\omega|\geq \triangle >0\), we have \(|\delta_T(\omega)|\leq\frac{2}{T\triangle^2}\) Fact 1. $$\frac{1}{2\pi} \int_{-\infty}^{\infty} \delta_T(\omega) \; d\omega=\frac{1}{2\pi} \int_{-\infty}^{\infty} 2T\underbrace{\left(\frac{\sin(\omega T)}{\omega T}\right)^2}_{\textrm{sinc}^2(\omega T)} \; d\omega.$$
Note that
$$\Lambda\left(\frac{t}{4T}\right) \longleftrightarrow 2T\textrm{sinc}^2(\omega T).$$
Therefore,
$$\begin{aligned}
\frac{1}{2\pi} \int_{-\infty}^{\infty} 2T\textrm{sinc}^2(\omega T)\;d\omega &= \frac{1}{2\pi} \int_{-\infty}^{\infty} 2T\textrm{sinc}^2(\omega T) e^{j \omega 0}\;d\omega\\
&= \Lambda\left(\frac{0}{4T}\right) = 1.
\end{aligned}$$
■ Fact 2. \(\delta_T(\omega)\) is symmetric, so, it is sufficient to check only one side:
$$\begin{aligned}
\int_{\triangle}^{\infty} \delta_T(\omega) \; d\omega &= \int_{\triangle}^{\infty} 2T \left(\frac{\sin(\omega T)}{\omega T}\right)^2 \; d\omega\\
&=\frac{2T}{T^2}\int_{\triangle}^{\infty} \frac{\sin^2(\omega T)}{\omega^2} \; d\omega\\
& \leq \frac{2}{T} \int_{\triangle}^{\infty} \frac{1}{\omega^2} \; d\omega \quad \quad \quad |\sin(.)|^2\leq1 \\
&= \frac{2}{T} \left[-\frac{1}{\omega}\right]_\triangle^\infty
= \frac{2}{T\triangle} \rightarrow 0 \quad \textrm{ as } \quad T \rightarrow \infty.
\end{aligned}$$
■ Fact 3.
$$\begin{aligned}
|\delta_T(\omega)| &= 2T \left(\frac{\sin(\omega T)}{\omega T}\right)^2 \leq 2T \left(\frac{1}{(\omega T)^2}\right) = \frac{2}{\omega^2 T} \leq \frac{2}{T \triangle^2}.
\end{aligned}$$
■ Lemma. Consider \(Q_X(\omega)\). By Property 1,
$$\begin{aligned}
Q_X(\omega)&= Q_X(\omega).\frac{1}{2\pi} \int_{-\infty}^\infty \delta_T(\omega-v)\;dv =\frac{1}{2\pi} \int_{-\infty}^\infty Q_X(\omega) \delta_T(\omega-v)\;dv.
\end{aligned}$$
Therefore,
$$\begin{aligned}
&\bigg|\frac{1}{2\pi} \int_{-\infty}^\infty Q_X(v) \delta_T(\omega-v)\;dv-Q_X(\omega)\bigg|\\
&=\bigg|\frac{1}{2\pi} \int_{-\infty}^\infty Q_X(v) \delta_T(\omega-v)\;dv-\frac{1}{2\pi} \int_{-\infty}^\infty Q_X(\omega) \delta_T(\omega-v)\;dv\bigg| \\
&= \frac{1}{2\pi} \bigg|\int_{-\infty}^\infty \left(Q_X(v)-Q_X(\omega)\right)\delta_T(\omega-v) \;dv\bigg| \\
&\leq \frac{1}{2\pi} \int_{-\infty}^\infty \big|Q_X(v)-Q_X(\omega)\big|\delta_T(\omega-v) \;dv.
\end{aligned}$$
For any \(\epsilon>0\), let \(\triangle\) be a constant such that $$|\omega-v|<\triangle \quad\textrm{ whenever }\quad |Q_X(v)-Q_X(\omega)|<\epsilon.$$
Then we can partition the above integral into
$$\begin{aligned}
\frac{1}{2\pi} \int_{-\infty}^\infty &\big|Q_X(\omega)-Q_X(v)\big|\delta_T(\omega-v) \;dv \\
&= \frac{1}{2\pi} \int_{\omega-\triangle}^{\omega+\triangle} \big|Q_X(\omega)-Q_X(v)\big|\delta_T(\omega-v) \;dv \quad (1) \\
&\quad+ \frac{1}{2\pi} \int_{\omega+\triangle}^{\infty} \big|Q_X(\omega)-Q_X(v)\big|\delta_T(\omega-v) \;dv \quad (2) \\
&\quad+ \frac{1}{2\pi} \int_{-\infty}^{\omega-\triangle} \big|Q_X(\omega)-Q_X(v)\big|\delta_T(\omega-v) \;dv. \quad (3)
\end{aligned}$$
Partition (1) above can be evaluated as follows:
$$\begin{aligned}
&\frac{1}{2\pi} \int_{\omega-\triangle}^{\omega+\triangle} \big|Q_X(\omega)-Q_X(v)\big|\delta_T(\omega-v) \;dv \\
&\qquad\leq \frac{1}{2\pi} \int_{\omega-\triangle}^{\omega+\triangle} \epsilon \delta_T(\omega-v) \;dv\\
&\qquad = \frac{\epsilon}{2\pi} \int_{\omega-\triangle}^{\omega+\triangle} \delta_T(\omega-v) \;dv\\
&\qquad\leq \frac{\epsilon }{2\pi} \int_{-\infty}^{\infty} \delta_T(\omega-v)\;dv = \epsilon,
\end{aligned}$$
where the last inequality holds because \(\delta_T(\omega-v)\geq 0\). Since \(\epsilon\) can be arbitrarily small, the only possibility for $$\frac{1}{2\pi} \int_{\omega-\triangle}^{\omega+\triangle} \big|Q_X(\omega)-Q_X(v)\big|\delta_T(\omega-v) \;dv$$ for all \(\epsilon\) is that the integral is \(0\).
Partition (2) above can be evaluated as follows:
$$\begin{aligned}
&\frac{1}{2\pi} \int_{\omega+\triangle}^{\infty} \big|Q_X(\omega)-Q_X(v)\big|\delta_T(\omega-v) \;dv \\
&\qquad\leq \frac{1}{2\pi} \int_{\omega+\triangle}^{\infty} \left(\big|Q_X(\omega)\big|+\big|Q_X(v)\big|\right)\delta_T(\omega-v) \;dv \\
&\qquad= Q_X(\omega) \frac{1}{2\pi} \int_{\omega+\triangle}^{\infty} \delta_T(\omega-v)\;dv + \frac{1}{2\pi} \int_{\omega+\triangle}^{\infty} Q_X(v) \delta_T(\omega-v)\;dv.
\end{aligned}$$
By Property 2, \(\frac{1}{2\pi} \int_{\omega+\triangle}^{\infty} \delta_T(\omega-v)\;dv \rightarrow 0\) as \(T\rightarrow \infty\). By Property 3,
$$\begin{aligned}
\frac{1}{2\pi} \int_{\omega+\triangle}^{\infty} Q_X(v) \delta_T(\omega-v)\;dv
&\leq \frac{1}{2\pi} \frac{2}{T\triangle^2} \underbrace{\int_{\omega+\triangle}^{\infty} Q_X(v) \;dv}_{<\infty,\; Q_X = \mathcal{F}[R_X] } \rightarrow 0.
\end{aligned}$$
Therefore, we conclude that
$$\frac{1}{2\pi} \int_{\omega+\triangle}^{\infty} Q_X(v) \delta_T(\omega-v)\;dv \rightarrow 0\quad \textrm{ as } \quad T\rightarrow \infty,$$
and hence (1), (2) and (3) all \(\rightarrow 0\) as \(T\rightarrow\infty\).
So we have
$$\begin{aligned}
&\bigg|\lim_{T\rightarrow\infty}\frac{1}{2\pi} \int_{-\infty}^{\infty} Q_X(v) 2T\left( \frac{\sin((\omega-v)T)}{(\omega-v)T}\right)^2 \;dv \\
&\qquad\qquad - Q_X(\omega)\bigg|\rightarrow 0 \quad\textrm{ as }\quad T\rightarrow\infty,
\end{aligned}$$
which completes the proof.
■ The Mean-Square Ergodic Theorem
The mean-square ergodic theorem states that for any WSS random process, the statistical average is the same as the temporal average . This provides an important tool in practice because finding the statistical average is typically very difficult. With the mean-square ergodic theorem, one can easily estimate the statistical average using the temporal average.
Theorem 10.11 (Mean-Square Ergodic Theorem) Let \(Y(t)\) be a WSS process, with mean \(\E[Y(t)] = m\) and autocorrelation function \(R_Y(\tau)\). Assume that the Fourier transform of \(R_Y(\tau)\) exists. Define
$$M_T \bydef \frac{1}{2T} \int_{-T}^T Y(t)\;dt.$$
Then \(\E\left[\big|M_T-m\big|^2\right]\rightarrow 0\) as \(T\rightarrow\infty\).
Mean-Square Ergodic Theorem. Let \(X(t)=Y(t)-m\). It follows that
$$\begin{aligned}
M_T-m &= \frac{1}{2T} \int_{-T}^T Y(t)\;dt - m
=\frac{1}{2T} \int_{-T}^T X(t)\;dt.
\end{aligned}$$
We define the finite-window approximation of \(X(t)\):
$$X_T(t)=\left\{
\begin{array}{cc}
X(t), & -T \leq t \leq T, \\
0, & \textrm{elsewhere}.
\end{array}
\right.$$
Then the difference \(M_T-m\) can be computed as
$$\begin{aligned}
M_T-m &= \frac{1}{2T} \int_{-T}^T X(t)\;dt
=\frac{1}{2T} \int_{-\infty}^\infty X(t)e^{-j 0 t}\;dt =\frac{1}{2T} \widetilde{X}_T(\omega)\big|_{\omega=0}
=\frac{\widetilde{X}_T(0)}{2T}.
\end{aligned}$$
Taking the expectation of the squares yields $$\E\left[|M_T-m|^2\right]=\frac{\E\left[\big|\widetilde{X}_T(0)\big|^2\right]}{4T^2}.$$
Recall from the Einstein-Wiener-Khinchin theorem,
$$\begin{aligned}
\frac{1}{2T} \E\left[\big|\widetilde{X}_T(\omega)\big|^2\right]
&=\frac{1}{2\pi} \int_{-\infty}^\infty S_X(v) 2T \left(\frac{\sin((\omega-v)T)}{(\omega-v)T}\right)^2 \;dv.
\end{aligned}$$
Putting the limit \(T\rightarrow \infty\), if we have that
$$\begin{aligned}
\lim_{T \rightarrow \infty} \frac{1}{2\pi} \int_{-\infty}^\infty S_X(v) 2T \left(\frac{\sin((\omega-v)T)}{(\omega-v)T}\right)^2 \;dv
&= S_X(\omega),
\end{aligned}$$
then we will have
$$\begin{aligned}
\frac{1}{2T} \E\left[\big|\widetilde{X}_T(\omega)\big|^2\right] \rightarrow S_X(\omega) \;\; \text{and} \;\;
\frac{1}{2T} \E\left[\big|\widetilde{X}_T(0)\big|^2\right] \rightarrow S_X(0).
\end{aligned}$$
Hence,
$$\begin{aligned}
\lim_{T\rightarrow \infty} \E\left[\big|M_T-m\big|^2\right]&= \lim_{T\rightarrow \infty} \frac{1}{2T} \cdot \frac{1}{2T}\E\left[\big|\widetilde{X}_T(0)\big|^2\right]
= \lim_{T\rightarrow \infty} \frac{1}{2T} S_X(0)
= 0.
\end{aligned}$$
This completes the proof.
■ ‹ Previous Summary