Why is a random walk not a stationary process? In the book Analysis of Financial Time Series by Rue Tsay, I read:

A time series $\{p_t\}$ is a random walk if it satisfies $p_t = p_{t−1} + a_t$ 
  where $p_0$ is a real number denoting the starting value of the process
  and $\{a_t\}$ is a white noise series. If $p_t$ is the log price of a
  particular stock at date $t$ , then $p_0$ could be the log price of the
  stock $a_t$ its initial public offering (IPO) (i.e., the logged IPO
  price). If $a_t$ has a symmetric distribution around zero, then
  conditional on $p_{t−1}$, $p_t$ has a 50–50 chance to go up or down, implying
  that $p_t$ would go up or down at random. If we treat the random-walk
  model as a special AR(1) model, then the coefficient of $p_{t−1}$ is unity,
  which does not satisfy the weak stationarity condition of an AR(1)
  model. A random-walk series is, therefore, not weakly stationary, and
  we call it a unit-root nonstationary time series.

If $p_t$ has a 50–50 chance of going up or down, then its mean is constant, correct? So if $p_0=1$, and it can go to 0 with a probability of 0.5 or go to 2 with probability of 0.5, then the mean is constant at 1.
So why is this not a stationary process?
 A: For stationarity, the entire distribution of $p_t$ has to be constant over time, not only its mean. And while the mean of $p_t$ is indeed constant, e.g., it’s standard deviation isn’t. The larger $t$, the higher is the standard deviation of $p_t$ (over all realisations of the random walk – which is what you have to consider for stationarity), since individual realisations of the random walk can stray further and further from $p_0$.
From another point of view, non-stationarity is tied to special points in time, and here $t=0$ is special, since $p_0$ is fixed to $1$.
To turn this into a stationary process, you would have to equally allow for all initial conditions – which is impossible as there is no uniform distribution on the real numbers.
A: It’s not stationary because if you assume $p_t = bp_{t-1} + a_t$, then the variance of this process is $\sigma^2_{p_t}$ = $\sigma^2_{a_t} / (1-b^2)$.  Hence when b = 1, the variance explodes, (i.e- the time series could be anywhere).  This violates the condition required to be stationary (constant variance)
