Intuitive understanding covariance, cross-covariance, auto-/cross-correliation and power spectrum density I'm currently studying for my finals in basic statistics for my ECE bachelor.
While I think I have the math mostly down, I lack intuitive understanding what the numbers actually mean.(Preamble: I'll use rather sloppy language).
I know the E[X] is the "weighted average" of all results of X weighted by their probability.
Var[X] gives gives the expected variance from that E[X] squared, so tells us something about the "blurriness" of the distribution.
The other properties I know the formulae, but lack any kind of intuition. Anyone have good explanations/resources to help with that?
 A: Covariance, as you might guess from the name, indicates the tendency of two variables to co-vary or "move" together. If cov($X$, $Y$) is positive, then larger values of $X$ are associated with larger values of $Y$ and smaller values of $X$ are associated with smaller values of $Y$. If the covariance is negative, the opposite holds: small $X$s are associated with larger $Y$s and vice versa. For example, we'd expect to see a high covariance between salary and years of experience, but a low or negative covariance between weight and top running speed.
Covariance is scale-dependent (e.g., you'll get a different covariance if weight is measured in kilograms or pounds) and the units are a little strange (dollar-years and kilogram-meters-per-second in our two examples), so we often normalize covariances by dividing by $\sigma_x \cdot \sigma_y$ to get correlation. Correlation is unitless and ranges from -1 to 1, which makes it a handy measure of linear associations. (That linear bit is a very imporant caveat!)
Now, suppose we have a series of values that are somehow ordered; these are often, but not always, a time series. The auto-correlation function is the correlation between the value at position/time $t$ is with values at other positions $(t-1)$, $(t-2)$, etc. High autocorrelations may indicate that the series changes slowly, or, equivalently, that the present value is predictable from previous values.  Although variance and covariance are scalars (i.e., single values), the auto-correlation is a vector--you get an autocorrelation value for each "lag" or "gap". White noise has a very flat autocorrelation function since it's random; natural images typically have broad spatial autocorrelations since nearby pixels are often of similar color and brightness. An echo might have a peak near the center (since the sounds are self-similar), a flat region during the silence, and then another peak that constitutes the echo itself.
Cross-correlation compares two series by shifting one of them relative to the other. Like auto-correlation, it produces a vector. The middle of the vector is just the correlation between $X$ and $Y$. The entry before that is correlation between a copy of $X$ shifted slightly one way and Y; the entry after the middle is the correlation between a copy of $X$ shifted slightly the other way and $Y$. (If you're familiar with convolution, this is very similar). If $X$ and $Y$ are (possibly-delayed) copies of each other, they'll have a cross-correlation function with a peak of 1.0 somewhere, with the location of the peak given by the delay.
The auto-covariance and cross-covariance functions are like their correlation equivalents, but unscaled; it's the same difference as between covariance and correlation.
A power spectral density tells you how the power of a signal is distributed over various frequencies. The PSD of pure tone (i.e., a sine wave) is flat except at the tone's frequency; Naturalistic signals and sounds have much more complicated PSDs with harmonics, overtones, resonance, etc. It's related to the other concepts because the Fourier transform of the autocorrelation function is the PSD.
