For purely descriptive purposes, is it okay to run correlations between two non stationary series? I have been obsessed with trying to conduct an analysis in the “correct” way. I read that no time series analysis should be conducted on non stationary series. I found both my series have a unit root and therefore are non stationary, so I’ve been running analyses on the first differenced data.
My final question here is the following: if I am purely interested in describing the past relationship, with no intent of predicting the future, is is “okay” if I run correlation analysis on two non stationary series to identify time lags? Or should I cross correlate the first differenced data and stop asking so many questions?
I have zero interest in inferring a causal “relationship.” I only want to explain the co-movement. I’m interested to know the lag which maximizes the correlation between two interest rates. I.e. interest rate A changes now, that is correlated most with changes 2 months from now for interest rate B.
 A: Imagine that you have 2 independent series $y_t$ and $x_t$, both random walks (non-stationary series with a unit root). You run the following regression: $y_t=\beta_1+\beta_2x_t+\epsilon_t$. Davidson & MacKinnon (see the red line in snapshot below from their textbook "Econometric Theory and Methods") did this simulation 1 million times for increasingly large sample sizes. With a sample size of 20, almost 50% of the time the t-statistic for $\beta_2$ rejected the null hypothesis $\beta_2=0$. We found a relationship that doesn't really exist because we know that $y_t$ and $x_t$ are independent by design; therefore we found a spurious relationship. What's more alarming, the larger the sample size, the higher the proportion of time that the regression result will be spurious, and this proportion converges to 1.
The phenomenon of spurious regressions is not something that afflicts non-stationary series only. Imagine repeating the above simulation, this time taking $y_t$ and $x_t$ to be independent stationary AR(1) processes with the autoregressive parameter $\rho_1=0.8$. Davidson & MacKinnon did just that; see the results depicted by the blue line. For most sample sizes, some 35% of the time we will find a spurious relationship. Fortunately, the problem does not get worse the larger the sample sizes, as it did when the 2 series were random walks. If we repeated the above experiment but this time using a lower value  for the autoregressive parameter $\rho_1$ we can expect to find a lower proportion of spurious relationships. For low enough a value of the autoregressive parameter the proportion of spurious relationships should be around 0.05 (the significance level).
Whether you are looking to explain the co-movement of the 2 series, as you put it, or to predict one using the other, you will want to estimate $\beta_2$ consistently.  If $y_t$ and $x_t$ are both random walks but instead of being independent (as in the first experiment above), they happen to be cointegrated then the OLS estimator of $\beta_2$ will be super-consistent (consistent and converges to the true value even faster).
So, in summary, if the 2 series are both Integrated of Order 1 and they are cointegrated, then yes you can regress one on the other to estimate their co-movement. If they are not cointegrated, you shouldn't (think of the first of the above experiments). In this case, you should difference them and regress the two in first-differences. Even then, keep in mind that if the true process (of the first-differenced series) was an AR(1) process with a root close to 1, spuriousness will still show up. Time series analysis is a tricky business.
Keep in mind that if $y_t$ is cointegrated with $x_t$ then $y_t$ will also be cointegrated with any lag of $x_t$. So, if you conclude that $x_t$ and $y_t$ are cointegrated, you are in business - and can run binary OLS regressions of $y_t$ on lagged $x_t$, and $\beta_2$ will be consistent. You can then use a fit statistic (such as $R^2$ or $AIC$) to compare the results of these binary regressions for different lags of $x_t$.

A: That is an interesting and nontrivial question. Based on some recent related threads by the OP, let me focus on the case where the correlation is spurious and its population counterpart is ill defined. This would be the case with a pair of unit-root, noncointegrated time series.
Whether dealing with description, prediction or something else, we routinely use statistical estimates to infer something about the corresponding estimands / population counterparts, thus to generalize.$^{*}$,$^{**}$ That shapes how we react to statistical estimates when presented with them. Given a sample correlation, people would often implicitly think about generalization and picture themselves a well-defined population counterpart.$^{***}$
Therefore, I would avoid reporting sample correlation when its population counterpart is ill defined – unless I knew very well what I am doing and had ample opportunity to explain myself in detail to the audience. Otherwise, the message that gets through might be quite different from the one you may be trying to send.
$^{*}$An exception could be information compression where we only care about the sample, trying to compress it without much loss in fidelity.
$^{**}$In the case of prediction, we do not focus on the estimands directly but employ knowledge about them to obtain predictions.
$^{***}$Most people would probably instinctively consider causation, too, but that is another topic.
