Recursion. After you first-order difference, what do you have? Another time series which is, under the right conditions, closer to stationary. If it's not close enough, you now have a time series that's not stationary and you want to move it closer to stationary, so you take a first-order difference. (Which happens to be a second-order difference of the original time series.) If the differenced time series isn't close enough to stationary, you ... [recurse] ...
Derivatives. Imagine you record your car's GPS location every 10 minutes. If I could take the GPS points from any two days and show them to you and you couldn't figure out which day it might've been -- perhaps couldn't even really tell one day from another -- your location data would be stationary.
But say you drove to a different nearby city each day for two weeks? You'd easily be able to tell the difference between the days -- perhaps even knowing exactly which day I was showing you. Not stationary.
Perhaps if you instead recorded your distance from home every 10 minutes, it would make your data more stationary. Distance doesn't include direction, so perhaps now your data for those two weeks would pretty much look the same? (Average location is home, for example.)
Say, instead, that you chose to drive from New York straight through to Los Angeles. The distance trick wouldn't work, since your distance would give you a pretty clear distinction between days.
But you might then choose to record your speed every 10 minutes instead. Driving cross country on the interstate system, you days would tend to look an a lot alike, speed-wise. That is, your speed would be stationary.
Say, location at time 0 is $L_0$, and 10 and 20 minutes later is $L_1$ and $L_2$, respectively. The distance traveled in each 10-minute interval would be $D_1 = L_1 - L_0$ and $D_2 = L_2 - L_1$, which, when divided by the time interval yields the velocity (same units as speed but with direction). The second differential, $A_2 = D_2 - D_1 = L_2 - 2 L_1 + L_0$ is the acceleration. If the speed is stationary, and the vehicle is perpetually moving, the differences in location would also be stationary.