1
$\begingroup$

I've recently studied Fourier transform and I've applied it on a time series data, since I am still confused between time and frequency domain I doubt the authenticity of my code to calculate Fourier transform. The data used in analysis is in the link.

Following is the code that I've used to compute and visualize the transform:

df_new = pd.read_csv('df_avn.csv',
                     usecols = ['Date', 'Close'], parse_dates = True,
                     index_col = ['Date'])
t = np.linspace(0,24*60*60, 55)
s = df_new.values
sns.set_style("darkgrid")
plt.ylabel("Amplitude")
plt.xlabel("Time [s]")
plt.plot(t, s)
plt.show()

fft = scipy.fftpack.fft(s)
T = t[1] - t[0]  # sample rate
N = s.size

# 1/T = frequency
f = np.linspace(0, 1 /(T), N)
f = f * 1000
sns.set_style("darkgrid")
plt.ylabel("Amplitude")
plt.xlabel("Frequency [mHz]")
plt.plot(f[:N // 2], np.abs(fft)[:N // 2]  * 1 / N)  
plt.show()

To explain what I've done in the above code, since the data is has of daily frequency for a total of 55 days I've converted days to seconds using 24*60*60 and plotted it against the original series. As shown here Original Data plotted against time in seconds.

Next I used scipy Fast Fourier Transform to transform the series and achieved a transformation as follows:enter image description here

Edit: As suggested by Dan scaling was done by using following piece of code f = np.linspace(0, 1/(2*T), N). Result obtained is as follows: enter image description here

My first concern is that the implementation of Fourier Transform as demonstrated above is it correct? If yes then we can move on to the following next two questions.

I am having trouble in explaining the three peaks in transformed graph. What I think is that these three peaks have highest significance, is it that the correct interpretation? Furthermore how would you explain the frequencies at which these three peaks occur to a layman.

Is it more appropriate to use a data converted in frequency domain for clustering (using e.g. Kmeans) or is it appropriate to use time series data without converting to frequency domain?

$\endgroup$

closed as unclear what you're asking by Michael Chernick, kjetil b halvorsen, mdewey, Xavier Bourret Sicotte, mkt Nov 7 '18 at 7:56

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    $\begingroup$ Are you sure your frequency domain axis scaling is right? If I eye-ball your time domain graph, you have big peaks every 20000 seconds. That should make a frequency domain peak at 0.00005Hz, but your lowest peak is at around 40Hz $\endgroup$ – Dan Oct 30 '18 at 18:25
  • $\begingroup$ Your sampling frequency (1/T) is 0.000625. So considering the nyquist sampling rate, you can't really measure frequencies more then half of that so 0.0003125. I think you just need to scale your frequency axis to correct for this. The shape of your curve won't change $\endgroup$ – Dan Oct 30 '18 at 18:29
  • $\begingroup$ But regarding the three peaks, one is from those large peaks hitting 4 which occur roughly every 20 000 seconds. Then there seems to be a peak or trough roughly every 5 000 - 6 000 seconds. So those should explain at least two of your peaks. Maybe once you have sorted out the scaling the third peak will make more sense $\endgroup$ – Dan Oct 30 '18 at 18:32
  • 1
    $\begingroup$ For the scaling, i think you want f = np.linspace(0, 1/(2*T), N) and then if you do f = f*1000 you've converted your units to mHz not kHz I think. dsp.stackexchange.com/a/40769/8153 $\endgroup$ – Dan Oct 30 '18 at 18:38
  • $\begingroup$ @Dan I've made changes as suggested, your explanation is logical but I am still trying to understand nyquist sampling rate and the change in kHz to mHz. I've edited the transformed data graph as suggested by you. $\endgroup$ – Furqan Hashim Oct 30 '18 at 18:47