2
$\begingroup$

I have a dataset or flight data with several columns including delay time, and date of departure.

There are several other parameters, and I would like to run some sort of PCA with SVD to see how those parameters are correlated with each other (more particularly the delay time).

The goal I have in mind, is to use this svd to highlight specific times of the year when there are more delayed flights (eg. around christmas, etc.)

So the problem is that I have a "datetime" field which I cannot use as is for a SVD. I was suggested that maybe I could transform this datetime field into 365 binary vectors, one for each day of the year, and do the SVD with this.

Same problem with hours. I could cut it into 24 binary vectors, or maybe time groups.

Is this decomposition going to be useful ? Are there better ways to analyse data with datetime fields ?

My matrix A is going to be decomposed into UDV^T and looking at the matrix V^T I can see which dayOfYear often see delays

Number of records : ~500k per month, data available for the last 10 years (so I can have up to 160k for each DayOfYear)

$\endgroup$
3
  • $\begingroup$ This is not going to achieve what I want. The higher the value in the to-SVD matrix, the "more important" that value is considered in the SVD output. But I want each dayOfYear to be considered as importantly as the other ones. It's like want to consider each dayOfTheYear as a "category". $\endgroup$ Jan 16, 2016 at 20:57
  • $\begingroup$ Oh right sorry, I didn't realize you meant to have both sin and cos. Yes it makes more sense now. About the quantity of data, I have up to 300-500k records per month, so about 1.6k entries per day. And I can gather up to 10 years of data. I can actually tweak the 1/0 flagging to take into account days that are close to each other. I liked this idea because I thought it would make the final matrix decomposition easier to analyse, as each dayOfYear would be represented as a column in the V^T Matrix of the A = UDV^t SVD $\endgroup$ Jan 17, 2016 at 12:29
  • $\begingroup$ Usually pca is used for dimensionality reduction (compression) in practice. It will describe your dataset with different features sorted by variance. There are more suitable methods for time series though because PCA is more suited for independent identically distributed random variable while time series are not characterized necessarily from this especially when there is periodicity like in your data. Maybe methods dedicated for correlation of multivariate time series like [Granger causality.](stats.stackexchange.com/questions/409407/… $\endgroup$
    – partizanos
    Nov 15, 2022 at 0:29

1 Answer 1

-1
$\begingroup$

Take the row number of the matrix as the day of the year. That is, 01 Jan is 1 (in any year), 02 Jan as 2 etc.

Column indices may be taken as: first year (say 2005), second year (2006), etc.

If $(i,j)$ are the row and column indices starting from $1$ in both cases. Then the matrix element $(i,j)$ can be set to the delay in the day $i$, in the year $j$.

You have think something special for leap years.

A similar scheme may be used for hourly data.

$\endgroup$
1
  • $\begingroup$ This is certainly a way to represent dates using numbers, by why is this a good way? What problem does it solve, and how does it relate to OP's stated question of interpreting this data using PCA or SVD? $\endgroup$
    – Sycorax
    Nov 14, 2022 at 21:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.