0
$\begingroup$

I have a 4d dataset.

The first two dimensions consist of latitude and longitude coordinates. Hence they will utilize the orthodromic distance.

Latitudes range from -90 to 90. Longitudes range from -180 to 180.

  • Example: 35.137879, -82.836914 (Somewhere on Google maps)

The third dimension consists of day of week. Monday would be number 0, Tuesday 1 etc. up until Sunday which would be number 6. Whole numbers. Euclidean distance.

  • Example: 5 (Would be Saturday)

The last dimension consists of time of day. Consists of a number which represents seconds which have past since 00:00:00 of that day. Euclidean distance. Interval 0 (00:00:00) to 86399 (23:59:59)

  • Example: 27060 (HH:MM:SS -> 07:31:00)

Since half of the dataset use orthodromic distance and the other half euclidean distance this complicates it slightly (I presume).

Another issue is circular variables. All of the dimensions represent circular variables, posing another issue.

I saw this: new value = (value-min)/(max-min) Could I first convert orthodromic to eucliden and then put it all through this formula?

How does one "normalize" this type of dataset? Are there any libraries which one can use? I keep reading about what normalizing is but never has anyone shown a practical example.

Improve this question
$\endgroup$
5
  • $\begingroup$ For what purpose are you "normalizing" the data? The word can mean different things in different contexts. Also, time-of-day, day-of-week, and longitude might need to be considered as circular rather than linear variables; for example, there is only 1 minute difference between 24-hour times of 23:59 and 00:00, but in your "Euclidean" distance measure they would be far apart. Similarly for Sunday versus Monday; longitudes of -180 and +180 are identical, not separated by 360 units. $\endgroup$
    – EdM
    Aug 30, 2015 at 18:41
  • $\begingroup$ @EdM Ah, good point. I'll read up and get back to you! (I am to throw the entire dataset into a clustering method) $\endgroup$
    – Karl Morrison
    Aug 30, 2015 at 18:42
  • $\begingroup$ Edit the question rather than just post another comment, as that brings the question back to the top of the queue for others to see. I know about the importance of circular variables, but have no experience in analyzing them. More details about what you are trying to accomplish, beyond "normalizing" the data, would help readers judge whether "normalizing" is important in your case or if there might be better approaches to addressing the scientific (as opposed to just statistical) issues underlying your clustering scheme, particularly if your data come from an extended time series. $\endgroup$
    – EdM
    Aug 30, 2015 at 18:50
  • $\begingroup$ You are correct on the circular variables as for example 23:59 distance wise is not close to 00:01 however it should be close when clustering. I'll include it. $\endgroup$
    – Karl Morrison
    Aug 30, 2015 at 18:51
  • $\begingroup$ To give useful answers, those using this site will need more information about what and why you are trying to cluster. Add that to the original question. It's particularly important to know if your data are from extended time series, as time-series data can have lots of potential "gotchas" that aren't always obvious at first glance. $\endgroup$
    – EdM
    Aug 30, 2015 at 21:51

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.