Timeline for Getting a meaningful metric for variation in this type of cyclical, panel data? WSS won't exactly cut it!
Current License: CC BY-SA 3.0
16 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Dec 12, 2014 at 16:44 | vote | accept | wolfsatthedoor | ||
| Dec 12, 2014 at 16:44 | |||||
| Dec 12, 2014 at 16:44 | review | Suggested edits | |||
| Dec 12, 2014 at 16:48 | |||||
| Dec 7, 2014 at 17:37 | comment | added | wolfsatthedoor | The point is that 11pm to midnight is not an outlier! The problem with your answer is that it gets confused by the cyclicality of the time-data. | |
| Dec 7, 2014 at 17:35 | comment | added | Andy | So then it's as I said: trim outliers if you worry about those (e.g. trips 2 standard deviations off the median time) otherwise these are features of the data. You asked for a summary statistics that captures individual within variation in trips and that's what my answer does. And this is it's purpose: it summarizes. | |
| Dec 7, 2014 at 17:32 | comment | added | wolfsatthedoor | No they are over many different days, but I only care about the hour in which they make the trip. | |
| Dec 7, 2014 at 17:27 | comment | added | Andy | But no single person would make 100 trips per day or would they? If so, then the question needs clarification. | |
| Dec 7, 2014 at 17:25 | comment | added | wolfsatthedoor | Yes this kind of thing could happen, especially with 150,000 people. There's substantial heterogeneity across people. | |
| Dec 7, 2014 at 17:22 | comment | added | Andy | Is that a realistic scenario? The description of the data doesn't tell me such things so my answer is based on what I could read from the question. Otherwise we may hope for someone else to have a better idea. | |
| Dec 7, 2014 at 16:59 | comment | added | wolfsatthedoor | That's not really much variation if 99/100 trips are made at 11pm and the first trip is made at midnight! That's definitely a problem. | |
| Dec 7, 2014 at 16:59 | comment | added | Andy | I don't see the problem with that. It's part of the variation in the data. If you are worried about outliers then you can as well trim them away. But as it stands you're asking for a summary measure that doesn't summarize everything. | |
| Dec 7, 2014 at 16:51 | comment | added | wolfsatthedoor | Andy the problem with that is that if the first purchase was at an unusual time, and all other trips are made at some other far off hour, then the variance by that metric will look large. | |
| Dec 7, 2014 at 16:39 | comment | added | wolfsatthedoor | If someone shops only at 11pm and midnight, then they will look like they have a lot if variance in their trip times because 11pm is 23 and midnight is 0. | |
| Dec 7, 2014 at 15:07 | comment | added | Andy | Alternatively, you could consider each individual-day combination and take the time distance of each purchase relative to the first purchase a person made. So if I shop first at 8am and then at 9am the distance is only 1 compared to an 8am purchase vs. 8pm purchase (distance = 12) and then take the average of these distances within day. Then take the average of those for each person over all the days in the sample. | |
| Dec 7, 2014 at 14:32 | comment | added | Andy | Not really, but you can tell me :) | |
| Dec 7, 2014 at 14:25 | comment | added | wolfsatthedoor | Suppose my variable is just hour of day. You see the problem with calculating the within sum of squares the normal way, right? | |
| Dec 7, 2014 at 13:29 | history | answered | Andy | CC BY-SA 3.0 |