0
$\begingroup$

I have data shown below, with a response variable tabulated by hour (x-axis) as well as numerous other factors: results larger chart available here

I want to see if the colored points (red, light and dark blue, orange) are statistically different from the grey points. Upon visual inspection it seems as though the colored points differ from the grey points at hours 6 and 7, so I tried to conduct a 2-sample t-test of colored points vs grey points for each of the 24 hours separately, and got the following results:

    Hour           p-value for 2-sample t test
[1] "00"             "0.226638013124"
[1] "01"             "0.691245588294"
[1] "02"             "0.840988947432"
[1] "03"             "0.147960817870"
[1] "04"             "0.673288043713"
[1] "05"             "0.0000640211129743"
[1] "06"             "0.000186641391119"
[1] "07"             "0.000417492339098"
[1] "08"             "0.000111826661499"
[1] "09"             "0.0000426931229112"
[1] "10"             "0.000274211619809"
[1] "11"             "0.0000342302853781"
[1] "12"             "0.00357160058606"
[1] "13"             "0.00108203195091"
[1] "14"             "0.00695746584398"
[1] "15"             "0.134903506946"
[1] "16"             "0.0792616667511"
[1] "17"             "0.258782638906"
[1] "18"             "0.265273732241"
[1] "19"             "0.0615368690249"
[1] "20"             "0.0738850389549"
[1] "21"             "0.215317462846"
[1] "22"             "0.0531097020924"
[1] "23"             "0.0344662311452"

The fact that many of these p-values are so low surprise me, given the points on the chart. Even with a simple Bonferroni correction for multiple comparisons, hours 5,6,7,8,9,10,11,13 and 14 are significant with a p-value < .05/24.

I can understand why this is so (large number of observations, small standard deviation), but intuitively it does not look right, especially for hours 9,10,11,13 and 14. I just wanted to check with people who are more knowledgeable to see if there is anything I am doing wrong or anything I have missed out. Is a 2-sample t-test on each hour the wrong procedure? Assuming the data is valid, would you be comfortable concluding that hours 9,10,11,13 and 14 are significant?

Any help would be greatly appreciated; I am more comfortable with data visualization than I am with statistics.

Thanks!

EDIT: added boxplot for the relevant hours. Using a 2-sample t-test with Bonferroni correction for multiple comparisons, all of the hours below show significant differences.

enter image description here

$\endgroup$
  • $\begingroup$ Also, I have checked the hourly data for autocorrelation and found nothing significant. For the 2-sample t-test, I am also using the Welch approximation for degrees of freedom to avoid assuming equal variances. $\endgroup$ – ethane Apr 28 '15 at 22:54
  • 2
    $\begingroup$ Your plot makes it impossible to distinguish point colors reliably. Why don't you make some sets of side-by-side boxplots for each hour so you can see what's going on in your data? $\endgroup$ – whuber Apr 28 '15 at 23:01
  • 1
    $\begingroup$ You seem to have a pretty large data set, so almost any difference will be shown to be statistically significant. What are the means and standard deviations of the different groups? Also, why do you lump together the different colored dots? I look at the plot and it seems that in each of the hours after 12 there is an orange dot near the low end of the observations. This looks like outliers to me, and a t-test is susceptible to outliers (by contaminating both the mean and standard deviation). $\endgroup$ – user3697176 Apr 28 '15 at 23:36
  • $\begingroup$ @whuber, done. as you can see I have misgivings about hours 8-14. $\endgroup$ – ethane Apr 29 '15 at 0:02
  • $\begingroup$ @user3697176, I have about 250 points for each hour. I just added a boxplot, as you can see I have misgivings about hours 8-14. The data actually shows occupancy at each hour over different days, and the color represents adverse weather conditions (rain=blue, orange/red=snow). There aren't enough red/orange points for me to comfortably work with so I lumped it in with the blue (rain), but even if I omit the red/orange points from the sample I will still face the same issue (how to determine statistical significance with a large n since sd will be tiny). $\endgroup$ – ethane Apr 29 '15 at 0:08
0
$\begingroup$

The data is likely to be not normally distributed. As mentioned by @ user3697176 , you should test with Wilcoxan test. Also it will be worthwhile to post mean/median values of dependent variable (cabsOccupied) in 2 groups. The P value will be significant if there is even a small difference if the sample size is large. But actual difference in outcome variable may not be much different. This can also be tested by 'effect size' for which there are several methods: https://en.wikipedia.org/wiki/Effect_size

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.