1
$\begingroup$

I have data that represents the length of certain events.

I have 13 months of data and want to know if there has been a significant increase over the last year in these lengths.

The first method I can think of seems crude: withhold each month's mean, find the average of the remaining and see if that month's mean value is significantly different from the average of the others (within 2 standard deviations).

oct 9.54
nov 9.77
dec 9.38
jan 9.59
feb 9.59
mar 9.41
apr 10.4
may 10.31
jun 10.33
jul 10.38
aug 10.41
sep 10.44
oct2 10.51

 9.54-average(9.77,9.38,9.59,9.59,9.41,10.4,10.31,10.33,10.38,10.41,10.44,10.51) = -0.05 

2*standard deviation of nov-oct2 = 0.86

0.05 < 0.86 therefore not significant?

If I do this for every month it looks as if none are significantly different. So each month fits in with the year and therefore not an increase.

I then decided I wasn't confident with that method at all and so I did some tests. To start, I did a KS test, a T test and a Wilcoxon test on the two October months given:

Mean 1: 9.54
Mean 2: 10.51
N1: 147770
N2: 138371
Std Dev.1: 8.32
Std Dev.2: 8.07

Results look like:

Two-sample Kolmogorov-Smirnov test

data:  dat$A and dat$B
D = 0.060383, p-value < 2.2e-16
alternative hypothesis: two-sided    

Wilcoxon rank sum test with continuity correction

data:  dat$A and dat$B
W = 9556400000, p-value < 2.2e-16
alternative hypothesis: true location shift is not equal to 0

    Welch Two Sample t-test

data:  dat$A and dat$B
t = -31.585, df = 285780, p-value < 2.2e-16
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -1.0276513 -0.9075632
sample estimates:
mean of x mean of y 
 9.543439 10.511046 

So that's telling me there are (massively) significant differences between the two months.

Also looking graphically there are fluctuations but there seems to be an increase across the year.

EDIT

Each month has roughly 140,000 observations. I can split these by other metrics but thought first to try a method on everything and then to extend it to each group to improve accuracy. I thought I would compare distributions as if the newest month is skewed more towards longer events then that shows an overall average increase in event lengths. I used R for the tests and simply uploaded a csv with two columns for each October as the data.frame dat. I thought I would look at each October as that cancels out seasonality. Each SD is from the vector of ~140,000 obs for each month. I have experience in using multiple regression for financial forecasting so I am open to those techniques. I'm not sure why I would want to make the period longer if I am only interested in this year. I suppose two years is a better choice to compare months with the same seasonality rather than just two Octobers.

Regression attempt using data from 2015, using daily averages rather than monthly and d being the length and using months and years as dummy variables:

Model:  Model 01                            
Dependent Variable:     d                       

Regression Statistics:    Model 10 for d    (12 variables, n=695)                               
    R-Squared   Adj.R-Sqr.  Std.Err.Reg.    Std. Dev.   # Cases # Missing   t(2.50%,682)    Conf. level
    0.198   0.184   0.745   0.824   695 0   1.963   95.0%

Coefficient Estimates:    Model 10 for d    (12 variables, n=695)                               
Variable    Coefficient Std.Err.    t-Stat. P-value Lower95%    Upper95%    Std. Dev.   Std. Coeff.
Constant    9.926   0.143   69.403  0.000   9.645   10.207      
month_EQ_1  -0.212  0.164   -1.290  0.197   -0.535  0.111   0.285   -0.073
month_EQ_10 0.203   0.164   1.233   0.218   -0.120  0.525   0.285   0.070
month_EQ_11 0.263   0.168   1.567   0.118   -0.067  0.593   0.268   0.085
month_EQ_2  -0.283  0.167   -1.696  0.090   -0.610  0.045   0.275   -0.094
month_EQ_3  -0.446  0.164   -2.714  0.007   -0.769  -0.123  0.285   -0.154
month_EQ_4  0.034   0.165   0.203   0.839   -0.291  0.358   0.281   0.011
month_EQ_5  0.004784    0.164   0.029   0.977   -0.318  0.327   0.285   0.002
month_EQ_6  0.123   0.165   0.742   0.458   -0.202  0.447   0.281   0.042
month_EQ_7  0.126   0.164   0.768   0.443   -0.196  0.449   0.285   0.044
month_EQ_8  0.080   0.164   0.485   0.628   -0.243  0.402   0.285   0.028
month_EQ_9  0.111   0.165   0.671   0.502   -0.214  0.435   0.281   0.038
year_EQ_2015    -0.615  0.058   -10.653 0.000   -0.728  -0.501  0.500   -0.373

enter image description here

Now obviously the R-squared is horrific, but I don't care about the model per se. Looking at the negative coefficient for 2015 (-0.615) implies this year there is an increase on last. But how do I know if that increase is deemed significant?

$\endgroup$
8
  • $\begingroup$ No need to apologise. We will be candid if we think you are wrong, but we won't be vicious. $\endgroup$
    – Nick Cox
    Commented Nov 24, 2016 at 13:16
  • 1
    $\begingroup$ With millions of observations, significance testing is arguably superfluous. It's a practical matter of calculating trend and seeing if that is a quantity you care about. $\endgroup$
    – Nick Cox
    Commented Nov 24, 2016 at 14:32
  • $\begingroup$ 2015 had 365 days. Why not use just daily date with specified origin as predictor? If you expect more complicated variations, use a scatterplot smoother. The coefficient of year == 2015 isn't a measure of trend. It just compares days in 2015 with whatever days were not in 2015 (330 of them?). $\endgroup$
    – Nick Cox
    Commented Nov 24, 2016 at 15:12
  • $\begingroup$ Predicted d = -62.817 + 0.00171*date implying as date increases length increaes however again Rsquared only .173 $\endgroup$
    – Olivia
    Commented Nov 24, 2016 at 15:21
  • $\begingroup$ I don't know what these data are (event lengths means nothing particularly to me), but you presumably do: so why are you, it seems, surprised by day to day variability as well as a trend? $\endgroup$
    – Nick Cox
    Commented Nov 24, 2016 at 15:31

1 Answer 1

3
$\begingroup$

Your methods don't seem directed at your question, but there are several puzzles over what your data are and what you have done.

First off, it seems that what you show are themselves means for each month of larger datasets. Whether that reduction discards useful detail is an open question.

Looking at each month relative to the others makes some sense but there are better ways to do it. I won't comment in detail on your procedure except that it is problematic in various ways.

I can't fully follow your Kolmogorov-Smirnov test, t test or Wilcoxon test.

The first (K-S) is a test of whether distributions differ and has nothing at all to do with testing trend in time.

Without explicit previous steps, your analysis doesn't seem reproducible even by those using the unnamed software you used (quite apart from the apparent reduction to monthly means).

I can't see the logic of comparing just the two Octobers while getting the standard deviations ... from where??? Looking for trend can't validly focus just on beginning and end periods. The fallacy lurking here is seen in its strongest form by imagining a test comparing the heights of very tall and very short people selected from a larger group.

A simple method that should be relevant is just to plot the monthly data (or means) and check for a trend. Conventional linear regression results lead to rejecting a null hypothesis at conventional levels and so supports the idea of an increase. But the $R^2$ (e.g.) is inflated by just looking at means, as within-month variation is suppressed, whereas conversely, almost any trend would be deemed significant if the raw data were analysed and the true sample size is much larger than 13.

enter image description here

It's not at all obvious that a straight-line fit is quite right, but without any ideas on the process here or a longer dataset, it seems prudent just to stop there, and let a graph tell us that there's an increase and that a straight line trend is a coarse summary. There has to be a higher-level reservation about dependence in time of the errors and what is being assumed.

Some might want to direct you to time series analysis here, and in principle they have a good case, but my prejudice with data like yours is that time series analysis would make a simple problem more complicated and that you have far too short a dataset to fit anything more than very simple summaries. If this is just a token problem and you have, or will have, much more data, then the balance tips in favour of that approach. However, time series analysis itself might need to give daily or monthly summaries if the raw data are unevenly spaced in time.

Somehow we seem to be assuming that season of year is irrelevant.

$\endgroup$
4
  • $\begingroup$ Edited in response $\endgroup$
    – Olivia
    Commented Nov 24, 2016 at 13:59
  • $\begingroup$ So I could take two or three years data and work out a regression model (as if I were to forecast) and if the model is positive (such that a forecast would be going up) then it shows an increase. $\endgroup$
    – Olivia
    Commented Nov 24, 2016 at 14:23
  • $\begingroup$ (Forgot to add a graph!) In short, yes indeed, except that you want a positive slope (or gradient) (not model; a model has no sign). $\endgroup$
    – Nick Cox
    Commented Nov 24, 2016 at 14:28
  • $\begingroup$ I had a go at some regression (using a very useful excel add in). The model isnt a great fit but it may tell the story Im looking for? $\endgroup$
    – Olivia
    Commented Nov 24, 2016 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.