# Regression of data that includes a date

I have a dataset that contains a few hundred transactions from a three suppliers operating in 100+ countries over a three year period.

We've found that the country of sales is not a significant factor in the prices achieved (the products are more or less global commodities). All of the prices have declined significantly over time. Any one day can have multiple transactions at different prices from the same supplier (i.e. in different countries).

I would like to test whether there is a statistically significant difference in the prices charged by the different suppliers.

The data look something like this:

    Country X  1/1/2010  $200 Supplier A Country Y 1/1/2010$209 Supplier A
Country Z  1/1/2010  $187 Supplier A Country A 1/1/2010$200 Supplier B
Country X  1/2/2010  \$188 Supplier B


Any ideas on how to do this?.....

• Longitudinal data analysis is a broad field. To give a good answer to this question, more info about what effect you expect time to have is necessary. Without this, it's not clear whether the answers you've received are good or not (this is why it's good to clarify the question before, not after, you answer...). I know you've said price decreases over time but, is there more to it? E.g. are repeated measurements on the same country expected to be correlated? If so, neither of the existing answers will do what you need. Sep 25 '13 at 16:51
• Great points! @Macro I would expect some correlation between multiple data points within the same country (prices can be a little sticky). Also, I ran the durban-watson test and found that the residuals are autocorrelated. Looks like this one may be beyond me. Might be time to call in a true statistician.
– Tom
Sep 25 '13 at 18:08
• if you are in R, there are functions for dealing with dates, and I wouldn't be surprised if R was able to handle data like this quite elegantly without you having to specify much more than you regularly would. See this Sep 24 '14 at 11:54
• You definitely need to take the auto-correlation into account and incorporate. It seems a time-series analysis is in order. +1 for recognizing the need to bring in "a true statistician." There are plenty of reputable statisticians available through the American Statistical Association's website and many can be found at local universities. May 14 '16 at 3:38

It sounds to like you need use time series methods, such as ARMA or ARIMA, that let you calculate a regression using time as an independent variable without violating the independent observations assumption of OLS.

You may want to try a two step analysis: - first use time as a single predictor variable and use a suitable time series method - two see if there is any meaningful difference in residuals between the two suppliers. (A simple t-test might be sufficient.)

There's several ways. An option is to convert dates into days after the very first day. Also, you could have have additional variables of days of the week (trends across the week) and the month (to see trends in certain times of the year). By doing so, you can use multiple regression.

To get the variable "# of days after the first day", I believe (both excel and R) you can simply subtract the earlier data from the latter date and get the day difference. So maybe try subtracting 1/1/2010 from all your dates. You should also tell R that the new value is numeric using as.numeric()

EDIT: R seems to read in the year first, so you may have to mess around the dates a bit. see this: https://stackoverflow.com/questions/2254986/how-to-subtract-days-in-r

Time series analysis is another approach, but I'm not too familiar with it.

I can advise you to use non-linear function for time variable because the prices fall is lesser with each additional time unit. Otherwise the price would finally fall below zero. Moreover, there may be periods when the trend changed up. Thus I recommend to use cubic splines for time variable.

Experience whispers me that I would check the following model:
Y = country_parameter * price(t) * e

where price(t) is a function, preferably cubic spline, but it may also be whatever, even linear trend. Note that there are multiplication signs, not sums, in the model.

Pick a reference date, say 1/1/2010, and make a new variable time that is the difference between the date and the reference date, where the difference is computed in, say, days.

Now run a linear regression (or something similar) with time and supplier as the two predictor variables and price as the response variable.

This is just a starting point.

• Hi I did this, but does it violate one of the assumptions of regression? ie the points are independent observations? the residual plot looks okay.
– Tom
Sep 25 '13 at 16:51
• Good point. It is always good to be suspicious of linear regression output, as the assumptions are never met 100%. In particular, the standard error could be vastly understated if the residuals are correlated, as you suggest. To check for independence, on option is to use the Durban Watson Test, like this: tc.umn.edu/~ryoox001/images/DurbinWatson_test.pdf Sep 25 '13 at 17:13
• Great points! I would expect some correlation between multiple data points within the same country (prices can be a little sticky). Also, I ran the durban-watson test and found that the residuals are autocorrelated. Looks like this one may be beyond me. @zkurtz Might be time to call in a true statistician.
– Tom
Sep 25 '13 at 18:11
• It's a misconception that regression observations need to be iid. The errors only need to be uncorrelated and with constant variance. If anything, adding a fixed effect for time could help ensure that this condition is met. The only thing that can go wrong here is overfitting and/or a loss in precision due to having too many parameters for the data. It's probably worth pointing out that the equations for fitting stationary (V)ARMA models reduce to OLS. Aug 25 '14 at 0:36
• In order to handle the serial correlation, you'll probably want to use a time series analysis approach or at a minimum a GEE or mixed effects model capable of handling the correlated nature of your data. Oct 28 '15 at 2:05