1
$\begingroup$

I have transactions which are produced on a daily basis. Each transaction has a price. I'd like to see, if the weekday has an influence on the average daily price, whether it is statistically significant.

I have tried an anova analysis:

stats.f_oneway(df['price'][df['weekday'] == 0], 
             df['price'][df['weekday'] == 1],
             df['price'][df['weekday'] == 2],
              df['price'][df['weekday'] == 3],
              df['price'][df['weekday'] == 4])
executed in 43ms, finished 21:33:16 2020-04-24
F_onewayResult(statistic=1.7483567589994404, pvalue=0.16221246736524217)

It seems that the H0 of means being equal cannot be rejected. Is this correct? Can we therefore conclude that the weekday has no effect? Is there a way to get an f-score for every categorie, so I can see which day has the most impact?

$\endgroup$
5
  • $\begingroup$ You said " I was thinking of simply doing a t-test of each day against each day and see if it's statistically significantly different for each permutation." This is what an ANOVA test is. $\endgroup$
    – Ron Jensen
    Apr 24 '20 at 20:22
  • $\begingroup$ is there an automated way to do that in python? $\endgroup$
    – Nickpick
    Apr 24 '20 at 20:26
  • $\begingroup$ I've sure there is a way to do it in Python, but I would have to Google for how, which is why I put it in a comment and not a full-fledged answer. $\endgroup$
    – Ron Jensen
    Apr 24 '20 at 20:32
  • $\begingroup$ updated the question with anova analysis $\endgroup$
    – Nickpick
    Apr 24 '20 at 20:35
  • $\begingroup$ you are testing all of the coefficients to be zero in the above test.. you can use statsmodels in python for anova. something like this stats.stackexchange.com/questions/462478/… $\endgroup$
    – StupidWolf
    Apr 24 '20 at 21:09
2
$\begingroup$

You can try something like belong, first simulate data where weekday == 2 has an effect:

import pandas as pd
import statsmodels.api as sm
from statsmodels.formula.api import ols
import numpy as np

df = pd.DataFrame({'price':np.random.normal(10,2,100),
                  'weekday':np.random.randint(0,7,100)})
df.loc[df['weekday']==2,'price'] = df['price'][df['weekday']==2]+np.random.normal(3,1,sum(df['weekday']==2))

Then test it with a linear model:

lmfit = ols('price ~ C(weekday)',data=df).fit()
lmfit.summary()

    coef    std err t   P>|t|   [0.025  0.975]
Intercept   9.9709  0.510   19.539  0.000   8.958   10.984
C(weekday)[T.1] 0.0055  0.687   0.008   0.994   -1.359  1.370
C(weekday)[T.2] 3.4443  0.709   4.860   0.000   2.037   4.852
C(weekday)[T.3] -0.6847 0.662   -1.034  0.304   -2.000  0.630
C(weekday)[T.4] 0.5838  0.798   0.732   0.466   -1.001  2.168
C(weekday)[T.5] 0.1265  0.687   0.184   0.854   -1.238  1.491
C(weekday)[T.6] -0.1915 0.722   -0.265  0.791   -1.625  1.242

Here day 0 is set as a reference, and you can see most other weekdays don't much differ in their means from day 0, except for weekday 2, and this has a low p.value.

you can change the regression to using other days as reference using:

from patsy.contrasts import Treatment
lmfit = ols('price ~ C(weekday,Treatment(reference=1))',data=df).fit()
lmfit.summary()

And also visualize it:

import seaborn as sns
sns.boxplot(data=df,x='weekday',y='price')

enter image description here

$\endgroup$
7
  • $\begingroup$ very useful! In my case there are at least 2 days that have a p-value of 0.00, no matter what reference I take. Is this enough evidence that the weekday matters? $\endgroup$
    – Nickpick
    Apr 24 '20 at 23:10
  • $\begingroup$ yes, there are at least 2 weekdays that are different. you can easily visualize it, see my edit $\endgroup$
    – StupidWolf
    Apr 24 '20 at 23:11
  • $\begingroup$ do I need to iterate over each day? Is there a way to do it all in one go? What about the one sided f-test? $\endgroup$
    – Nickpick
    Apr 24 '20 at 23:50
  • $\begingroup$ Ok.. you don't need it.. it is pretty obvious from the result if 2 days are different. For example if you set reference at 0, day 2 and day 5 have p < 0.05, the rest have coefficients close to zero, means, 0,1,3,4,6 are very similar $\endgroup$
    – StupidWolf
    Apr 24 '20 at 23:54
  • 1
    $\begingroup$ Great help. Thanks $\endgroup$
    – Nickpick
    Apr 25 '20 at 0:11
0
$\begingroup$

you don't want to simply introduce 1 series into the regression model with values [1,2,3,4,5,6,7,1,2,3.....] which assumes linearity between days ' BUT rather introduce 6 dummy 0/1 indicators providing 6 estimable contrasts.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.