P-Values based on Experiment Data Aggregation I'm trying to understand which interpretation of p-values for a linear regression model is correct given different levels of aggregation.
Let's say I am running a pricing experiment in a store-chain that sells widget A. I have 4 locations and I want to change the price of A in 2 of them with the objective of measuring the impact of the test on revenue. The experiment lasts 5 weeks. Let's assume there's some noise with revenue, but the true effect of the price test actually raised revenue by $5 per week.
I can either aggregate the data to the level of the test (5 week period) or at the weekly grain and run a regression to determine the effect size. I ran a simulation of this scenario while fitting a linear model to both levels of aggregations, and the coefficients are the same for both levels of aggregations, but the p-values differ.

*

*How do I interpret the p-values for each level of aggregations?

*How do I know which level of aggregation makes more sense?

See below for the simulation I ran:
set.seed(100)
# simulate locations at the 1 week level
# a numeric location is used to generate revenue numbers unique to the location
(location = c(rep(0,5),rep(1,5),rep(2,5),rep(3,5)))
(y = 100 + 5*location + rnorm(20,0,2)) 

# add treatment effect at 5 week level
(t.c = c(rep(1,10),rep(0,10)))
(y.64 = y + 5*t.c)

mat=cbind.data.frame(y.64,location,t.c)

#  analyze at the aggregated level
annual.mat=aggregate(mat$y.64,by=list(mat$location,mat$t.c),FUN=mean)
names(annual.mat)=c("location","t.c","y.annual")

fit1=lm(y.annual~location+t.c, data=annual.mat)
summary(fit1)

#  analyze at the week level
fit2=lm(y.64~location + t.c,data=mat)
summary(fit2)

 A: Yes, the p-value is changing due to the degrees of freedom is being reduced by the aggregation because there is less information available to make the prediction.
In the first case you are reducing the number of data points for each store from 5 to 1, after the aggregation. Now you only have four pieces of information to predict the three parameters of the regression.  The process is throwing out all the information associated with the variance of each location.
In the second case you have 20 pieces of information to predict the value of 3 parameters. Thus providing higher confidence that the fit is not due to chance and resulting in a lower p-value.
Generally, one should analyze the entire dataset and take full advantage of ALL the available information.
With that being said.  Your model is incorrect, location (and possibly t.c.) should not be a numeric variable but a factor, because location is not a continuous variable.  Location 3.5 is just nonsense.
A t.test or ANOVA is probably a better statistical test for the data presented in the example.
