1
$\begingroup$

I have 3000 independent time series samples (customers) where I fit a dynamic regression model with ARIMA errors to each sample and estimate regression coefficient of interest (intervention), $B_1{_i}$ from the following model
$Y{_i} = B_0{_i} + X_1{_i}B_1{_i} + .... + e{_i}$

where $Y_i$ is sales per customer $i$.

I used ARIMA to take into account any seasonality and trend and am okay with ARIMA terms capturing other unexplained variance.

I end up with 2 vectors of size 3000; one for the $B_1{_i}$ and another for their standard errors, $SE_1{_i}$. Some coefficients are significant and others are not.

An overall estimate of $B_1$ is needed so I use a weighted average (a weight has been derived based on prior year sales proportion out of the total) to calculate the overall estimate and use wtd.t.test from the weights package in R to test for the significance of the overall estimate.

My questions are

  • Is it valid to test the significance of the overall estimate using a weighted one-sample t-test?
  • Or do I need to combine the standard error estimates, $SE_1{_i}$ from all the models and calculate an overall standard error?
  • And how would i calculate the overall standard error integrating the weights?
Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

If I understand correctly, you are interested in estimating some overall population effect. Instead of first doing a regression and then trying to do subsequent weighted tests, there is a more efficient way to use your data to achieve your objective.

A (a.k.a. hierarchical or mixed-effects model) allows you to estimate your overall population effect while accounting for deviations from the population effect at an individual level using random effects. Random effects (intercepts and slopes) characterise the variation around the population effect. The mixed modelling approach benefits from partial pooling, in which each customer's regression is informed/constrained by the data from other customers; this is described as the individual-level random effects being 'shrunk' towards zero. This might seem unintuitive, but it has good theoretical and empirical support.

Improve this answer
$\endgroup$
7
  • $\begingroup$ I understand that mixed-models are one way to achieve my objective, but my questions are regarding the an alternative method of modeling each customer separately instead of building one model. $\endgroup$
    – Moe Sam
    Commented Sep 16, 2019 at 13:53
  • $\begingroup$ @MoeSam Why do you want to use a t-test instead? $\endgroup$
    – mkt
    Commented Sep 16, 2019 at 14:45
  • $\begingroup$ that is my question, is there anything invalid about using t-test? $\endgroup$
    – Moe Sam
    Commented Sep 16, 2019 at 14:50
  • $\begingroup$ @MoeSam As I've said here, it is an inefficient use of data and provides results that are likely to be less accurate than mixed models. Isn't that good enough reason to avoid a t-test? If not, you should define what criterion you are using to decide on a method. $\endgroup$
    – mkt
    Commented Sep 16, 2019 at 14:53
  • $\begingroup$ Since each sample is a time series, I am using dynamic regression with ARIMA errors to take into account any seasonality and trend. I am only interested in the effect of the intervention, $B_1$, and would be okay with ARIMA terms capturing other unexplained variance. If I go with a mixed-model, it would further complicate the analysis having to deal with omitted variable bias. $\endgroup$
    – Moe Sam
    Commented Sep 16, 2019 at 15:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.