5
$\begingroup$

I am estimating a time series model. For simplicity, let it be:

$y_t=\beta_0+\beta_1y_{t-1}+\epsilon_t$.

I estimate this model for a whole panel of $n$ sections, i.e. I end up with a tuple of coefficients $\beta_{1,1},...,\beta_{1,n}$.

I also have a slightly adjusted version of the same panel. The adjustment is not of statistical nature. I am dealing with accouning data and the adjusted version is simply some accounting adjustment that is being introduced. More specifically, the adjustments occur at specific points in time, i.e. not every quantity of a time series is adjusted. I again estimate a time series model:

$p_t=\gamma_0+\gamma_1p_{t-1}+\omega_t$,

where $p$ denotes the adjusted $y$ and $k$ denotes the adjusted $x$. I thus get a tuple of coefficients $\gamma_{1,1},...,\gamma_{1,n}$.

I now want to test wether the independent variable has a significantly larger effect on the dependent variable in the adjusted panel than in the unadjusted panel, i.e. I want to compare the persistence (coefficient of an AR1 model) of the two panels. How do I do this? That is, how do I determine which coefficient of the two models applied to different sets of data is of significantly higher value.

Specifically, two issues have to be considered:

(1) I dont have two values which I want to compare. I have two tuples of values. I was thinking about a decision threshhold. For example, if more than half of the $\beta_{1,1},...,\beta_{1,n}$ are higher than their adjusted counterpart $\gamma_{1,1},...,\gamma_{1,n}$, the effect of the independent variable on the dependent variable seems to be higher in the unadjusted dataset. Is this reasonable?

(2) I need to test wether the the proposed difference between the coefficients is significant. How do I test wether the regression coefficients from two models applied to different data are significantly different? Is the Chow-Test appropriate here?

How do I tackle these issues at once?

$\endgroup$
4
  • $\begingroup$ How are you "adjusting" the $y$s and $x$s? If you have a vector of values, you'll have to compare them element by element. In general, the approach to deriving statistical tests is to write down the distributions of the independent and dependent variables, manipulate them through your adjustment process, and see the resulting distributions at the comparison stage. Start with strong distributional assumptions -- normality, independence, etc., wherever applicable -- that's OK -- and then try to see what you can relax. $\endgroup$
    – Vimal
    Apr 16, 2020 at 13:49
  • 1
    $\begingroup$ I am looking at accounting information. The original versions are originally reported accounting information. The adjusted versions are retrospectively readjusted reported accounting information. $\endgroup$
    – shenflow
    Apr 17, 2020 at 8:43
  • $\begingroup$ What formula are you using to adjust your $y$? For instance, are you mean-centering them? This matters because if you are mean-centering them using means computed from the data, then it will affect the distribution of the adjusted $y$. $\endgroup$
    – Vimal
    Apr 17, 2020 at 12:43
  • $\begingroup$ Its just an accounting "thing". Accounting numbers might get restated for several reasons. Its not a statistical "thing". $\endgroup$
    – shenflow
    Apr 18, 2020 at 14:05

4 Answers 4

1
+100
$\begingroup$

Let us define $\delta_i = \beta_{1,i} - \gamma_{1,i}$, with i indexing the samples. You would ideally regress $\delta_i$ over 1: $\delta_i = b.1 +\eta_i $. If I make no mistake, your question can indeed be rephrased:: What is the value of $b$? Is $b$ significantly different from 0?

However, as noticed by @F.Tusel, there is some uncertainty regarding $\delta_i$; this will bias upwards the variance associated with $b$, and this could cause your result to be (wrongly) non-significant. If your result is significantly different from zero, stop here as what is below would only increase significance.

If not: Having uncertainty on the value of the dependent variable is however a classical problem. A clear explanation on how to deal with it in classical cases can be found in [1].

Is it simple in your case? It probably depends. Are your samples i.i.d. ?

  • If yes: the variance-covariance matrix of $\delta_i$ is simply the diagonal matrix with $\sigma_{\delta_i}$ on the diagonal. And $\sigma_{\delta_i}$ can itself be infered from the matrixes of variance-covariance of $\left(\array{\beta_{1,i} \\ \gamma_{1,i} }\right)$. The latter can be estimated if you jointly estimate: $$ \left(\array{y_{t,i} \\ p_{t,i}}\right) = \left(\array{\beta_{0,i} \\ \gamma_{0,i} }\right) + \left(\array{y_{t-1,i} \ \ 0 \\ 0 \ \ p_{t-1,i}}\right)\left(\array{\beta_{1,i} \\ \gamma_{1,i} }\right) + \left(\array{\epsilon_{t,i} \\ \omega_{t,i} }\right) $$.

  • If not, more thinking is required. Please add more information in the Question regarding your samples.


Reference: [1] Lewis, Jeffrey B., and Drew A. Linzer. "Estimating regression models in which the dependent variable is based on estimates." Political analysis 13.4 (2005): 345-364. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.318.7018&rep=rep1&type=pdf

$\endgroup$
1
  • $\begingroup$ I make a second try! (again, this comes without guarantee) $\endgroup$ Apr 23, 2020 at 20:53
3
$\begingroup$

I fully concur with the last paragraph of @AlexC-L's answer which is in essence a paired comparisons method. I have a feeling, though, that you do not want to look at the raw differences $\delta_i = \beta_i - \gamma_i$. The $\beta_i$ and $\gamma_i$ are presumably estimated by regression and are affected by uncertainty: does $\hat\beta_i=0.8$ with standard deviation of 0.6 imply higher persistence that $\hat\gamma_i=0.5$ with standard deviation 0.1? I would think the second more indicative of persistence than the first, which is not even significantly different from zero.

The choice depends on your problem, but I think you might at least consider to take not the raw estimated coefficients, but rather the coefficients measured in standard deviations when you compute the differences.

$\endgroup$
6
  • $\begingroup$ In fact, I am interested in the raw difference. However, you are touching an issue I am currently also thinking about. What if the differences are insignificant? This would deem any subsequent analysis, such as a KS test, irrelevant. stats.stackexchange.com/questions/93540/… describes an approach of how the significance could be tested. However, I am unsure of how to integrate results of such an analysis into the method suggested by @AlexC-L. $\endgroup$
    – shenflow
    Apr 23, 2020 at 9:26
  • $\begingroup$ The problem with @rusellpierce proposal as applied to your problem is that $\hat\beta_i$, $\hat\gamma_i$ seem likely correlated --for $p_t$ and $y_t$ as I understand refer to the same firms and can be expected to be correlated. So dividing the differences by $sd( \hat\beta_i) + sd(\hat\gamma_i)$ neglects the term in covariance among them. $\endgroup$
    – F. Tusell
    Apr 23, 2020 at 11:44
  • $\begingroup$ I am aware of that. I would need to estimate a model of the form: $\left(\array{y_t \\ p_t}\right) = \left(\array{y_{t-1} \ \ 0 \\ 0 \ \ p_{t-1}}\right)\left(\array{\beta_1 \\ \gamma_1 }\right) + \left(\array{\epsilon \\ \omega }\right) $ for each $i$ to derive the covariance matrix for each $i$, right? This would allow me to test for significance while not having to impose the restrictive limitation you were mentioning. Yet, even if I am able to do that, I do not know how to integrate those results into the analysis suggested by @AlexC.-L. $\endgroup$
    – shenflow
    Apr 23, 2020 at 11:52
  • $\begingroup$ Once you are satisfied with the standardization, I guess all that is left is to test mean zero of a set of observations with a predefined alternative. $\endgroup$
    – F. Tusell
    Apr 23, 2020 at 12:04
  • $\begingroup$ I am sorry, but I dont understand your last comment @F. Tusell. Could you maybe explain it to me? $\endgroup$
    – shenflow
    Apr 23, 2020 at 12:41
2
$\begingroup$

I assume when saying "test wether the independent variable has a significantly larger effect on the dependent variable in the adjusted panel than in the unadjusted panel", you are actually trying to find out which model can better describe the uncertainties and relations among the observations.

So instead of comparing the difference of the coefficients, a better approach is to perform model selection on your models. Since model selection has to be done on the same set of samples, you need to some how tweak your models to make them applying to the same sample set:

Model 1: $$ \begin{align}y & = \beta_0 + \beta_1x_1 + ... + \beta_n x_n + \epsilon \\\Rightarrow y &\sim F(y|x_{1:n},\theta_1) \\\end{align} $$ Where $\theta_1 = \{\beta_{0:n} \text{ and all the other parameters}\}$, you can understand $F(y|x_{1:n},\theta_1)$ as a distribution of $y$ conditioned on $(x_{1:n},\theta_1)$. For example when the model is a simple linear regression $y = \beta_0 + \beta_1x_1 + ... + \beta_n x_n + \epsilon,\epsilon \sim N(0,\sigma^2)$, then $F(y|x_{1:n},\theta_1)$ will be a normal distribution with mean $\beta_0+\beta_1x_1+...+\beta_nx_n$ and variance $\sigma^2$, i.e. $F(y|x_{1:n},\theta_1) = N(y|\beta_0+\beta_1x_1+...+\beta_nx_n, \sigma^2)$

Model 2:

No matter how you "adjust" your samples, there must be a way to represent the adjustment with a function, say $p = h(y)$, $k=g(x)$. For example if the adjustment is discounting future payment $y$ into current value $p$, then the function $h(y)$ will be something like $h(y) = \frac{y}{(1+r)^m}$. With this idea in mind, your second model $p = \gamma_0 + \gamma_1k_1+,...,+\gamma_nk_n+\epsilon_2$ can be rewritten as: $$ \begin{align}y & = h^{-1}(\gamma_0 + \gamma_1g(x_1)+,...,+\gamma_ng(x_n)+\epsilon_2) \\\Rightarrow y &\sim G(y|x_{1:n},\theta_2) \\\end{align} $$ Where $\theta_2 = \{\gamma_{0:n} \text{ and all the other parameters involved in h() and g()}\}$.

Now that both the models are put to the same set of samples, you can start comparing them. There are two common ways to perform the comparison:

  • method1: If $F()$ and $G()$ are Bayesian models, you can compare their marginal likelihood (the higher the better) or BIC (the lower the better).
  • method2: Use cross validation and compare their cross validated expected-prediction-error.
$\endgroup$
3
  • 1
    $\begingroup$ Thats a clever method. However, the question I am trying to answer can be more precisely defined as follows: Is the persistence of some metric (e.g. cash flow), i.e. the coefficient in an AR1 model, higher in the adjusted dataset or the unadjusted dataset. The way you rephrased it does not take into account the actual value of the coefficient that I am interested in, but rather the goodness of fit. Is this correct? Because then, it would not provide an answer to the question of which coefficient is significantly higher. $\endgroup$
    – shenflow
    Apr 21, 2020 at 15:22
  • 1
    $\begingroup$ Moreover, the adjustment is not of the nature you were suggesting. I am talking about accounting adjustments such as a firm might readjust its previous financial statements to ensure comparability across periods, e.g. in the case of M&A activity. So the readjustments occur at specific points in time and the time series is not readjusted as a whole. Thus, it is practically not possible (at least in my opinion) to formulate the readjustment process as a single function. I edited the question so it is more clear. $\endgroup$
    – shenflow
    Apr 21, 2020 at 15:37
  • $\begingroup$ Am I correct? Or am I wrong with what I was saying in regards to your approach against the background of what I am interested in? $\endgroup$
    – shenflow
    Apr 23, 2020 at 8:33
1
$\begingroup$

Note: This answer does not take into consideration that $\beta_i$ and $\gamma_i$ are themselves estimated (thank @Turell for pointing that out). I make another try in another answer.


You have n $\beta_i$ and n $\gamma_i$ that you want to compare. If n is large enough, you might turn this problem into the comparison between two distributions.

You may use the Kolmogorov–Smirnov test to determine if those two distributions are significantly different from each other. If this is the case, graphical inspection may then be used to determine if beta are generally higher than the gamma. To go beyond this simple graphical inspection, you may look at the differences between the quantiles of two distributions following Doksum, and Wilcox.

One issue I notice yet: above, this would not compare $\beta_{i0}$ and $\gamma_{i0}$ one to one. This can be fixed by defining: $\delta = \beta - \gamma$, and by comparing $\delta$ with a distribution of zeros.

$\endgroup$
8
  • $\begingroup$ NB: a quick suggestion from a non-expert $\endgroup$ Apr 19, 2020 at 20:35
  • $\begingroup$ Good idea! I was thinking about that too. It would actually solve both issues, since I could thereby compare two vectors of values (by looking at their distribution) while doing testing for significant differences. However, is there a statistical method directly determining wether one distribution is "shifted" relative to the other, i.e. skipping the KS Test + graphical inspection and doing both in one step? $\endgroup$
    – shenflow
    Apr 19, 2020 at 21:58
  • $\begingroup$ If there is a significant difference between the two distributions, you may then look at the quantiles. Are those of one distribution always lower than for the other distribution ? I would imagine this would fit your purpose. $\endgroup$ Apr 20, 2020 at 6:27
  • $\begingroup$ I otherweise added a blog reference that illustrates the last sentence of my answer: garstats.wordpress.com/2016/07/12/shift-function $\endgroup$ Apr 20, 2020 at 6:32
  • $\begingroup$ Thanks, I will reward the bounty to your answer throughout the next days, if no other answers are being posted. $\endgroup$
    – shenflow
    Apr 20, 2020 at 7:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.