Statistical test with violation of independence assumption

Question

My objective is Checking of the significance of differences of two linear regression methods on one database based on the robustness measure I used (which I called robustness as only a name).

The robustness measure used as follows.

I am working with two different linear regression models on one dataset.

First, I standardised all the variables (independent/dependent) to zero mean and unit variance.

With the linear regression model 1, I performed 30-fold split for the dataset. So, I have model coefficients for each fold. I calculated the variance for each variable within 30 models. Finally, I sum all the variances in one total value.

For example, I have 30 coefficients for a variable (X1), then I calculate the variance for 30 coefficients and the same for all the remaining variables and Finally, I sum all the variances in one total value.

I did this process with two models and one dataset. So I end up with a small matrix contains the Sum of variances values (its rows refer to the dataset used and its columns for linear models used).

I want a statistical test to evaluate the significance of differences.

UPDATE 1

I will recall the definition of the robustness indicator I used again.

The first requirement: all features (input and output) are standardised (have zero mean and unit variance).

I have $d$ regression coefficients $C_1,…C_d$ (there is no intercept because data are standardised).

For each coefficient, we have k estimations $C_i^1,…,C_i^k$ by one linear method (I consider $k=30$, because of the usage of 30-fold CV).

I calculate variance for each coefficient $σ_i=var(C_i )$. So, I define an indicator of robustness as $R= \sum_{i=1}^{d} σ_i$

UPDATE 2

I created below table to have an idea of the problem.

To make it more clear, linear regression Model 1 could be Ordinary least square regression and linear regression Model 2 is, for example, Weighted linear regression. I can also use Huber, Quantile, Least absolute deviation regression and so on.

I want to check the significance of differences between the two methods on one database. Robustness Indicators will be different but are them significantly different?

Answer 1

I have many questions about the proposed approach, mostly because it is not clearly described. It's unlikely this question can be edited to give us much insight into the setting and rationale for this approach. Nonetheless, a few points can be addressed.

Sums of variances obtained by randomly partitioning a dataset 30-fold cannot be called "robustness". A random partition shouldn't be referred to as a "dataset".

Calculating the variance of partitions of an independent sample are not interesting because

$$ \text{var}(\sum_{i=1}^n X_i) = \sum_{i=1}^n \text{var}( X_i)$$

when $X$ is mutually independent.

However, if there is a lack of independence, the expression generalized as $$ \text{var}(\sum_{i=1}^n X_i) = \sum_{i=1}^n \text{var}( X_i) + \sum_{i \ne j}\text{cov}(X_i, X_j)$$

By partitioning the dataset, the working covariance matrix forces off-diagonal blocks to be 0 valued, which may produce interesting if imprecise differences from the total sample variance. Of course, comparing to other random partitions will give the same-ish answer each time, with little other than random variation contributing to differences, and no valid inference about covariance structure.

Even if you made this total-sample vs. partitioned sample comparison, you would have a very imprecise test. You are ignoring a whole family of regression models for correlated data that provide estimates of covariance directly in the form of autoregression or exchangeable correlation structures. Variogram models are also of interest. So as far as tests to detect autodependence, while this approach is close to something that can be used, it is not worth reinventing the wheel when numerous approaches with proven statistical properties are already out there.

It is a pitfall of "robust" statistics to presume you must test to determine the presence (or lack) of a particular assumption. The point of robust statistics is to identify an omnibus which reduces the complicated nature of testing assumptions (and risking false positives or false negatives) and recommending final analyses based on interim/penultimate results.

In particular, if the desire is for a test of regression parameter differences that is robust to undetected dependence within sample observations, the generalized estimating equation (GEE) provides consistent and unbiased inference with sandwich variance estimation.

Answer 2

If I understand you correctly, you are asking about testing the significance of the difference between two regressions on the same data set. This is not a standard significance testing problem. A standard significance testing problem states a null hypothesis ($H_0$) about how the data were generated, and uses the distribution assuming this $H_0$ of a certain test statistic measuring in some sense the difference between the data and the $H_0$ to see whether the data are "too far away" from the $H_0$, in which case one would say that there is evidence that the data were in fact not generated by the $H_0$.

However, it is not clear what your $H_0$ is supposed to be. You seemingly have in mind that "the two regressions are equal", but as they are computed in different ways, their results will (almost) always be different on the same data, so in fact we know that they are not the same, we don't need to test this. "The regressions are the same" is not a proper $H_0$ about the generation of the data.

The only sensible $H_0$ I can imagine here is the model underlying one of the regressions (maybe the Gaussian model underlying the least squares regression), and then one can imagine using the difference between the two regressions for testing this, with the idea in mind that if the model holds they should be rather similar, whereas if the model doesn't hold, they can be much further from each other. This depends on how exactly the model is violated though. A significant result would then not mean that there is evidence that the "two regressions are in fact different", but rather that there is evidence against the Gaussian regression model. Not sure whether this is what you actually want to know, but it may well be (though there's the sensible answer of AdamO who doubts that this is a good question to ask when using robust methods).

Even then this is a nonstandard test and I'm not aware of any standard software that does this. There may be specialist literature flying around exploring such a test (I haven't put time into searching), but surely it will depend on what precisely your regressions are, there's no general answer without knowing that. Chances are something can be done using bootstrap (maybe parametric bootstrap or Monte Carlo), but I'd not expect that anybody here could tell you straight away how to solve this (and as said before, I'm not even sure whether this is what you really want); it would be something of a project.