I have a very high dimensional (p=946, n =123) dataset. For the one's i have checked, the covariates do not really have a relationship with the response so i have ruled out linear regression. Then, i tried to find the distribution of Y as plotted below. I found that Weibull distributuon fits this dataset best followed by Gamma second based off AIC/BIC.

I thought I would go for Gamma as this might be easy to implement using glmnet in R but this package doesn't support the Gamma distribution. So i'm wondering, for high dimensional datasets, what regression method could i use in which an R package may support it? I'm not sure what model i could use with a Weibull distribution. In particular i'd like to use a package where I can. use screening methods for variable selection.

The data can be found by the following commands


enter image description here


1 Answer 1


You say "Then, i tried to find the distribution of Y as plotted below. I found that Weibull distributuon fits this dataset best followed by Gamma second based off AIC/BIC."

I say nonsense "The distribution of Y is not a concern. All of the assumptions in regression are about the distribution of the error terms which is in effect a conditional distribution of Y."

Paraphrasing "There is nonsense and there is nonsense but the most nonsensical nonsense of them all is statistical nonsense"


Time series analysis allowing for the effect of auto-correlated data i.e. non-independent samples requires certain procedures to be used in identifying a useful model . Here is a discussion that I authored some 20-25 years ago https://autobox.com/pdfs/regvsbox-old.pdf . Your data is cross-sectional thus "shuffling the cards" is not necessary as the 123 observations are free of auto-correlation because they were independently collected.

Cross-sectional regression (your problem) is a particular case of a SARMAX problem https://autobox.com/pdfs/SARMAX.pdf without the ARMA component or differencing or lagged structure in the X's.

Cross-sectional data and time series data have a number of commonalities viz..

1) the need to identify pulses i.e. an instantaneous adjustment to the expected value.

2) the need to consider when you have more input series(140) than obervations(123) and how to obtain the optimal/parsimonious/sufficient set of predictors

3) what kind of power transformations are necessary to form a useful model

With these common objectives in place , I introduced your data to AUTOBOX ( a generalized regression package that I have helped to develop ) specifying that order in the observations was not of interest or concern by declaring the "seasonality to be 1" and supressed lag structure in the predictors AND lag structure in the memory of Y.

Here is a plot of the original data enter image description here suggesting an outlier/pulse at period 10 which might be due to one or more of the 140 causal series . Nonetheless it certainly would thwart any attempt to specify a stabilizing xform . Not treating that one observation can easily lead to a spurious conclusion about the need for some sort of transformation.

AUTOBOX has a feature to do a sub-set elect and concludes that 12 of the 140 are both sufficient and statistically significant predictors while treating the errant 10th observation.

Following is a plot of the residuals from that model suggesting model sufficiency.

enter image description here

The optimal sub-set model is here enter image description here AND here in two parts enter image description here enter image description here

The model statistics are here enter image description here

The Actual/Fit graph is here

enter image description here and the cleansed graph is here

enter image description here

The histogram of the residual series is here enter image description here and in contrast the histogram of the original series is here showing the effect of the anomaly at period 10 enter image description here and the omission of the selected 12 input series.

There is no evidence of any needed power transformation

enter image description here

In closing "All models are wrong but some are useful" from G.E.P Box . This example should be useful to all those studying how to form a possible model.

  • $\begingroup$ So i need to run each possible regression model first and then diagnose the residuals to see if it fits? For GLM & other models what do i need to look for? $\endgroup$
    – Btzzzz
    Commented Apr 18, 2020 at 20:22
  • $\begingroup$ after developing a model and treating anomalies (albeit insufficient model!) I suggest that you look at stats.stackexchange.com/questions/18844/…. I tried to download your data but couldn't read it perhaps you need to save it as a csv file or a text file. $\endgroup$
    – IrishStat
    Commented Apr 18, 2020 at 22:34
  • $\begingroup$ I have tried some transformations but this doesn't make my data normal. I thought for a GLM that each outcome Y of the dependent variables is assumed to be generated from a particular distribution in an exponential family, therefore i need to check the distribution in this case rather than residuals? $\endgroup$
    – Btzzzz
    Commented Apr 19, 2020 at 5:06
  • $\begingroup$ i've also edited the question so the data should be (easily) downloadable now $\endgroup$
    – Btzzzz
    Commented Apr 19, 2020 at 5:21
  • $\begingroup$ In this case AFTER you have identified the BEST SUBSET of time series less than the # of observations AND any anomalies pulses IN THE ERROR PROCESS ( 1 in this example) and incorporated an appropriate dummy indicator , the solution is MLR with STEPDOWN . AUTOBOX is available in R . $\endgroup$
    – IrishStat
    Commented Apr 20, 2020 at 8:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.