2
$\begingroup$

I currently have a data.frame with 273 variables and 94 rows:

      Month count_n hhgroup0 hhgroup1 hhgroup2 hhgroup3 hhgroup4 hhgroup5 ... variable273
1  FEB 2008     477        0       75       67       86       35       35
2  MAR 2008    1760        6      327      266      259      129      106
3  APR 2008    2598       11      525      358      401      177      161
4  MAY 2008    3182       12      626      437      496      206      203
5  JUN 2008    3528       14      690      476      552      239      227
...
94 NOV 2015

The dependent variable is count_n. It is set up in time-series fashion. I wanted to reduce the dimensions to improve model performance. There many portions like the example above. They are not all a part of hhgroupxx. It is just one of the variables HouseholdIncomeGrp that has been split sparsely. There are also variables with maritalstatus, age, gender, etc...

The variables are HIGHLY correlated with each other. And this phenomenon repeats with other sets of variables. I want to reduce multi-collinearity by eliminating unnecessary features.

cor(d[-(1:2)])
          hhgroup0  hhgroup1  hhgroup2  hhgroup3  hhgroup4  hhgroup5
hhgroup0 1.0000000 0.9965717 0.9915431 0.9910599 0.9940124 0.9886601
hhgroup1 0.9965717 1.0000000 0.9959187 0.9981747 0.9955502 0.9969752
hhgroup2 0.9915431 0.9959187 1.0000000 0.9931393 0.9978899 0.9927226
hhgroup3 0.9910599 0.9981747 0.9931393 1.0000000 0.9934632 0.9997952
hhgroup4 0.9940124 0.9955502 0.9978899 0.9934632 1.0000000 0.9930959
hhgroup5 0.9886601 0.9969752 0.9927226 0.9997952 0.9930959 1.0000000

Should I randomly select one of these groups since they all move similarly? Or should I not break up the set of variables since they all come from the same category? Is the reason I am seeing such high correlations the fact that this company is growing over time? It seems obvious that as the company gets bigger, all of the variables will grow. My hope is to predict the count_n in a future period based on a selection (or all) of the other variables.

$\endgroup$
3
  • $\begingroup$ Did you mean to say 273 rows? Also, is count_n a sum of which hhgroupx is a part? $\endgroup$
    – HStamper
    Feb 11 '16 at 15:06
  • $\begingroup$ There are 273 columns. This is a subset for example purposes. count_n is not a sum of any other columns. It is an independent measure of "Total number of active agents in the field". The client I am working with is attempting to predict the agent count based on the hundreds of other variables. $\endgroup$
    – Pierre L
    Feb 11 '16 at 15:10
  • $\begingroup$ I think I have an idea after thinking about it yesterday. Should I try to take out the effect of time with detrending or seasonality to use in an OLS model? That way the values are not growing from month to month. I could possibly use the change from month to month instead of the overall values. $\endgroup$
    – Pierre L
    Feb 12 '16 at 14:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.