# Regression with independent variables as level (Low=1, Medium=2, High=3) [duplicate]

I have the following data :

Yield   Rain    PH  EC  OC  N   P   K   S   Zn  Fe  Cu  Mn  Hydro
7778.1  103.03  3   1   2   1   1   2   1   1   2   3   2   1
5929.4  91.23   3   1   2   1   1   2   3   2   2   3   3   1
2872.3  109.47  3   1   2   1   1   2   2   1   2   3   3   2
4332.7  92.58   3   1   2   1   1   2   3   1   2   3   3   2
5236    100.37  3   1   2   1   1   2   1   1   2   3   3   2
3086.5  103.68  3   1   2   1   1   1   1   1   2   3   3   2
4526.1  83.38   3   1   1   1   1   2   2   2   2   3   3   1


Where Yield is my dependent variable. and Rain , PH, EC ... are independent.

I have to use regression analysis for predicting the Yield.

But, I notice that accept Rain variables all others are ordinal levels. (Like High=3, Medium=2, Low=1) where PH, EC , N, P, Fe, Cu are same values for each observation

Which regression model is the best to predict the yield? I am using python with scikit-learn.

## marked as duplicate by ttnphns, mdewey, Peter Flom♦ regression StackExchange.ready(function() { if (StackExchange.options.isMobile) return; $('.dupe-hammer-message-hover:not(.hover-bound)').each(function() { var$hover = $(this).addClass('hover-bound'),$msg = $hover.siblings('.dupe-hammer-message');$hover.hover( function() { $hover.showInfoMessage('', { messageElement:$msg.clone().show(), transient: false, position: { my: 'bottom left', at: 'top center', offsetTop: -7 }, dismissable: false, relativeToBody: true }); }, function() { StackExchange.helpers.removeMessages(); } ); }); }); Mar 8 '17 at 11:22

• Is it correct to assume that that is a subsample of your data? If not, you have a problem of having more predictors than observations and shouldn't model it this way. If this is a subsample, you can run multiple ones and observe any number of performance measures. Consider testing the resulting models, checking the predictors for what makes sense (coefficients are in the right "direction", their coefficients are significant, the standard errors are within reason, etc.). You're looking to make sure that the regression assumptions are met. – Alex Firsov Mar 8 '17 at 6:55
• @Alex Firsov, this is complete data , not a subsample:( , can you tell me in this case with model we use ? – e4e5 Mar 8 '17 at 7:09
• It doesn't matter. Your question is about the problem of ordinal independent variables. The dependent - if it is interval - you use linear regression (say); if it is binary, nominal or ordinal - you use proper logistic regression model. In all these regressions the problem of ordinal independent variables is alike. – ttnphns Mar 8 '17 at 8:15
• You mention, this is complete data , not a subsample. You have only 7 cases at 13 predictors? If yes you have problem of singularity (multicollinearity). It can be handled specially, but that would be not the best way. The best way is to collect considerably more cases than there are variables. – ttnphns Mar 8 '17 at 8:19
• No. You didn't understand me. 1) Collect much more cases (n). Or reduce the number of independent variables (p); n>p is your way, for me. (Or use use special forms of regulazized regressions, but it is more tricky.) 2) Model: use multiple linear regression, not logistic. 3) If insist to regard your ordinal predictors as ordinal, use suggestions I've given in the linked answer above. – ttnphns Mar 8 '17 at 8:30

Having only 7 cases, there isn't likely much you can do. Parametric models will overfit the data, as there are more predictors than cases. As such, your best bet may be exploratory analysis--try plotting variables against each other and seeing how they behave, and "feeling things out" with whatever domain expertise you may have. Also consider the use of non-parametric procedures, since you don't have a representative sample to tell you about the distribution of the population.

If you are dead-set on regression, you can try to use techniques like stepwise variable selection or MCA to first reduce the number of variables, and then run some regression models. You will most likely find this unstable/ineffective. You can also try to penalize the regressors by using ridge or lasso regression, but these are also very unlikely to be useful here.

In short, stick to exploratory analysis, or collect more data if possible.

• I have use all independent variables, and check which p value are bigger, delete that variables in the model, again run the new model, remove bigger p value variables, and finally getting model where p values are less than 0.05 , so I have use coefficients values for final model for predict yield , is that correct way? – e4e5 Mar 8 '17 at 7:43
• Firsov ,Ridge or lasso not useful here ? – e4e5 Mar 8 '17 at 7:48
• The short answer is no, they probably will not. There's a good discussion of the effectiveness of this here, though the sample size discussed there is considerably larger. – Alex Firsov Mar 8 '17 at 7:56
• if i collect more data like 40 -45 observation , than which model I can use ? – e4e5 Mar 8 '17 at 8:05
• If you're able to collect that, you could try running lasso or MCA with a plain old multiple regression, but be careful with the results. Some general rules for minimum multiple regression sample size are outlined here. If MCA gets you down to few enough predictors, you should be closer to something reasonable. Be sure to check for normality though, and examine the output carefully. Also be careful coding you variables. See ttnphns's comment on your question for more on that. – Alex Firsov Mar 8 '17 at 8:23

If you have enough data you can use an ordered logit model or an ordered probit model. The difference between the two is the IIA assumption.

Here is a good description of the IIA to assumption and the difference between multinomial logit and multinomial probit. The difference between ordered logit and ordered probit can be described analogously.

So, why should we apply multinomial probit rather than multinomial logit? What is the advantage of relaxing IIA? To answer the question, we should understand IIA axiom. Wikipedia entry on IIA provides a nice summary. To illustrate the issue, blue bus/red bus problem is given as an example (based on McFadden, 1973). So, suppose that we need to choose between two forms of transportation, car and red bus, and suppose that we choose these two options with equal probability, 0.5. If we introduce a blue bus as an additional alternative, under the assumption of IIA, we should have a new probability, 0.33, for each option. However, this is not very intuitive as two of our options (red bus and blue bus) are quite similar. Another, and maybe more realistic, example could be a choice between four alternative modes of travel: plane, train, car, and bus. Now, under IIA we consider these alternatives independent or distinct, but three of these options can be grouped as ground transportation. Thus, if we estimate a model, we might want to have correlated errors. In this and similar cases, alternative-specific multinomial probit model can be preferred.

Maybe also classifiers such as Bayesian networks, Neural Networks or SVMs works in this case.

• Bayesian networks:

Bayesians networks such as Naive Bayes can also be used for classification. However usually they are applied to unordered dependent variables.

• Neural Networks:

Neural Networks work similarly to (multinomial/ordered) logistic regressions, but they can capture any type of non-linearity.

• Support vector machines:

SVMs are binary classifiers. They can be extended to classifications with many classes however I would not use them in our case.

I attached coding examples in the Hyperlinks. Unfortunately these coding examples are in R. As far as I know there ordered logit and ordered probit are not implemented in scikit-learn.

• I have just 7 observations, and predict yield , I am confusing in lasso and forward-backword, can i use SVM in this case? where I have repeated observation in some response variable? – e4e5 Mar 8 '17 at 7:46
• Okay so I need to use Multinomial logistic regression ? – e4e5 Mar 8 '17 at 8:03
• You can't really train anything with a sample that size though. Additionally, the question doesn't pose a classification problem. – Alex Firsov Mar 8 '17 at 8:03
• I propose an ordered logit. However the sample size might be a big problem. – Ferdi Mar 8 '17 at 8:18