Regression with independent variables as level (Low=1, Medium=2, High=3) [duplicate]

Question

I have the following data :

Yield   Rain    PH  EC  OC  N   P   K   S   Zn  Fe  Cu  Mn  Hydro
7778.1  103.03  3   1   2   1   1   2   1   1   2   3   2   1
5929.4  91.23   3   1   2   1   1   2   3   2   2   3   3   1
2872.3  109.47  3   1   2   1   1   2   2   1   2   3   3   2
4332.7  92.58   3   1   2   1   1   2   3   1   2   3   3   2
5236    100.37  3   1   2   1   1   2   1   1   2   3   3   2
3086.5  103.68  3   1   2   1   1   1   1   1   2   3   3   2
4526.1  83.38   3   1   1   1   1   2   2   2   2   3   3   1

Where Yield is my dependent variable. and Rain , PH, EC ... are independent.

I have to use regression analysis for predicting the Yield.

But, I notice that accept Rain variables all others are ordinal levels. (Like High=3, Medium=2, Low=1) where PH, EC , N, P, Fe, Cu are same values for each observation

Which regression model is the best to predict the yield? I am using python with scikit-learn.

Answer 1

Having only 7 cases, there isn't likely much you can do. Parametric models will overfit the data, as there are more predictors than cases. As such, your best bet may be exploratory analysis--try plotting variables against each other and seeing how they behave, and "feeling things out" with whatever domain expertise you may have. Also consider the use of non-parametric procedures, since you don't have a representative sample to tell you about the distribution of the population.

If you are dead-set on regression, you can try to use techniques like stepwise variable selection or MCA to first reduce the number of variables, and then run some regression models. You will most likely find this unstable/ineffective. You can also try to penalize the regressors by using ridge or lasso regression, but these are also very unlikely to be useful here.

In short, stick to exploratory analysis, or collect more data if possible.

Answer 2

If you have enough data you can use an ordered logit model or an ordered probit model. The difference between the two is the IIA assumption.

Here is a good description of the IIA to assumption and the difference between multinomial logit and multinomial probit. The difference between ordered logit and ordered probit can be described analogously.

So, why should we apply multinomial probit rather than multinomial logit? What is the advantage of relaxing IIA? To answer the question, we should understand IIA axiom. Wikipedia entry on IIA provides a nice summary. To illustrate the issue, blue bus/red bus problem is given as an example (based on McFadden, 1973). So, suppose that we need to choose between two forms of transportation, car and red bus, and suppose that we choose these two options with equal probability, 0.5. If we introduce a blue bus as an additional alternative, under the assumption of IIA, we should have a new probability, 0.33, for each option. However, this is not very intuitive as two of our options (red bus and blue bus) are quite similar. Another, and maybe more realistic, example could be a choice between four alternative modes of travel: plane, train, car, and bus. Now, under IIA we consider these alternatives independent or distinct, but three of these options can be grouped as ground transportation. Thus, if we estimate a model, we might want to have correlated errors. In this and similar cases, alternative-specific multinomial probit model can be preferred.

Maybe also classifiers such as Bayesian networks, Neural Networks or SVMs works in this case.

• Bayesian networks:

Bayesians networks such as Naive Bayes can also be used for classification. However usually they are applied to unordered dependent variables.

• Neural Networks:

Neural Networks work similarly to (multinomial/ordered) logistic regressions, but they can capture any type of non-linearity.

• Support vector machines:

SVMs are binary classifiers. They can be extended to classifications with many classes however I would not use them in our case.

I attached coding examples in the Hyperlinks. Unfortunately these coding examples are in R. As far as I know there ordered logit and ordered probit are not implemented in scikit-learn.