3
$\begingroup$

I am a newbie in data mining world. I have a general question. I have a data set which has 10 independent variables and one target variable named as category which has 9 values like: 1, 2, 3, 4, 5, 6, 7, 8, 9. the 10 independent variables have different kind of range of values. some of them have values between 0 - 5000, some have big range like 5,000,000 - 100,000,000 etc.

Moreover there is no specific relation (linear etc.) existing between target and independent variables.

I am basically trying to predict the target variable category by using all of these independent variables.

Can someone suggest what should be my approach? I am very confused. Should I use regression models, decision trees or cluster analysis?

Improve this question
$\endgroup$
1
  • $\begingroup$ Note that cluster analysis does not predict. So it will not get you very far alone. It may however improve the result of the others sometimes. $\endgroup$
    – Has QUIT--Anony-Mousse
    Dec 17 '12 at 16:05
2
$\begingroup$

You can use whatever multi-category supervised classification algorithm you like, for example multinomial logistic regression or trees (but not linear regression or binary logistic regression). Make sure you use training/evaluation/test set or a type of cross validation, though.

(Also, if there is no specific relationship between predictors and target as you say, then your classification will most likely perform poorly).

Improve this answer
$\endgroup$
3
  • $\begingroup$ You can use logistic regression to solve this problem, you just have to do it more than once. For example, you could model $q_1=Pr(y=1|y\in\{1,...,9\})$, and then remove the ones, and then model $q_2=Pr(y=2|y\in\{2,...,9\})$, and then remove the ones and twos, then model $q_3=Pr(y=3|y\in\{3,...,9\})$, and so on. Then your prediction is given by $Pr(y=k)=q_1\dots q_k$ $\endgroup$
    – probabilityislogic
    Nov 23 '12 at 7:20
  • $\begingroup$ Sure you can, but I don't consider this or one vs. all good practise. $\endgroup$
    – Momo
    Nov 23 '12 at 7:40
  • $\begingroup$ Thanks momo, But honestly I didn't get is what do you mean by "(Also, if there is no specific relationship between predictors and target as you say, then your classification will most likely perform purely). " Also please have a look at my complete problem here: stats.stackexchange.com/questions/43040/… $\endgroup$
    – user16603
    Nov 24 '12 at 13:47
5
$\begingroup$
  1. You probably need to first normalize your data [A]
  2. You should never just use just one classifier and end it there.
  3. Certain choices of classifiers are also done based on the data characteristics. For example, if your data has quite a bit of missing attribute values, decision trees almost always tend to do better than any other model.

The order in which I use the classifiers:
1] SVMs [B]
2] Logistic Regression/Multinomial Logistic Regression [C]
3] Decision Trees [D]

I usually first try SVMs, if I get my results with the standard implementations of SVM then great! If not, then try to use LR/MLR from Weka. The only reason I try SVM first is because Weka is not great at handling large datasets for training. But this might not be the case for you, so you can try LR/MLR first as well if you wish. If neither SVM nor LR give the desired result, then I simply move to Decision Trees. After running all of them, I just pick the one that performed the best.

[A] http://www.quora.com/What-are-the-ways-to-normalize-the-features-for-statistics-or-machine-learning-software
[B] http://svmlight.joachims.org/svm_multiclass.html or http://www.csie.ntu.edu.tw/~cjlin/libsvm/
[C] Weka
[D] An in-house implementation of Gradient Boosted Decision Tree.

Improve this answer
$\endgroup$
0
$\begingroup$

First of all. 9 categories sound like a lot. How big is you sample?

Start by assessing the independent variables. You might want to remove the ones which have a score on 3 standard deviations above them mean. This should reduce you range. Next you want to assess if the independent variables follows a normal distribution. If not, you want to apply some transformation to make them normal distributed. It is not a necessity but it is good common practice.

The way you describe it, logistic regression is the way to go with the methods you are mentioning. It all comes out on the characteristics of your dependent variable Cluster analysis is an explorative/undirected technique so that can not be used for prediction. K-means clustering can however be used for prediction...

Improve this answer
$\endgroup$
10
  • 1
    $\begingroup$ Why should the independent variables be normally distributed? Most inferential procedures are based on errors or residuals being normally distributed. $\endgroup$
    – Stephan Kolassa
    Nov 23 '12 at 7:29
  • $\begingroup$ Because it is an assumption of regression. Non normal distributed variables and outliers can have a negative impact on the different tests you run to assess your model. However i am not sure it is as important if we are talking machine learning. Non the less it is always just good common practice to "tidy up" your data before doing analysis. $\endgroup$
    – Kasper Christensen
    Nov 23 '12 at 9:52
  • 5
    $\begingroup$ Uh... no. Regression does not require independent variables to be normally distributed. Regression (more specifially: the various theorems on the distributional properties of parameter estimates) require that errors are normally distributed: en.wikipedia.org/wiki/Ordinary_least_squares I will happily concede, though, that outliers in independent variables should be examined critically and obvious errors should be corrected. $\endgroup$
    – Stephan Kolassa
    Nov 23 '12 at 9:58
  • 1
    $\begingroup$ The source you mention has it wrong. Don't trust everything on the internet... ;-) One log-transforms (or does other transformations, e.g., Box-Cox) independent variables if the relationship with the dependent one is not additive-linear on the non-transformed data. That said, you are certainly right that highly skewed independent variables may lead to different data points having different influence on parameter estimates, so one should look at this. $\endgroup$
    – Stephan Kolassa
    Nov 23 '12 at 11:08
  • 1
    $\begingroup$ When you have continuous and/or discrete variables you can calculates descriptive statistics, as for example the mean, standard deviation, range etc. One of the things one can do to "clean" up ones dataset is to remove outliers. One can define an outliers in many ways but a good rule of thump is to remove observations which have a standard deviation "far away" from the mean as those cases will normally be bad representations of what you are trying to capture and they can affect your calculations. $\endgroup$
    – Kasper Christensen
    Nov 25 '12 at 17:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy