As a final assignment on Data Mining course I'm taking, we are given a training set and a test set.
We are requested to generate two different models of label prediction and run them on the test set.
I'm looking for some advice on what's the best way to approach the problem, in terms of getting a good model(s) fit.
Models\Algorithms I'm allowed to use: (Learnt in course)
- Association Rules
- K-Means \ K-Medoids
- Linear Regression
- Logistic Regression
- Decision Trees (CART)
- Bayes Classifier
- Random forests with Bagging \ Boosting
Training set observations:
Contains 1460 observations, which represent sales of individual houses in some US state, in 2006-2010.
Training set features:
The number of explanatory variables for each observation ("features") is 79.
• 43 Factorial (categorical) variables
• 36 numbered (int, continuous) variables
All errors in the data were corrected prior to the data publication, and also 5 samples that were considered as outliers (partial data, or unusual sales) were pre-removed from the data.
Also, no missing values in the data.
Build 2 models which predict Sale Price label - a continuous 'label', for each of the samples in the test set. The training set, of course, contains the Sale Price label for each observation in it.
- Need to address overfitting problem and how I've handled it
My way of thinking:
I was thinking of generating two models.
The 1st would be using Linear Regression - Predicting response variable SalePrice based on samples features linear combination.
The 2nd one would be Random Forest with Boosting while using regression trees of course.
Before applying any of the above-stated algorithms, I was thinking running PCA and choose some K < 79 principal components to build my model on, and run the models with cross-validation method.
I would like to get some advice -
Am I thinking in the correct way in regards to the given problem? Would you use, given the algorithms learned limitation, the same algorithms I've chosen?
How do I penalize linear regression \ random forest model, so my model won't suffer from overfitting? Theoretically, I think I know the answer - by adding some bias variable to the linear regression model, for example. Not sure how to achieve this in R.
What's the proper way of performing cross-validation in R?
If using PCA - How to choose a 'proper' K < 79 principal components to work with? Should I take 10, or 70? Is there any 'thumb rule' to work upon here?
Any other pieces of advice, Technical or theoretical are most welcome.