# Prediction of continuous variable - CV, Overfitting, Choosing K principal components

As a final assignment on Data Mining course I'm taking, we are given a training set and a test set.

We are requested to generate two different models of label prediction and run them on the test set.

I'm looking for some advice on what's the best way to approach the problem, in terms of getting a good model(s) fit.

Models\Algorithms I'm allowed to use: (Learnt in course)

1. Association Rules
2. K-Means \ K-Medoids
3. Linear Regression
4. Logistic Regression
5. PCA
6. LDA
7. SVM
8. Decision Trees (CART)
9. Bayes Classifier
10. Random forests with Bagging \ Boosting

Training set observations:

Contains 1460 observations, which represent sales of individual houses in some US state, in 2006-2010.

Training set features:

The number of explanatory variables for each observation ("features") is 79.

• 43 Factorial (categorical) variables

• 36 numbered (int, continuous) variables

Outliers:

All errors in the data were corrected prior to the data publication, and also 5 samples that were considered as outliers (partial data, or unusual sales) were pre-removed from the data.

Also, no missing values in the data.

Problem\Requirement:

Build 2 models which predict Sale Price label - a continuous 'label', for each of the samples in the test set. The training set, of course, contains the Sale Price label for each observation in it.

• Need to address overfitting problem and how I've handled it

My way of thinking:

I was thinking of generating two models.

The 1st would be using Linear Regression - Predicting response variable SalePrice based on samples features linear combination.

The 2nd one would be Random Forest with Boosting while using regression trees of course.

Before applying any of the above-stated algorithms, I was thinking running PCA and choose some K < 79 principal components to build my model on, and run the models with cross-validation method.

I would like to get some advice -

1. Am I thinking in the correct way in regards to the given problem? Would you use, given the algorithms learned limitation, the same algorithms I've chosen?

2. How do I penalize linear regression \ random forest model, so my model won't suffer from overfitting? Theoretically, I think I know the answer - by adding some bias variable to the linear regression model, for example. Not sure how to achieve this in R.

3. What's the proper way of performing cross-validation in R?

4. If using PCA - How to choose a 'proper' K < 79 principal components to work with? Should I take 10, or 70? Is there any 'thumb rule' to work upon here?

5. Any other pieces of advice, Technical or theoretical are most welcome.

Thanks

• There are too many questions and the topic too broad to be a good Cross Validated question. Putting that aside, my quick reaction is that linear regression is often a great place to start and you may want to lookup the caret package in R. There are several tutorials out there. I'm guessing you covered ridge regression, lasso, and possibly elastic net? The parameters $\lambda$ etc... for regularization are to prevent overfitting and can be chosen through CV. – Matthew Gunn Feb 6 '17 at 3:11
• I'll try to open seperate threads, with more specific questions, Thanks. Regarding your guess - I'm afraid you're guessing wrong :) unfortunatley, we've learned only the listed algorithms – Adiel Feb 6 '17 at 7:06