5
In your two-way split, as you also mentioned, your validation set is actually your test set. In your way, you haven't mentioned about hyperparameter optimisation (HPO), but it's a key step in many machine learning algorithms. When you need HPO, you'll either need to have a separate validation set to tune the HPs or tune them using cross validation over the ...
3
The two methods you are describing are essentially the same thing. When you describe using cross validation, this is analogous to using a train test split just repeated multiple times. Train/validation/test and train/test with cross validation on the training set are exactly the same but using cross validation repeats for different splits of train/test.
2
Polynomial regression is linear regression! Linear means linear in the unknown parameters, that you use some non-linear transformation of the known regressor values (in this case a polynomial) is immaterial.
So the answer to your question is yes, the formula is valid.
2
In general, you need a split of your data set into test/training whenever there is danger of using information from the test set on the trainig set. The information might flow through you as a modeller. Let this sink in for a moment.
For example, if you build one model with one method, use a 80/20 split or cross validation. If you compare many methods on a ...
2
“All models are wrong, but some are useful”. George E. P. Box
When we build a model, our main materials is data. But it is ultimate goal that probably will never be achieved is to use our model to different data and performs like we trained with our trained data. But we can obviously try and then comes this split.
Now I will not discuss the size of the split....
2
Right, but the number of epochs is your choice. You can decrease it a bit if you want LOOCV. Since you're going to train the model from scratch for each training fold, it'll be $e\times n$ looping over, where $e$ is your epoch number.
Yes, you'll have n different set of parameters. Averaging is not good, even if the problem is convex like this one. For ...
2
Question 1: Yes, you do train the model $n$ times, each time on an $n-1$-sized dataset. In some cases it is possible to save computational resources by using algebraic tricks to relate the estimates in each of the $n$ cases. You can do this e.g. in linear regression so that effectively your computational cost is that of a single fit (!); i.e. you are roughly ...
2
He basically just verified if the data is more or less linear
Granted I know nothing about the course in question and its purpose, in real life assumptions are rarely if ever fully satisfied so realizing that the data is more or less linear may be enough to proceed provided you are aware of the limitations.
I suppose many of the 6-months-DS-without-stat/...
1
If you want to only evaluate using cross-validation, you need to do it like this:
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler
pipeline = Pipeline([
('scaler', StandardScaler()),
('reg', LinearRegression())
])
k = ... # define k
cross_val_score(pipeline, X, y, cv=k).mean()
Notes:
...
1
Your understanding is correct, the tuning process doesn't improve from nested CV as it is handed by the "inner loop". What nested CV does is that it helps the final evaluation be more robust and objective, as it is done based on multiple test sets.
1
Yes it is reasonable. I don't think lambda.1se adds anything.
For any cross validation metric it is recommended to run multiple cross validation runs.
Similarly how many folds is a generic problem.
However I would point out your reasoning is wrong: loo has high variance and low bias, basically because the estimators are so correlated, so noise doesn't ...
1
Answer is no. In cross validation, each set takes a turn being the "validation" set. In a 2 fold CV, there would be 2 such sets, so there would be two validation sets.
1
From your description, everything seems to work "as designed".
Machine learning algorithms work by finding patterns in the data. The patterns are used to make such classifications that achieve the best value of cost function on the training data. If your data is random*, the better the performance on training set, the more your algorithm is ...
1
To test my model should I split my data into 3: training, validation, test?
If you have sufficient data to do so, then this might be reasonable. I think there may be some arguments to be made that cross validation is preferable to a one time data splitting. See Frank Harrell's Regression Modelling Strategies chapter. 5.3.4 for more if you are interested ...
1
Neither are correct estimates of performance.
Your 1.0 is obtained by scoring clf.best_estimator_ on test folds, which it has already seen; it is the refit model (trained on all of X) from the grid search. You refit rfc in the loop, but that seems to be with default parameters, and you don't use it to score anyway. Even if you replaced clf.best_estimator_ ...
1
I think this:
The other possible case is when the validation accuracy tracks the training accuracy fairly well. This case indicates that your model capacity is not high enough: make the model larger by increasing the number of parameters.
Is getting at the idea that training a model, in general, should yield overfitting. That is basically its design -- ...
1
Generally if your training error is much lower than the test error, it indeed suggests overfitting. However, the training error would almost always be smaller than the test error, and Random Forest generally doesn't overfit (as long as the bootstrap samples and mtry ratio are good- rule of thumb- 2/3, and sqrt(# variables) respectively).
You could try ...
Only top voted, non community-wiki answers of a minimum length are eligible
