17
I can't say with 100% certainty, but I can give you my two cents.
Let's start with the difference in philosophy between statistics (as practised in the books mentioned) and machine learning. The former is (usually, but not always) concerned with some sort of inference. There is usually a latent parameter, like the sample mean or the effect of a novel drug, ...
8
Cross-validation is a means of estimating the performance of a method for fitting a model, rather than of the model itself, so all steps in fitting the model (including feature selection and optimising the hyper-parameters) need to be performed independently in each fold of the cross-validation procedure. If you don't do this, then you will end up with an ...
8
The main reason is that almost all book authors are in inference statistics. In particular, bio statistics is heavy on this aspect. A lot of stats used in regulated industries, such as banking is guilty of this too. Question like "what caused this? Did this cause that?" are usually asked from the inference point of view.
Cross-validation is of ...
6
One of the most challenging elements of this problem is knowing when it's OK to put unsupervised learning steps outside of the CV loop and when they should be fully penalized for by including them inside the loop. Generally speaking, unsupervised learning procedures such as principal components analysis can be unstable, i.e., the loadings of the first ...
5
The thread you mentioned already discusses it in great detail, so I'd skip the parts that were already mentioned there. Answering your question, it depends on what you mean by "one shot feature selection before cv if we do not take response variable into account". For example, if you look at the data and discover that some of the features are very ...
3
Why would you want to exponentiate the errors? That will not give you anything useful. To see that, think about the equation you are fitting:
$$\log(y_t) = \beta_0 + \beta_1 x_t + \varepsilon_t$$
The errors on the transformed scale are simply the innovation residuals $\hat{\varepsilon}_t$. The errors on the original scale (called response residuals) are $y_t ...
3
Probability
If you consider a really strict delineation of probability and statistics, the former is about mathematically describing how likely it is for an event to occur, or a proposition to be true. You can have a textbook or a course that is about probability, without entering the field of statistics at all.
Classical examples include drawing different ...
3
In principle, if you want your CV validation scores to reflect what applying an algorithm trained in the manner you did will do on new data, then everything should be part of the cross-validation (or bootstrapping, or whatever else you do along those lines). It's of course the most concerning when we need to make decision on this basis (e.g. Which of several ...
3
You are correct, choosing between two algorithms based on their test set evaluations causes contamination of your test set. What you should do is select the best performing algorithm during the CV stage and evaluate only that algorithm on the test set. Think about it this way, when you are choosing between a random forest with tree depth x vs tree depth y ...
2
You are doing $k$-fold cross-validation, so $k$ times you train a model on different train set and validate on different test set. After each such run, you calculate an error metric, in this case RMSE. So, you end up with $k$ values of RMSE, the table shows average RMSE and standard deviation of the metric. Standard deviation tells you how variable, or ...
2
Ok. I got it. It's my roc_auc_score. The correct code should use pipe.predict_proba(X_test).
1
To add a bit to @usεr11852's answer:
variance due to finite test sample size in each fold:
A fraction of tested cases such as accuracy follows a binomial distribution. You can therefore do a rough check how much such variance you'd expect given the number of tested cases for each fold $n_{f}$: if the true accuracy is $p$, the variance of observed variance ...
answered Oct 23 at 12:35
cbeleites unhappy with SX
33.6k3 gold badges66 silver badges131 bronze badges
1
Percentiling is meaningless in this context. That involves incomplete conditioning, e.g., computing things like P(Y=1 | X > c) instead of P(Y=1 | X = c), the latter being proper complete conditioning, i.e., full use of information. The task instead is to estimate P(Y=1 | X=c) as a function of height c. This is readily accomplished by fitting a binary ...
1
If you are tuning computationally intensive algorithms like neural networks, you wouldn't usually use $k$-fold cross-validation, because the computations would take too long. Instead, you would use held-out validation and test sets, so you would train the algorithm only once and validate only on a single test dataset. In fact, this is what Andrew Ng ...
Only top voted, non community-wiki answers of a minimum length are eligible