985 views

### How does cross validation works for feature selection (using stepwise regression)?

I have used the MATLAB regression learner application to do some stepwise regression with a 10-fold cross validation for feature selection. But now I want to code it myself and I'm confused about the ...
• 59
5k views

### How to measure the goodness-of-fit of a nonlinear model? Is $R^2$ useful?

Well… I did search for a while before asking and noticed perhaps my question itself has something basically wrong after reading this and this but still not sure so decided to cry out loud :). As ...
• 261
6k views

### Machine learning without test and validation data

All mainstream machine learning approaches I've seen depend on a test and usually a validation dataset to measure model accuracy during and after training. This seems like it uses up quite a lot of ...
• 131
2k views

### $R^2$ on out-sample data set

The conventional definition of $R^2$ is: $R^2 = 1-SSE/SST$, where SSE denotes sum of squared errors and SST is total sum of squares ($n\times variance$, n being number of sample points in train set). ...
• 317
2k views

### Decision Tree Quality Metric

For regression models we have the AIC to use as a quality metric. It has a high score for models that use fewer exogenous variables compared to models with many exogenous variables (holding the ...
120 views

### Evaluating the classifier on K validation sets, but training it on a fixed training set, when data is imbalanced

I'm training a binary classifier on imbalanced data (The real/production data has ~%2 of positive labels). Besides the questionable efficiency of oversampling/undersampling technique, I have a lot of ...
• 57
246 views

### The validation set includes few positive labels

I'm training a classifer on an unbalanced dataset. The test dataset's positive proportion is 0.02%. For that reason, the validation data set labels proportions are the same. Because the validation set ...
• 57
98 views

### Can I skip test set and train on 100% of data?

Is it a viable solution to train on the whole dataset without splitting the data into 'train' and 'test' sets? In other words, is it okay to skip offline evaluation and only perform online evaluation (...
106 views

### Accuracy score change a lot by changing random seed in train/test split

I'm running a ML algorithm on some data, and I noticed that if I change the random state inside the train_test_split function, accuracy score change in a quite wide range. For example, with random ...
• 123
61 views

### Multiclass Classification for Multiple Minority Classes

I've been working on a multiclass problem (5 classes) and having some challenges on Feature Selection and Class Imbalance. I have around 1,000 rows and 2,000 features (which I also generated ...
1 vote
39 views

### Refitting model on entire dataset to use in production to make prediction on latest data?

I have a question regarding calibration (train) and holdout (test) periods.I have time based data. I split my data into two sets. Fit/Train (first 2/3 of data) and validation/testing (latest third). ...
• 33
1 vote
31 views

### Comparison of the performances of Regression Models and ANN models

Background I use R for statistical computations. I am working on data obtained from Lathe CNC Machining Systems there is a tool and there is a workpiece that needs to be operated on. Lathe Machining ...
1 vote
32 views

### Does a newly constructed ML dataset need to have an official train-dev-test split? Should the test set be intentionally balanced?

I have constructed a novel ML (NLP) dataset for classification and labeled it with three classes. The dataset is rather small with about 700 examples, out of which the classes have about 400, 200, and ...
• 11
1 vote