Linked Questions

5 votes
3 answers
985 views

How does cross validation works for feature selection (using stepwise regression)?

I have used the MATLAB regression learner application to do some stepwise regression with a 10-fold cross validation for feature selection. But now I want to code it myself and I'm confused about the ...
Azarang's user avatar
  • 59
15 votes
3 answers
5k views

How to measure the goodness-of-fit of a nonlinear model? Is $R^2$ useful?

Well… I did search for a while before asking and noticed perhaps my question itself has something basically wrong after reading this and this but still not sure so decided to cry out loud :). As ...
xzczd's user avatar
  • 261
2 votes
3 answers
6k views

Machine learning without test and validation data

All mainstream machine learning approaches I've seen depend on a test and usually a validation dataset to measure model accuracy during and after training. This seems like it uses up quite a lot of ...
yters's user avatar
  • 131
7 votes
3 answers
2k views

$R^2$ on out-sample data set

The conventional definition of $R^2$ is: $R^2 = 1-SSE/SST$, where SSE denotes sum of squared errors and SST is total sum of squares ($n\times variance$, n being number of sample points in train set). ...
Maaz's user avatar
  • 317
2 votes
1 answer
2k views

Decision Tree Quality Metric

For regression models we have the AIC to use as a quality metric. It has a high score for models that use fewer exogenous variables compared to models with many exogenous variables (holding the ...
Arash Howaida's user avatar
3 votes
2 answers
120 views

Evaluating the classifier on K validation sets, but training it on a fixed training set, when data is imbalanced

I'm training a binary classifier on imbalanced data (The real/production data has ~%2 of positive labels). Besides the questionable efficiency of oversampling/undersampling technique, I have a lot of ...
Amit S's user avatar
  • 57
2 votes
3 answers
246 views

The validation set includes few positive labels

I'm training a classifer on an unbalanced dataset. The test dataset's positive proportion is 0.02%. For that reason, the validation data set labels proportions are the same. Because the validation set ...
Amit S's user avatar
  • 57
2 votes
2 answers
98 views

Can I skip test set and train on 100% of data?

Is it a viable solution to train on the whole dataset without splitting the data into 'train' and 'test' sets? In other words, is it okay to skip offline evaluation and only perform online evaluation (...
asparagus's user avatar
2 votes
1 answer
106 views

Accuracy score change a lot by changing random seed in train/test split

I'm running a ML algorithm on some data, and I noticed that if I change the random state inside the train_test_split function, accuracy score change in a quite wide range. For example, with random ...
Federicofkt's user avatar
0 votes
1 answer
61 views

Multiclass Classification for Multiple Minority Classes

I've been working on a multiclass problem (5 classes) and having some challenges on Feature Selection and Class Imbalance. I have around 1,000 rows and 2,000 features (which I also generated ...
easymoneysniper's user avatar
1 vote
1 answer
39 views

Refitting model on entire dataset to use in production to make prediction on latest data?

I have a question regarding calibration (train) and holdout (test) periods.I have time based data. I split my data into two sets. Fit/Train (first 2/3 of data) and validation/testing (latest third). ...
Birk's user avatar
  • 33
1 vote
1 answer
31 views

Comparison of the performances of Regression Models and ANN models

Background I use R for statistical computations. I am working on data obtained from Lathe CNC Machining Systems there is a tool and there is a workpiece that needs to be operated on. Lathe Machining ...
Suddhasheel Ghosh's user avatar
1 vote
1 answer
32 views

Does a newly constructed ML dataset need to have an official train-dev-test split? Should the test set be intentionally balanced?

I have constructed a novel ML (NLP) dataset for classification and labeled it with three classes. The dataset is rather small with about 700 examples, out of which the classes have about 400, 200, and ...
Arno's user avatar
  • 11
1 vote
0 answers
30 views

Best Strategy for Model Training & Selection (Spoiler: Should I Re-Train?)

After a discussion with some colleagues, I've realized we've different views on which is the go-to strategy for model training. Strategy A: Train-Validation-Test Split and Final Model Selection ...
rusiano's user avatar
  • 566