Questions tagged [train-test-split]
The train-test split is a method used to estimate the performance of machine learning algorithms that are applicable for prediction-based Algorithms/Applications.
130 questions
0
votes
0
answers
11
views
Identify predictors for clustering output?
I have a dataset with variables collected years ago, and many variables collected this year as outcome variables. I want to combine all the variables collected this year to get one outcome, e.g. ...
- 43
0
votes
0
answers
24
views
Model Stacking - Out of Fold Procedures
I am attempting to use a model stacking procedure where I am using a time-series split on a set of data I have (around 5000 entries). The goal is binary classification.
After obtaining hyper ...
- 47
2
votes
1
answer
36
views
Model Stacking Train Test Split Methdods
I am trying to validate my processes in terms of how I am engaging in model stacking for binary classification. Say I have two models as my base models, models A and B both with different classifiers ...
- 47
1
vote
2
answers
46
views
Purpose of test set in cross-validation
How does the test set in k-fold cross-validation have any purpose? The most common argument in favor of a test set I can find is to not have any data leakage between training and testing. But you don'...
- 11
8
votes
1
answer
286
views
Should out-of-sample validation also be out-of-time for time-series?
Introduction
When training a model a "sample" usually refers to the data used to fit the model, so...
Sample: Data used for training model
Out-of-sample: Data not used for training model
Out-...
- 245
0
votes
0
answers
6
views
What could be causing test-loss to consistently outperform eval-loss?
I'm training a number of different models, all of them XGBoost/LightGBM type models, and thus they require an eval set for early stopping.
Nonetheless, unless I've done something careless when doing ...
- 2,482
0
votes
0
answers
19
views
Splitting training and test set on a time series problem
I have an OHLCV* dataset that starts on 01-01-2000 and ends on 31-12-2003 and I want to evaluate a model, say an SVM regressor. In other words, given some daily features describing the dynamics of the ...
- 1
0
votes
0
answers
17
views
Encode & normalize features limited in range before or after split
I want to train a classifier on music data which contains a limited set of features which are all constrained in range:
...
- 1
1
vote
0
answers
28
views
What are the appropriate data splitting techniques for time-dependent sequential datasets, such as breakdown records over time?
I am working with a time-dependent sequential dataset, specifically a record of machine breakdowns over a period of time. My dataset includes data from the sensors of several machines until they fail ...
- 11
3
votes
2
answers
684
views
test & train for very very small data
I have just 25 observations. I'm not sure would it possible to test & train the data. For example 15 observations for train and 10 observations for test set. 15 observations is so small for ...
- 179
2
votes
1
answer
45
views
What is the performance of a "meta" learner that performs internally CV for model selection?
I am trying to understand the proof that reporting CV performance during model selection as performance estimate is optimistically biased. The steps in the proof are the following:
Let $p_i, \pi_i$ ...
- 621
1
vote
1
answer
44
views
R2_test>R2_train from a published paper. How can it be consistently possible?
This image is from a paper where the author has trained and then tested different models on a small dataset (consisting of 117 samples in total). I had the following observation and their questions ...
- 11
0
votes
0
answers
27
views
Test/validation set
I've been having a discussion with colleagues and wanted to seek your input. If I'm using holdout and cross-validation to build and test my models. In this process, the training set is used to tune ...
- 101
0
votes
1
answer
25
views
Using a model to evaluate over or under-priced rental prices for the same apartments used in training
If I have a machine learning model which predicts the rental prices of apartments, can I use the model once complete to analyse the prediction for the same apartments I used to train the model so I ...
- 83
1
vote
1
answer
43
views
How should I split my dataset if I am applying oversampling?
Related: How can I apply multiple sampling tenchiques to a single dataset?
Suppose I have a dataset called my_dataset.dat with a length of 1079134 rows.
This ...
- 2,125
3
votes
2
answers
89
views
Can an Anomaly Detector be Tested with Data that it Labeled?
Is it wrong to leverage a model to label data, then perform a train/test split to evaluate the performance of said model?
Assume I have an unlabeled data set where the missing labels are a binary ...
- 285
1
vote
0
answers
33
views
What to do when you realize you've overfit?
This is hypothetical and I would like to hear what people do when the get to the test set and realize they've overfit. Of course, preventing overfitting in the first place is ideal.
You're working on ...
- 362
2
votes
1
answer
23
views
weighted random forest with train/test datasets
I have a dataset where the sample distribution does not match the population distribution, but I have weights that can be applied to address that issue. I have randomly partitioned the original ...
- 23
0
votes
0
answers
19
views
How to Train a Model on the Whole Outer-loop Training Set in Nested Cross-validation?
I'm implementing nested cross-validation for a machine learning project and need some clarity on the training process using the outer-loop training set. Here’s a summary of my process:
Outer-loop ...
0
votes
0
answers
15
views
Is GroupKFold needed if some samples have some of their feature values equal?
I am given a dataset $D$ of 10k enzyme-substrate complexes having a lock-key relationship, with each sample (complex) being characterized by enzyme features $x_e$ and substrate features $x_s$. That is,...
- 621
1
vote
0
answers
37
views
How should you split up data in a train-test-validation split
I've seen it is generally recommended when using a train-test-validation data split, to first split your data into train and test datasets, and then furtherly split the train dataset into a train and ...
- 21
0
votes
0
answers
23
views
Lasso regression test MSE lower than train MSE
Im currently using Lasso to build a predictive model for numeric variable .
Before scaling the features I split the data for train test and validation . I have a feature named 'year' and i wanted the ...
4
votes
2
answers
97
views
How to approach dataset splitting for building time-series models?
Suppose I have 100 observations of time series data $x_1,...,x_{100}$, and that I want to split the data into a train set, a validation set, and a test set. I know that the train set must have smaller ...
- 1,256
0
votes
1
answer
37
views
Random forest cross-validaton by patient
I have a dataset of various features from 10 patients and 10 controls. Each patient has many data points.
Random forest does an amazing job in predicting whether a data point is from a patient or a ...
- 127
1
vote
1
answer
247
views
How do you train-test split an imbalanced dataset?
I have an imbalanced dataset and I'm trying to predict a binary target. The minority class amounts to approximately 0.4% of all observations (60 million observations from which 250K belong to the ...
- 569
2
votes
2
answers
221
views
Can I skip test set and train on 100% of data?
Is it a viable solution to train on the whole dataset without splitting the data into 'train' and 'test' sets? In other words, is it okay to skip offline evaluation and only perform online evaluation (...
- 21
0
votes
0
answers
24
views
Scaling data to a sample that is neither training nor validation. Is this data leakage?
TL;DR: Data being scaled to a sample that is neither training nor validation. Is this data leakage?
Hi,
I have a data set from samples that are distributed on a plate. Precisely, there are 96 wells ...
- 65
1
vote
0
answers
36
views
Choosing a suitable sample size for a Random Forest Model
I know that this isn't always a straight forward question to answer, but I am working on a provincial wide wetland classification model that has $7$ classes and $32$ or so explanatory variables. In my ...
- 11
6
votes
4
answers
2k
views
Is it really so bad to do SMOTE on the training set before crossvalidation?
I understand that doing this leads to data leakage, but if I get better performance on the test set does it really matter?
I tried using caret with ...
1
vote
0
answers
21
views
Train/validate/test subsets of the data; which one do we use?
We are writing a scientific paper using a large dataset of healthcare data. We split our data into three sets: a train set (60%), a validation set (20%), and a test set (20%).
My question is what do ...
- 251
0
votes
0
answers
70
views
Data leakage in time series forecasting framed as a supervised learning problem
Suppose that I have a simple univariate time series. My goal is to use the value of 3 consecutive days to predict the value of the fourth day.
I built my dataset by applying a rolling window that ...
- 11
1
vote
1
answer
41
views
Impute missing value [closed]
In machine learning when I impute missing values which of the following I perform :
1-Impute data set and then split it?
2-Split dataset to Training and testing datasets and then Impute each datasets ...
- 335
1
vote
0
answers
58
views
Should I delete samples from the training data that are present in the testing data by accident?
I classify pairs of entities, let's say dog-cat pairs, whether there is association between them (positive class) or there is not (negative class). I have a moderately sized positive dataset (~130k ...
- 185
1
vote
1
answer
291
views
Should we use train, validation, or test data when creating PR/AUC curves to optimize the decision threshold?
It makes sense to me that we can use the ROC-AUC and PR-AP scores of the validation sets during CV to tune our model hyperparameter selection. And when reporting the models final performance, it makes ...
- 107
1
vote
0
answers
15
views
Determining Optimal Data Period / Time Span for Model Training
I'm seeking advice on determining the ideal time span for optimizing a weather forecast strategy using historical data without overfitting/underfitting our model. In pursuit of optimal performance and ...
- 111
2
votes
1
answer
208
views
Accuracy score change a lot by changing random seed in train/test split
I'm running a ML algorithm on some data, and I noticed that if I change the random state inside the train_test_split function, accuracy score change in a quite wide range.
For example, with random ...
- 123
1
vote
1
answer
48
views
CV score vastly different from Train-Test score
I'm working on a multi-class classification task. I'm currently trying to tune a LGB model but have encountered a behavior that I do not understand. First, my data is from 1996 to 2015 so I split my ...
- 11
4
votes
1
answer
86
views
Split-sample analysis as a way of avoiding p-hacking?
In my lab we pre-register our hypotheses prior to running an experiment, and we also attempt to fully specify our exclusion criteria, coding practices for small categories and edge cases, etc. However,...
- 634
0
votes
0
answers
39
views
Is it normal to have a sharp increase in validation error when using 10% of the data instead of something like 20-30%?
Scenario:
I'm training a relatively simple neural network to classify pairs of tabular datapoints (~150k), lets say drugs and diseases, whether they are related (positive) or not (negative). As I only ...
- 185
2
votes
0
answers
37
views
Data leakage: Train test split before or after data preprocessing? [duplicate]
A while ago I came across the word "data leakage" for the first time, and after some research, I found that it is a common mistake among data science/machine learning practitioners. But the ...
- 81
1
vote
1
answer
39
views
train / validation / test split problem
Suppose that I have created train/validation/test splits for model building.
I optimized the hyperparameters using the validation set and chose the parameter values which gave the highest accuracy. To ...
- 1,302
1
vote
1
answer
202
views
How reliable is ```train_test_split```? Is there a way to optimize it?
Using train_test_split is a common practice while building a Machine Learning model. Nevertheless, partitioning your dataset to get train and test samples is an ...
5
votes
2
answers
860
views
Deleting outliers prior to data splitting or only in the training set?
I'm working on a dataset with some outliers in the response variable which are actually natural results (not errors). I want to calibrate a model which could then be used to predict on populations ...
1
vote
0
answers
84
views
How to create representative training, validation, and test sets when working with time series data?
In my application, I am working with a relatively long time series of daily market index percentage returns (many years) and am trying to model the dependence structure of the returns from a pure time ...
- 451
1
vote
1
answer
147
views
Multi-label stratified split
I am working on a multilabel text classification problem.
The text data is called 'cleaned_text' and has shape (92259, 1) and the one-hot encoded label data is called 'labels' and has shape (92259, 32)...
- 670
0
votes
0
answers
26
views
Is it common the performance in validation set being lower than the performance in test set during the training of a classifier?
I'm evaluating some settings for using graph convolutional networks for image classification.
I'm using train/validation/test split of 64%/16%/20%, and adopting
5-fold cross-validation. That is, in ...
- 1,008
1
vote
0
answers
316
views
random split vs temporal/time based split
Some background: I want to train a regression model to predict future prices for used cars. I have about 85,000 observations collected from November 2022 to June 2023 and have around 80 different ...
- 267
1
vote
0
answers
52
views
Splitting strategy for performing hyperparameter tuning, algorithm comparison and model validation in one experiment
Let's say that for a supervised machine learning experiment I am using a fixed learning algorithm (e.g. Random Forest), and I want to achieve the following:
Choose optimal hyper-parameters for the ...
- 21
3
votes
1
answer
311
views
Why should I split the data when searching for outliers? (pyod)
I am using pyod to detect outliers in data, and I came across this official example: https://github.com/yzhao062/pyod/blob/master/examples/comb_example.py
I have a question regarding the need to split ...
- 125
4
votes
3
answers
781
views
Is it necessary to do train-test split when we are interested in understanding the model rather than predicting?
In machine learning we are taught to always do validation of some sort, for instance by creating a hold out validation set that is used to test the performance of the model.
However, in some use cases ...
- 247