Questions tagged [train-test-split]
The train-test split is a method used to estimate the performance of machine learning algorithms that are applicable for prediction-based Algorithms/Applications.
130 questions
0
votes
0
answers
11
views
Identify predictors for clustering output?
I have a dataset with variables collected years ago, and many variables collected this year as outcome variables. I want to combine all the variables collected this year to get one outcome, e.g. ...
- 43
8
votes
1
answer
286
views
Should out-of-sample validation also be out-of-time for time-series?
Introduction
When training a model a "sample" usually refers to the data used to fit the model, so...
Sample: Data used for training model
Out-of-sample: Data not used for training model
Out-...
- 245
0
votes
0
answers
24
views
Model Stacking - Out of Fold Procedures
I am attempting to use a model stacking procedure where I am using a time-series split on a set of data I have (around 5000 entries). The goal is binary classification.
After obtaining hyper ...
- 47
2
votes
1
answer
36
views
Model Stacking Train Test Split Methdods
I am trying to validate my processes in terms of how I am engaging in model stacking for binary classification. Say I have two models as my base models, models A and B both with different classifiers ...
- 46k
1
vote
1
answer
482
views
Why can stratified sampling to testing/training sets on strata that contain less than 10% of the entire dataset be statistically risky?
I'm trying to split my data into a testing and a training set. There are lots of variables that I want to ensure are well represented in both the training and testing sets (say, 15 covariates). But ...
CommunityBot
- 1
1
vote
2
answers
46
views
Purpose of test set in cross-validation
How does the test set in k-fold cross-validation have any purpose? The most common argument in favor of a test set I can find is to not have any data leakage between training and testing. But you don'...
- 42k
1
vote
1
answer
44
views
R2_test>R2_train from a published paper. How can it be consistently possible?
This image is from a paper where the author has trained and then tested different models on a small dataset (consisting of 117 samples in total). I had the following observation and their questions ...
- 67k
3
votes
1
answer
117
views
What is the difference between spliting the dataset into training and testing or collecting the training and testing data seperately?
I am working on active learning and I was wondering about the difference if we split the dataset into training and testing or collecting and labeling the training and testing datasets separately. ...
- 65.8k
0
votes
0
answers
6
views
What could be causing test-loss to consistently outperform eval-loss?
I'm training a number of different models, all of them XGBoost/LightGBM type models, and thus they require an eval set for early stopping.
Nonetheless, unless I've done something careless when doing ...
- 2,482
1
vote
1
answer
537
views
How to properly impute values on the test set using imputer (missForest)
I'm trying to impute some missing values on my dataset $X$. So first I shuffle and split data to obatin the train set X_train and the test set ...
CommunityBot
- 1
0
votes
0
answers
19
views
Splitting training and test set on a time series problem
I have an OHLCV* dataset that starts on 01-01-2000 and ends on 31-12-2003 and I want to evaluate a model, say an SVM regressor. In other words, given some daily features describing the dynamics of the ...
- 1
0
votes
0
answers
17
views
Encode & normalize features limited in range before or after split
I want to train a classifier on music data which contains a limited set of features which are all constrained in range:
...
- 4,487
1
vote
1
answer
611
views
How to interpret Isolation Forest results on variations of train/test sets?
I have a labelled dataset, originally intended for classification or clustering tasks, whose minority class is at 10%. I am investigating whether this problem can be tackled with anomaly detection ...
CommunityBot
- 1
1
vote
0
answers
28
views
What are the appropriate data splitting techniques for time-dependent sequential datasets, such as breakdown records over time?
I am working with a time-dependent sequential dataset, specifically a record of machine breakdowns over a period of time. My dataset includes data from the sensors of several machines until they fail ...
- 11
2
votes
1
answer
45
views
What is the performance of a "meta" learner that performs internally CV for model selection?
I am trying to understand the proof that reporting CV performance during model selection as performance estimate is optimistically biased. The steps in the proof are the following:
Let $p_i, \pi_i$ ...
- 1,210
3
votes
2
answers
684
views
test & train for very very small data
I have just 25 observations. I'm not sure would it possible to test & train the data. For example 15 observations for train and 10 observations for test set. 15 observations is so small for ...
- 98.4k
1
vote
0
answers
48
views
Classifier score is higher on test set than on training set. Is this an error?
I am currently getting used to scikit-learn and I trained a simple logistic regression model on the iris dataset. One interesting thing I noticed was when I looked at the classifier.score, namely:
<...
- 82.8k
1
vote
0
answers
100
views
Splitting Data Into Training and Test set [closed]
I am trying to split a data set into training and test set with these codes
...
- 82.8k
0
votes
1
answer
25
views
Using a model to evaluate over or under-priced rental prices for the same apartments used in training
If I have a machine learning model which predicts the rental prices of apartments, can I use the model once complete to analyse the prediction for the same apartments I used to train the model so I ...
- 10.4k
0
votes
0
answers
27
views
Test/validation set
I've been having a discussion with colleagues and wanted to seek your input. If I'm using holdout and cross-validation to build and test my models. In this process, the training set is used to tune ...
- 67k
5
votes
2
answers
860
views
Deleting outliers prior to data splitting or only in the training set?
I'm working on a dataset with some outliers in the response variable which are actually natural results (not errors). I want to calibrate a model which could then be used to predict on populations ...
- 18.4k
3
votes
2
answers
89
views
Can an Anomaly Detector be Tested with Data that it Labeled?
Is it wrong to leverage a model to label data, then perform a train/test split to evaluate the performance of said model?
Assume I have an unlabeled data set where the missing labels are a binary ...
- 4,092
1
vote
1
answer
43
views
How should I split my dataset if I am applying oversampling?
Related: How can I apply multiple sampling tenchiques to a single dataset?
Suppose I have a dataset called my_dataset.dat with a length of 1079134 rows.
This ...
1
vote
0
answers
33
views
What to do when you realize you've overfit?
This is hypothetical and I would like to hear what people do when the get to the test set and realize they've overfit. Of course, preventing overfitting in the first place is ideal.
You're working on ...
- 362
3
votes
1
answer
805
views
Machine Learning for Time Series: Train and test set overlap due to lagged target as feature – problem of data leakage?
Situation:
My objective is to apply Machine Learning (for regression problems). Therefore, I have a panel dataset of time series with daily fund data from 2018-01-01...
CommunityBot
- 1
2
votes
1
answer
23
views
weighted random forest with train/test datasets
I have a dataset where the sample distribution does not match the population distribution, but I have weights that can be applied to address that issue. I have randomly partitioned the original ...
- 4,603
1
vote
1
answer
375
views
How to deal with groups when splitting a data into train and test?
Say i have a dataset with groups that i want to use for a Regression problem that looks like the following where feature1 is the group column:
...
CommunityBot
- 1
0
votes
0
answers
15
views
Is GroupKFold needed if some samples have some of their feature values equal?
I am given a dataset $D$ of 10k enzyme-substrate complexes having a lock-key relationship, with each sample (complex) being characterized by enzyme features $x_e$ and substrate features $x_s$. That is,...
- 82.8k
0
votes
0
answers
19
views
How to Train a Model on the Whole Outer-loop Training Set in Nested Cross-validation?
I'm implementing nested cross-validation for a machine learning project and need some clarity on the training process using the outer-loop training set. Here’s a summary of my process:
Outer-loop ...
1
vote
0
answers
37
views
How should you split up data in a train-test-validation split
I've seen it is generally recommended when using a train-test-validation data split, to first split your data into train and test datasets, and then furtherly split the train dataset into a train and ...
- 21
4
votes
2
answers
97
views
How to approach dataset splitting for building time-series models?
Suppose I have 100 observations of time series data $x_1,...,x_{100}$, and that I want to split the data into a train set, a validation set, and a test set. I know that the train set must have smaller ...
- 991
0
votes
0
answers
23
views
Lasso regression test MSE lower than train MSE
Im currently using Lasso to build a predictive model for numeric variable .
Before scaling the features I split the data for train test and validation . I have a feature named 'year' and i wanted the ...
0
votes
1
answer
37
views
Random forest cross-validaton by patient
I have a dataset of various features from 10 patients and 10 controls. Each patient has many data points.
Random forest does an amazing job in predicting whether a data point is from a patient or a ...
- 4,603
6
votes
4
answers
2k
views
Is it really so bad to do SMOTE on the training set before crossvalidation?
I understand that doing this leads to data leakage, but if I get better performance on the test set does it really matter?
I tried using caret with ...
- 5,051
1
vote
1
answer
247
views
How do you train-test split an imbalanced dataset?
I have an imbalanced dataset and I'm trying to predict a binary target. The minority class amounts to approximately 0.4% of all observations (60 million observations from which 250K belong to the ...
- 1,658
2
votes
2
answers
221
views
Can I skip test set and train on 100% of data?
Is it a viable solution to train on the whole dataset without splitting the data into 'train' and 'test' sets? In other words, is it okay to skip offline evaluation and only perform online evaluation (...
- 21
0
votes
0
answers
24
views
Scaling data to a sample that is neither training nor validation. Is this data leakage?
TL;DR: Data being scaled to a sample that is neither training nor validation. Is this data leakage?
Hi,
I have a data set from samples that are distributed on a plate. Precisely, there are 96 wells ...
- 65
1
vote
0
answers
36
views
Choosing a suitable sample size for a Random Forest Model
I know that this isn't always a straight forward question to answer, but I am working on a provincial wide wetland classification model that has $7$ classes and $32$ or so explanatory variables. In my ...
- 18.4k
1
vote
0
answers
21
views
Train/validate/test subsets of the data; which one do we use?
We are writing a scientific paper using a large dataset of healthcare data. We split our data into three sets: a train set (60%), a validation set (20%), and a test set (20%).
My question is what do ...
- 82.8k
0
votes
0
answers
70
views
Data leakage in time series forecasting framed as a supervised learning problem
Suppose that I have a simple univariate time series. My goal is to use the value of 3 consecutive days to predict the value of the fourth day.
I built my dataset by applying a rolling window that ...
- 82.8k
-1
votes
2
answers
2k
views
Should I use transformer.fit_transform(X_test, y_test) or not?
tl-dr: The function model.fit() is different from transformer.fit(). My idea is to make all transformations needed on the training set and after that on the test set with fit_transform in both.
Hi!
I'...
- 103
1
vote
1
answer
41
views
Impute missing value [closed]
In machine learning when I impute missing values which of the following I perform :
1-Impute data set and then split it?
2-Split dataset to Training and testing datasets and then Impute each datasets ...
- 131k
1
vote
0
answers
58
views
Should I delete samples from the training data that are present in the testing data by accident?
I classify pairs of entities, let's say dog-cat pairs, whether there is association between them (positive class) or there is not (negative class). I have a moderately sized positive dataset (~130k ...
- 185
0
votes
1
answer
580
views
Is it sufficient to report only the result of cross validation in research paper?
I'm working on my master's in developing a machine-learning model to predict classes of biomedical images from a microscope. These images are
collected separately from each patient. For example, I ...
4
votes
3
answers
781
views
Is it necessary to do train-test split when we are interested in understanding the model rather than predicting?
In machine learning we are taught to always do validation of some sort, for instance by creating a hold out validation set that is used to test the performance of the model.
However, in some use cases ...
1
vote
1
answer
291
views
Should we use train, validation, or test data when creating PR/AUC curves to optimize the decision threshold?
It makes sense to me that we can use the ROC-AUC and PR-AP scores of the validation sets during CV to tune our model hyperparameter selection. And when reporting the models final performance, it makes ...
- 129
1
vote
0
answers
15
views
Determining Optimal Data Period / Time Span for Model Training
I'm seeking advice on determining the ideal time span for optimizing a weather forecast strategy using historical data without overfitting/underfitting our model. In pursuit of optimal performance and ...
- 82.8k
2
votes
1
answer
208
views
Accuracy score change a lot by changing random seed in train/test split
I'm running a ML algorithm on some data, and I noticed that if I change the random state inside the train_test_split function, accuracy score change in a quite wide range.
For example, with random ...
- 67k
1
vote
1
answer
48
views
CV score vastly different from Train-Test score
I'm working on a multi-class classification task. I'm currently trying to tune a LGB model but have encountered a behavior that I do not understand. First, my data is from 1996 to 2015 so I split my ...
- 1,455
2
votes
1
answer
845
views
How to distinguish two versions of R-squared calculated on test set?
I've come across two ways that people calculate R-squared on a test set:
Calculate the square of the correlation between predictions and actual values (in practice, I've seen people do this in R by ...
CommunityBot
- 1