Skip to main content

Questions tagged [train-test-split]

The train-test split is a method used to estimate the performance of machine learning algorithms that are applicable for prediction-based Algorithms/Applications.

Filter by
Sorted by
Tagged with
0 votes
0 answers
19 views

Can an Anomaly Detector be Tested with Data that it Labeled?

Is it wrong to leverage a model to label data, then perform a train/test split to evaluate the performance of said model? Assume I have an unlabeled data set where the missing labels are a binary ...
noNameTed's user avatar
  • 153
1 vote
0 answers
29 views

What to do when you realize you've overfit?

This is hypothetical and I would like to hear what people do when the get to the test set and realize they've overfit. Of course, preventing overfitting in the first place is ideal. You're working on ...
user2330624's user avatar
2 votes
1 answer
16 views

weighted random forest with train/test datasets

I have a dataset where the sample distribution does not match the population distribution, but I have weights that can be applied to address that issue. I have randomly partitioned the original ...
Sarah Hardy's user avatar
0 votes
0 answers
16 views

How to Train a Model on the Whole Outer-loop Training Set in Nested Cross-validation?

I'm implementing nested cross-validation for a machine learning project and need some clarity on the training process using the outer-loop training set. Here’s a summary of my process: Outer-loop ...
Surayuth Pintawong's user avatar
0 votes
0 answers
12 views

Is GroupKFold needed if some samples have some of their feature values equal?

I am given a dataset $D$ of 10k enzyme-substrate complexes having a lock-key relationship, with each sample (complex) being characterized by enzyme features $x_e$ and substrate features $x_s$. That is,...
ado sar's user avatar
  • 471
1 vote
0 answers
33 views

How should you split up data in a train-test-validation split

I've seen it is generally recommended when using a train-test-validation data split, to first split your data into train and test datasets, and then furtherly split the train dataset into a train and ...
sammcm998's user avatar
0 votes
0 answers
17 views

Lasso regression test MSE lower than train MSE

Im currently using Lasso to build a predictive model for numeric variable . Before scaling the features I split the data for train test and validation . I have a feature named 'year' and i wanted the ...
liza read's user avatar
4 votes
2 answers
58 views

How to approach dataset splitting for building time-series models?

Suppose I have 100 observations of time series data $x_1,...,x_{100}$, and that I want to split the data into a train set, a validation set, and a test set. I know that the train set must have smaller ...
David's user avatar
  • 1,276
0 votes
1 answer
36 views

Random forest cross-validaton by patient

I have a dataset of various features from 10 patients and 10 controls. Each patient has many data points. Random forest does an amazing job in predicting whether a data point is from a patient or a ...
SuperDuperMario's user avatar
1 vote
1 answer
48 views

How do you train-test split an imbalanced dataset?

I have an imbalanced dataset and I'm trying to predict a binary target. The minority class amounts to approximately 0.4% of all observations (60 million observations from which 250K belong to the ...
Arturo Sbr's user avatar
2 votes
2 answers
95 views

Can I skip test set and train on 100% of data?

Is it a viable solution to train on the whole dataset without splitting the data into 'train' and 'test' sets? In other words, is it okay to skip offline evaluation and only perform online evaluation (...
asparagus's user avatar
0 votes
0 answers
21 views

Scaling data to a sample that is neither training nor validation. Is this data leakage?

TL;DR: Data being scaled to a sample that is neither training nor validation. Is this data leakage? Hi, I have a data set from samples that are distributed on a plate. Precisely, there are 96 wells ...
Luiz Gustavo's user avatar
1 vote
0 answers
30 views

Choosing a suitable sample size for a Random Forest Model

I know that this isn't always a straight forward question to answer, but I am working on a provincial wide wetland classification model that has $7$ classes and $32$ or so explanatory variables. In my ...
John Gallop's user avatar
6 votes
4 answers
1k views

Is it really so bad to do SMOTE on the training set before crossvalidation?

I understand that doing this leads to data leakage, but if I get better performance on the test set does it really matter? I tried using caret with ...
maglorismyspiritanimal's user avatar
1 vote
0 answers
20 views

Train/validate/test subsets of the data; which one do we use?

We are writing a scientific paper using a large dataset of healthcare data. We split our data into three sets: a train set (60%), a validation set (20%), and a test set (20%). My question is what do ...
Stata_user's user avatar
0 votes
0 answers
33 views

Data leakage in time series forecasting framed as a supervised learning problem

Suppose that I have a simple univariate time series. My goal is to use the value of 3 consecutive days to predict the value of the fourth day. I built my dataset by applying a rolling window that ...
Ray's user avatar
  • 11
1 vote
1 answer
26 views

Impute missing value [closed]

In machine learning when I impute missing values which of the following I perform : 1-Impute data set and then split it? 2-Split dataset to Training and testing datasets and then Impute each datasets ...
zhyan's user avatar
  • 335
1 vote
0 answers
39 views

Should I delete samples from the training data that are present in the testing data by accident?

I classify pairs of entities, let's say dog-cat pairs, whether there is association between them (positive class) or there is not (negative class). I have a moderately sized positive dataset (~130k ...
oliver.c's user avatar
  • 185
0 votes
0 answers
10 views

How does the training set size affect the uncertainty (variance) of performance estimation?

I am reading this paper which discusses the factors that affect the uncertainty (variance) in the performance estimation of a learner. The authors say (p. 2, "The monotonicity of the learning ...
ado sar's user avatar
  • 471
1 vote
1 answer
111 views

Should we use train, validation, or test data when creating PR/AUC curves to optimize the decision threshold?

It makes sense to me that we can use the ROC-AUC and PR-AP scores of the validation sets during CV to tune our model hyperparameter selection. And when reporting the models final performance, it makes ...
another_student's user avatar
1 vote
0 answers
14 views

Determining Optimal Data Period / Time Span for Model Training

I'm seeking advice on determining the ideal time span for optimizing a weather forecast strategy using historical data without overfitting/underfitting our model. In pursuit of optimal performance and ...
RezAm's user avatar
  • 111
0 votes
0 answers
63 views

Using pycaret's preprocessing on unseen dataset

I'm thoroughly enjoying pycaret to handle much of the legwork in my analysis. I'm making heavy use of the setup() method in preprocessing to handle normalization, ...
neanderslob's user avatar
2 votes
1 answer
104 views

Accuracy score change a lot by changing random seed in train/test split

I'm running a ML algorithm on some data, and I noticed that if I change the random state inside the train_test_split function, accuracy score change in a quite wide range. For example, with random ...
Federicofkt's user avatar
1 vote
1 answer
33 views

CV score vastly different from Train-Test score

I'm working on a multi-class classification task. I'm currently trying to tune a LGB model but have encountered a behavior that I do not understand. First, my data is from 1996 to 2015 so I split my ...
jauyjad's user avatar
  • 11
4 votes
1 answer
77 views

Split-sample analysis as a way of avoiding p-hacking?

In my lab we pre-register our hypotheses prior to running an experiment, and we also attempt to fully specify our exclusion criteria, coding practices for small categories and edge cases, etc. However,...
octern's user avatar
  • 624
0 votes
0 answers
10 views

Is a chronological testing split required for this problem?

I'm training a regression model (probably gradient boosting based) that has some time based features. Mainly in the form of the day, month, year and day_of_week. Results change based on whether its a ...
FAD's user avatar
  • 31
0 votes
0 answers
37 views

Is it normal to have a sharp increase in validation error when using 10% of the data instead of something like 20-30%?

Scenario: I'm training a relatively simple neural network to classify pairs of tabular datapoints (~150k), lets say drugs and diseases, whether they are related (positive) or not (negative). As I only ...
oliver.c's user avatar
  • 185
2 votes
0 answers
35 views

Data leakage: Train test split before or after data preprocessing? [duplicate]

A while ago I came across the word "data leakage" for the first time, and after some research, I found that it is a common mistake among data science/machine learning practitioners. But the ...
jairiidriss's user avatar
1 vote
1 answer
35 views

train / validation / test split problem

Suppose that I have created train/validation/test splits for model building. I optimized the hyperparameters using the validation set and chose the parameter values which gave the highest accuracy. To ...
Sanyo Mn's user avatar
  • 1,262
1 vote
1 answer
141 views

How reliable is ```train_test_split```? Is there a way to optimize it?

Using train_test_split is a common practice while building a Machine Learning model. Nevertheless, partitioning your dataset to get train and test samples is an ...
Juan Flautista De Torrepacheco's user avatar
5 votes
2 answers
639 views

Deleting outliers prior to data splitting or only in the training set?

I'm working on a dataset with some outliers in the response variable which are actually natural results (not errors). I want to calibrate a model which could then be used to predict on populations ...
Renaud Bied-charreton's user avatar
1 vote
0 answers
75 views

How to create representative training, validation, and test sets when working with time series data?

In my application, I am working with a relatively long time series of daily market index percentage returns (many years) and am trying to model the dependence structure of the returns from a pure time ...
QMath's user avatar
  • 451
1 vote
1 answer
82 views

Multi-label stratified split

I am working on a multilabel text classification problem. The text data is called 'cleaned_text' and has shape (92259, 1) and the one-hot encoded label data is called 'labels' and has shape (92259, 32)...
Steven Gubkin's user avatar
0 votes
0 answers
25 views

Is it common the performance in validation set being lower than the performance in test set during the training of a classifier?

I'm evaluating some settings for using graph convolutional networks for image classification. I'm using train/validation/test split of 64%/16%/20%, and adopting 5-fold cross-validation. That is, in ...
Zaratruta's user avatar
  • 1,018
0 votes
0 answers
20 views

Normalizing data when retraining on the train and validation set

It is often good practice to normalize training data for numerical stability and faster convergence. When using a train-validation-test split, it is recommended to calculate normalization parameters (...
Vityou's user avatar
  • 11
1 vote
0 answers
247 views

random split vs temporal/time based split

Some background: I want to train a regression model to predict future prices for used cars. I have about 85,000 observations collected from November 2022 to June 2023 and have around 80 different ...
Jash Shah's user avatar
  • 267
0 votes
0 answers
11 views

When to Split Data for Time Series Forecasting using Ensemble Empirical Mode Decomposition?

I want to forecast a time series data by using Ensemble Empirical Mode Decomposition (EEMD) and LSTM. However, I'm unsure about when to split the data into training and test sets. Should the data be ...
bella_pa's user avatar
  • 103
1 vote
0 answers
36 views

Splitting strategy for performing hyperparameter tuning, algorithm comparison and model validation in one experiment

Let's say that for a supervised machine learning experiment I am using a fixed learning algorithm (e.g. Random Forest), and I want to achieve the following: Choose optimal hyper-parameters for the ...
saveturn's user avatar
3 votes
1 answer
273 views

Why should I split the data when searching for outliers? (pyod)

I am using pyod to detect outliers in data, and I came across this official example: https://github.com/yzhao062/pyod/blob/master/examples/comb_example.py I have a question regarding the need to split ...
JAdel's user avatar
  • 125
4 votes
3 answers
596 views

Is it necessary to do train-test split when we are interested in understanding the model rather than predicting?

In machine learning we are taught to always do validation of some sort, for instance by creating a hold out validation set that is used to test the performance of the model. However, in some use cases ...
lalaland's user avatar
  • 247
1 vote
1 answer
54 views

Model complexity and number of examples

Is there a measure for model complexity? For given units of this measure how many examples do we need to train a network to get the model right and generalize? In essence what is the relation between ...
Justaperson's user avatar
2 votes
0 answers
46 views

The train/validation/test split does not make sense to me in cases where they all originate from a single dataset

I read everywhere that the ideal way of training a model would be to e.g.: run k-fold learning for hyperparameter optimization on 80-90% of the dataset, then test the best model on the rest. As far as ...
oliver.c's user avatar
  • 185
1 vote
1 answer
270 views

How to deal with groups when splitting a data into train and test?

Say i have a dataset with groups that i want to use for a Regression problem that looks like the following where feature1 is the group column: ...
Hamza Adnan's user avatar
3 votes
2 answers
2k views

Is it a bad practice to learn hyperparameters from the training data set?

I am working on a project where I am evaluating different machine learning models to be used as scoring functions during in-silico docking. It is a regression problem where the 3D structure data of a ...
fairy_bluebirb's user avatar
0 votes
1 answer
456 views

Is it sufficient to report only the result of cross validation in research paper?

I'm working on my master's in developing a machine-learning model to predict classes of biomedical images from a microscope. These images are collected separately from each patient. For example, I ...
Surayuth Pintawong's user avatar
0 votes
1 answer
34 views

Train Test leakage

Imagine the following dataset. 1 --> Person buys this product sometimes 0 --> Person never buys this product: ...
Tarquinius's user avatar
0 votes
1 answer
42 views

Can this approach be used for machine learning using train-test split?

So let's say I have a dataset with 1000 samples, 20 cols. Regression problem. I use train-test split, say 80-20% I create a Model, lets say Random Forest. I use gridsearchCV to find the best model ...
Sharan Shetty's user avatar
0 votes
0 answers
43 views

generalizability of training model to testing set

Let's say mtcars contain 3 independent datasets: cyl=4, cyl=6 and ...
locus's user avatar
  • 1,633
1 vote
1 answer
288 views

Why can stratified sampling to testing/training sets on strata that contain less than 10% of the entire dataset be statistically risky?

I'm trying to split my data into a testing and a training set. There are lots of variables that I want to ensure are well represented in both the training and testing sets (say, 15 covariates). But ...
Aegis's user avatar
  • 183
0 votes
1 answer
105 views

Capping before or after splitting the data into train and test?

I have a data set with N ~ 9000 and about 50% missing on at least one important variable. There are more than 50 continuous variables and for each variable, the values after 95th percentile seems ...
Ritik P. Nayak's user avatar