Questions tagged [train-test-split]

The train-test split is a method used to estimate the performance of machine learning algorithms that are applicable for prediction-based Algorithms/Applications.

Filter by
Sorted by
Tagged with
0 votes
0 answers
11 views

multiple datasets train-val-test split for time series

Suppose that I have data with dimension $(N,H,F)$, where $N$ represents the number of different datasets, $H$ is the history size and $F$ is the input size. how would you split it into a train-...
Hadar's user avatar
  • 125
0 votes
0 answers
25 views

At what point during model development can model calibration be applied?

I have been working on prediction models in R studio based on a rather small data set. There is a total of ~ 1200 cases with 150 to 400 positive cases depending on which of the different outcomes is ...
mmo's user avatar
  • 11
0 votes
0 answers
14 views

training vs test data sizes

If I have a training set of 10k samples (balanced--two classes) and 20k test samples (imbalanced-- 5% versus 95%), could I compare their ROCs for overfitting/underfitting? I guess not bu I need more ...
user3441553's user avatar
0 votes
0 answers
16 views

What is the explanation of splitting an Out-Of-Time Sample into train and test?

I'm working with a dataset to train a credit risk model. The dataset already has flags for how it should be split so results can be reproduced, it has a column with two values ...
sgaseretto's user avatar
0 votes
0 answers
33 views

train, test and validation data set configuration to compare a machine learing method and ARMA method

I want to compare performance a machine learning method and Autoreressive Moving Average-ARMA(p,q) for time series data. I do such a configuration: First I divide data into three part: Trainig data(...
oercim's user avatar
  • 659
2 votes
0 answers
40 views

The train/validation/test split does not make sense to me in cases where they all originate from a single dataset

I read everywhere that the ideal way of training a model would be to e.g.: run k-fold learning for hyperparameter optimization on 80-90% of the dataset, then test the best model on the rest. As far as ...
oliver.c's user avatar
0 votes
0 answers
5 views

Item based collaborative filtering. Since is an unsupervised task should I do training/test split or not?

I'm learning about recommentation systems and I'm trying to build one using item based collaborative filtering approach. I have this dataset in which the lines correspond the items and columns the ...
Antonio Caipora's user avatar
0 votes
0 answers
15 views

cross-validation and test-set: variance estimate

i am confused at how to estimate the variance of a classifier. Currently, i have split my data into training and test sets and used the training data with a k-fold cross-validation strategy to get the ...
Hattori's user avatar
  • 151
0 votes
0 answers
46 views

How to deal with groups when splitting a data into train and test?

Say i have a dataset with groups that i want to use for a Regression problem that looks like the following where feature1 is the group column: ...
Hamza Adnan's user avatar
3 votes
2 answers
1k views

Is it a bad practice to learn hyperparameters from the training data set?

I am working on a project where I am evaluating different machine learning models to be used as scoring functions during in-silico docking. It is a regression problem where the 3D structure data of a ...
fairy_bluebirb's user avatar
0 votes
0 answers
47 views

Is it sufficient to report only the result of cross validation in research paper?

I'm working on my master's in developing a machine-learning model to predict classes of biomedical images from a microscope. These images are collected separately from each patient. For example, I ...
Surayuth Pintawong's user avatar
0 votes
1 answer
21 views

Train Test leakage

Imagine the following dataset. 1 --> Person buys this product sometimes 0 --> Person never buys this product: ...
Tarquinius's user avatar
0 votes
0 answers
17 views

Make predictions with Time Series, starting from the date that the training set starts

Good evening everyone, I have a question about time series, I have a challenge that was given to me at work to work with a dataset, with a very short time series, of 365 days, I am using the darts ...
eddi's user avatar
  • 21
0 votes
0 answers
62 views

Train/Test split before or after CountVectorizer and TfidfVectorizer?

I'm learning about machine learning and I have two questions about nlp. Considering a dataset with many texts. Should I split it in train/test set before or after use CountVectorizer? I'm asking this ...
Antonio Caipora's user avatar
0 votes
1 answer
23 views

Can this approach be used for machine learning using train-test split?

So let's say I have a dataset with 1000 samples, 20 cols. Regression problem. I use train-test split, say 80-20% I create a Model, lets say Random Forest. I use gridsearchCV to find the best model ...
Sharan Shetty's user avatar
0 votes
0 answers
27 views

Timeseries regression model forecasting including test data?

I have a multiple timeseries on which I want to train a timeseries regression model. There are past covariates from the data that will be used for model training. I am confused how the flow for ...
albin02t's user avatar
0 votes
0 answers
40 views

generalizability of training model to testing set

Let's say mtcars contain 3 independent datasets: cyl=4, cyl=6 and ...
locus's user avatar
  • 1,159
1 vote
1 answer
94 views

Why can stratified sampling to testing/training sets on strata that contain less than 10% of the entire dataset be statistically risky?

I'm trying to split my data into a testing and a training set. There are lots of variables that I want to ensure are well represented in both the training and testing sets (say, 15 covariates). But ...
Aegis's user avatar
  • 173
0 votes
0 answers
21 views

Should I force duplicates into the same bucket when performing a train/test split?

I'm embarking on an NLP project building a classifier - predict the category based on the description. The text descriptions I have are very short and there are frequent duplicates amongst them. After ...
Tom Wagstaff's user avatar
0 votes
1 answer
42 views

Capping before or after splitting the data into train and test?

I have a data set with N ~ 9000 and about 50% missing on at least one important variable. There are more than 50 continuous variables and for each variable, the values after 95th percentile seems ...
Ritik P. Nayak's user avatar
1 vote
1 answer
49 views

Is my regression network overfitting or is my simulated MRI training data unrepresentative of real MRI val/test data?

I have been working on a regression task for the past one year and I am stuck. My project is to simulate voxel-wise MRI data using a physics-based function and train a neural network using that data ...
Krithika Balaji's user avatar
1 vote
0 answers
34 views

Correct way to split data for propensity modeling

For generic propensity (purchase, churn etc.) modeling a lot of typical references / examples available use randomized splitting for train / eval / test sets. For propensity modeling in practice ...
permustats's user avatar
0 votes
1 answer
19 views

Is using the same person as data observation in different time stamps a way to produce data leakage in a Machine Learning model?

Let's assume we are going to train a regression model (could be any ML tabular solution for regression. Ex.: LGBM, XGBoost, Perceptron, ...) to predict a customer profit in the next month. While ...
Matheus Nascimento's user avatar
0 votes
0 answers
16 views

Post training calibration of BNN and dataset split

since i am working with a small dataset (1048 rows) with much of the data concentrated in the region for which i have no interest, i was wondering if for Bayesian neural networks it is necessary to ...
Alucard's user avatar
  • 305
0 votes
0 answers
55 views

Recursive time series forecasting test set

Problem: I am building a multivariate model for recursive time series forecasting, where the goal is to make a 4-step-ahead forecast. The actual data for the forecasting period is available. As far as ...
Matthew.M's user avatar
0 votes
0 answers
64 views

Grouped stratified train-val-test split for a multilabel dataset

I was wondering if there is a fast heuristic algorithm for performing grouped stratified dataset split on a multilabel dataset. Question originally posted on Data Science stackexcahnge here. ...
jasperhyp's user avatar
1 vote
0 answers
59 views

Data augmentation before the test/train split on a small dataset

First of all im working with a project that is not mine and cointains 0 explanatios about why things were done and I fear some of them are just mistakes. I found that for the trainig of a ...
Alvar's user avatar
  • 11
1 vote
1 answer
45 views

Do I remove outliers within training set or duplicate of original?

I want to predict on a test set. I have created a binary logistic regression using my current training set and have predicted on the test set. The dataset I used to split has 299 observations. What if ...
Antonio's user avatar
  • 403
0 votes
0 answers
9 views

Using rerandomization/minimization for train/test splits?

When splitting data into train/test subsets, simple randomization will make the data distributions the same on average, but in finite samples, we may want to do more to ensure similarity. Stratified ...
BeingQuisitive's user avatar
0 votes
0 answers
54 views

Train/test split for model stacking

I am quite new to model stacking. Suppose that I have a have a model stack with two sub-models and one meta-model. I know that the meta-model will make predictions based on the outputs of the sub-...
DavidSilverberg's user avatar
0 votes
1 answer
175 views

Does survival analysis require you to split the data?

I have some data which I'm using to predict the potential outcome of a new applicant defaulting on their credit loan. For Kaplan-Meier, I don't believe there would be a need for splitting the data as ...
Antonio's user avatar
  • 403
1 vote
0 answers
45 views

Forecasting accuracy of VECM using train and test

I am forecasting using VECM and I plan to do it on train and test split data. My data is 132 monthly observations. My VECM is lag 3 with unrestricted constant. All diagnostic test are passed. I plan ...
Sandy's user avatar
  • 11
0 votes
0 answers
15 views

Comparing models using different training sets

I try to compare the forecasting performance of several models. I do it for two situations: normal and extreme cases. My dataset set is not big. One of the models (gradient boosting) requiers a ...
Cateded Ur's user avatar
0 votes
1 answer
29 views

Size of Final TestSet

is there a rule of thumb of how large the final test set has to be in Machine Learning? Assumed I have 1.000 images how many images do I ignore and use only in the final run? My proposal: Select ...
SchwarzbrotMitHummus's user avatar
0 votes
0 answers
83 views

Splitting the dataset before GridsearchCV

I have a dataset of 390 rows. I have created a pipeline consisting of scaling, selecting features, and modelling. I am using GridSearchCV (k=5) to find the optimal scaling method and to tune the ...
MxML's user avatar
  • 11
1 vote
0 answers
15 views

Which one to choose: subject-wise or record-wise

I want to do some classifications on open access "Activity Recognition Using Wearable Physiological Measurements" data set. In this dataset, there are 40 subjects whose Electrocardiogram (...
Sara's user avatar
  • 111
0 votes
1 answer
18 views

What is the purpose of separate validation/testing subsets in k-fold cross validation?

So it turns out I have misunderstood what k-fold CV actually does. I had originally thought that (e.g.) 5-fold CV splits the whole dataset into 5 subsets, then on each iteration the model is trained ...
T.Murray's user avatar
0 votes
0 answers
14 views

Accelerated failure time model: train & prediction data leakage?

I want to predict survival time for each subject by collecting all subjects since the very beginning to now for training. However, I will continue to predict on subjects that are alive in my ...
Yi Mao's user avatar
  • 1
2 votes
1 answer
70 views

Scaling the same continuous feature both in train and test

I'm building a classification model to predict some target variable. I have only one continuous feature (age) that I am interested in scaling. I split my data into train and test sets, I scale this ...
Programming Noob's user avatar
0 votes
2 answers
564 views

Should I use transformer.fit_transform(X_test, y_test) or not?

tl-dr: The function model.fit() is different from transformer.fit(). My idea is to make all transformations needed on the training set and after that on the test set with fit_transform in both. Hi! I'...
Antonio Caipora's user avatar
0 votes
0 answers
20 views

How come model performance score approaches perfection as test size decreases?

When I test my random forest classifier with the following test sizes: [0.001, 0.05, 0.1, 0.25, 0.4] I get the following macro-F1 scores: ...
Oskar's user avatar
  • 1
1 vote
3 answers
237 views

What is the benefit of k-fold cross validation?

I understand that using 100% of the dataset and doing k-fold cross validation instead of train_test_split would eliminate that randomness the latter method have in splitting and thus potentially avoid ...
MxML's user avatar
  • 11
0 votes
0 answers
23 views

Trained EfficientNetB0 does not seem to make an impact on evaluation

I am using EfficientNetB0 to detect steganography in an image. It has been previously described in a research paper that under certain conditions it can work. The parameters for the training are $...
Martin Benes's user avatar
1 vote
1 answer
17 views

Training models with estimated features

A model (model X) is being trained with a set of features. One of the features (say, feature A) is not available on test data. But it can be accurately estimated using another model (model Y) using ...
Pedro Schuller's user avatar
0 votes
0 answers
23 views

Usability and bias of test set for limited data

I understand that optimizing model parameters, for example in k-fold cross-validation should be done in absence of a final test set in order not to overfit the model. Only at the very end, the model ...
mirei's user avatar
  • 63
1 vote
1 answer
247 views

Machine Learning for Time Series: Train and test set overlap due to lagged target as feature – problem of data leakage?

Situation: My objective is to apply Machine Learning (for regression problems). Therefore, I have a panel dataset of time series with daily fund data from 2018-01-01...
Maxzl's user avatar
  • 11
0 votes
0 answers
13 views

Bulding a labelling dataset and modelling categorical features

Summary: How do I ensure that sample I create for labelling is representative enough and would be appropriate for modelling, given I cannot include all feature combinations in it. I have a tabular ...
Dimitar Argirov's user avatar
0 votes
1 answer
60 views

rep k fold cross validation, train test split and overfitting

I've recently gotten into ML and I'm a bit confused about rep k fold cross validation, train, test split and overfitting. I have already read some of the posts in this forum, but none of them could ...
Domi's user avatar
  • 1
1 vote
3 answers
200 views

On the usage of many train/test split ratios

Is there any benefit in evaluating model performance under different train/test ratios? To me, this sounds kind of nonsense because of these reasons: I cannot access such a parameter when the model ...
Marco Repetto's user avatar
1 vote
3 answers
80 views

How to design cross-validation and testing scheme when N is small?

I have a binary classification problem with 60 samples (N=60). 40 are responders (+) and 20 are non-responders (-) to a drug. There will be ~20 measured features (p=20) per sample with which to make ...
zachursinus's user avatar