Questions tagged [train-test-split]
The train-test split is a method used to estimate the performance of machine learning algorithms that are applicable for prediction-based Algorithms/Applications.
130 questions
1
vote
1
answer
77
views
Model complexity and number of examples
Is there a measure for model complexity?
For given units of this measure how many examples do we need to train a network to get the model right and generalize?
In essence what is the relation between ...
- 121
2
votes
0
answers
47
views
The train/validation/test split does not make sense to me in cases where they all originate from a single dataset
I read everywhere that the ideal way of training a model would be to e.g.: run k-fold learning for hyperparameter optimization on 80-90% of the dataset, then test the best model on the rest. As far as ...
- 185
1
vote
1
answer
375
views
How to deal with groups when splitting a data into train and test?
Say i have a dataset with groups that i want to use for a Regression problem that looks like the following where feature1 is the group column:
...
- 11
3
votes
2
answers
2k
views
Is it a bad practice to learn hyperparameters from the training data set?
I am working on a project where I am evaluating different machine learning models to be used as scoring functions during in-silico docking. It is a regression problem where the 3D structure data of a ...
0
votes
1
answer
580
views
Is it sufficient to report only the result of cross validation in research paper?
I'm working on my master's in developing a machine-learning model to predict classes of biomedical images from a microscope. These images are
collected separately from each patient. For example, I ...
0
votes
1
answer
36
views
Train Test leakage
Imagine the following dataset. 1 --> Person buys this product sometimes 0 --> Person never buys this product:
...
- 101
0
votes
1
answer
49
views
Can this approach be used for machine learning using train-test split?
So let's say I have a dataset with 1000 samples, 20 cols. Regression problem.
I use train-test split, say 80-20%
I create a Model, lets say Random Forest. I use gridsearchCV to find the best model ...
0
votes
0
answers
45
views
generalizability of training model to testing set
Let's say mtcars contain 3 independent datasets: cyl=4, cyl=6 and ...
- 1,629
1
vote
1
answer
482
views
Why can stratified sampling to testing/training sets on strata that contain less than 10% of the entire dataset be statistically risky?
I'm trying to split my data into a testing and a training set. There are lots of variables that I want to ensure are well represented in both the training and testing sets (say, 15 covariates). But ...
- 183
0
votes
1
answer
129
views
Capping before or after splitting the data into train and test?
I have a data set with N ~ 9000 and about 50% missing on at least one important variable. There are more than 50 continuous variables and for each variable, the values after 95th percentile seems ...
- 313
1
vote
1
answer
63
views
Is my regression network overfitting or is my simulated MRI training data unrepresentative of real MRI val/test data?
I have been working on a regression task for the past one year and I am stuck. My project is to simulate voxel-wise MRI data using a physics-based function and train a neural network using that data ...
1
vote
0
answers
56
views
Correct way to split data for propensity modeling
For generic propensity (purchase, churn etc.) modeling a lot of typical references / examples available use randomized splitting for train / eval / test sets. For propensity modeling in practice ...
- 11
0
votes
1
answer
28
views
Is using the same person as data observation in different time stamps a way to produce data leakage in a Machine Learning model?
Let's assume we are going to train a regression model (could be any ML tabular solution for regression. Ex.: LGBM, XGBoost, Perceptron, ...) to predict a customer profit in the next month. While ...
0
votes
0
answers
102
views
Recursive time series forecasting test set
Problem: I am building a multivariate model for recursive time series forecasting, where the goal is to make a 4-step-ahead forecast. The actual data for the forecasting period is available.
As far as ...
0
votes
0
answers
452
views
Grouped stratified train-val-test split for a multilabel dataset
I was wondering if there is a fast heuristic algorithm for performing grouped stratified dataset split on a multilabel dataset. Question originally posted on Data Science stackexcahnge here.
...
- 21
1
vote
0
answers
88
views
Data augmentation before the test/train split on a small dataset
First of all im working with a project that is not mine and cointains 0 explanatios about why things were done and I fear some of them are just mistakes.
I found that for the trainig of a ...
- 11
1
vote
1
answer
251
views
Do I remove outliers within training set or duplicate of original?
I want to predict on a test set.
I have created a binary logistic regression using my current training set and have predicted on the test set. The dataset I used to split has 299 observations.
What if ...
- 673
1
vote
1
answer
804
views
Does survival analysis require you to split the data?
I have some data which I'm using to predict the potential outcome of a new applicant defaulting on their credit loan.
For Kaplan-Meier, I don't believe there would be a need for splitting the data as ...
- 673
1
vote
0
answers
90
views
Forecasting accuracy of VECM using train and test
I am forecasting using VECM and I plan to do it on train and test split data. My data is 132 monthly observations. My VECM is lag 3 with unrestricted constant. All diagnostic test are passed.
I plan ...
- 11
0
votes
0
answers
24
views
Comparing models using different training sets
I try to compare the forecasting performance of several models. I do it for two situations: normal and extreme cases. My dataset set is not big.
One of the models (gradient boosting) requiers a ...
- 111
0
votes
1
answer
29
views
Size of Final TestSet
is there a rule of thumb of how large the final test set has to be in Machine Learning?
Assumed I have 1.000 images how many images do I ignore and use only in the final run?
My proposal:
Select ...
1
vote
0
answers
19
views
Which one to choose: subject-wise or record-wise
I want to do some classifications on open access "Activity Recognition Using Wearable Physiological Measurements" data set. In this dataset, there are 40 subjects whose Electrocardiogram (...
- 111
2
votes
1
answer
95
views
What is the purpose of separate validation/testing subsets in k-fold cross validation?
So it turns out I have misunderstood what k-fold CV actually does. I had originally thought that (e.g.) 5-fold CV splits the whole dataset into 5 subsets, then on each iteration the model is trained ...
- 81
2
votes
1
answer
386
views
Scaling the same continuous feature both in train and test
I'm building a classification model to predict some target variable.
I have only one continuous feature (age) that I am interested in scaling. I split my data into train and test sets, I scale this ...
- 689
-1
votes
2
answers
2k
views
Should I use transformer.fit_transform(X_test, y_test) or not?
tl-dr: The function model.fit() is different from transformer.fit(). My idea is to make all transformations needed on the training set and after that on the test set with fit_transform in both.
Hi!
I'...
2
votes
3
answers
953
views
What is the benefit of k-fold cross validation?
I understand that using 100% of the dataset and doing k-fold cross validation instead of train_test_split would eliminate that randomness the latter method have in splitting and thus potentially avoid ...
- 71
1
vote
1
answer
27
views
Training models with estimated features
A model (model X) is being trained with a set of features. One of the features (say, feature A) is not available on test data. But it can be accurately estimated using another model (model Y) using ...
- 326
3
votes
1
answer
805
views
Machine Learning for Time Series: Train and test set overlap due to lagged target as feature – problem of data leakage?
Situation:
My objective is to apply Machine Learning (for regression problems). Therefore, I have a panel dataset of time series with daily fund data from 2018-01-01...
- 31
0
votes
0
answers
14
views
Bulding a labelling dataset and modelling categorical features
Summary: How do I ensure that sample I create for labelling is representative enough and would be appropriate for modelling, given I cannot include all feature combinations in it.
I have a tabular ...
0
votes
1
answer
149
views
rep k fold cross validation, train test split and overfitting
I've recently gotten into ML and I'm a bit confused about rep k fold cross validation, train, test split and overfitting. I have already read some of the posts in this forum, but none of them could ...
- 1
1
vote
3
answers
314
views
On the usage of many train/test split ratios
Is there any benefit in evaluating model performance under different train/test ratios?
To me, this sounds kind of nonsense because of these reasons:
I cannot access such a parameter when the model ...
1
vote
3
answers
138
views
How to design cross-validation and testing scheme when N is small?
I have a binary classification problem with 60 samples (N=60). 40 are responders (+) and 20 are non-responders (-) to a drug. There will be ~20 measured features (p=20) per sample with which to make ...
- 25
2
votes
1
answer
282
views
Non-uniform data: how to build a prediction model with high performance on unseen groups of data?
How to deal with non-uniform data?
There are groups in my data. Samples in the same group share the majority of feature values.
What approach to model training can I use to guarantee high performance ...
- 33
1
vote
1
answer
51
views
How many independet test-train splits (with independent training) should I perform?
I recently read some literature by Bagnall et al https://arxiv.org/pdf/1602.01711.pdf "The Great Time Series Classification Bake Off". If I understand them correctly, they advise to perform ...
- 1,783
0
votes
1
answer
110
views
Train/Test splitting for seasonal adjustment
At work, I just started dealing with seasonal adjustment of monthly time series on credit data, so being new to the topic it is quite possible that my question is pretty trivial.
From what I've read ...
- 1
1
vote
0
answers
102
views
0
votes
1
answer
696
views
Should AIC be reported on training or test data? [duplicate]
I have a handful of logistic regression models and I would like to report AIC. Should I report it on training or test data? I have quite a big dataset and a maximum of 10 predictors in any of the ...
- 139
1
vote
1
answer
537
views
How to properly impute values on the test set using imputer (missForest)
I'm trying to impute some missing values on my dataset $X$. So first I shuffle and split data to obatin the train set X_train and the test set ...
- 380
2
votes
2
answers
161
views
Train test split based on statistics
I would like to know if there is a method of splitting that is not random but based on the distribution of the train and test data samples feauture values.
Currently I am splitting randomly but ...
- 33
0
votes
0
answers
25
views
Running model on full dataset or just test set?
I'm new to using XGBoost and I'm confused about how we should obtain the XGBoost predicted values for each data point. For example, the process for fitting and evaluating an XGBoost model is:
...
- 65
1
vote
1
answer
953
views
Why don’t we split the dataset into training and testing set if the sample size is small?
I learned in school that we don't split the dataset into training and testing sets if the sample size is less than 30. I wonder why we don't?
- 181
2
votes
2
answers
418
views
few images in validation and test set
I've a dataset with about 123 images (two categories, 19 defect and 104 no defect).
I've to implement a classifier so I've decided to split my data in train (70% of all data), validation (20% of all ...
- 41
3
votes
1
answer
663
views
Train/test split on time-based data with lagged features
I am working with data on bank transactions, and am using RFM (recency/frequency/monetary value) features like days since last transaction, number of transactions last n days, average value of ...
- 31
0
votes
0
answers
84
views
Statistical comparison of two probabilistic classifiers
TL;DR There are well-known tests to compare classifiers. How can we generalize to classifiers with random training steps?
I am comparing two classification algorithms (A and B). I can see that ...
1
vote
1
answer
355
views
Is cross-validation with no data leakage sufficient to replace train-test split?
I would like to seek expert advice on the topic above.
I was taught to follow this workflow:
Split dataset into training and testing
Use training dataset to develop model
Set hyperparameter in model
...
- 13
1
vote
0
answers
41
views
How do I perform a train-validation split on data with class imbalance such that the class imbalance ratio is preserved?
My data has class imbalance-- that is, some classes have significantly fewer training samples than the others. I want to perform a train-validation split in such as way that the class ratios are ...
- 23
1
vote
0
answers
123
views
Strategy for Train/Test-Split on Video Sequences
My dataset consists of 15 video sequences, each sequence showing a different movement. I want to train a CNN to detect poses (e.g. standing, sitting, ...) on single frames of this dataset but struggle ...
- 11
2
votes
1
answer
110
views
How does changing the training/test ratio, affect the RSE ans R2 metrics?
If I change the training/test ratio from 90/10 to say 80/20 or 70/30, how does that affect the RSE and R2 metrics? I see changes in the results of the RSE and R2 when I implement the model, but I ...
- 21
2
votes
1
answer
845
views
How to distinguish two versions of R-squared calculated on test set?
I've come across two ways that people calculate R-squared on a test set:
Calculate the square of the correlation between predictions and actual values (in practice, I've seen people do this in R by ...
- 4,404
-2
votes
2
answers
1k
views
Why don’t we train on the test dataset in machine learning?
Can somebody explain why we can not train the dataset on the test dataset?
- 111