Questions tagged [train-test-split]
The train-test split is a method used to estimate the performance of machine learning algorithms that are applicable for prediction-based Algorithms/Applications.
99
questions
0
votes
0
answers
11
views
multiple datasets train-val-test split for time series
Suppose that I have data with dimension $(N,H,F)$, where $N$ represents the number of different datasets, $H$ is the history size and $F$ is the input size. how would you split it into a train-...
- 125
0
votes
0
answers
25
views
At what point during model development can model calibration be applied?
I have been working on prediction models in R studio based on a rather small data set. There is a total of ~ 1200 cases with 150 to 400 positive cases depending on which of the different outcomes is ...
- 11
0
votes
0
answers
14
views
training vs test data sizes
If I have a training set of 10k samples (balanced--two classes) and 20k test samples (imbalanced-- 5% versus 95%), could I compare their ROCs for overfitting/underfitting? I guess not bu I need more ...
- 11
0
votes
0
answers
16
views
What is the explanation of splitting an Out-Of-Time Sample into train and test?
I'm working with a dataset to train a credit risk model. The dataset already has flags for how it should be split so results can be reproduced, it has a column with two values ...
- 121
0
votes
0
answers
33
views
train, test and validation data set configuration to compare a machine learing method and ARMA method
I want to compare performance a machine learning method and Autoreressive Moving Average-ARMA(p,q) for time series data.
I do such a configuration:
First I divide data into three part:
Trainig data(...
- 659
2
votes
0
answers
40
views
The train/validation/test split does not make sense to me in cases where they all originate from a single dataset
I read everywhere that the ideal way of training a model would be to e.g.: run k-fold learning for hyperparameter optimization on 80-90% of the dataset, then test the best model on the rest. As far as ...
- 65
0
votes
0
answers
5
views
Item based collaborative filtering. Since is an unsupervised task should I do training/test split or not?
I'm learning about recommentation systems and I'm trying to build one using item based collaborative filtering approach. I have this dataset in which the lines correspond the items and columns the ...
0
votes
0
answers
15
views
cross-validation and test-set: variance estimate
i am confused at how to estimate the variance of a classifier. Currently, i have split my data into training and test sets and used the training data with a k-fold cross-validation strategy to get the ...
- 151
0
votes
0
answers
46
views
How to deal with groups when splitting a data into train and test?
Say i have a dataset with groups that i want to use for a Regression problem that looks like the following where feature1 is the group column:
...
3
votes
2
answers
1k
views
Is it a bad practice to learn hyperparameters from the training data set?
I am working on a project where I am evaluating different machine learning models to be used as scoring functions during in-silico docking. It is a regression problem where the 3D structure data of a ...
0
votes
0
answers
47
views
Is it sufficient to report only the result of cross validation in research paper?
I'm working on my master's in developing a machine-learning model to predict classes of biomedical images from a microscope. These images are
collected separately from each patient. For example, I ...
0
votes
1
answer
21
views
Train Test leakage
Imagine the following dataset. 1 --> Person buys this product sometimes 0 --> Person never buys this product:
...
- 101
0
votes
0
answers
17
views
Make predictions with Time Series, starting from the date that the training set starts
Good evening everyone, I have a question about time series, I have a challenge that was given to me at work to work with a dataset, with a very short time series, of 365 days, I am using the darts ...
- 21
0
votes
0
answers
62
views
Train/Test split before or after CountVectorizer and TfidfVectorizer?
I'm learning about machine learning and I have two questions about nlp.
Considering a dataset with many texts. Should I split it in train/test set before or after use CountVectorizer? I'm asking this ...
0
votes
1
answer
23
views
Can this approach be used for machine learning using train-test split?
So let's say I have a dataset with 1000 samples, 20 cols. Regression problem.
I use train-test split, say 80-20%
I create a Model, lets say Random Forest. I use gridsearchCV to find the best model ...
0
votes
0
answers
27
views
Timeseries regression model forecasting including test data?
I have a multiple timeseries on which I want to train a timeseries regression model. There are past covariates from the data that will be used for model training. I am confused how the flow for ...
- 23
0
votes
0
answers
40
views
generalizability of training model to testing set
Let's say mtcars contain 3 independent datasets: cyl=4, cyl=6 and ...
- 1,159
1
vote
1
answer
94
views
Why can stratified sampling to testing/training sets on strata that contain less than 10% of the entire dataset be statistically risky?
I'm trying to split my data into a testing and a training set. There are lots of variables that I want to ensure are well represented in both the training and testing sets (say, 15 covariates). But ...
- 173
0
votes
0
answers
21
views
Should I force duplicates into the same bucket when performing a train/test split?
I'm embarking on an NLP project building a classifier - predict the category based on the description. The text descriptions I have are very short and there are frequent duplicates amongst them. After ...
- 245
0
votes
1
answer
42
views
Capping before or after splitting the data into train and test?
I have a data set with N ~ 9000 and about 50% missing on at least one important variable. There are more than 50 continuous variables and for each variable, the values after 95th percentile seems ...
- 163
1
vote
1
answer
49
views
Is my regression network overfitting or is my simulated MRI training data unrepresentative of real MRI val/test data?
I have been working on a regression task for the past one year and I am stuck. My project is to simulate voxel-wise MRI data using a physics-based function and train a neural network using that data ...
1
vote
0
answers
34
views
Correct way to split data for propensity modeling
For generic propensity (purchase, churn etc.) modeling a lot of typical references / examples available use randomized splitting for train / eval / test sets. For propensity modeling in practice ...
- 11
0
votes
1
answer
19
views
Is using the same person as data observation in different time stamps a way to produce data leakage in a Machine Learning model?
Let's assume we are going to train a regression model (could be any ML tabular solution for regression. Ex.: LGBM, XGBoost, Perceptron, ...) to predict a customer profit in the next month. While ...
0
votes
0
answers
16
views
Post training calibration of BNN and dataset split
since i am working with a small dataset (1048 rows) with much of the data concentrated in the region for which i have no interest, i was wondering if for Bayesian neural networks it is necessary to ...
- 305
0
votes
0
answers
55
views
Recursive time series forecasting test set
Problem: I am building a multivariate model for recursive time series forecasting, where the goal is to make a 4-step-ahead forecast. The actual data for the forecasting period is available.
As far as ...
0
votes
0
answers
64
views
Grouped stratified train-val-test split for a multilabel dataset
I was wondering if there is a fast heuristic algorithm for performing grouped stratified dataset split on a multilabel dataset. Question originally posted on Data Science stackexcahnge here.
...
- 21
1
vote
0
answers
59
views
Data augmentation before the test/train split on a small dataset
First of all im working with a project that is not mine and cointains 0 explanatios about why things were done and I fear some of them are just mistakes.
I found that for the trainig of a ...
- 11
1
vote
1
answer
45
views
Do I remove outliers within training set or duplicate of original?
I want to predict on a test set.
I have created a binary logistic regression using my current training set and have predicted on the test set. The dataset I used to split has 299 observations.
What if ...
- 403
0
votes
0
answers
9
views
Using rerandomization/minimization for train/test splits?
When splitting data into train/test subsets, simple randomization will make the data distributions the same on average, but in finite samples, we may want to do more to ensure similarity. Stratified ...
- 116
0
votes
0
answers
54
views
Train/test split for model stacking
I am quite new to model stacking. Suppose that I have a have a model stack with two sub-models and one meta-model. I know that the meta-model will make predictions based on the outputs of the sub-...
- 924
0
votes
1
answer
175
views
Does survival analysis require you to split the data?
I have some data which I'm using to predict the potential outcome of a new applicant defaulting on their credit loan.
For Kaplan-Meier, I don't believe there would be a need for splitting the data as ...
- 403
1
vote
0
answers
45
views
Forecasting accuracy of VECM using train and test
I am forecasting using VECM and I plan to do it on train and test split data. My data is 132 monthly observations. My VECM is lag 3 with unrestricted constant. All diagnostic test are passed.
I plan ...
- 11
0
votes
0
answers
15
views
Comparing models using different training sets
I try to compare the forecasting performance of several models. I do it for two situations: normal and extreme cases. My dataset set is not big.
One of the models (gradient boosting) requiers a ...
- 111
0
votes
1
answer
29
views
Size of Final TestSet
is there a rule of thumb of how large the final test set has to be in Machine Learning?
Assumed I have 1.000 images how many images do I ignore and use only in the final run?
My proposal:
Select ...
0
votes
0
answers
83
views
Splitting the dataset before GridsearchCV
I have a dataset of 390 rows. I have created a pipeline consisting of scaling, selecting features, and modelling.
I am using GridSearchCV (k=5) to find the optimal scaling method and to tune the ...
- 11
1
vote
0
answers
15
views
Which one to choose: subject-wise or record-wise
I want to do some classifications on open access "Activity Recognition Using Wearable Physiological Measurements" data set. In this dataset, there are 40 subjects whose Electrocardiogram (...
- 111
0
votes
1
answer
18
views
What is the purpose of separate validation/testing subsets in k-fold cross validation?
So it turns out I have misunderstood what k-fold CV actually does. I had originally thought that (e.g.) 5-fold CV splits the whole dataset into 5 subsets, then on each iteration the model is trained ...
- 31
0
votes
0
answers
14
views
Accelerated failure time model: train & prediction data leakage?
I want to predict survival time for each subject by collecting all subjects since the very beginning to now for training. However, I will continue to predict on subjects that are alive in my ...
- 1
2
votes
1
answer
70
views
Scaling the same continuous feature both in train and test
I'm building a classification model to predict some target variable.
I have only one continuous feature (age) that I am interested in scaling. I split my data into train and test sets, I scale this ...
- 409
0
votes
2
answers
564
views
Should I use transformer.fit_transform(X_test, y_test) or not?
tl-dr: The function model.fit() is different from transformer.fit(). My idea is to make all transformations needed on the training set and after that on the test set with fit_transform in both.
Hi!
I'...
0
votes
0
answers
20
views
How come model performance score approaches perfection as test size decreases?
When I test my random forest classifier with the following test sizes:
[0.001, 0.05, 0.1, 0.25, 0.4]
I get the following macro-F1 scores:
...
- 1
1
vote
3
answers
237
views
What is the benefit of k-fold cross validation?
I understand that using 100% of the dataset and doing k-fold cross validation instead of train_test_split would eliminate that randomness the latter method have in splitting and thus potentially avoid ...
- 11
0
votes
0
answers
23
views
Trained EfficientNetB0 does not seem to make an impact on evaluation
I am using EfficientNetB0 to detect steganography in an image. It has been previously described in a research paper that under certain conditions it can work.
The parameters for the training are
$...
- 81
1
vote
1
answer
17
views
Training models with estimated features
A model (model X) is being trained with a set of features. One of the features (say, feature A) is not available on test data. But it can be accurately estimated using another model (model Y) using ...
- 316
0
votes
0
answers
23
views
Usability and bias of test set for limited data
I understand that optimizing model parameters, for example in k-fold cross-validation should be done in absence of a final test set in order not to overfit the model. Only at the very end, the model ...
- 63
1
vote
1
answer
247
views
Machine Learning for Time Series: Train and test set overlap due to lagged target as feature – problem of data leakage?
Situation:
My objective is to apply Machine Learning (for regression problems). Therefore, I have a panel dataset of time series with daily fund data from 2018-01-01...
- 11
0
votes
0
answers
13
views
Bulding a labelling dataset and modelling categorical features
Summary: How do I ensure that sample I create for labelling is representative enough and would be appropriate for modelling, given I cannot include all feature combinations in it.
I have a tabular ...
0
votes
1
answer
60
views
rep k fold cross validation, train test split and overfitting
I've recently gotten into ML and I'm a bit confused about rep k fold cross validation, train, test split and overfitting. I have already read some of the posts in this forum, but none of them could ...
- 1
1
vote
3
answers
200
views
On the usage of many train/test split ratios
Is there any benefit in evaluating model performance under different train/test ratios?
To me, this sounds kind of nonsense because of these reasons:
I cannot access such a parameter when the model ...
1
vote
3
answers
80
views
How to design cross-validation and testing scheme when N is small?
I have a binary classification problem with 60 samples (N=60). 40 are responders (+) and 20 are non-responders (-) to a drug. There will be ~20 measured features (p=20) per sample with which to make ...
- 25