Questions tagged [train-test-split]
The train-test-split tag has no usage guidance.
43
questions
1
vote
0
answers
12
views
How do I perform a train-validation split on data with class imbalance such that the class imbalance ratio is preserved?
My data has class imbalance-- that is, some classes have significantly fewer training samples than the others. I want to perform a train-validation split in such as way that the class ratios are ...
- 23
0
votes
0
answers
10
views
After training a model, how does test set error inform decision making?
I split a data set into three subsets: training, validation, and test sets. I use my training data for fitting and validation to check for overfitting. I then have a final model that I then propose to ...
- 1,063
0
votes
0
answers
10
views
Strategy for Train/Test-Split on Video Sequences
My dataset consists of 15 video sequences, each sequence showing a different movement. I want to train a CNN to detect poses (e.g. standing, sitting, ...) on single frames of this dataset but struggle ...
- 1
0
votes
0
answers
5
views
Does combining train and test data introduces any potential bias?
Given an estimator $f$ and a dataset $D$ with $k>2$ classes where $S_{train}$ denotes the train set and equivalently $S_{test}$ the test set.
Suppose we want to transform the problem and instead of ...
- 102
2
votes
1
answer
25
views
How does changing the training/test ratio, affect the RSE ans R2 metrics?
If I change the training/test ratio from 90/10 to say 80/20 or 70/30, how does that affect the RSE and R2 metrics? I see changes in the results of the RSE and R2 when I implement the model, but I ...
- 21
1
vote
0
answers
33
views
How to distinguish two versions of R-squared calculated on test set?
I've come across two ways that people calculate R-squared on a test set:
Calculate the square of the correlation between predictions and actual values (in practice, I've seen people do this in R by ...
- 3,768
-2
votes
2
answers
107
views
Why don’t we train on the test dataset in machine learning?
Can somebody explain why we can not train the dataset on the test dataset?
- 11
4
votes
1
answer
112
views
Rule based label - random split vs time-based split
We have a dataset of 977 records (77:23 class ratio) where we try to predict a binary outcome using random forests and neural networks. whether supplier met the target or not. However, we didn't have ...
- 1,804
2
votes
0
answers
34
views
How to name train + validation set
Usually in machine learning pipelines, we use a train set, a validation set and a test set. Quite often, we first split the test set from the rest, and then we split the "rest" into train ...
- 21
1
vote
0
answers
40
views
Bootstrapped Latin Partition
I'm having trouble understanding the Bootstrapped Latin partition method (as presented in Statistical validation of classification and calibration models using bootstrapped Latin partitions and ...
- 11
0
votes
0
answers
10
views
What are the uses of of sample splitting in removing statistical bias of variance estimators?
Take some iid sample of size N drawn from a distribution with unknown mean and variance (both finite).
I understand that Bessel's correction is the use of N − 1 instead of N in the formula for the ...
0
votes
0
answers
17
views
Cross-Validation - Datasplit train/test/holdout or train/holdout
I haven't seen a topic on that subject yet so I will ask my question here.
When using cross-validation to train a model, should we split the data into two datasets (train/holdout) or three datasets (...
- 21
0
votes
0
answers
15
views
How should we divide the data into training and testing sets in a specific way instead of probabilities of 70-30%?
The dataset contains 5 stages of the disease. The respective blood concentrations are 12.5, 25, 50, 62, and 75 mg/ml. The intensity is measured at each concentration, which will be used as a response ...
1
vote
1
answer
210
views
random split vs time based split of train and test data
I have been working on binary classification problem using algorithms such as Random Forest, Boosting methods, neural networks and logistic regression. I have data from Jan 2017 to Jan 2022. We wish ...
- 1,804
1
vote
0
answers
33
views
How to guarantee the test set is "independent"?
In Machine Learning (ML) tasks, one splits the dataset into training and test sets. We train the ML model based on the training test, and then we evaluate the performance of the model with the test ...
- 111
0
votes
0
answers
14
views
Is there a way to measure or find the optimal accuracy of an association rule model which is to know if it is overfit or not
I'm currently new at data mining and have a problem understanding that most model such as classification and regression techniques are mostly using test and validation sets, which they can also apply ...
- 1
2
votes
1
answer
64
views
Validity of basic train - test - split for a time series using a RNN
I am trying to determine if a simple train-test-split is valid for a time series if I use a Recurrent Neural Network (LSTM). Lets say I have samples (x) which consist of 2 days values (time steps) ...
- 23
1
vote
1
answer
427
views
If my test size is small, should the validation set be the same size?
I know there is a rule of thumb to split the data to 70%-90% train data and 30%-10% validation data.
But if my test size is small, for example: its size is 5% of the size of the train set, and I can't ...
- 67
1
vote
0
answers
11
views
Scematic of ML model training and testing process?
I'm currently getting confused by how to train a model and then to cross validate it.
Many tutorials seem to show that the process is as follows:
Define model e.g model = LogisticRegression()
Split ...
- 361
0
votes
0
answers
16
views
Train-test-split , 'out of time' samples
When should I approach a problem by splitting the data by time instead of random splits?
E.g. In the case of credit card fraud detection. Where I have data of transactions from multiple different ...
2
votes
1
answer
19
views
What's the official name of the "crop test"?
I call "crop test" or whether my model passed the "crop test" when I remove data from my dataset, conveniently before some events in the data to check whether the historical ...
- 885
1
vote
1
answer
19
views
k-Fold Cross-Validation across classes
Suppose I want to do k-fold cross-validation across a set of data points. Each of these data points is a member of exactly one of n classes. (For example, suppose each of the data points is a person ...
- 827
0
votes
0
answers
26
views
(Stratified) 5-fold cross validation for 2D tensor and real-valued target regression using sklean train_test_split method
In classification problem, when we want to do stratified 5-fold cross validation, we pick the target and use train_test_split using something like below:
...
- 514
0
votes
0
answers
20
views
When do we use stratified splitting?
I have a problem with my data. I only have 59 recordings and that is not optimal for a machine learning regression pipeline. Anyway, I have multiple independent variables, most of them are in this ...
- 55
1
vote
1
answer
396
views
Stratify a train / test split according to some categorical variables
I would like to train / test split a dataset in such a way that all categories of categorical variables are in both train and test split.
I tried ( using sickit learn ) :
...
3
votes
1
answer
333
views
Why does error rate of kNN increase when k approaches size of training set?
I've been experimenting with the effect that different values of k have on the generalisation error of kNN classifier, and I've gotten some unexpected results towards the end when k approaches the ...
- 53
0
votes
1
answer
40
views
Can the number of Target features (y) exceed the number of Input features (X)?
I am trying to perform a train_test_split() on a dataset. Before doing so and while I am assigning data to X and y variables, I realized I have 8 Input features a.k.a. Independent variables and 45 ...
- 141
4
votes
2
answers
3k
views
What is the role of 'shuffle' in train_test_split()?
Wondering what shuffle does if I set it to True when splitting a dataset into train and test splits. Can I use shuffle on a dataset which is ordered by dates?
...
- 141
0
votes
0
answers
17
views
cross validation for training and testing with distinct dataset?
I have seen a poster dissertation about an NLP-based system to rank the preferences of a particular product by some customers. The classification variable could only contain two values, "...
- 579
10
votes
4
answers
2k
views
I've already used my entire dataset in a regression, should I not use that as a prediction model?
At the hospital I work at we were writing a paper on what variables about a patient predict whether they'll return for a follow-up visit. We included variables such as age, gender, distance from ...
- 291
1
vote
0
answers
61
views
train/validate/test split for time series anomaly detection
I'm trying to perform a multivariate time series anomaly detection. I have training data that consists of "normal" data. I train on this data and detect anomalies on the test set that ...
- 11
0
votes
1
answer
68
views
How to interpret Isolation Forest results on variations of train/test sets?
I have a labelled dataset, originally intended for classification or clustering tasks, whose minority class is at 10%. I am investigating whether this problem can be tackled with anomaly detection ...
0
votes
0
answers
45
views
Data split using model's error
For splitting of the data into train/test/val I use stratified sampling. Then I confirm that metadata distributions represent the original dataset well enough.
I want to start considering error of the ...
0
votes
0
answers
122
views
recommended ways of splitting train/validation/test in time series in neural networks [duplicate]
for time series using neural networks with 500000 samples I am looking for tips for recommended percentage of spliting samples into train/validation/test and intervals of splitting. I absolutely am ...
- 141
0
votes
0
answers
24
views
Data Split Strategy
In case of very less data set for training and testing, can I remove validation set and keep only train and test set for training a DL network for segmentation? If yes, then how to get the best model ...
1
vote
1
answer
31
views
Train/dev/test split with limited and skewed positive labels
(Because of the sensitive nature of the actual project, I am using an analogy here. I hope it's clear, if not, please let me know!)
My goal is to classify images as cats or dogs (binary classification)...
- 13
1
vote
0
answers
16
views
Python library to split dataset with token-level labels into dev, train and test [closed]
I have two datasets and I need to split them into dev, train and test. The first one is a regular dataset where in each line there is a sentence along with a label. I found the following code in a ...
- 111
0
votes
1
answer
45
views
Comparison of multiple train/test ratios
I have a dataset of 1000 elements. I am doing random subsampling validation with different ratios for the train/test sets (90/10%, 80/20%, 10/90%), for each ratio I generate 100 train/test samples. My ...
- 13
0
votes
0
answers
14
views
Should there be a preference in adding new training samples?
In a binary example, if you have a training and testing set of samples, should you attempt to add additional samples similar to test set observations with predicted class probabilities near 50%?
Would ...
- 21
1
vote
0
answers
93
views
What is the difference between spliting the dataset into training and testing or collecting the training and testing data seperately?
I am working on active learning and I was wondering about the difference if we split the dataset into training and testing or collecting and labeling the training and testing datasets separately. ...
- 111
0
votes
0
answers
10
views
Any issues with conducting stratafied train/test splitting based on the distribution of a categorical predictor?
I am building a xgboost regressor for a dataset that includes a categorical feature with a very large number of levels (on average, each level has an observation frequency of only about .2%). However, ...
- 1
0
votes
1
answer
85
views
Scaling a gene expression data generates NA values
I would like to analyze the prostate gene expression data which has a link named 12859_2005_967_MOESM4_ESM.tgz in the site here.
In a paper I read, the author scaled the predictors in the training set ...
- 263
2
votes
1
answer
366
views
Should outlier detected before or after train test split
Outliers are usually first detected using Boxplot, then the suspicious observations may be sent to experts for justification - justify whether they are true outliers (contaminated data) or leverage ...
- 643