Questions tagged [train-test-split]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
0 answers
12 views

How do I perform a train-validation split on data with class imbalance such that the class imbalance ratio is preserved?

My data has class imbalance-- that is, some classes have significantly fewer training samples than the others. I want to perform a train-validation split in such as way that the class ratios are ...
user avatar
0 votes
0 answers
10 views

After training a model, how does test set error inform decision making?

I split a data set into three subsets: training, validation, and test sets. I use my training data for fitting and validation to check for overfitting. I then have a final model that I then propose to ...
user avatar
  • 1,063
0 votes
0 answers
10 views

Strategy for Train/Test-Split on Video Sequences

My dataset consists of 15 video sequences, each sequence showing a different movement. I want to train a CNN to detect poses (e.g. standing, sitting, ...) on single frames of this dataset but struggle ...
user avatar
0 votes
0 answers
5 views

Does combining train and test data introduces any potential bias?

Given an estimator $f$ and a dataset $D$ with $k>2$ classes where $S_{train}$ denotes the train set and equivalently $S_{test}$ the test set. Suppose we want to transform the problem and instead of ...
user avatar
2 votes
1 answer
25 views

How does changing the training/test ratio, affect the RSE ans R2 metrics?

If I change the training/test ratio from 90/10 to say 80/20 or 70/30, how does that affect the RSE and R2 metrics? I see changes in the results of the RSE and R2 when I implement the model, but I ...
user avatar
  • 21
1 vote
0 answers
33 views

How to distinguish two versions of R-squared calculated on test set?

I've come across two ways that people calculate R-squared on a test set: Calculate the square of the correlation between predictions and actual values (in practice, I've seen people do this in R by ...
user avatar
  • 3,768
-2 votes
2 answers
107 views

Why don’t we train on the test dataset in machine learning?

Can somebody explain why we can not train the dataset on the test dataset?
user avatar
  • 11
4 votes
1 answer
112 views

Rule based label - random split vs time-based split

We have a dataset of 977 records (77:23 class ratio) where we try to predict a binary outcome using random forests and neural networks. whether supplier met the target or not. However, we didn't have ...
user avatar
  • 1,804
2 votes
0 answers
34 views

How to name train + validation set

Usually in machine learning pipelines, we use a train set, a validation set and a test set. Quite often, we first split the test set from the rest, and then we split the "rest" into train ...
user avatar
1 vote
0 answers
40 views

Bootstrapped Latin Partition

I'm having trouble understanding the Bootstrapped Latin partition method (as presented in Statistical validation of classification and calibration models using bootstrapped Latin partitions and ...
user avatar
  • 11
0 votes
0 answers
10 views

What are the uses of of sample splitting in removing statistical bias of variance estimators?

Take some iid sample of size N drawn from a distribution with unknown mean and variance (both finite). I understand that Bessel's correction is the use of N − 1 instead of N in the formula for the ...
user avatar
0 votes
0 answers
17 views

Cross-Validation - Datasplit train/test/holdout or train/holdout

I haven't seen a topic on that subject yet so I will ask my question here. When using cross-validation to train a model, should we split the data into two datasets (train/holdout) or three datasets (...
user avatar
  • 21
0 votes
0 answers
15 views

How should we divide the data into training and testing sets in a specific way instead of probabilities of 70-30%?

The dataset contains 5 stages of the disease. The respective blood concentrations are 12.5, 25, 50, 62, and 75 mg/ml. The intensity is measured at each concentration, which will be used as a response ...
user avatar
1 vote
1 answer
210 views

random split vs time based split of train and test data

I have been working on binary classification problem using algorithms such as Random Forest, Boosting methods, neural networks and logistic regression. I have data from Jan 2017 to Jan 2022. We wish ...
user avatar
  • 1,804
1 vote
0 answers
33 views

How to guarantee the test set is "independent"?

In Machine Learning (ML) tasks, one splits the dataset into training and test sets. We train the ML model based on the training test, and then we evaluate the performance of the model with the test ...
user avatar
  • 111
0 votes
0 answers
14 views

Is there a way to measure or find the optimal accuracy of an association rule model which is to know if it is overfit or not

I'm currently new at data mining and have a problem understanding that most model such as classification and regression techniques are mostly using test and validation sets, which they can also apply ...
user avatar
  • 1
2 votes
1 answer
64 views

Validity of basic train - test - split for a time series using a RNN

I am trying to determine if a simple train-test-split is valid for a time series if I use a Recurrent Neural Network (LSTM). Lets say I have samples (x) which consist of 2 days values (time steps) ...
user avatar
  • 23
1 vote
1 answer
427 views

If my test size is small, should the validation set be the same size?

I know there is a rule of thumb to split the data to 70%-90% train data and 30%-10% validation data. But if my test size is small, for example: its size is 5% of the size of the train set, and I can't ...
user avatar
  • 67
1 vote
0 answers
11 views

Scematic of ML model training and testing process?

I'm currently getting confused by how to train a model and then to cross validate it. Many tutorials seem to show that the process is as follows: Define model e.g model = LogisticRegression() Split ...
user avatar
0 votes
0 answers
16 views

Train-test-split , 'out of time' samples

When should I approach a problem by splitting the data by time instead of random splits? E.g. In the case of credit card fraud detection. Where I have data of transactions from multiple different ...
user avatar
2 votes
1 answer
19 views

What's the official name of the "crop test"?

I call "crop test" or whether my model passed the "crop test" when I remove data from my dataset, conveniently before some events in the data to check whether the historical ...
user avatar
  • 885
1 vote
1 answer
19 views

k-Fold Cross-Validation across classes

Suppose I want to do k-fold cross-validation across a set of data points. Each of these data points is a member of exactly one of n classes. (For example, suppose each of the data points is a person ...
user avatar
0 votes
0 answers
26 views

(Stratified) 5-fold cross validation for 2D tensor and real-valued target regression using sklean train_test_split method

In classification problem, when we want to do stratified 5-fold cross validation, we pick the target and use train_test_split using something like below: ...
user avatar
0 votes
0 answers
20 views

When do we use stratified splitting?

I have a problem with my data. I only have 59 recordings and that is not optimal for a machine learning regression pipeline. Anyway, I have multiple independent variables, most of them are in this ...
user avatar
1 vote
1 answer
396 views

Stratify a train / test split according to some categorical variables

I would like to train / test split a dataset in such a way that all categories of categorical variables are in both train and test split. I tried ( using sickit learn ) : ...
user avatar
3 votes
1 answer
333 views

Why does error rate of kNN increase when k approaches size of training set?

I've been experimenting with the effect that different values of k have on the generalisation error of kNN classifier, and I've gotten some unexpected results towards the end when k approaches the ...
user avatar
0 votes
1 answer
40 views

Can the number of Target features (y) exceed the number of Input features (X)?

I am trying to perform a train_test_split() on a dataset. Before doing so and while I am assigning data to X and y variables, I realized I have 8 Input features a.k.a. Independent variables and 45 ...
user avatar
  • 141
4 votes
2 answers
3k views

What is the role of 'shuffle' in train_test_split()?

Wondering what shuffle does if I set it to True when splitting a dataset into train and test splits. Can I use shuffle on a dataset which is ordered by dates? ...
user avatar
  • 141
0 votes
0 answers
17 views

cross validation for training and testing with distinct dataset?

I have seen a poster dissertation about an NLP-based system to rank the preferences of a particular product by some customers. The classification variable could only contain two values, "...
user avatar
  • 579
10 votes
4 answers
2k views

I've already used my entire dataset in a regression, should I not use that as a prediction model?

At the hospital I work at we were writing a paper on what variables about a patient predict whether they'll return for a follow-up visit. We included variables such as age, gender, distance from ...
user avatar
1 vote
0 answers
61 views

train/validate/test split for time series anomaly detection

I'm trying to perform a multivariate time series anomaly detection. I have training data that consists of "normal" data. I train on this data and detect anomalies on the test set that ...
user avatar
0 votes
1 answer
68 views

How to interpret Isolation Forest results on variations of train/test sets?

I have a labelled dataset, originally intended for classification or clustering tasks, whose minority class is at 10%. I am investigating whether this problem can be tackled with anomaly detection ...
user avatar
0 votes
0 answers
45 views

Data split using model's error

For splitting of the data into train/test/val I use stratified sampling. Then I confirm that metadata distributions represent the original dataset well enough. I want to start considering error of the ...
user avatar
0 votes
0 answers
122 views

recommended ways of splitting train/validation/test in time series in neural networks [duplicate]

for time series using neural networks with 500000 samples I am looking for tips for recommended percentage of spliting samples into train/validation/test and intervals of splitting. I absolutely am ...
user avatar
0 votes
0 answers
24 views

Data Split Strategy

In case of very less data set for training and testing, can I remove validation set and keep only train and test set for training a DL network for segmentation? If yes, then how to get the best model ...
user avatar
1 vote
1 answer
31 views

Train/dev/test split with limited and skewed positive labels

(Because of the sensitive nature of the actual project, I am using an analogy here. I hope it's clear, if not, please let me know!) My goal is to classify images as cats or dogs (binary classification)...
user avatar
1 vote
0 answers
16 views

Python library to split dataset with token-level labels into dev, train and test [closed]

I have two datasets and I need to split them into dev, train and test. The first one is a regular dataset where in each line there is a sentence along with a label. I found the following code in a ...
user avatar
  • 111
0 votes
1 answer
45 views

Comparison of multiple train/test ratios

I have a dataset of 1000 elements. I am doing random subsampling validation with different ratios for the train/test sets (90/10%, 80/20%, 10/90%), for each ratio I generate 100 train/test samples. My ...
user avatar
  • 13
0 votes
0 answers
14 views

Should there be a preference in adding new training samples?

In a binary example, if you have a training and testing set of samples, should you attempt to add additional samples similar to test set observations with predicted class probabilities near 50%? Would ...
user avatar
  • 21
1 vote
0 answers
93 views

What is the difference between spliting the dataset into training and testing or collecting the training and testing data seperately?

I am working on active learning and I was wondering about the difference if we split the dataset into training and testing or collecting and labeling the training and testing datasets separately. ...
user avatar
  • 111
0 votes
0 answers
10 views

Any issues with conducting stratafied train/test splitting based on the distribution of a categorical predictor?

I am building a xgboost regressor for a dataset that includes a categorical feature with a very large number of levels (on average, each level has an observation frequency of only about .2%). However, ...
user avatar
  • 1
0 votes
1 answer
85 views

Scaling a gene expression data generates NA values

I would like to analyze the prostate gene expression data which has a link named 12859_2005_967_MOESM4_ESM.tgz in the site here. In a paper I read, the author scaled the predictors in the training set ...
user avatar
  • 263
2 votes
1 answer
366 views

Should outlier detected before or after train test split

Outliers are usually first detected using Boxplot, then the suspicious observations may be sent to experts for justification - justify whether they are true outliers (contaminated data) or leverage ...
user avatar