Questions tagged [data-leakage]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
26 views

Data leakage in time series forecasting framed as a supervised learning problem

Suppose that I have a simple univariate time series. My goal is to use the value of 3 consecutive days to predict the value of the fourth day. I built my dataset by applying a rolling window that ...
Ray's user avatar
  • 11
1 vote
1 answer
48 views

EDA and Model Selection for Forecasting while avoiding Data Leakage

How to do EDA and model selection for time series forecasting without data leakage? Im assuming just checking for missing values is ok. But is graphing the entire time series considered data leakage? ...
pandashelp's user avatar
1 vote
0 answers
34 views

Should I delete samples from the training data that are present in the testing data by accident?

I classify pairs of entities, let's say dog-cat pairs, whether there is association between them (positive class) or there is not (negative class). I have a moderately sized positive dataset (~130k ...
oliver.c's user avatar
2 votes
0 answers
32 views

Data leakage: Train test split before or after data preprocessing? [duplicate]

A while ago I came across the word "data leakage" for the first time, and after some research, I found that it is a common mistake among data science/machine learning practitioners. But the ...
jairiidriss's user avatar
0 votes
0 answers
20 views

Nested Cross Validation performance on each Sequential Feature Selection subset

I want to get cross-validated performance values of my model after hyperparameter tuning and sequential feature selection on each feature subset. Following this example, I want to use an outer-CV ...
Charlie's user avatar
  • 132
0 votes
0 answers
10 views

DNN: Does mini-batching of grouped data during training introduce information leakage?

I am trying to replicate a deep neural network from this notebook, which works with the French MTPL dataset. The NB is in R and mine is in Python. The dataset contains policy entries with the same ...
JGM's user avatar
  • 1
0 votes
0 answers
23 views

Temporal leakage or different phenomena?

I've following problem/toy-example: every week I sample data describing users (one row is one user) I want to predict that in next three weeks user will be a fraud 1 or not 0, so basically binary ...
Quant Christo's user avatar
2 votes
1 answer
56 views

Data leakage or not?

The goal is to predict whether an employee will leave the company: yes or no. I have a dataframe with information about employees. There are 30 independent features and one dependent feature (Left: ...
Milvhb's user avatar
  • 21
0 votes
0 answers
36 views

Simple demonstration of imputation data leakage?

I'm aware that it's best practice to do all pre-processing within train-test splits, including data imputation. At least, it's recommended not to use the test data to generate the imputation model for ...
Evan's user avatar
  • 145
0 votes
0 answers
34 views

Splitting data when object of study are correlated (based on "origin" group)

everyone! Imagine I have a dataset which I'd like to use it to train a churn model (fot example: logistic regression, xgboost binary classifier, lgbm binary classifier, etc.). The structure of my ...
Gabriel Monteiro's user avatar
0 votes
0 answers
22 views

The validation loss and training loss are low and close to each other but there exists high test loss?

I have been using CNN-LSTM for action recognition with dataset split 70% for training , 20% for validation , 10% for testing and after training , the validation loss and training loss were very close ...
The Limit Breaker's user avatar
1 vote
0 answers
19 views

Data augmentation specific per class

I have a database of defects on plastic films. Defects are burns, holes, and similar things. Some defects are direction specific. A vertical sign on the material represents a scratch done on the ...
Jonny_92's user avatar
2 votes
1 answer
281 views

Why doesn't CatBoost Encoding cause target leakage?

I'm currently working on a fraud detection problem with a dataset of 300,000 rows and 500 columns, 70 of which are categorical with over 10 categories each. I'm facing memory constraints and exploring ...
Connor's user avatar
  • 565
0 votes
0 answers
6 views

Time series -- peeking for truncation of outlying values displaced from mean under some structural assumptions

Per https://stats.stackexchange.com/a/204977/384097 the suggestion was made that, given a known distribution, applying a cutoff as means of dealing with outliers is not data leakage. I feel ...
bmf's user avatar
  • 1
1 vote
1 answer
107 views

Does feature selection and model testing have to be coupled in each fold of the cross-validation?

Quick overview of my data and aims: I have two groups, 50 samples per group, and 6000 features. I want to find the minimal amount of features capable of distinguishing both groups. I know the sample ...
Luiz Gustavo's user avatar
2 votes
3 answers
174 views

What is "information leak from test to train" ? Is stratification by target a leak?

It's common practice to do procedures such as standardization and even missing value imputation (commonly based on some means) after train/test split - otherwise it is treated as information leak from ...
Ars ML's user avatar
  • 31
0 votes
1 answer
23 views

Is using the same person as data observation in different time stamps a way to produce data leakage in a Machine Learning model?

Let's assume we are going to train a regression model (could be any ML tabular solution for regression. Ex.: LGBM, XGBoost, Perceptron, ...) to predict a customer profit in the next month. While ...
Matheus Nascimento's user avatar
0 votes
1 answer
109 views

Preprocessing on training set only or both training & test set? Seems like there would be errors for both answers

Let's say I have a dataset that hasn't been split into train/test yet. Upon loading it, I discover that there are columns where there are nulls that need to be filled in, some quadratic relationships ...
Katsu's user avatar
  • 861
5 votes
1 answer
3k views

Do we One Hot Encode (create Dummy Variables) before or after Train/Test Split?

I've seen quite a lot of conflicting views on if one-hot encoding (dummy variable creation) should be done before/after the training/test split. Responses seem to state that one-hot encoding before ...
Beans On Toast's user avatar
4 votes
0 answers
148 views

Examples of Leakages in the Training Data

I was wondering about Data Leakage in the data preparation phase during the training of a model. By definition, data leakage happens when information is revealed to the model giving it an unrealistic ...
Denis Mazzucato's user avatar
3 votes
2 answers
383 views

Is data leakage from time series autocorrelation actual data leakage?

That's the question: Is data leakage from time series autocorrelation actual data leakage? To explain it with an example (I will separate the example in numbers to give more structure to the ...
Chris's user avatar
  • 525
1 vote
1 answer
25 views

How to choose train, test and validation data for two-staged experiments?

I have a question concerning the possibility of the train-test-validation split in a staged experiment setup. The data I used is split up into 3 parts: train, test and validation data. Then I try to ...
user19452872's user avatar
1 vote
1 answer
141 views

Is data leakage a concern when using an ensemble of leave-one-out predictions?

I am new to stacking. I have a dataset with N samples and 7 tables corresponding to different data types, plus a binary label. Some tables have dozens of features, other have many thousands. I train ...
SebDL's user avatar
  • 221
1 vote
1 answer
471 views

Is shuffling timeseries data, then separating into training/testing sets a form of data leakage?

I am building a multiple regression MLP (Multi-Layer Perceptron), the input is 8 weather variables collected from October-February, and the output is another weather variable. The assumption is that ...
schmibbler's user avatar
3 votes
1 answer
398 views

Lagged variables, data leakage and machine learning

I am reading a paper that fits a random forest (RF) to some data that is grouped by company and quarter. In the data engineering stage, the authors include 'lagged' variables of many of the ...
thebabystatistician's user avatar
3 votes
1 answer
477 views

Train/test split on time-based data with lagged features

I am working with data on bank transactions, and am using RFM (recency/frequency/monetary value) features like days since last transaction, number of transactions last n days, average value of ...
mdouglas81's user avatar
3 votes
0 answers
114 views

Avoiding data leakage in preprocessing and handling unseen values in test data

I've been reading up on avoiding data leakage in the preprocessing step of a machine-learning/data-science pipeline, specifically that it is wrong to apply preprocessing to both training and test data ...
njp's user avatar
  • 131
0 votes
1 answer
62 views

how to deal with data leakage in historical data

I have a dataset containing matches from 2000 TO 2018 and I am asked to predict match outcomes for the year 2017 to avoid data leakage I am going to just train my model from 2000 to 2016. in the ...
Mohamed Amine's user avatar
1 vote
0 answers
169 views

Cross validation within a bootstrap sample: is leakage a problem here?

I would like to calculate the sampling distribution for logistic LASSO coefficients. One approach to calculating this sampling distribution is described on page 143 of "Statistical Learning with ...
D I's user avatar
  • 11
2 votes
0 answers
76 views

Quote on too good to be true model performance

I seem to recall that there is a nice quote by some (well known?) machine learning expert about too good to be true model performance. The quote is something like "If your model performance looks ...
Björn's user avatar
  • 31.3k
1 vote
0 answers
499 views

Normalization and RidgeCV in Sklearn Pipeline - possible data leakage?

To avoid data leakage between the train and test set, I'm using sklearn's Pipeline as follows: ...
flanders's user avatar
1 vote
0 answers
18 views

Preprocessing for the final model to be deployed [duplicate]

Typically for a ML workflow, we import the data (X and y), split the X and ...
spectre's user avatar
  • 350
1 vote
0 answers
222 views

Many Preprocessing steps can cause data leakage, then how should we perform EDA?

For the past week, I have been constantly checking with people on this sub on how to avoid data leakage during preprocessing like feature selection and/or scaling etc here and here. I understand most ...
nan's user avatar
  • 795
1 vote
1 answer
301 views

Data Leak or Feature Engineering in regression problem?

I recently worked on a housing price dataset, where the goal is to predict sale prices. I had the idea to construct a feature on the training set, which would be dependent on the target variable and ...
Nils Lcrx's user avatar
0 votes
1 answer
123 views

Modeling length of stay with (Cox) regression with censored observations

I'm attempting to model length of stay (LOS) in a psychiatric child/adolescent setting. My LOS is censored for a few patients because they are required to leave the facility when they turn 18. I was ...
Chiel's user avatar
  • 1
2 votes
1 answer
108 views

Data Leakage Concerns

I've come across the concept of data leakage in which optimistically biased generalisation errors occur due to test data in some sense 'seeing' the training data. For instance, normalisation on an ...
N Blake's user avatar
  • 579
1 vote
1 answer
834 views

Avoiding data leakage in preprocessing

I'm a data science newbie and a bit confused with the following: I usually do the preprocessing on all predictors of a dataset, meaning I create X by concatenating <...
LeLuc's user avatar
  • 651
2 votes
1 answer
812 views

What is the difference between standardizing time series data and non-time series data?

From reading some answers on this site (1, 2, 3 and 4) I found that, on time series data, standardization must be applied separately on the train and test sets to avoid data leakage. So the train data ...
Marcus's user avatar
  • 255
0 votes
1 answer
43 views

Avoiding Data Leakage from Bucketed Features During Cross-Validation

I am working on a classification problem and have engineered a few categorical features with high cardinality by dummying out the most frequently occuring values and then using the response variable ...
Jake Niederer's user avatar
0 votes
0 answers
818 views

Does k-fold cross-validation induce data leakage in time series data?

I created a predicative model using neural networks and applied in on a time series. This is how I split my data: ...
Marcus's user avatar
  • 255
9 votes
1 answer
3k views

Does using a random train-test split lead to data leakage?

I am trying to understand data leakage in modeling practice. If we had a dataset of patient instances from 2000-2018 (with all patient visits included), and used a randomly selected train-test split (...
AmeySMahajan's user avatar
1 vote
0 answers
185 views

Doubly Robust Estimator

When use Doubly Robust Estimator we train m0/m1 models and propensity score model to be used by the estimator. Is it OK to use the same dataset to train those models and then use them to measure ATE ...
Dennis Lyubyvy's user avatar
1 vote
1 answer
622 views

Machine Learning + Hyperparameter Tuning + Data Leakage : Is my procedure free of data leakage?

I'm trying to classify 8 types of hand gestures with EMG signals. For that I followed these steps: Split the entire data into training data and test data For training data I extracted features. Here ...
Debbie's user avatar
  • 119
5 votes
1 answer
97 views

what does it mean that there is leakage of information when one uses a test set?

I have read about the term "leakage of information" that occurs when one tries to estimate the generalization error by using a test set in Machine Learning models. However, I was not able to ...
Layla's user avatar
  • 621
0 votes
1 answer
213 views

Data leakage with clustered observations

I have (what I call) a clustered dataset, that is: for one client, I can have multiple observations that will have some variables in common and some variables will be specific to each observation. ...
amestrian's user avatar
  • 255
3 votes
1 answer
116 views

How to split multiple measurements of the same sample between folds

I'm solving the spectroscopy problem. Based on reflectivity values for wavelengths from the spectrum, I build a regression to find a target for the sample. I have 30 samples. For each sample I take ...
Mishin V.'s user avatar
0 votes
1 answer
2k views

What is temporal leakage?

So I've been trying to work out exactly what temporal leakage is for a while now and I'm getting nowhere. I'm not necessarily looking to code or anything, I'm more so interested in what it actually is ...
Fluffyrox4's user avatar
1 vote
0 answers
50 views

LabelBinarizer gives too many features on test

Let's say I have a Dataset with a coulum called countries. Lots of the values are usa and there is a small amount of values wich are either ...
jan-seins's user avatar
  • 111
0 votes
0 answers
184 views

How to avoid data snooping when doing a leave-one-subject-out train-test split?

I have 13 subjects, and I am trying to create a model which can do well on unseen subjects. The problem is that, when I leave one subject for the test set and train the model on others, the accuracy ...
ThePortakal's user avatar
3 votes
1 answer
799 views

data leakage when scaling time series

Suppose I want to forecast future values of $y$ past values of features $x$. In this example I am using: the training set goes from $t_0$ to $t_{15}$ values from $x_{t_0}$ to $x_{t_{10}}$ to forecast ...
gioxc88's user avatar
  • 1,190