Questions tagged [data-imputation]
Refers to a general class of methods used to "fill in" missing data. Methods used for doing this typically are related to interpolation (http://en.wikipedia.org/wiki/Interpolation) and require assumptions about why the data is missing (e.g. "missing at random")
671
questions
2
votes
2
answers
29
views
Does a relationship between missing values indicate MNAR?
I have a dataset where several features are missing values only if another feature is also missing a value. Does this indicate that the missing data is missing not at random (MNAR)? Additionally, how ...
- 55
0
votes
0
answers
9
views
Choosing Between Training on Historical Data vs. Imputed Data for Time Series Nowcasting
I have several time series datasets which we can label as y, x1, x2, ..., ...
- 285
0
votes
0
answers
13
views
Incremental Users On Platform Method
I have the following data (aggregated, user level data is not available).
date | platform | total_watch_time
platform = [mobile, mobile + desktop, mobile + desktop + tv]
I want to determine the ...
- 11
0
votes
0
answers
24
views
Bayesian imputation model for day of month when the year, month, and day of week are available
I'm considering including vehicle incident data paired to date, however the most extensive data for the region I am interested in does not have dates. I would like to include it in a multivariable ...
- 6,988
0
votes
0
answers
13
views
Congeniality between imputation model and analysis model
I have a question about congeniality between imputation model and analysis model. Suppose:
Research goal: To estimate the prevalence of disease A among a population as well as among different ...
- 132
2
votes
1
answer
31
views
proof of missing at random
I have an longitudinal data with measurements of hand grip strength measured in kg by a hand-held dynamometer, using a maximum of three measurements taken with the strongest hand in an old population....
- 119
1
vote
0
answers
23
views
Multilevel multiple imputation in practice using R
I'm currently involved in a project where I want to address missing data using multiple imputation. I'm using healthcare data in a longitudinal setting with 16 time points, where observations are ...
0
votes
0
answers
11
views
Need guidance in missing value imputation in count variables
I have a few count variables that are part of a larger dataset (the rest of the variables are either numerical or ordinal). Their distributions can be seen below.
My goal is to impute the missing ...
- 3
0
votes
0
answers
16
views
Include fully-observed variables as a response or a covariate in multivariate normal imputation?
I have a few questions regarding the specification of multivariate normal imputation model (MVNI):
Suppose I have the following substantive analysis model, and there is missing values in X1. I want to ...
- 132
0
votes
0
answers
7
views
Compare LASSO regressions as missing data increases
I'm comparing three LASSO-regression models for classifying two patient types. Each model has increasingly complex variables, which are less likely to be available. Consequently, the last model has ...
- 39
0
votes
0
answers
14
views
How to impute the std deviation of a sample using the pooled mean and standard deviation of similar samples?
I want to perform a meta-regression based on the mean changes from baseline values.In one of the studies, the standard deviation has not been mentioned. How can I impute the value of the missed ...
- 53
4
votes
1
answer
43
views
Creating deliberate missing data to impute a counter factual
This question is more out of curiosity than necessity. Any useful resources/literature will be massively appreciated.
Recently started research in a new field that I’m unfamiliar with: economic policy ...
1
vote
0
answers
37
views
How to perform inference on regression coefficients when some X values are interpolated?
This should be a simple question but I'm having a hard time finding an answer.
Assume I have some data $y$ with covariates $X$. This set of data was obtained via a simple random sample. I want to ...
- 119
2
votes
0
answers
20
views
Is it possible to define a specific criterion prior to imputing missing values? [closed]
it's hard to summarise the problem in a question so I'll explain it here.
I have a variable called "Plant Species" that is missing a lot of values because of the sampling method used. For ...
- 21
1
vote
1
answer
87
views
How to calculate a linear combination of regression coefficients after multiple imputation?
A method to handle missing data is with multivariate imputation by chained equations. During this process, we create many datasets (as many as you specify) with imputed values for the missing data. ...
- 13
1
vote
0
answers
33
views
Imputation method for missing values that are irrelevant
I have a data set $\mathbf X$, with around 20 predictors, which is a matrix of parameters of a surrogate model. For each observation $\mathbf i$ of $\mathbf X$, the surrogate model was trained to ...
- 153
0
votes
0
answers
6
views
How can I learn sequence-to-sequence imputation model (SSIM and dual SSIM)?
I have read papers that have used this model for the imputation method.
Link: https://ivivan.com/papers/IOTJ2019.pdf
I want to read more and be able to use these models for the imputation of my own ...
3
votes
1
answer
134
views
Multiple Imputation in R package Amelia with wide vs. long data
I am imputing missing values in a longitudinal dataset using the Amelia package in R. Does it matter if I have the data in long format (with id, time, and value in each row) or in wide format (with id,...
- 313
0
votes
0
answers
31
views
Simple demonstration of imputation data leakage?
I'm aware that it's best practice to do all pre-processing within train-test splits, including data imputation. At least, it's recommended not to use the test data to generate the imputation model for ...
- 145
1
vote
0
answers
14
views
Can kriging method be used for dataset that is not spatial?
Sorry if the question sounds stupid. I am new to this. Consider the following dataset:
...
0
votes
0
answers
24
views
Compensating for anomalies in training data by diluting them with older data?
I'm having trouble finding references to (and the name of) an obvious correction procedure that surely must have been used extensively in time series forecasting, but my Google searches keep turning ...
0
votes
1
answer
41
views
MICE multiple imputation in R - imputation number
I'm running MICE for 100 imputations with big data (~600k rows). Due to storage restrictions at work (which I am not permitted to change), I can't save all 100 imputations in one go, and I'd hit ...
- 1
0
votes
0
answers
33
views
How to perform MCMC posterior sampling of $X+Y$ random variable?
Let $X$ follow normal distribution with mean $\mu_X=10$ and variance $V_X=20$. Let $Y$ follow truncated normal distribution with mean $\mu_Y=-10,V_Y=8$, where $Y\leq 0$ by truncation.
Let $Z=X+Y$. ...
- 1,365
1
vote
0
answers
17
views
Is there any issue in imputing missing observations when the missing observations are related to each other?
Say that we have a cross-sectional dataset with two variables, A and B.
Also suppose that A and B are related to each other in some way. Now, there are some rows for which only A is missing, and some ...
- 11
0
votes
0
answers
17
views
Questionnaire validity with imputed data, possible?
I would like to validate a 30-item questionnaire. There is a a lot of missing data. How do I handle missing data.
My questions are the following:
a) Do I have to use only complete case data to ...
- 1
0
votes
0
answers
17
views
My KNN result not as expected for all numerical data
I have developed the following KNN code and tested it against several datasets with >90% accuracy (the wdbc dataset from UCI for one, granted it was a categorical result), however when using the ...
- 1
1
vote
1
answer
69
views
when working with missing data, what percentage of data is considered too much missing before implementing something like imputation?
I am looking for advice (do not have a specific example regarding data) but am wondering, when working with any dataset that is missing, at what point/percentage would you consider using something ...
- 145
0
votes
0
answers
43
views
Imputation process before LDA
I want to perform LDA in my cohort which is based on 140 inviduals distributed according in 3 groups. These individuals have undergone an analysis of 50 variables (gene expression). So my dataset is ...
1
vote
0
answers
69
views
Numerical data imputation: Generative Adversarial Imputation Nets (GAIN) not reproducible?
I am interested in numerical data imputation problems: how to properly estimate missing values in a tabular data set (rows and columns) with missing numerical values?
In 2018, Yoon et al. proposed the ...
- 153
3
votes
0
answers
18
views
How to pool estimates from multiply-imputed datasets with complex sampling designs?
Analysts often use Rubin's rule (RR) to obtain a pooled estimate of a popular quantity from multiple (imputed) datasets. While popular statistical software (such as the R ...
- 691
0
votes
0
answers
29
views
General guidance on missing data
I'm analysing some non-randomised time to event data. I have 5 controls (demographic variables) and a variable of interest treatment. I only have missing data for 3/5 of the controls. The variables ...
- 521
0
votes
0
answers
4
views
Searching for references that show imputation methods use RMSE as distance measures for ordinal data
RMSE or MAD are used as distance measures more for the continuous data. What will be good distance measures for ordinal data?
Are you aware of any good references that show imputation methods also ...
- 1
2
votes
1
answer
168
views
Use the target variable during imputation?
There is a quite old yet very good question about the proper way for using rfImpute but to me the question raised by Doug7 (whether the target variable y gets used for the imputation of the features ...
- 43
2
votes
0
answers
27
views
Theoretical Results for MICE Imputation
Is there any literature exploring convergence guarantees of the MICE imputation method for missing data? In practice, the method seems to work pretty reliably with different regressor but I can't seem ...
- 21
0
votes
1
answer
94
views
Can you impute (predict) missing continuous data using categorical data as the predictor?
I just read that to use MICE Imputation, variables with missing values need to have a relationship to other variables. In my case, I will anonymize the variable just for convenience purposes:
...
1
vote
1
answer
15
views
Imputing missing values for a RM-ANOVA
I have 4 experimental groups/conditions and 5 measurement times for each group/condition. Each participant only took part in one of the 1 conditions. In total there are 27 participants and each ...
- 11
0
votes
0
answers
28
views
Should I use MICE Imputation or Other Method in this Case?
I'm building a beginner data analysis project and have been stuck on this problem for almost a month. I'm analyzing a TV Brand E-commerce dataset from Kaggle with several missing values. The dataset ...
0
votes
0
answers
32
views
Missing at Random / Missing not at Random
I plan a study where I predict variables with machine learning methods integrating data from multiple surveys. The questionnaires assessed with each survey slightly differ, so that I have a lot of ...
0
votes
0
answers
11
views
I want to impute missing values of SNP data from the DArTseq . Which software and dosage model should I use?
The data was initially sent in a numerical format, with the SNP being coded as 0, 1, and 2. Using the provided reference allele, each SNP was now translated into its corresponding variants. Which ...
- 1
1
vote
0
answers
32
views
Constrained imputation in Python
I actually have two original datasets (each one for a departure that are related to each one in a specific way , but it's not important to know how exactly) , but these 2 datasets contain some ...
- 11
1
vote
1
answer
67
views
Proper imputation of missing values for machine learning
I always struggled to get imputation for missing values right, and it doesn't help that you can find contradictory opinions online about it.
Say I have data X that I split into X_train and X_test, and ...
- 73
1
vote
0
answers
51
views
Clustering mixed data - SPSS [closed]
On my current project, I have to form four or five clusters describing different types of banking customers. The data is based on a survey of around 3500 participants and contains more than 250 ...
- 11
0
votes
0
answers
37
views
Aggregating physical science data from multiple sources for time series
I’m attempting to build a time series model using water quality data. The problem is, for any given site where this data exists, it is irregular time intervals and sparse data (anywhere from 20-70% ...
- 13
0
votes
0
answers
13
views
MICE for Boolean variables
I have a dataset with numerical and boolean values. But when I use MICE to impute the missing data, I get numerical values for missing boolean data. Here is my code
...
- 1
8
votes
2
answers
103
views
Choosing $m$ value when using multiple imputation (MI)
Multiple imputation creates $m$ new imputed datasets by taking each missing value and replacing it by analyzing the $m$ imputed values (for example: using the mean). Is there a rule of thumb or a ...
- 105
0
votes
0
answers
37
views
Data Imputation Without Ground-truth
recently I am dealing with a project of data imputation. I use the probabilistic imputation (multiple imputation) methods. As is known, the real data do not contain the REAL values for the missing ...
- 85
1
vote
1
answer
30
views
Definition of an imputation in statistics
I recently used the terminology imputation by zero, because the cause of the loss to follow-up were well known in ourstudy, since they were failures. Somebody pointed out to me that the terminology is ...
- 735
2
votes
0
answers
14
views
MNAR Imputation On Time Series Data in Ongoing Study
I am working on an ongoing study that tracks subjects through time. Each subject is enrolled for months, where upon exiting I calculate their length-of-stay.
Since the study is ongoing, I have both ...
- 21
0
votes
0
answers
33
views
imbalance and imputation
It has been answered in https://stats.stackexchange.com/q/380668 that class imbalance should have nothing to do with imputation. I agree with the post, but I am concerned about class imbalance.
I am ...
- 1,365
2
votes
0
answers
77
views
Effective way to down sample or up sample signals without losing information?
I have an edf data here that I got from this website. The data was supposed to be fed into an ML model. The data is taken from a sleep study (polysomnography). However, the data for some of the ...
