Questions tagged [data-imputation]

Refers to a general class of methods used to "fill in" missing data. Methods used for doing this typically are related to interpolation (http://en.wikipedia.org/wiki/Interpolation) and require assumptions about why the data is missing (e.g. "missing at random")

685 questions
Filter by
Sorted by
Tagged with
103 views

Best way to impute missing values in a time series

I have a camera that detects every time it views a car. Each detection is recorded in a database. I then simulate this behavior as a time series by doing an each hour count of the records. The problem ...
19 views

How can I combine fully measured with partially measured outcomes in an IPD (Network) Meta-Analysis?

I was asked to help in an Individual-Patient Data (IPD) Network Meta-Analysis (NMA). Outcomes are supposed to measure combined scores of condition 1 and condition 2 (thus: $Score=Score_1 + Score_2$), ...
• 1,212
1 vote
20 views

After the mutiple imputation (MICE package in R), I still found that some variables are still with missing values. How to deal with it?

I have a relatively large data set with around 12000 samples with 550 variables. Originally, I have around 800 variables, I used a rule that if missing rate in each variable is larger than 80% I will ...
32 views

Imputing missing observations of zip code level data

I am looking for a sufficient imputation method for missing observations in my zip code level data, using R. I have a random sample consisting of households which live in different zip codes within ...
23 views

Vectorized Linear Regression with missing values

I have a few questions about handling missing data during matrix/array operations. Essentially, I'm doing a vectorized linear regression to perform a bunch of linear regressions in a few matrix/array ...
27 views

Should EM algorithm's final imputed mean match the initial parameter?

I am running a manual EM (expectation-maximization) algorithm in r. My code is the following: ...
• 111
1 vote
30 views

Manually program EM in r to updated multiple parameters and solve missing data [closed]

I am trying to use EM (Expectation-maximization) to fill in missing data in R, but am not sure how to model/code it for my specific case. I am generally trying to follow the example format used in ...
• 111
1 vote
21 views

Impute missing value [closed]

In machine learning when I impute missing values which of the following I perform : 1-Impute data set and then split it? 2-Split dataset to Training and testing datasets and then Impute each datasets ...
• 335
1 vote
30 views

median imputation or case deletion for small samples

I have a dilemma. I have a data set of different proteins from 4 groups of animals characterized by two factors: age (3 or 12 months) and presence of a certain gene (KO or WT). For each combination of ...
26 views

Complete case analysis, multiple imputation, or instrumental variable analysis?

I ran a small study where all participants attended a psychological intervention, and completed measures at pre, post, and follow-up. Approx. 40 participants took part, of which approx. 5 did not ...
• 21
1 vote
19 views

Interpret Mean vs. SMAPE results [duplicate]

I have following dataframe: ...
51 views

Is it possible to replace missing data if 50% of total data are missing?

In my study, 40K completed household surveys. However, when we suggested visiting a nearby health center to measure their physical parameters (height, weight, blood pressure, and blood glucose), only ...
22 views

What is the best method for imputing binary (or integral/count) data?

I have a longitudinal dataset, and I want to create a composite score by including five healthy lifestyles to measure the overall lifestyle over time (use as predictor). Each lifestyle is a binary ...
• 175
17 views

References for effects of constant imputation

I have recently seen people simply adding a constant value in missing data before using regular statistical procedures, but those choices are not argued or do not have any resemblance to the rest of ...
12 views

Alternative to hot deck imputation

I use hot deck imputation for a project that I work on. But I'm dissatisfied that it can't come up with new values. If my doner set had thousands of values in it, and they were nicely distributed, ...
1 vote
89 views

How to use priors to impute values at an individual level and replicate a distribution of the population?

I am trying to correct a variable from a survey that has measurement error. To do this, I have been taking this column as if it was missing and imputing new values based on the predictions of an ...
40 views

Gaussian Processes regression for missing data?

Suppose I have data with three dimensions and the joint is $P(X,Y,Z)$. Were Z to be missing on occasion, Gaussian Process regression could infer distributions over these missing values. But suppose ...
• 2,828
51 views

Would it be preferable to use statistical imputation instead of a subject matter expert's subjective estimate for missing data?

I'm working on a project where I need a variable for the total number of medications a patient is on. The PI is a clinician and I feel they would be able to use the resources at hand - case note and ...
• 591
45 views

Missing data and missing not at random

My outcome is a child cognition scores (y). I will skip the exposure for now. My covariates are regular features like , ...
• 513
26 views

How can I perform an analysis of the NHIS imputed income variables?

I have downloaded five family income variables from https://nhis.ipums.org/nhis-action/variables/group?id=economic_income (INCPPOINT1, INCPPOINT2, INCPPOINT3, INCPPOINT4, INCPPOINT5) for they years ...
55 views

Does a relationship between missing values indicate MNAR?

I have a dataset where several features are missing values only if another feature is also missing a value. Does this indicate that the missing data is missing not at random (MNAR)? Additionally, how ...
• 55
13 views

Choosing Between Training on Historical Data vs. Imputed Data for Time Series Nowcasting

I have several time series datasets which we can label as y, x1, x2, ..., ...
• 205
13 views

Incremental Users On Platform Method

I have the following data (aggregated, user level data is not available). date | platform | total_watch_time platform = [mobile, mobile + desktop, mobile + desktop + tv] I want to determine the ...
26 views

Bayesian imputation model for day of month when the year, month, and day of week are available

I'm considering including vehicle incident data paired to date, however the most extensive data for the region I am interested in does not have dates. I would like to include it in a multivariable ...
• 8,145
43 views

Congeniality between imputation model and analysis model

I have a question about congeniality between imputation model and analysis model. Suppose: Research goal: To estimate the prevalence of disease A among a population as well as among different ...
• 172
44 views

proof of missing at random

I have an longitudinal data with measurements of hand grip strength measured in kg by a hand-held dynamometer, using a maximum of three measurements taken with the strongest hand in an old population....
• 129
1 vote
51 views

Multilevel multiple imputation in practice using R

I'm currently involved in a project where I want to address missing data using multiple imputation. I'm using healthcare data in a longitudinal setting with 16 time points, where observations are ...
16 views

Need guidance in missing value imputation in count variables

I have a few count variables that are part of a larger dataset (the rest of the variables are either numerical or ordinal). Their distributions can be seen below. My goal is to impute the missing ...
18 views

Include fully-observed variables as a response or a covariate in multivariate normal imputation?

I have a few questions regarding the specification of multivariate normal imputation model (MVNI): Suppose I have the following substantive analysis model, and there is missing values in X1. I want to ...
• 172
7 views

Compare LASSO regressions as missing data increases

I'm comparing three LASSO-regression models for classifying two patient types. Each model has increasingly complex variables, which are less likely to be available. Consequently, the last model has ...
• 39
20 views

How to impute the std deviation of a sample using the pooled mean and standard deviation of similar samples?

I want to perform a meta-regression based on the mean changes from baseline values.In one of the studies, the standard deviation has not been mentioned. How can I impute the value of the missed ...
72 views

Creating deliberate missing data to impute a counter factual

This question is more out of curiosity than necessity. Any useful resources/literature will be massively appreciated. Recently started research in a new field that I’m unfamiliar with: economic policy ...
50 views

How to perform inference on regression coefficients when some X values are interpolated?

This should be a simple question but I'm having a hard time finding an answer. Assume I have some data $y$ with covariates $X$. This set of data was obtained via a simple random sample. I want to ...
• 209
20 views

Is it possible to define a specific criterion prior to imputing missing values? [closed]

it's hard to summarise the problem in a question so I'll explain it here. I have a variable called "Plant Species" that is missing a lot of values because of the sampling method used. For ...
• 21
1 vote
221 views

How to calculate a linear combination of regression coefficients after multiple imputation?

A method to handle missing data is with multivariate imputation by chained equations. During this process, we create many datasets (as many as you specify) with imputed values for the missing data. ...
• 13
1 vote
34 views

Imputation method for missing values that are irrelevant

I have a data set $\mathbf X$, with around 20 predictors, which is a matrix of parameters of a surrogate model. For each observation $\mathbf i$ of $\mathbf X$, the surrogate model was trained to ...
• 153
9 views

How can I learn sequence-to-sequence imputation model (SSIM and dual SSIM)?

I have read papers that have used this model for the imputation method. Link: https://ivivan.com/papers/IOTJ2019.pdf I want to read more and be able to use these models for the imputation of my own ...
328 views

Multiple Imputation in R package Amelia with wide vs. long data

I am imputing missing values in a longitudinal dataset using the Amelia package in R. Does it matter if I have the data in long format (with id, time, and value in each row) or in wide format (with id,...
• 333
36 views

Simple demonstration of imputation data leakage?

I'm aware that it's best practice to do all pre-processing within train-test splits, including data imputation. At least, it's recommended not to use the test data to generate the imputation model for ...
• 145
1 vote
18 views

Can kriging method be used for dataset that is not spatial?

Sorry if the question sounds stupid. I am new to this. Consider the following dataset: ...
25 views

Compensating for anomalies in training data by diluting them with older data?

I'm having trouble finding references to (and the name of) an obvious correction procedure that surely must have been used extensively in time series forecasting, but my Google searches keep turning ...
59 views

MICE multiple imputation in R - imputation number

I'm running MICE for 100 imputations with big data (~600k rows). Due to storage restrictions at work (which I am not permitted to change), I can't save all 100 imputations in one go, and I'd hit ...
33 views

How to perform MCMC posterior sampling of $X+Y$ random variable?

Let $X$ follow normal distribution with mean $\mu_X=10$ and variance $V_X=20$. Let $Y$ follow truncated normal distribution with mean $\mu_Y=-10,V_Y=8$, where $Y\leq 0$ by truncation. Let $Z=X+Y$. ...
• 1,406
1 vote
18 views

Is there any issue in imputing missing observations when the missing observations are related to each other?

Say that we have a cross-sectional dataset with two variables, A and B. Also suppose that A and B are related to each other in some way. Now, there are some rows for which only A is missing, and some ...
• 11
17 views

Questionnaire validity with imputed data, possible?

I would like to validate a 30-item questionnaire. There is a a lot of missing data. How do I handle missing data. My questions are the following: a) Do I have to use only complete case data to ...
22 views

My KNN result not as expected for all numerical data

I have developed the following KNN code and tested it against several datasets with >90% accuracy (the wdbc dataset from UCI for one, granted it was a categorical result), however when using the ...
246 views

when working with missing data, what percentage of data is considered too much missing before implementing something like imputation?

I am looking for advice (do not have a specific example regarding data) but am wondering, when working with any dataset that is missing, at what point/percentage would you consider using something ...
• 345
50 views

Imputation process before LDA

I want to perform LDA in my cohort which is based on 140 inviduals distributed according in 3 groups. These individuals have undergone an analysis of 50 variables (gene expression). So my dataset is ...