Questions tagged [data-imputation]
Refers to a general class of methods used to "fill in" missing data. Methods used for doing this typically are related to interpolation (http://en.wikipedia.org/wiki/Interpolation) and require assumptions about why the data is missing (e.g. "missing at random")
694 questions
0
votes
1
answer
24
views
Reverse Coding Likert Data for Cronbach's Alpha
I am completing an exam whereby we were given Likert data in which 2 questions were negatively worded. The data contained missing values which are MCAR (listwise deletion isn't appropriate it loses 3/...
0
votes
0
answers
26
views
Efficient Imputation method for big longitudinal dataset in R
I have very big dataset of around 3 million rows and 50 variables of different types. The dataset is longitudinal in long format (around 350 000 unique individuals). I want to impute missing data ...
- 33
0
votes
0
answers
12
views
mice multilevel imputation: does specifying cluster variable ("-2" in predictor Matrix) without multilevel methods lead to cluster robust imputation?
In short: Are mice's imputations cluster robust when I only specify the cluster variable with "-2" in the predictor matrix but do not use multilevel models during imputation?
For clustered ...
- 1
0
votes
0
answers
14
views
How to reduce prediction error in a long-term time series prediction
In this project, I aim to predict a missing segment of 100,000 points in the middle of a sequence using a total of 100,000 points before and after the gap. I have already tried polynomial fitting and ...
- 1
2
votes
1
answer
39
views
Best Practices for Imputing Missing Data in Trade Data (Linear Interpolation and Random Volume)
I am working on a dataset containing trade data, and my goal is to impute the missing data for a period of around 24 hours. Here's a sample of the trade data I'm working with:
timestamp
symbol
price
...
- 21
1
vote
0
answers
37
views
One logistic regression model w/ multiple predictors or multiple models with one predictor each?
I imagine this is a sloppy solution to my problem and perhaps even completely invalid, but I'm trying fit a logistic regression model where I have some empty values (...
- 405
1
vote
0
answers
28
views
Can cold-deck imputation used to address class imbalance?
If you have experience on using cold-deck imputation to address class imbalance problem in the dataset, could you please share more details or reference papers?
Edit to add:
The modeling methodology ...
- 11
1
vote
1
answer
34
views
Models with Uncertainty in Y
Let's say I have data with predictors x1, x2, x3...xn for a variable y. I have essentially imputed y using a Bayesian analysis, which means I have a posterior distribution for each value of y. To ...
- 107
0
votes
1
answer
36
views
Is there any way to can deal with missing data without imputation ? model considering NA values [closed]
My question is how to model data with NA values and without imputing. Is there any possibility? and what is the advantage and disadvantage? The problem is classification.
- 179
6
votes
2
answers
539
views
Justification for imputation with over 50% missing data
I'm looking to get some advice/thoughts on the following situation: let's say I have a prospective, observational study that was designed to assess change in BMI over two years of follow-up (primary ...
- 173
1
vote
1
answer
53
views
Multiple imputation with some missing baseline data and with some missing longitudinal data
How shall I impute the data in the following situation: I have some baseline covariates collected and longitudinal data. Both baseline covariates and longitudinal data have some missing data.
Shall I ...
- 337
0
votes
0
answers
52
views
Difference between copy increments and copy reference imputation
I'm reading up on reference-based imputation (Carpenter et al. 2013, but the explanation is a little bit too mathematical for me) and I'm not quite sure if I understand the difference between copy ...
0
votes
0
answers
35
views
Multiple imputation mice pmm
I need to impute 20 yes/no variables in my dataset. In this dataset, 35% of the individuals have all 20 of these variables missing, while the remaining 65% have complete data for these variables. ...
- 11
0
votes
1
answer
25
views
How does KNNImputer stores fitted values of the train set?
If someone here is familiar with the KNNImputer implementation of Scikit-learn, I would be eager to learn this from him.
When you fit an Imputer transformer on your ...
- 53
0
votes
1
answer
16
views
When imputing a numeric column, does it matter if the imputed value is impossible?
Suppose we have a table with flight data, and one of the numeric columns denotes airline ID. The same airline appears many times in the table, so this column can potentially have predictive value.
...
- 1,453
0
votes
0
answers
26
views
2
votes
2
answers
206
views
Multiple Imputation for missing outcomes in Cox regression
Imagine an RCT with a time-to-event outcome which is analyzed using a Cox regression. There are four assessments (T1=before randomization, T2=3 weeks, T3=6 weeks, T4=12 weeks). Under the censoring at ...
- 149
0
votes
0
answers
10
views
Handling valid missing data in prediction model features
I'm developing an elastic net model using caret, with k-nearest neighbours to handle missing data in the features.
Some features are conditional on others such that a missing value is valid e.g., ...
- 3
1
vote
1
answer
84
views
What is the name/terminology for this application of OLS regression
I don't come from a statistics background and was instructed to follow these steps to fill in missing data. I'm wondering if there is a name for this specific method so that I can learn more of it and ...
- 13
0
votes
0
answers
62
views
How to deal with missing values in a panel survey (for Propensity score matching analysis)
I would like to know what is the recommended method for data imputation for propensity score matching in panel survey data. This survey has 4 waves and I am examining the treatment effect between the ...
0
votes
0
answers
25
views
Feature Selection Before or After kNN Imputation?
From my understanding, kNN imputation is dependent on the variables where any two cases do not have missing values. Thus, would it be ideal to do feature selection before or after imputation?
- 1
0
votes
0
answers
29
views
Large Scale Missing Data & Imputation of Time Series Data in Neural Networks [duplicate]
I know there has already been a lot of discussion about this topic, but I have reasons to believe it still remains unanswered and lacks several justifications.
Suppose we have an time series feature ...
- 101
1
vote
0
answers
24
views
Imputing a Continuous P-value Distribution from Discrete P-values [closed]
I'm exploring methods to create a continuous-like P-value distribution from discrete P-values. For example, consider the process where a balanced coin is tossed four times per experiment to note the ...
- 143
1
vote
1
answer
58
views
Question on Best Transformation (Negative, Zero, Positive Values) + Missing Data
I have a dataset with $5000$ observations, and 10 explanatory and 1 response variable (binary 0 or 1), and my task is to make a logistic regression model for prediction (but also needs to provide some ...
- 23
5
votes
1
answer
167
views
Best way to impute missing values in a time series
I have a camera that detects every time it views a car. Each detection is recorded in a database. I then simulate this behavior as a time series by doing an each hour count of the records.
The problem ...
- 141
0
votes
0
answers
22
views
How can I combine fully measured with partially measured outcomes in an IPD (Network) Meta-Analysis?
I was asked to help in an Individual-Patient Data (IPD) Network Meta-Analysis (NMA). Outcomes are supposed to measure combined scores of condition 1 and condition 2 (thus: $Score=Score_1 + Score_2$), ...
- 1,262
1
vote
0
answers
75
views
After the mutiple imputation (MICE package in R), I still found that some variables are still with missing values. How to deal with it?
I have a relatively large data set with around 12000 samples with 550 variables. Originally, I have around 800 variables, I used a rule that if missing rate in each variable is larger than 80% I will ...
- 61
0
votes
0
answers
63
views
Imputing missing observations of zip code level data
I am looking for a sufficient imputation method for missing observations in my zip code level data, using R.
I have a random sample consisting of households which live in different zip codes within ...
0
votes
1
answer
33
views
Vectorized Linear Regression with missing values
I have a few questions about handling missing data during matrix/array operations. Essentially, I'm doing a vectorized linear regression to perform a bunch of linear regressions in a few matrix/array ...
0
votes
0
answers
28
views
Should EM algorithm's final imputed mean match the initial parameter?
I am running a manual EM (expectation-maximization) algorithm in r. My code is the following:
...
- 65
1
vote
0
answers
54
views
Manually program EM in r to updated multiple parameters and solve missing data [closed]
I am trying to use EM (Expectation-maximization) to fill in missing data in R, but am not sure how to model/code it for my specific case. I am generally trying to follow the example format used in ...
- 65
1
vote
1
answer
41
views
Impute missing value [closed]
In machine learning when I impute missing values which of the following I perform :
1-Impute data set and then split it?
2-Split dataset to Training and testing datasets and then Impute each datasets ...
- 335
1
vote
0
answers
33
views
median imputation or case deletion for small samples
I have a dilemma. I have a data set of different proteins from 4 groups of animals characterized by two factors: age (3 or 12 months) and presence of a certain gene (KO or WT). For each combination of ...
- 33
2
votes
1
answer
82
views
Complete case analysis, multiple imputation, or instrumental variable analysis?
I ran a small study where all participants attended a psychological intervention, and completed measures at pre, post, and follow-up. Approx. 40 participants took part, of which approx. 5 did not ...
- 21
1
vote
0
answers
20
views
0
votes
1
answer
64
views
Is it possible to replace missing data if 50% of total data are missing?
In my study, 40K completed household surveys. However, when we suggested visiting a nearby health center to measure their physical parameters (height, weight, blood pressure, and blood glucose), only ...
- 1
0
votes
0
answers
32
views
What is the best method for imputing binary (or integral/count) data?
I have a longitudinal dataset, and I want to create a composite score by including five healthy lifestyles to measure the overall lifestyle over time (use as predictor). Each lifestyle is a binary ...
- 347
2
votes
1
answer
107
views
How to use priors to impute values at an individual level and replicate a distribution of the population?
I am trying to correct a variable from a survey that has measurement error. To do this, I have been taking this column as if it was missing and imputing new values based on the predictions of an ...
3
votes
0
answers
136
views
Gaussian Processes regression for missing data?
Suppose I have data with three dimensions and the joint is $P(X,Y,Z)$. Were Z to be missing on occasion, Gaussian Process regression could infer distributions over these missing values.
But suppose ...
- 3,520
2
votes
3
answers
65
views
Would it be preferable to use statistical imputation instead of a subject matter expert's subjective estimate for missing data?
I'm working on a project where I need a variable for the total number of medications a patient is on. The PI is a clinician and I feel they would be able to use the resources at hand - case note and ...
- 761
0
votes
1
answer
39
views
How can I perform an analysis of the NHIS imputed income variables?
I have downloaded five family income variables from https://nhis.ipums.org/nhis-action/variables/group?id=economic_income (INCPPOINT1, INCPPOINT2, INCPPOINT3, INCPPOINT4, INCPPOINT5) for they years ...
2
votes
2
answers
151
views
Does a relationship between missing values indicate MNAR?
I have a dataset where several features are missing values only if another feature is also missing a value. Does this indicate that the missing data is missing not at random (MNAR)? Additionally, how ...
- 55
0
votes
0
answers
13
views
Incremental Users On Platform Method
I have the following data (aggregated, user level data is not available).
date | platform | total_watch_time
platform = [mobile, mobile + desktop, mobile + desktop + tv]
I want to determine the ...
- 11
0
votes
0
answers
30
views
Bayesian imputation model for day of month when the year, month, and day of week are available
I'm considering including vehicle incident data paired to date, however the most extensive data for the region I am interested in does not have dates. I would like to include it in a multivariable ...
- 9,660
2
votes
1
answer
54
views
proof of missing at random
I have an longitudinal data with measurements of hand grip strength measured in kg by a hand-held dynamometer, using a maximum of three measurements taken with the strongest hand in an old population....
- 139
2
votes
0
answers
100
views
Multilevel multiple imputation in practice using R
I'm currently involved in a project where I want to address missing data using multiple imputation. I'm using healthcare data in a longitudinal setting with 16 time points, where observations are ...
- 128
0
votes
0
answers
43
views
How to impute the std deviation of a sample using the pooled mean and standard deviation of similar samples?
I want to perform a meta-regression based on the mean changes from baseline values.In one of the studies, the standard deviation has not been mentioned. How can I impute the value of the missed ...
- 63
4
votes
1
answer
97
views
Creating deliberate missing data to impute a counter factual
This question is more out of curiosity than necessity. Any useful resources/literature will be massively appreciated.
Recently started research in a new field that I’m unfamiliar with: economic policy ...
2
votes
0
answers
75
views
How to perform inference on regression coefficients when some X values are interpolated?
This should be a simple question but I'm having a hard time finding an answer.
Assume I have some data $y$ with covariates $X$. This set of data was obtained via a simple random sample. I want to ...
- 471
2
votes
0
answers
20
views
Is it possible to define a specific criterion prior to imputing missing values? [closed]
it's hard to summarise the problem in a question so I'll explain it here.
I have a variable called "Plant Species" that is missing a lot of values because of the sampling method used. For ...
- 21