Questions tagged [data-imputation]

Refers to a general class of methods used to "fill in" missing data. Methods used for doing this typically are related to interpolation (http://en.wikipedia.org/wiki/Interpolation) and require assumptions about why the data is missing (e.g. "missing at random")

Filter by
Sorted by
Tagged with
27
votes
5answers
54k views

Imputation of missing values for PCA

I used the prcomp() function to perform a PCA (principal component analysis) in R. However, there's a bug in that function such that the ...
14
votes
4answers
11k views

How to handle with missing values in order to prepare data for feature selection with LASSO?

My situation: small sample size: 116 binary outcome variable long list of explanatory variables: 44 explanatory variables did not come from the top of my head; their choice was based on the ...
7
votes
1answer
1k views

Can I replace NAs based on response variable?

My data consists of 1 response variable 'Age' and 1 feature (beta). The feature contains some missing values (NA) so I want to replace them. I've been replacing them with the median of the feature. ...
4
votes
1answer
861 views

Problems with Missing values

I have a data set for a predictive model(predicting survival rate with certain acute medical condition on some animals) with 25 predictors where around 30% of the predictors are complete, 3 predictors ...
16
votes
3answers
4k views

Methods to work around the problem of missing data in machine learning

Virtually any database we want to make predictions using machine learning algorithms will find missing values ​​for some of the characteristics. There are several approaches to address this problem, ...
10
votes
1answer
7k views

Multiple imputation for missing count data in a time series from a panel study

I am trying to tackle a problem which deals with the imputation of missing data from a panel data study(Not sure if I am using 'panel data study' correctly - as I learned it today.) I have total death ...
10
votes
2answers
17k views

Imputation with Random Forests

I have two questions on using random forest (specifically randomForest in R) for missing value imputation (in the predictor space). 1) How does the imputation algorithm work - specifically how and ...
10
votes
3answers
9k views

How to perform SVD to impute missing values, a concrete example

I have read the great comments regarding how to deal with missing values before applying SVD, but I would like to know how it works with a simple example: ...
12
votes
5answers
10k views

How to perform imputation of values in very large number of data points?

I have a very large dataset and about 5% random values are missing. These variables are correlated with each other. The following example R dataset is just a toy example with dummy correlated data. <...
7
votes
2answers
18k views

Fast missing data imputation in R for big data that is more sophisticated than simply imputing the means?

I need a package for missing data imputation in R. But since I am dealing with big data, the number of missing data entries can also be high. The packages which impute using mean or median are of ...
1
vote
4answers
463 views

Good references on learning how to deal with missing data/imputation

Could you recommend up-to-date and well-supported references on the topic of data imputation?
6
votes
4answers
3k views

Imputing missing observation in multivariate time series

Suppose I have a dataframe consisting of six time series. In this dataframe, some observations are missing, meaning at some timepoints all time series contain a NA-value. In R, one possible imputation ...
5
votes
3answers
2k views

A data set with missing values in multiple variables

I'm trying to analyze a set of data related to the health area but I'm not sure how to proceed with the missing values. Objective: To adjust a model with a discrete response, to study the influence ...
5
votes
1answer
4k views

Hot deck imputation, ''it preserves the distribution of the item values'', how can that be?

I read in this link, under section 2, first paragraph about hot deck that ''it preserves the distribution of item values''. I do not understand that, if one and the same donor is used for a lot of ...
7
votes
1answer
567 views

Missing data at random

How does one tell if a dataset is missing data at random? I've been reading up on how to impute missing values, and was wondering what techniques can be used to tell if data is really missing at ...
5
votes
3answers
3k views

Stepwise regression modeling using multiply imputed data sets

After multiply imputing data, it is natural to estimate regression models on the data. When multiple predictors are available, sometimes stepwise regression is used for model building (forward ...
2
votes
2answers
350 views

Trouble with imputed data set

1) I had a dataset with missing data for baseline variables and outcome variables. Through multiple imputation in SPSS (10 imputations, 50 iterations, PMM for scale variables) I imputed the missing ...
16
votes
2answers
14k views

How to fill in missing data in time series?

I have a large set of pollution data that has been recorded every 10 minutes for the course of 2 years, however there are a number of gaps in the data (including some that go for a few weeks at a time)...
9
votes
2answers
5k views

Using multiple imputation for Cox proportional hazards, then validating with rms package?

I've been researching the mice package, and I haven't yet discovered a way to use the multiple imputations to make a Cox model, then validate that model with the rms package's ...
15
votes
1answer
13k views

How do the number of imputations & the maximum iterations affect accuracy in multiple imputation?

The help page for MICE defines the function as: ...
14
votes
1answer
30k views

XGBoost can handle missing data in the forecasting phase

Recently I have reviewed XGBoost algorithm and I have noticed that this algorithm can handle missing data (without requiring imputation) in the training phase. I was wondering if XGboost can handle ...
10
votes
3answers
4k views

Combining two time-series by averaging the data points

I would like to combine the forecasted and backcasted (viz. the predicted past values) of a time-series data set into one time-series by minimizing the Mean Squared Prediction Error. Say I have time ...
9
votes
1answer
13k views

How does the mice imputation function work?

I was wondering if anyone had experience using the mice function, as described in mice: Multivariate Imputation by Chained Equations in R (JSS 2011 45(3))? I have a dataset with a number of variables, ...
24
votes
6answers
3k views

What are the disadvantages of using mean for missing values?

I have an assignment (Data Mining course) and there is a part which asks: "What are the disadvantages of using mean for missing values?" in Missing Value section. ...
8
votes
2answers
16k views

How to combine multiple imputed datasets?

I need a single imputed dataset (e.g. to create a country group dummy from the imputed country per capita income data). R offers packages package for creating multiple imputed data (e.g. Amelia) and ...
3
votes
4answers
2k views

Missing data and imputation in general

Handling missing data is a bit confusing for me. My questions are: Is it better to calculate imputations than simply leave out NAs and leave it to the (appropriate) model to handle it? Is there a ...
21
votes
3answers
3k views

How to combine confidence intervals for a variance component of a mixed-effects model when using multiple imputation

The logic of multiple imputation (MI) is to impute the missing values not once but several (typically M=5) times, resulting in M completed datasets. The M completed datasets are then analyzed with ...
16
votes
5answers
46k views

KNN imputation R packages

I am looking for a KNN imputation package. I have been looking at imputation package (http://cran.r-project.org/web/packages/imputation/imputation.pdf) but for some reason the KNN impute function (...
13
votes
2answers
531 views

using neighbor information in imputing data or find off-data (in R)

I have dataset with assumption that nearest neighbors are best predictors. Just a perfect example of two-way gradient visualized- Suppose we have case where few values are missing, we can easily ...
6
votes
2answers
4k views

Should I use missing value using imputation or listwise or pairwise deletion methods?

I have 60,000 data and around 45% of them is missing and the missing values are random. Can I simply use listwise or pairwise deletion or do I have to use imputation? If imputation is recommended ...
6
votes
1answer
3k views

Imputation methods for time series data

I have some network data which measures the noise levels in a cellular network. On a typical mast there are generally 3 sectors or antennas which point in different directions. Within one of these ...
1
vote
1answer
4k views

Adjustment for missing values of the categorical variables in a data set

I Have a data set containing about 40 categorical variables. I am trying to factor analyze them. But each categorical variable contains a good number of missing values. Some of them are simply because ...
10
votes
3answers
1k views

What is the advantage of imputation over building multiple models in regression?

I wonder if someone could provide some insight into if an why imputation for missing data is better than simply building different models for cases with missing data. Especially in the case of [...
8
votes
1answer
1k views

How to use restricted cubic splines with the R mice imputation package

I am wondering how to integrate restricted cubic splines (such as in the rms package) in the imputation models within R mice imputation package. Context: I am doing biomedical research and have ...
4
votes
2answers
4k views

Simulate MAR (Missing at Random) data

I am trying to generate MCAR, MAR and MNAR data. MCAR and MNAR are relatively easy. However I am struggling with MAR data. I generate 500 observations with 2 variables (Y and X) out of a ...
15
votes
1answer
586 views

Pooling calibration plots after multiple imputation

I would like advice on pooling the calibration plots/statistics after multiple imputation. In the setting of developing statistical models in order to predict a future event (e.g. using data from ...
8
votes
2answers
4k views

Missing rates and multiple imputation

Is there a limit which is the least acceptable when using multiple imputation (MI)? For example can I use MI if the missing values in a variable are the 20% of the cases while and other variables ...
4
votes
2answers
4k views

How to impute an ordinal variable with MICE but prevent it from taking one value?

I have an ordinal variable, overall_tumor_grade, that can take on values of 1, 2, ...
2
votes
1answer
1k views

Predictive Mean Matching as Single Imputation?

Multiple imputation is known to be advantageous compared to single imputation. However, in practice there are often non-statistical reasons why multiple imputation can not be used (e.g. the data ...
2
votes
0answers
85 views

Method for predicting price based on Geographical market, Product, and Company

I have a dataset which tracks the prices of 21 products, charged by 24 companies, in 150 different cities across the globe. However, the data set has missing values--that is, I might have Company X's ...
9
votes
1answer
125 views

Does imputation introduce unacceptable bias?

I have recently come to know about imputation techniques, which, in short, "guess" realistic values with which to replace missing values in a dataset. My big issue with this is that we are ...
5
votes
4answers
3k views

Using information on both sides of a 'gap' in time series data for imputation

As with my previous question, I'm looking at ways to impute missing data in a hierarchical time series data. With al my other procedures, including the experimentation of imputation packages (...
5
votes
3answers
5k views

Multiple imputation on single subscale item or subscale scores?

Recently I am conducting a research on the relationship between motivation/attitude variables (Gardner's model) and English language proficiency in the Philippines. I encountered a problem: missing ...
5
votes
1answer
783 views

Missing data not at random - Advice needed on method

I have been developing a logistic regression model based on retrospective data from a national trauma database of head injury in the UK. The key outcome is 30 day mortality (denoted as "Survive" ...
3
votes
2answers
4k views

how to check missing data is missing at random or not?

I have a survey data, in which there are some missing data (not answered questions). I threw away those where the whole page(s) questions were missed, but there are still some with unanswered ...
2
votes
2answers
48 views

Missing values in a variable depending on the values of another variable

I'm working on a public procurement dataset where I have information on all the participants that presented offers in 358 tenders. I'm analysing relationships between all the companies of the dataset (...
2
votes
1answer
631 views

Imputing missing outcome data

I saw the other link (Multiple imputation for outcome variables) discussing missing outcome data imputation for complete case analysis. However, I have missing outcome data as well as missing ...
2
votes
1answer
1k views

Missing data - Regression imputation

I want to produce imputations for the missing values using a naive imputation method "Regression imputation " . The first step involves building a model from the observed data then predictions for the ...
1
vote
2answers
294 views

Framework of multiple imputation

I read this paper about ("Multiple Imputation For Missing Data: What Is It And How Can I Use It?")(http://www.csos.jhu.edu/contact/staff/jwayman_pub/wayman_multimp_aera2003.pdf) Does any one have ...
1
vote
1answer
103 views

Proper Imputation and bias-correction on degrading signal with Kalman Filtering?

A signal degrades in its quality. Some signals are far more robust to degradation while others are not. We will simulate degradation by randomly removing values from a function and then applying ...