Questions tagged [data-imputation]

Refers to a general class of methods used to "fill in" missing data. Methods used for doing this typically are related to interpolation (http://en.wikipedia.org/wiki/Interpolation) and require assumptions about why the data is missing (e.g. "missing at random")

Filter by
Sorted by
Tagged with
1 vote
1 answer
148 views

Multiple imputation in R with mice package

I have conducted a multiple imputation in R with 5 imputations and 50 iterations using the function mice() from the corresponding mice package. Now that I have ...
user avatar
1 vote
3 answers
844 views

Variable-specific random sample imputation. Is it a valid method of imputation?

Is random sample imputation a valid method of imputation for categorical variables? Not randomly drawing from any old uniform or normal distribution, but drawing from the specific distribution of the ...
user avatar
  • 113
1 vote
0 answers
22 views

Dropping Missing Observations under MAR Assumption

Some of the outcome data in my RCT data set are missing. I believe that the missing data mechanism is missing at random (MAR) as the observed characteristics significantly differ between the missing ...
user avatar
  • 11
0 votes
0 answers
16 views

Can I impute a variable using MICE so that I can use the value(s) from this imputed variable to then code another variable?

I am working with pregnancy data where I would like to impute a variable called LABOR PRESENTATION (nomical var. with 5 categories) from several other variables but then create a variable called ...
user avatar
0 votes
0 answers
26 views

Drop or impute predictor that is missing around 70% of values, but which is known to be highly relevant?

Suppose we have a medical dataset and we are interested in predicting blood pressure using the following variables: age, sex, weight, height, volume of circulating blood, cardiac output, parent with ...
user avatar
0 votes
1 answer
21 views

Can I build an equation from just few observations?

I have the following data that has 5 observations and 2 missed observatiosn. I want to build an equation that can help estimate the missed observations. ...
user avatar
3 votes
1 answer
227 views

How to deal with missing values of explanatory variables when comparing models

I want to compare several logistic regression models. The different models are built using the same initial dataset. The models differ with respect to the explanatory variables included. Many of the ...
user avatar
  • 153
1 vote
2 answers
40 views

How does one effectively deal with data imbalance while working on a NLP problem without dropping data points?

I am working with a data set of fake job postings and it has the columns following columns: ...
user avatar
  • 11
2 votes
1 answer
263 views

Is it better to impute a feature with single value or impute based on frequency?

Last time I posted a question in Stackoverflow how to fill nans based on frequency. I got some comments about whether it is a good idea or not. So I am seeking some suggestions if this is actually a ...
user avatar
1 vote
1 answer
301 views

Multiple imputation and normality assumption

I am reading the an online book by Stef Van Buuren (link at bottom) regarding multiple imputation. In Section 3.2.1 he lists 4 different approaches to multiple imputation: Later on in Section 3.3 he ...
user avatar
  • 614
10 votes
1 answer
264 views

Does imputation introduce unacceptable bias?

I have recently come to know about imputation techniques, which, in short, "guess" realistic values with which to replace missing values in a dataset. My big issue with this is that we are ...
user avatar
0 votes
1 answer
957 views

How to use missForest in R for test data imputation?

I want to use the R missForest() function at work to perform missing value imputation. However, after reading up on the algorithm more, I can't decide how to impute ...
user avatar
1 vote
1 answer
145 views

Imputing missing values for linear regression model, using linear regression

I scraped a real estate website and would like to impute missing data on total area (about 40% missing) using linear regression. I achieve the best results using price, number of rooms, bedrooms, ...
user avatar
2 votes
2 answers
187 views

Imputing data using covariance?

Suppose I have some samples of sensor data, where each row has ten measurements from various sensors. And suppose I know what the covariances are among these sensor measurements. Are there any ...
user avatar
  • 21
0 votes
0 answers
23 views

how to estimate the precipitation value from other relative data

I have climate data but the precipitation values are missing, I would like to know if there is any formula used to estimate the precipitation value from the other recorded climate data: temperature, ...
user avatar
  • 153
0 votes
0 answers
22 views

Missing data roughly proportional to the clusters, does this indicate MAR?

I have data in which the number of missing values per cluster (in this case, zip-code), are proportional to the population. Does this indicate Missing at Random (MAR)? Third column with missing ...
user avatar
3 votes
1 answer
203 views

How to use multiple imputed data for survey estimation?

I'm trying to calculate population mean, median, (etc, descriptive analysis) using multiple imputed data. However, the example that I found in sources were regression and then pool them into one ...
user avatar
1 vote
2 answers
217 views

When is it OK not to keep a testing/holdout set?

I am performing data imputation on a large matrix [100000,34] of past measurements that contains missing values (rows are time-steps and columns are stations). So far I've used several machine-...
user avatar
  • 117
0 votes
0 answers
2k views

How to check whether the missing data is MCAR, MAR, or MNAR? [duplicate]

I read few responses close to the question and was suggested in using t-test or chi-sq test. However, the pattern between variables can also involve more than 2 variables (e.g. data at x tend to be ...
user avatar
0 votes
1 answer
46 views

How reliable must a regressor be for data imputation?

I have a dataset that seemingly has missingness not at random and am thinking on using regressor to fill the missing values. I know that complete case analysis is on the table; however almost 20% ...
user avatar
1 vote
0 answers
52 views

Can KNNImputer be used in filling time series data?

I was wondering whether I could fill some null values in a time series dataset about the air quality of India, with knnimputer. Because it seems reasonable to say some days are similar to each other ...
user avatar
6 votes
2 answers
234 views

At what point in analysis do you perform imputation for missing variables?

There is a dataset with 30 variables and over 5 million observations. We plan to use a subsample of the data for analysis. Around .02 - 2.5% of EACH variable are missing. I plan imputation in Stata ...
user avatar
  • 211
1 vote
1 answer
21 views

Should we try imputation on cases with slightly problematic datasets or prefer ommiting observations

I would like to ask a general question which makes me worry when I try to impute NA values. We know that most of the imputation methods are based on the rest non-NA values. However, if we know that ...
user avatar
  • 1,679
0 votes
0 answers
42 views

Imputation where some data points should not be imputed

I am trying to impute the RSSI (Received Signal Strength Indicator) measurement from different WiFi access points. In this instance, data can be missing for two reasons: either the laptop did not look ...
user avatar
  • 101
0 votes
0 answers
41 views

replacing nas when 50% or more of the data is missing

I have to perform a linear regression on a dataset. However, I am having trouble figuring out what type of imputation I should do on the data because in some cases the majority of the the data is ...
user avatar
  • 1
0 votes
1 answer
28 views

Imputation: which method is this

I'm working on a paper on machine learning. Suppose I have three classes (A, B, C). Suppose now I have a model that outputs ...
user avatar
1 vote
0 answers
26 views

How should I use multiple imputation when certain questions only appear in later survey waves?

I have what I'll call L-shaped panel data: ...
user avatar
1 vote
0 answers
27 views

How to identify whether my data follows "Missing At Random" (MAR) mechanism or not? [closed]

I was having two similar studies with two variables (anti-gE and anti-VZV (continuous variable)) linearly related to each other (with same relationship between both the studies in anti-gE and anti-VZV)...
user avatar
0 votes
1 answer
79 views

Iterative Imputor gives the same output for all the values it has to impute

I have a df named so as follows: ...
user avatar
  • 125
0 votes
1 answer
75 views

MissMDA Bayesian MIPCA help/MIFAMD

While looking into methods relating to principal components related imputation, I came across MissMDA package in R. Could anyone please tell me what the Bayes in the MIPCA means in layman terms? I ...
user avatar
2 votes
1 answer
2k views

How to decide whether missing values are MAR, MCAR, or MNAR

I have a large proteomics dataset. In the rows I have the proteins , and in the rows I have the samples.The dataset contains a lot of missing values. I would like to know I can find out whether ...
user avatar
1 vote
0 answers
118 views

MCAR tests on mixed variable datasets [findings + discussion]

I have been spending a whole week on researching about testing on missingness mechanism in datasets and I thought it will be helpful to share what I found out about the 2 available MCAR test packages ...
user avatar
1 vote
1 answer
179 views

Will use of imputed values by regression imputation cause multicollinearity?

If I want to regress y on some variables, one of which contains missing values. For example, regress the price of a rental house on its size, location, Wifi(dummy), (number of) bedrooms, neighbours' ...
user avatar
3 votes
1 answer
675 views

Is it advisable to impute missing values and scale features before computing the Variance Inflation Factor (VIF)?

As far as scaling, Wikipedia says: Finally, note that the VIF is invariant to the scaling of the variables (that is, we could scale each variable Xj by a constant cj without changing the VIF). ...
user avatar
  • 259
0 votes
0 answers
14 views

Impute Continuous Predictor which 0 or median is not an option

I have a dataframe of the following patients: PatientID Days.To.Develop.Symptoms 1 0 2 1 3 3 4 NA ...
user avatar
1 vote
1 answer
73 views

Is Matrix Factorization also going to work with one feature?

I need to fill missing values. I have found that there are many approaches such as the mean and the median of the feature as well as using Matrix Factorization. However, I am kind of wondering if I ...
user avatar
  • 2,583
24 votes
6 answers
5k views

What are the disadvantages of using mean for missing values?

I have an assignment (Data Mining course) and there is a part which asks: "What are the disadvantages of using mean for missing values?" in Missing Value section. ...
user avatar
  • 241
2 votes
1 answer
156 views

Imputing Missing Value with Response Variable?

I have a dataset with 12 predictor variables and a binary response variable. There's 5960 observations. One of the predictor variables has 1,260 missing values so I'm using k-nearest neighbours to ...
user avatar
1 vote
1 answer
64 views

Fill or not to fill? That's the question

I have a dataset which shows the expense of users in a specific expense category daily, along the time. I am building a time series in order to predict whether this person will buy this some product ...
user avatar
  • 11
2 votes
0 answers
165 views

Should I involve the dependent variable when imputing missing values for an independent variable?

I am building a model for predicting whether my users would recommend my wine. In the final model, the dependent variable, $Y_{ni}$ is whether user $n$ recommends wine variant $i$. However, in some ...
user avatar
  • 101
1 vote
0 answers
72 views

What affect does multiple imputation have on Type 1 error and should I subset data before MI?

I'm conducting a hierarchical linear regression on a relatively small data set (n = 41) where missing data ranges from 0 to 47%. I have two questions about how multiple imputation (MI) may influence ...
user avatar
3 votes
1 answer
1k views

Can VAR models handle time series with different lengths?

I'm trying to better understand how Vector Auto-Regression models work in practice. It seems to me that a real world time series data set is likely to have time series of different lengths, but I can'...
user avatar
4 votes
1 answer
251 views

How to impute right-censored data

I have a dataset of vectors representing movement with various characteristics. Some vectors represents the movement that was stopped by external factor and therefore, observed value for length of ...
user avatar
  • 185
0 votes
0 answers
80 views

Do you clean the data before calculating MASE (Mean Absolute Scaled Error)

The denominator in the MASE calculation for seasonal data is the MAE of the seasonal naive forecast calculated in-sample. Is it common to do imputation before calculating the seasonal naive MAE or ...
user avatar
  • 943
1 vote
0 answers
26 views

How to handle missing data in a small sample

We've conducted a cohort research (single group), with 30 subjects. The study is an observational study, with two measurements (at the beginning and at the end of the study period). 5 of them were ...
user avatar
4 votes
0 answers
38 views

Auxiliary variables for imputation in survey data

I have a situation where I have a survey that contains two parts, everyone answers all of the questions on the first part, and a sample (10 percent) of the people are selected to answer some further ...
user avatar
  • 1,408
0 votes
0 answers
14 views

Analysis of imputed data [duplicate]

I have 3 questions regarding the analysis of imputed data. I have an idea how to do the analysis, but want to confirm with you guys that it's the correct way. 1) I had a dataset with missing data for ...
user avatar
  • 21
2 votes
2 answers
599 views

Trouble with imputed data set

1) I had a dataset with missing data for baseline variables and outcome variables. Through multiple imputation in SPSS (10 imputations, 50 iterations, PMM for scale variables) I imputed the missing ...
user avatar
  • 21
1 vote
1 answer
73 views

Machine learning for imputation?

I am currrently working on a paper where we have two datasets, where I wish to impute variables from one dataset onto the other. The way that I have been currently thinking about this is to use ...
user avatar
0 votes
0 answers
85 views

Do I impute missing values with the response?

I have a dataset with missing values in both predictors and the response. As far as I know, the data are missing not at random, so I cannot simply use listwise deletion. Instead, I employed the EM ...
user avatar
  • 1,385

1 2
3
4 5
13