Questions tagged [data-imputation]

Refers to a general class of methods used to "fill in" missing data. Methods used for doing this typically are related to interpolation (http://en.wikipedia.org/wiki/Interpolation) and require assumptions about why the data is missing (e.g. "missing at random")

Filter by
Sorted by
Tagged with
0
votes
0answers
12 views

Analysis of imputed data [duplicate]

I have 3 questions regarding the analysis of imputed data. I have an idea how to do the analysis, but want to confirm with you guys that it's the correct way. 1) I had a dataset with missing data for ...
2
votes
0answers
33 views

Trouble with imputed data set

1) I had a dataset with missing data for baseline variables and outcome variables. Through multiple imputation in SPSS (10 imputations, 50 iterations, PMM for scale variables) I imputed the missing ...
1
vote
1answer
45 views

Machine learning for imputation?

I am currrently working on a paper where we have two datasets, where I wish to impute variables from one dataset onto the other. The way that I have been currently thinking about this is to use ...
0
votes
0answers
15 views

Do I impute missing values with the response?

I have a dataset with missing values in both predictors and the response. As far as I know, the data are missing not at random, so I cannot simply use listwise deletion. Instead, I employed the EM ...
1
vote
0answers
20 views

Hot Deck Imputation

I am currently facing a set of data with missing values. I would like to impute these values with a Hot Deck. I have read upon the Hot Deck methods and decided to use Nearest Neigbour Hot Deck (NNHD) ...
0
votes
0answers
5 views

Mismatched sampling rates between predictors & response plus measurement errors on categories

Background I'm unsure how to best model data from a widget manufacturing process with measurement "uncertainties" on categorical variables (relative to an ordered indexing variable) and an overall ...
0
votes
1answer
33 views

What to assume if exact lower limit data values are not available

I have some data - about 100 values, maximum value 100. However, values below 10 are just written as <10, exact values for these are not available. How can I impute these data points. What value ...
3
votes
1answer
68 views

When performing imputation on categorical variables, does the data lose meaning?

Suppose I have a data set with several variables where one of my variables is categorical. For instance, a rating from 1 to 10. Suppose it has missing values. I want to impute this data via ...
3
votes
2answers
40 views

Dataset having about 1% missing fraction, missing completely at random. How would I address the missingness of the dataset?

I have a dataset comprising of 1% missing fraction where data is missing completely at random. How would I address the missingness of the dataset with a multiple imputation technique? For example, is ...
1
vote
0answers
18 views

Are my data a good candidate for EM imputation followed by exploratory factor analysis?

I am doing Exploratory Factor Analysis (EFA) in R, using principal axis factoring in the psych package. I have missing data that prevent me getting factor scores, so I am imputing data. I am using ...
0
votes
0answers
8 views

Does missing completely at random and imputing the mean together affect the bias or efficiency of the OLS estimator?

Suppose I am given a response variable $y$ and two predictors $A$, $B$. The correlations between these variables are given by $\rho_{AB} = 0.4395$, $\rho_{Ay} = 0.3141$, and $\rho_{By} = 0.9587$. ...
1
vote
0answers
21 views

Filling in Missing Data for Biological Experiment

I am trying to implement a semi-supervised learning model with biological data. In my case, I'm using features from DNA. I have a number of experiments each with many observations. Each observation ...
0
votes
1answer
32 views

Dataset that has approximately 1% missing values completely at random?

I have a dataset with approximately 1% of values missing completely at random. I have thought about using the Multiple Imputation technique but I am not sure if this would be the best solution. ...
1
vote
0answers
5 views

Doing multi-value imputation to maintain the same distribution

I currently learning value imputation. The popular methods that I've seen such as mean, median, arbitrary value etc, impute all missing values with a single calculated value. Each of these methods can ...
0
votes
1answer
29 views

Imputation: why impute missing test set values from train set?

I've consulted other posts here regarding imputation and the recommendation seems to be to not impute on unpartitioned data. Data should be partitioned and missing values in both train and test set ...
0
votes
0answers
57 views

Impute Missing Data Values with Mixture Models

Suppose I have a dataset with $o$ representing a collection of data dimensions with observed values and $d$ representing missing dimensions. The mixture model consists of discrete variables $Z = {1, .....
2
votes
1answer
48 views

Why is a sample covariance not even semi-positive definite with missing data?

I am trying to estimate a sample covariance when I have less observations $n$ than variables $p$ ($n<p$). This will serve later on as basis for a shrinkage estimator. We know (see this post) that ...
1
vote
0answers
21 views

Imputation of a single long period of missing values in a multivariate time series

I am working with a multivariate time series (with daily data) of approximately 40 years. It consists of the data of two stations (dataset A & dataset B) measuring both the river level and the ...
0
votes
0answers
16 views

Missing data not at random imputation for a ranking

Let's say I am interested in the differences in intelligence of 200 individuals, based on a huge database which ranks almost the entire population. The database do not record intelligence scores, but ...
0
votes
1answer
26 views

Baseline carried forward missing data assumptions

Researchers assessed the effectiveness of a range of weight management programmes for weight loss. A randomised controlled trial study design, incorporating eight treatment arms, was used. Each ...
1
vote
0answers
177 views

How to implement MICE in data imputation using sklearn IterativeImputer? [closed]

I'm interested in learning how to implement MICE in imputing missing values in my datasets. I'd like to use sklearn IterativeImputer for the following reason (source from sklearn docs): Our ...
0
votes
0answers
15 views

How to implement single Imputation from conditional distribution?

In [*] page 264, a method of drawing a missing value from a conditional distribution P(X_mis|X_obs;Theta) which is defined as: I did not find any code ...
1
vote
0answers
17 views

Multiple imputation for count data

Date, Transect, Year, Species, Tr.Length, Month, numb.on.transect I am working with historical sea-survey data.... boat transects counting birds on water. only there are some transects ...
1
vote
0answers
25 views

Missing Data imputation on one continuos column which depends on another feature and which does make sense only when such feature is positive

For each row (open contract) of my dataset, I have got a certain number of orders. I have created some features related to such orders; let's take for instance the average and the std deviation of the ...
1
vote
0answers
11 views

Correcting for known imputation bias

I have a survey sample which includes income values by type of income for a significant number of high-income households. Some of the income data is measured, i.e. present in the responses of the ...
1
vote
0answers
19 views

How to impute a categorical variable with MICE but prevent it from taking some values?

I have a categorical variable, var1 , that can take on values of W, B, A, M, N or P. There are some NAs that I want to impute using the mice package in R, but I know that the missing values cannot be "...
1
vote
0answers
15 views

Imputation that takes into account both relationships among variables and spatial adjacency?

I have a dataset with 13 variables and 50 observations representing the U.S. states. The variables represent the land use intensity of different agricultural industries in each state. Of those 650 ...
3
votes
1answer
74 views

How should I calculate a confidence interval for a gap-filled data set?

I have a data set of half hourly CO2 and CH4 fluxes (emission/uptake) from a landscape along with a set of environmental variables (Photon flux density (PPFD), temperature etc.). I'm using Neural ...
0
votes
0answers
8 views

Likelihood of having a set of attributes given knowledge of a subset of the attributes?

I am working on a problem that I am not sure how to model where we know a subset of and object attributes, but cannot be certain that we know all of the attributes. My goal would be to estimate the ...
2
votes
1answer
13 views

Visualized Imputation of 2D Sample Space With Saturation Corresponding to Confidence?

I want to visualize a 2D sample space imputed from a 2D scatter of measurements. Something like a 2D hexbin color histogram would work if it were enhanced in the ...
1
vote
1answer
32 views

How can I adjust for negative eigenvalues?

I wish to run a path analysis from a pooled correlation matrix that I have imputed using the maximum-likelihood procedure. There was considerable missing data. The resulting correlation matrix is: <...
0
votes
1answer
43 views

Correlation between two matrices and linear regression

I have two matrices with numeric data and a lot of missing values. The missing values are not completely coherent in both matrices. My aim is to fill the missing values so I can perform further ...
0
votes
0answers
22 views

Longitudinal analysis with high level of attrition

I have an initial dataset of 500 individuals at Time 1; 350 at Time 2; 150 at Time 3; 150 at Time 4. It's a bit of a headscratcher for me to figure out how to deal with this high level of loss to ...
1
vote
0answers
25 views

How to deal with features that are only possible for some sample in Tree based algorithms?

If I have a data set where, for instance, I am predicting something about Bank customers. Some customers have mortgages and some don't. For those with mortgages I want to include how long until ...
1
vote
0answers
31 views

Interpolate/Impute Time Series Data from another Time Series

I have a dataset of multiple lakes with water level elevations through time. The observations are not regularly spaced and have many large gaps. Further, some of the older observations may be of lower ...
0
votes
0answers
41 views

Item/Question left off Likert Scale. What can I do about “missing” responses?

I am using a certain 7 item Likert scale in my research and have collected my data - however I've now realised that I left one of the items (Item 7) off my initial survey. I have since collected all ...
0
votes
1answer
39 views

How to impute missing values not at random?

My data consists of 202 cases, each stand for a single interview. The variables reflect the interviewers' and interviewees' behaviours during four different parts of the interview: p1, g1, pA, gA. in ...
1
vote
1answer
23 views

EM Imputation and missing age data

I've completed an EM imputation to replace a small amount of missing data. One of the missing data was within my age variable (age range of 20 - 54). My question is... the new score that has been ...
1
vote
1answer
29 views

Any suggestion for imputing missing values in the following case?

I am doing a 0-1 classification problem. One of the features has a really high missing ratio(over 95%). So usually I will drop it. But for the rest 5%, almost all corresponding dependent variable is ...
4
votes
0answers
54 views

How much missing data can you replace with data imputation? [duplicate]

I have a data set with 40 participants that underwent 20 different tests. For most of these tests all the data is there. For 2 of the 20 tests, 2 out of the 40 values are missing. However, for 5 of ...
1
vote
3answers
59 views

Best approach for dealing with continuous predictors with missing data in random forests

I was thinking about a problem I'm facing: I have wage data that I want to add to my model, but it's incomplete (data for about 70% of my observations). So, I was ...
0
votes
0answers
55 views

What are the options to cross-validate imputed data generated with the MICE package?

I am currently imputing missing values in a dataset with the help of the MICE package. The dataset contains different types of variables (binary , continous numerical and ordinal variables). The code ...
2
votes
1answer
45 views

Missing data imputation when all variables have some missing data?

I am working with my resident survey data (n=8356) including 59 items, most of which are ordinal variables scored from 1 to 7, and others are continuous variables (e.g., age, residency length). ...
1
vote
2answers
29 views

Pre-processing - Filling missing values in supervised learning across test/train datasets

I have 2 datasets, testdf and traindf, they both contain missing values in some features. When filling in the missing values ...
1
vote
3answers
348 views

Binary Classification in Imbalanced Data; Oversampling and Imputation

Together with two friends I participate in a university course about data mining in R and we chose the topic of bankruptcy prediction. We started with some "clean" data found on an "In class" kaggle ...
0
votes
1answer
124 views

Pooling F-Values in Multiple Regression in a Multiply Imputed Database

When working with a dataset created via multiple imputation, SPSS pools some values but not others. For example, in multiple regression, I can get coefficients, t-tests for the coefficients, t-values ...
0
votes
1answer
40 views

'runif imputation' in R with mice package

I would like to do some imputation in a situation like this: Knowing that a value is missing is highly informative itself: if a certain variable has a missing value, it must be between zero and some ...
0
votes
0answers
9 views

Values are imputed on data outside the model

I have a dataset where one of the variables contains imputed values. The values are imputed from a multiple imputation model (missForest), where the outcome variable and some of the explaining ...
1
vote
0answers
27 views

Multivariate linear regression with missing points without imputation

I am currently trying to perform a vanilla multivariate linear regression- imagine the equations are $y_1 = a_0x_{0,1} + a_1x_{1,1}... a_nx_{n-1,1}$ $y_2 = a_0x_{0,2} + a_1x_{1,2}... a_nx_{n-1,2}$ .....
3
votes
1answer
92 views

What is the difference between Imputation and Prediction?

Background I am currently working on a project that has me stumped about the difference between imputing a "missing value" and predicting an "unknown value". So far I would understand imputation as ...