Questions tagged [data-imputation]

Refers to a general class of methods used to "fill in" missing data. Methods used for doing this typically are related to interpolation (http://en.wikipedia.org/wiki/Interpolation) and require assumptions about why the data is missing (e.g. "missing at random")

Filter by
Sorted by
Tagged with
0 votes
1 answer
26 views

Imputed binary response outcome

I am currently working on a group project with COVID data. The response variable is a binary outcome (positive/negative) from swab test; and let's only consider age groups (0-18, 18-60, 60+) for ...
user avatar
0 votes
1 answer
17 views

Why is Listwise Deletion Standard Error Too Small?

I am going over Stef Van Burren's "Flexible Imputation of Missing Data" and don't understand why Table 1.1 from section 1.3 below gives the standard error for listwise deletion as too large. ...
user avatar
  • 3
0 votes
0 answers
18 views

Should I normalize (preprocess/scale/log-transform) my data "before" imputing missing values with missRanger?

I am trying to impute missing values using missRanger package. missRanger is apparently much faster than missForest. I would like to know: 1 - Are these two packages any different in their imputation ...
user avatar
1 vote
0 answers
12 views

Missing data, unable to measure

A certain industry specializes in detecting harmful chemicals in drinking water. Let us assume that the goal is to detect lead in drinking water. The instrument used to detect lead has a lower limit ...
user avatar
0 votes
1 answer
41 views

How do you choose the imputation technique?

scikit-learn provides three imputation strategies: SimpleImputer(), IterativeImputer(), and ...
user avatar
0 votes
0 answers
22 views

How to deal with low survey response rate and hypothetical question about influence of people not responding?

Background: Imagine you are running a survey, asking 1000 random people: "do you like blue marbles?" Now, 200 people respond. 150 say yes, 50 say no. If I had only sent that survey to only ...
user avatar
  • 271
2 votes
1 answer
44 views

How do i choose which imputation itteration to use for my data after using pmm in R?

I have 40 continuous variables with missing data. I imputed the missing data using predictive mean matching through the MICE package in R, I did 5 iterations for each imputation. My question is how do ...
user avatar
0 votes
1 answer
29 views

How to use General Additive Model to impute missing data?

I am new to GAMs and R. I have a long lists of time series of different lengths. Many of them are incomplete. I have fitted a GAM and use the predict.gam function(mgcv package in R) to each time ...
user avatar
0 votes
0 answers
29 views

How to insert values for MNAR numeric data?

I have a dataframe with customer information: ...
user avatar
  • 1
0 votes
0 answers
83 views

Dealing with missing values for categorical variables

I am currently working on the dataset HR Analytics: Job Change of Data Scientists in Kaggle. The dataset is about a company wanting to know which of the candidates that took part in their training ...
user avatar
  • 1
1 vote
1 answer
37 views

Path analysis with missing data and dichotomous exogenous variables in lavaan

I am currently working on a path analysis in lavaan with many predictors, some of which are dichotomous. My sample size is 386 and I'm doing an exploratory study with many potential predictors ...
user avatar
  • 11
0 votes
0 answers
22 views

Besides computational feasibility, what kind of problems may occur when imputing missing data by exhaustive permutation?

I don't know if "exhaustive permutation" is the correct term, but happy to be corrected if it isn't. Here's an example, which hopefully will clarify what I mean. Let's say I have the ...
user avatar
  • 127
3 votes
1 answer
37 views

Missingness of data due to network issues

I have a time-series dataset that has 120 missing rows due to consecutive network issues and I am trying to impute these values using MICE in Python. As the source of missingness is a total ...
user avatar
  • 135
0 votes
0 answers
32 views

Imputation on a finite population

I have a finite population of samples defined on a feature space. Some of these samples have missing values. Now suppose that I want to fit a classifier on a labelled subset of this finite population ...
user avatar
0 votes
0 answers
29 views

What is the best imputation method for missing values in compositional data?

I'm a little newbie at statistics, so I'm sorry if my question is too dumb. I'm working with compositional datasets with several chemical elements (continuous data; arranged in columns), but also some ...
user avatar
0 votes
1 answer
50 views

How to do data imputation and normalization when using polynomial regression?

The question is about the practical use of polynomial regression. Let's say there is a dataset with columns A, B, T where T is a dependent variable, A and B are independent variables. A and B contain ...
user avatar
  • 13
0 votes
0 answers
23 views

Questions regarding Post-Stratification and Imputation

This is my first try in making a reproducible sample, so I hope it goes well and is comprehensive. Based on a (German) survey among young people of Generation Z, I ask: Does volunteering have a ...
user avatar
  • 11
0 votes
1 answer
41 views

Choosing MICE multiple datasets

I have read and watched several tutorials about MICE. My confusion is about step 1: creating several copies of the original dataset and imputing different values in each copy. In some tutorials, I ...
user avatar
  • 135
1 vote
0 answers
24 views

Single/ multiple imputation in post-selection/-regularization context

Context of problem: In some situations researchers face high-dimensional problems with $p > n$, where $p$ is the number of covariates to be considered in a regression model and $n$ is the sample ...
user avatar
  • 327
0 votes
1 answer
6 views

Rescaling for Imputation under the normal linear model

I am currently working through Stef van Buuren's book Flexible Imputation of Missing Data and I am currently at Chapter 3.2: https://stefvanbuuren.name/fimd/sec-linearnormal.html. The setup consists ...
user avatar
  • 143
0 votes
0 answers
16 views

How to deal with a predictor that is 25% unknown and the rest discrete? [duplicate]

For a sales prediction task, I have 2 datasets, one pertaining to the sales with each store occurring multiple times and the other that has extra information on each of the 1115 distinct stores. The ...
user avatar
1 vote
0 answers
15 views

using predictors and responses from one random forest as predictors for a subsequent random forest

I am planning to do a model (stage 1) for imputing missing data in a variable that will be used as a predictor in a subsequent model (stage 2). The overall goal of this project is having the best ...
user avatar
1 vote
0 answers
31 views

How to perform multiple imputation in time series to ensure sensible values per subject id

I have a time series of binary outcomes (case yes/no) and several covariates. The goal is to estimate incidence and prevalence of the outcome. I would like to perform multiple imputation for both ...
user avatar
0 votes
2 answers
40 views

Impute Nulls in Train but remove Nulls in Test

I have a continuous variable, call it A, with missing data. In my model, I have created a dummy variable, call it B, where 1 indicates missing in A and 0 otherwise. I impute missing values in A with ...
user avatar
  • 185
0 votes
0 answers
13 views

How to impute missing data and model entire dataset while preserving the original structure of the data?

Assume I am building a model to predict whether a customer buys an item (product 3), using a couple of binary variables reflecting existing item ownership (product 1 and product 2). Products 1 and 2 ...
user avatar
1 vote
0 answers
35 views

What is the best method of imputation if the data is missing at random

I am investigating sex differences in treatment response. Patients are defined as "responders" or "non-responders" at 0, 6, 12 and 24 months. My problem is that there is a huge ...
user avatar
  • 67
0 votes
0 answers
20 views

How to implement 'Multiple Complete Random Imputation' as described by Kroh (2006)?

I am a PhD student trying to perform an exploratory factor analysis on response data from a set of 5 attitudinal Strongly disagree/Strongly agree Likert items for a survey study I am working on. The ...
user avatar
1 vote
1 answer
82 views

How do we check MCAR, MAR in general?

Most of methods of imputations requires either MAR or MCAR. How do we check the assumption on MAR or MCAR in general? In how to check missing data is missing at random or not?, Turgeon said $H_0:MCAR$ ...
user avatar
  • 839
1 vote
1 answer
23 views

How to compare row entries in a sparse table with lots of missing values?

I have a dataset with ~1000 laptops and performance results across ~100 different benchmarks. Using the benchmark results, I want to give each laptop a single composite performance score, and rank the ...
user avatar
0 votes
0 answers
29 views

How to deal with imputed values outside the definition range? (Using softImpute e.g.)

When using imputation algorithms, we sometimes know that the missing data is subject to certain properties. For example, we might have the typical Netflix problem, where we have a sparse matrix of ...
user avatar
  • 1
0 votes
0 answers
44 views

Imputed data: how to check residuals of the pooled analysis

I used mice (Multiple imputation by chained equations) to impute my dataset with missing variables. Next, I did a linear regression on the imputed datasets (m=50) and pooled the results according to ...
user avatar
  • 169
1 vote
1 answer
68 views

When there are NaN values for a column of data, why is it okay to fill the values with the median or mean of that column?

Suppose I have a dataset with 100 rows, but for one of my columns, titled 'Age', there are NaN values for 14 of the rows. A common approach to dealing with this is filling up those NaN values with the ...
user avatar
  • 121
1 vote
0 answers
171 views

Imputation with MICE biased under MCAR in extreme case of missingness

I am currently using mice package to impute missing values for a simulation project. I use different performance metrics (eg. bias, precision etc) to compare different methods(eg. complete cases, ...
user avatar
0 votes
0 answers
71 views

Missforest imputation algorithm and categorical variables

I am trying to implement MissForest imputation algorithm on a data set of mostly categorical variables. I've run into the problem however of not knowing whether to use one-hot-encoding or label ...
user avatar
  • 103
1 vote
0 answers
94 views

K-fold cross validation for kNN Imputer in Python [closed]

I have a dataset with columns, say, y, x1, x2, x2 and a lot of missing values in x1, x2, x3. I decided to use ...
user avatar
  • 249
0 votes
0 answers
13 views

Multiple Imputation - can it address the missing data from the sub-cohort?

Background: I am currently working on a project looking at exploring potential predictors for the likelihood that students pass the exam. Participants were recruited from 3 schools, School A (N =25), ...
user avatar
1 vote
4 answers
95 views

How to round mean values when imputing missing questionnaire items

When there are missing items (answers) in a questionnaire it is sometimes usual to impute them with the mean of the valid items. For example here are 3 items/question which can have a four values from ...
user avatar
  • 63
0 votes
1 answer
114 views

Can AIC be used to select the best model with multiply imputed data (MICE)?

I have used the mice() package in R to impute some missing values and create a pooled linear regression model. I have also created another version but this time ...
user avatar
1 vote
1 answer
86 views

Constraints for Multiple imputation for missing values (MICE)

I am imputing average follow-up for a meta-analysis using MICE. In studies reporting a maximum follow-up duration, I attempted to constrain imputations so average follow-up is always less than the ...
user avatar
  • 21
0 votes
0 answers
12 views

Is there a such thing as Imputation Probability Scores?

When faced with missing data, imputation algorithms allow you to assign probable values to the missing data. I had the following question relating to this topic: I have seen different types of ...
user avatar
  • 5,754
1 vote
1 answer
40 views

Missing value imputation in (almost) balanced data set

Suppose we have a categorical field in our data set with two classes. The field contains 5% missing values. Of the remaining 95% values, 47.4% belong to class A and 47.6% belong to class B. Normally, ...
user avatar
0 votes
0 answers
29 views

Combining datasets by imputing variables unique to each set, incorporating set variation and treatment groups

I have two datasets collected from two runs of the same experimental set up (one dataset per run). The two datasets overlap for about 20 measured variables, but have 20 to 40 variables that are unique....
user avatar
  • 1
1 vote
0 answers
49 views

Should you train/test split when using softimpute/matrix completion?

I am trying to impute missing values on a large dataset. After reading the paper(s) introducing matrix completion via soft-SVD thresholding, as well as the softImpute R package vignetter by Hastie (...
user avatar
  • 11
0 votes
0 answers
32 views

Weighted imputation for ranking based on multiple variables

I need a suggestion for imputation of a data where there are missing values. yet I need to rank the items based on the data. Here's an example table to demonstrate this problem. There are 5 birds, ...
user avatar
0 votes
0 answers
18 views

How to incorporate MICE (i.e. impute missing data) into my logistic regression?

Can someone please guide me on how to properly impute missing data (I have missing data in both of my predictors) in my logistic regression analysis? I used the MICE to handle missingness but I am not ...
user avatar
0 votes
0 answers
16 views

Imputing Missing Data from Clusters of Environmental Sensors

I am working with a large environmental dataset with 15-minute resolution timeseries data from a variety of sensors. We collected data for multiple years and experienced some data loss as sensors ...
user avatar
  • 1
1 vote
1 answer
129 views

Does a larger sample size increase multi-collinearity between predictors, after imputation of missing data?

I have two datasets that have exactly the same 1701 predictors, but one has 936 subjects and the other has 547 subjects. (The initial rationale for creating these two different datasets was to see ...
user avatar
1 vote
0 answers
27 views

How do I identify what kind of missing-data I have?

If I have missing values in a dataset, I can't just blindly impute them with mean/median/mode or any other technique. I have to identify what kind of missing values they are, namely: MCAR (missing ...
user avatar
  • 255
1 vote
0 answers
20 views

Where and how to show imputed data in manuscript?

What are your thoughts on where or how to show (non-)imputed data? Please regard this question as a more a general question. I am in the field of medical clinical research, where missing data is very ...
user avatar
  • 13
9 votes
4 answers
1k views

Can I delete missing data?

I have a dataset (4,898 X 17,000) that follows 4898 mothers, fathers, and their children over a period of 15 years. The interviews have been conducted at baseline (when the child was born), year-1, ...
user avatar

1
2 3 4 5
13