Questions tagged [data-imputation]

Refers to a general class of methods used to "fill in" missing data. Methods used for doing this typically are related to interpolation (http://en.wikipedia.org/wiki/Interpolation) and require assumptions about why the data is missing (e.g. "missing at random")

Filter by
Sorted by
Tagged with
1
vote
0answers
7 views

methodological question: matching/imputation based on two datasets

I struggle to find the right method for what I want to do using two household surveys. I have two datasets: X dataset with socio-econ info (A1) and Z info Y dataset with socio-econ info (A2) The Y ...
0
votes
0answers
4 views

First MICE then Correlation analysis?

I am having a hard time finding the correct order of steps of imputation with MICE and correlation analysis: Imputation with MICE and afterwards exclude the highly correlating features. Exclude ...
1
vote
0answers
11 views

What are appropriate imputation methods for handling missing data in the case of censored distribution?

I measured antibodies in blood with a test kit that gives quantitative values between 2 and 20 mg/mL. All values below 2 are indicated as <2 and values above 20 are indicated as >20 (the test ...
0
votes
0answers
15 views

Correlation Analysis on a MICE Dataset

This question is similar to one that was asked almost 10 years ago but never got answered. I want to perform a CFA on a data set mainly consisting of categorical data. It also contains around 2-3 ...
0
votes
0answers
9 views

Fill NA's using a panel regression of specific AR(1) form

I need to fill NA's using a panel regression of this specific form (AR(1): $$\hat{y}_{i, t + h} - \bar{y}_{t + h} = \beta_i (\hat{y}_{i, t + h - 1} - \bar{y}_{t + h - 1}) + \epsilon_{i, t + h}$$ $\...
0
votes
0answers
16 views

Imputation methods for time series data (non-stationary)

I am looking for an impute method for non-stationary time series (financial indeces). From https://pypi.org/project/impyute/ (https://buildmedia.readthedocs.org/media/pdf/impyute/latest/impyute.pdf) I ...
0
votes
1answer
22 views

Impute missing values of dummy variables, using R's {caret} package: predicted values in between {0;1}?

I'm using {caret} to impute missing data resulting from non-response to survey questions. All of these variables are defined as numeric, though most are dummies. ...
0
votes
0answers
11 views

How to perform Multiple Imputation (MI) for longitudinal survey data, using {caret} in R?

I have a large dataset comprising survey responses throughout a time period of three to four decades (or in other words, slightly above 20 waves). Keep in mind that these are longitudinal data and not ...
0
votes
0answers
16 views

Multiple imputation using proc mi EM method in SAS

I'd like to clarify how to use SAS for (multiple) imputation in SAS, specifically the EM method option for proc mi. Do I need to analyze multiple imputed sets using proc mianalyze as mentioned in ...
0
votes
0answers
20 views

Distribution that relates mean and standard deviation with positive support

I am working on a meta analysis that is looking at change in dopamine levels between two populations. I have the mean and standard deviation for each group along with some trials which are missing the ...
0
votes
0answers
8 views

Imputation method for weather data

I have weather data (time series) which has variables like rainfall, air temperature, relative humidity etc. About 6% of the total records are missing (2191 total records). After data cleaning, what I ...
0
votes
0answers
9 views

Imputing MCAR Data

In practice, MCAR data cant be imputed. I have a data set which spans 2 years of individuals who joined a website. During the past 4 months a survey was given to individuals registering on the site. ...
0
votes
0answers
16 views

What is the difference between Mean Imputation (vs) Item Mean Imputation (vs) Person Mean Imputation?

I just learned that we can handle missing data using Imputation techniques, I would like to know difference between Mean Imputation & Item Mean Imputation & Person Mean Imputation and when to ...
0
votes
0answers
82 views

Methods for gappy and mis-recorded dates - with complete time index (edited)

I have some data where subject IDs and day-of-record index are complete, but the recorded dates themselves can be missing or otherwise wrongly recorded (meaning not exactly consistent with the day-of-...
1
vote
0answers
17 views

How to impute missing values at the beginning of a time series using GARCH model?

I want to use a GARCH(1,1) model to impute the missing data at the beginning of a time series, but GARCH model is normally used for predicting future data, is there a way to modify GARCH(1,1) so that ...
2
votes
1answer
35 views

Time series imputation of univariate data with ARIMA

My data is constructed from monthly mean Sea level measurements from 1950 till 2020. Univariate data. Some of the years/ months in 1960s / 1970s are missing and I would like to impute them. We know ...
1
vote
1answer
14 views

What are the consequences of using a dummy-based imputation in a Cox proportional hazard model?

We are doing a survival analysis with a Cox proportional hazard model. We are using retrospective data, and the analysis is underpowered (rare disease). We pretty much have a complete case situation, ...
0
votes
0answers
12 views

Testing for MCAR in univariate time-series

I have a large dataset consisting of hourly energy measurements over a period of 2 years for 1500 buildings. Since the energy consumption of one building can be assumed to be independent of the ...
3
votes
0answers
20 views

Theory behind Multivariate Imputation with Chained Equations

Can anyone provide a reference to the theory that supports multivariate imputation with chained equations (MICE). I know Rubin has provided this for MI but MICE is a Gibbs sampler (I have never seen ...
0
votes
1answer
11 views

Categorizing Date in a registry data base of cancer

I happen to work with a breast cancer database which is not available online. I want to follow these steps and report the results: Remove data of people who are young( less than 18 yo) and who are ...
1
vote
0answers
25 views

Lagged features in time series feature engineering

I understand the value of using lagged features in time series analyses, but conceptually I've still quite a bit to learn. When creating lagged features for modeling purposes, should I remove the rows ...
1
vote
0answers
33 views

How bad does MAR have to get to be inevitably of concern?

The textbook literature says that multiple imputation is valid if data is MCAR or MAR. MAR means that the probability of missingness is dependent of observed information, but not unobserved ...
0
votes
0answers
37 views

Mean Absolute Scaled Error as a measure for imputation

To evaluate different imputation methods with cross-validation I am searching for an appropriate accuracy measure. My cross-validation sample consists of 100 univariate time-series of equal length (...
1
vote
1answer
65 views

Rubin's rule, applied to absolute effect size or relative effect size (Cohen's d)?

Cohen's d is a way to describe the effect size relative to the standard deviation of the data. For instance in the case of the difference between the means of two populations $$\begin{array}{} \text{...
1
vote
1answer
37 views

Will Multiple Imputation (MICE) work on dataset with missing data on only one feature?

Based on this article, it is apparent that MICE works with the following logic: Fill missing values in every column apart from the column in question with either random or the mean of the given ...
1
vote
1answer
41 views

Multiple imputation in R with mice package

I have conducted a multiple imputation in R with 5 imputations and 50 iterations using the function mice() from the corresponding mice package. Now that I have ...
0
votes
0answers
132 views

Calculating Cohen's d with confidence intervals (CI) after multiple imputation

I have a dataset with missings and I was told by my supervisor to run multiple imputation. This is done. Now I need to calculate Cohen's d effect size for the mean differences between the experimental ...
0
votes
3answers
119 views

Variable-specific random sample imputation. Is it a valid method of imputation?

Is random sample imputation a valid method of imputation for categorical variables? Not randomly drawing from any old uniform or normal distribution, but drawing from the specific distribution of the ...
1
vote
0answers
18 views

Dropping Missing Observations under MAR Assumption

Some of the outcome data in my RCT data set are missing. I believe that the missing data mechanism is missing at random (MAR) as the observed characteristics significantly differ between the missing ...
0
votes
0answers
13 views

Can I impute a variable using MICE so that I can use the value(s) from this imputed variable to then code another variable?

I am working with pregnancy data where I would like to impute a variable called LABOR PRESENTATION (nomical var. with 5 categories) from several other variables but then create a variable called ...
0
votes
0answers
23 views

Drop or impute predictor that is missing around 70% of values, but which is known to be highly relevant?

Suppose we have a medical dataset and we are interested in predicting blood pressure using the following variables: age, sex, weight, height, volume of circulating blood, cardiac output, parent with ...
0
votes
1answer
21 views

Can I build an equation from just few observations?

I have the following data that has 5 observations and 2 missed observatiosn. I want to build an equation that can help estimate the missed observations. ...
0
votes
0answers
7 views

Data imputation for number of rooms and square meters of residential units

I have a dataset where each observation is a residential unit. The units are observed on two characteristics $$\mathbf x_i =(x_{1i},x_{2i}),$$ where the first is the size of the residential unit ...
0
votes
0answers
26 views

Missing value treatment for neural networks by imputing large negative numbers

I have a dataset with nearly 150 features and 87k records. Each record indicates a person. A few features have 2-10k missing values. These features encode the behavior of a person in the first few ...
3
votes
1answer
56 views

How to deal with missing values of explanatory variables when comparing models

I want to compare several logistic regression models. The different models are built using the same initial dataset. The models differ with respect to the explanatory variables included. Many of the ...
0
votes
0answers
20 views

Assessment of bias by imputation

Lets say i have a large survey dataset which i want to use as a source for income reporting of the population, e.g. parameters of the distribution, poverty and inequality. Due to item nonresponse on ...
1
vote
2answers
32 views

How does one effectively deal with data imbalance while working on a NLP problem without dropping data points?

I am working with a data set of fake job postings and it has the columns following columns: ...
0
votes
0answers
11 views

In a per-protocol analysis, should I impute values using alla available information, or only the per-protocol ones?

I have performed the main analyses of a RCT (where outcomes are measured repeatedly, starting from baseline), using an "Intention to Treat" approach. Since outcomes are scales from ...
1
vote
1answer
76 views

Is it better to impute a feature with single value or impute based on frequency?

Last time I posted a question in Stackoverflow how to fill nans based on frequency. I got some comments about whether it is a good idea or not. So I am seeking some suggestions if this is actually a ...
0
votes
1answer
74 views

Multiple imputation and normality assumption

I am reading the an online book by Stef Van Buuren (link at bottom) regarding multiple imputation. In Section 3.2.1 he lists 4 different approaches to multiple imputation: Later on in Section 3.3 he ...
9
votes
1answer
108 views

Does imputation introduce unacceptable bias?

I have recently come to know about imputation techniques, which, in short, "guess" realistic values with which to replace missing values in a dataset. My big issue with this is that we are ...
0
votes
0answers
12 views

Imputing values in new samples

For the dataset, I know that: for missing values in training dataset (and therefore for validation datasets for CV) we impute values using training samples for missing values in test dataset we ...
0
votes
0answers
135 views

How to use missForest in R for test data imputation?

I want to use the R missForest() function at work to perform missing value imputation. However, after reading up on the algorithm more, I can't decide how to impute ...
1
vote
1answer
49 views

Imputing missing values for linear regression model, using linear regression

I scraped a real estate website and would like to impute missing data on total area (about 40% missing) using linear regression. I achieve the best results using price, number of rooms, bedrooms, ...
0
votes
0answers
55 views

imputing high percentage of missing data in multivariate time series

In a dataset with time-series, that is dependant on a given input, which the time-series are given only on an irregular cycle of 10-12 time steps that makes lots of missing observations what is the ...
2
votes
2answers
48 views

Imputing data using covariance?

Suppose I have some samples of sensor data, where each row has ten measurements from various sensors. And suppose I know what the covariances are among these sensor measurements. Are there any ...
0
votes
0answers
22 views

how to estimate the precipitation value from other relative data

I have climate data but the precipitation values are missing, I would like to know if there is any formula used to estimate the precipitation value from the other recorded climate data: temperature, ...
0
votes
0answers
19 views

Missing data roughly proportional to the clusters, does this indicate MAR?

I have data in which the number of missing values per cluster (in this case, zip-code), are proportional to the population. Does this indicate Missing at Random (MAR)? Third column with missing ...
3
votes
1answer
94 views

How to use multiple imputed data for survey estimation?

I'm trying to calculate population mean, median, (etc, descriptive analysis) using multiple imputed data. However, the example that I found in sources were regression and then pool them into one ...
1
vote
2answers
102 views

When is it OK not to keep a testing/holdout set?

I am performing data imputation on a large matrix [100000,34] of past measurements that contains missing values (rows are time-steps and columns are stations). So far I've used several machine-...

1
2 3 4 5
11