Questions tagged [data-imputation]

Refers to a general class of methods used to "fill in" missing data. Methods used for doing this typically are related to interpolation (http://en.wikipedia.org/wiki/Interpolation) and require assumptions about why the data is missing (e.g. "missing at random")

Filter by
Sorted by
Tagged with
0
votes
0answers
6 views

Data imputation for number of rooms and square meters of residential units

I have a dataset where each observation is a residential unit. The units are observed on two characteristics $$\mathbf x_i =(x_{1i},x_{2i}),$$ where the first is the size of the residential unit ...
0
votes
0answers
16 views

Missing value treatment for neural networks by imputing large negative numbers

I have a dataset with nearly 150 features and 87k records. Each record indicates a person. A few features have 2-10k missing values. These features encode the behavior of a person in the first few ...
3
votes
1answer
31 views

How to deal with missing values of explanatory variables when comparing models

I want to compare several logistic regression models. The different models are built using the same initial dataset. The models differ with respect to the explanatory variables included. Many of the ...
0
votes
0answers
18 views

Assessment of bias by imputation

Lets say i have a large survey dataset which i want to use as a source for income reporting of the population, e.g. parameters of the distribution, poverty and inequality. Due to item nonresponse on ...
1
vote
2answers
27 views

How does one effectively deal with data imbalance while working on a NLP problem without dropping data points?

I am working with a data set of fake job postings and it has the columns following columns: ...
0
votes
0answers
11 views

In a per-protocol analysis, should I impute values using alla available information, or only the per-protocol ones?

I have performed the main analyses of a RCT (where outcomes are measured repeatedly, starting from baseline), using an "Intention to Treat" approach. Since outcomes are scales from ...
1
vote
1answer
30 views

Is it better to impute a feature with single value or impute based on frequency?

Last time I posted a question in Stackoverflow how to fill nans based on frequency. I got some comments about whether it is a good idea or not. So I am seeking some suggestions if this is actually a ...
0
votes
1answer
28 views

Multiple imputation and normality assumption

I am reading the an online book by Stef Van Buuren (link at bottom) regarding multiple imputation. In Section 3.2.1 he lists 4 different approaches to multiple imputation: Later on in Section 3.3 he ...
9
votes
1answer
91 views

Does imputation introduce unacceptable bias?

I have recently come to know about imputation techniques, which, in short, "guess" realistic values with which to replace missing values in a dataset. My big issue with this is that we are ...
0
votes
0answers
9 views

Imputing values in new samples

For the dataset, I know that: for missing values in training dataset (and therefore for validation datasets for CV) we impute values using training samples for missing values in test dataset we ...
0
votes
0answers
32 views

How to use missForest in R for test data imputation?

I want to use the R missForest() function at work to perform missing value imputation. However, after reading up on the algorithm more, I can't decide how to impute ...
1
vote
1answer
35 views

Imputing missing values for linear regression model, using linear regression

I scraped a real estate website and would like to impute missing data on total area (about 40% missing) using linear regression. I achieve the best results using price, number of rooms, bedrooms, ...
0
votes
0answers
25 views

imputing high percentage of missing data in multivariate time series

In a dataset with time-series, that is dependant on a given input, which the time-series are given only on an irregular cycle of 10-12 time steps that makes lots of missing observations what is the ...
2
votes
2answers
41 views

Imputing data using covariance?

Suppose I have some samples of sensor data, where each row has ten measurements from various sensors. And suppose I know what the covariances are among these sensor measurements. Are there any ...
0
votes
0answers
22 views

how to estimate the precipitation value from other relative data

I have climate data but the precipitation values are missing, I would like to know if there is any formula used to estimate the precipitation value from the other recorded climate data: temperature, ...
0
votes
0answers
19 views

Missing data roughly proportional to the clusters, does this indicate MAR?

I have data in which the number of missing values per cluster (in this case, zip-code), are proportional to the population. Does this indicate Missing at Random (MAR)? Third column with missing ...
3
votes
1answer
27 views

How to use multiple imputed data for survey estimation?

I'm trying to calculate population mean, median, (etc, descriptive analysis) using multiple imputed data. However, the example that I found in sources were regression and then pool them into one ...
0
votes
2answers
80 views

When is it OK not to keep a testing/holdout set?

I am performing data imputation on a large matrix [100000,34] of past measurements that contains missing values (rows are time-steps and columns are stations). So far I've used several machine-...
0
votes
0answers
47 views

How to check whether the missing data is MCAR, MAR, or MNAR? [duplicate]

I read few responses close to the question and was suggested in using t-test or chi-sq test. However, the pattern between variables can also involve more than 2 variables (e.g. data at x tend to be ...
0
votes
1answer
20 views

How reliable must a regressor be for data imputation?

I have a dataset that seemingly has missingness not at random and am thinking on using regressor to fill the missing values. I know that complete case analysis is on the table; however almost 20% ...
0
votes
0answers
7 views

Comparing Quality of Imputation

I have missing values in category type variable. I have used Mode Imputation as One approach. KNN Imputation as another approach. Is there any way on how to compare the quality of imputation made by ...
1
vote
0answers
11 views

Can KNNImputer be used in filling time series data?

I was wondering whether I could fill some null values in a time series dataset about the air quality of India, with knnimputer. Because it seems reasonable to say some days are similar to each other ...
5
votes
2answers
73 views

At what point in analysis do you perform imputation for missing variables?

There is a dataset with 30 variables and over 5 million observations. We plan to use a subsample of the data for analysis. Around .02 - 2.5% of EACH variable are missing. I plan imputation in Stata ...
0
votes
1answer
19 views

Should we try imputation on cases with slightly problematic datasets or prefer ommiting observations

I would like to ask a general question which makes me worry when I try to impute NA values. We know that most of the imputation methods are based on the rest non-NA values. However, if we know that ...
0
votes
0answers
16 views

Best practices and procedures for the imputation of missing data in a repeated measures design

I have a dataset from an experiment I conducted that examined the accuracy and precision of eye movements. It was a repeated measures, 3 x 4 x 10 design with no nesting of subjects and no between-...
0
votes
0answers
20 views

Imputation where some data points should not be imputed

I am trying to impute the RSSI (Received Signal Strength Indicator) measurement from different WiFi access points. In this instance, data can be missing for two reasons: either the laptop did not look ...
0
votes
0answers
15 views

replacing nas when 50% or more of the data is missing

I have to perform a linear regression on a dataset. However, I am having trouble figuring out what type of imputation I should do on the data because in some cases the majority of the the data is ...
0
votes
1answer
20 views

Imputation: which method is this

I'm working on a paper on machine learning. Suppose I have three classes (A, B, C). Suppose now I have a model that outputs ...
0
votes
0answers
27 views

imputing missing values in a binary variable using sklearn IterativeImputer (MICE imputation)

I tried using MICE (Iterative Imputer from SKlearn) for imputing missing binary variables. It works but imputed values are non-binary. it seems to be populating with probability (impressive that ...
1
vote
0answers
17 views
1
vote
0answers
22 views

How to identify whether my data follows “Missing At Random” (MAR) mechanism or not? [closed]

I was having two similar studies with two variables (anti-gE and anti-VZV (continuous variable)) linearly related to each other (with same relationship between both the studies in anti-gE and anti-VZV)...
0
votes
1answer
20 views
0
votes
0answers
21 views

Invalid values during data imputation

I have a dataset in which several values are missing (NA). To fill the missing values, I used a couple of data imputation techniques like softImpute and MICE. While analyzing the imputed datasets, I ...
0
votes
0answers
34 views

Modelling panel data - approach for dealing with missing values when analysing in wide format

My question concerns the appearance of missing values in panel data when it is converted from long to wide format. The model I am fitting (non linear distributed lag model using R package dlnm) ...
0
votes
1answer
19 views

MissMDA Bayesian MIPCA help/MIFAMD

While looking into methods relating to principal components related imputation, I came across MissMDA package in R. Could anyone please tell me what the Bayes in the MIPCA means in layman terms? I ...
1
vote
1answer
52 views

How to decide whether missing values are MAR, MCAR, or MNAR

I have a large proteomics dataset. In the rows I have the proteins , and in the rows I have the samples.The dataset contains a lot of missing values. I would like to know I can find out whether ...
0
votes
0answers
29 views

MCAR tests on mixed variable datasets [findings + discussion]

I have been spending a whole week on researching about testing on missingness mechanism in datasets and I thought it will be helpful to share what I found out about the 2 available MCAR test packages ...
1
vote
1answer
44 views

Will use of imputed values by regression imputation cause multicollinearity?

If I want to regress y on some variables, one of which contains missing values. For example, regress the price of a rental house on its size, location, Wifi(dummy), (number of) bedrooms, neighbours' ...
1
vote
1answer
64 views

Is it advisable to impute missing values and scale features before computing the Variance Inflation Factor (VIF)?

As far as scaling, Wikipedia says: Finally, note that the VIF is invariant to the scaling of the variables (that is, we could scale each variable Xj by a constant cj without changing the VIF). ...
0
votes
0answers
14 views

Impute Continuous Predictor which 0 or median is not an option

I have a dataframe of the following patients: PatientID Days.To.Develop.Symptoms 1 0 2 1 3 3 4 NA ...
1
vote
1answer
18 views

Is Matrix Factorization also going to work with one feature?

I need to fill missing values. I have found that there are many approaches such as the mean and the median of the feature as well as using Matrix Factorization. However, I am kind of wondering if I ...
0
votes
0answers
13 views

Should I impute the missing values of timeseries data?

I have the following task - predicting the next 12 hours of PM10 particles based on historical data of previous 24 hours of PM10, O3 (ozone), CO (carbon monoxide), and others (not included) using RNN'...
23
votes
6answers
2k views

What are the disadvantages of using mean for missing values?

I have an assignment (Data Mining course) and there is a part which asks: "What are the disadvantages of using mean for missing values?" in Missing Value section. ...
2
votes
1answer
21 views

Imputing Missing Value with Response Variable?

I have a dataset with 12 predictor variables and a binary response variable. There's 5960 observations. One of the predictor variables has 1,260 missing values so I'm using k-nearest neighbours to ...
0
votes
0answers
11 views

How PMM imputation works?

I am currently using the PMM method in my df. My doubt is, how it works. My df is composed of two categorical variables (such as family and genera, these are my explanatory variables) and one ...
1
vote
1answer
61 views

Fill or not to fill? That's the question

I have a dataset which shows the expense of users in a specific expense category daily, along the time. I am building a time series in order to predict whether this person will buy this some product ...
0
votes
0answers
46 views

interpreting error in imputation: missForest

Are there norms for how much error in imputation indicates too much error? I'm using missForest and missRanger to impute ...
0
votes
0answers
10 views

Identifying Gap Regions in Time Series Data [duplicate]

I have order millions of time series and order tens of thousands of them have sudden drops to near-zero and then return to legitimacy at random times. A couple of examples: First, my challenge is ...
0
votes
0answers
7 views

How to approach modelling based on question trees

I'm trying to cluster individuals based on survey data. The thing is, this survey sometimes has question trees, like this: Do you have kids? (Q1) Do all the kids go to school? (Q1.1) What's the ...
1
vote
0answers
41 views

Should I involve the dependent variable when imputing missing values for an independent variable?

I am building a model for predicting whether my users would recommend my wine. In the final model, the dependent variable, $Y_{ni}$ is whether user $n$ recommends wine variant $i$. However, in some ...

1
2 3 4 5
11