Questions tagged [data-imputation]

Refers to a general class of methods used to "fill in" missing data. Methods used for doing this typically are related to interpolation (http://en.wikipedia.org/wiki/Interpolation) and require assumptions about why the data is missing (e.g. "missing at random")

Filter by
Sorted by
Tagged with
1
vote
0answers
92 views

Space-Time Principal Component Analysis with Missing Lat/Long Data

Thank you for your help, I am looking to run a space-time Principal Component Analysis on Shotspotter data from Brockton, MA: http://justicetechlab.org/data/. Shotspotter sensors record the timing, ...
2
votes
0answers
344 views

Overcome NA's in Random Forest and SVM?

I am using clinical data for prediction purposes with SVM and RF. Two of my columns are as following: ...
3
votes
1answer
430 views

Forecast (impute) missing discrete values in multiple time series

I'm looking to forecast (impute) missing discrete values in multiple time series in order to reach a target volume in a consolidated time serie. The context: I have salesmen that are selling ...
2
votes
1answer
72 views

Drop data Vs fill data. Which one least hurts the integrity of the data?

I have a dilemma for an analysis I'm currently on. I doing some GARCH modelling of bitcoin and a fiat currency. There are some null values with the fiat datasets in comparison with bitcoin data as ...
1
vote
1answer
411 views

How does imputation work? I'm struggling to understand it

I have a short question. I am implementing Scikit-Learn in Typescript and currently blocked at understanding & implementing imputer (mean and regression strategies). Based on the example given ...
1
vote
0answers
120 views

Troubles with weekly data: reconciling weeks with months and years

I am working with weekly data and I am running into troubles when analyzing them because of their inconsistent nature. By inconsistent nature, I mean that not every month is made of the same number of ...
1
vote
0answers
19 views

Modeling when missingness will go away?

Are there any guidelines for how to create a model using data whose features have some missing values that will predict well when data quality improves and there is no longer missing values? I have ...
0
votes
0answers
18 views

Handle missing features [duplicate]

I am participating in a Kaggle competition and I would like to know what is the best way to handle missing attribute values in test data set. For example, if the train data set contains the attributes ...
3
votes
0answers
722 views

Imputing missing values and SVD

Similar questions have been asked a lot of times but I have not found an answer that gives an intuitive explanation as to why this works. For reference I have read the answers here and here. As I ...
0
votes
2answers
313 views

Can I use the 'mice' library in R to impute missing data separately for each of two groups in the same dataset?

I would like to use the 'mice' library in R to impute data from a clinical trial, in which I have two groups (i.e. var="group" [0=control; 1=intervention]). I want to impute the missing data ...
2
votes
0answers
86 views

To impute or not - community consensus for reporting accuracy of an imputed model

I have a model generated using an imputed data set with imputation accuracy of 75%. If the model using imputed data has an accuracy of 80% What would be the community consensus to report the ...
2
votes
0answers
93 views

Impute missing population values in Census data

I have population data from Census.gov: Total US population by age by year from 1940 through 2010 Depending on the range of decades, the data is missing discrete population values for ages greater ...
1
vote
0answers
31 views

Multiple imputation when have more than 1 outcome variable

Is there a good paper or reference for doing multiple imputation when there is more than one outcome variable? Anything that specifically addresses building the imputation model or software to use for ...
2
votes
1answer
93 views

One problem with imputing the missing value by the sample mean

Today I read an interesting statement, which I am not sure about its correctness. Assume we are looking at a data set with one column $H$ that is numeric. It has a bunch of missing values. Now let's ...
1
vote
0answers
69 views

Choosing, evaluating, and reporting data imputation

I have read about model checking for multiple imputation MI (https://ete-online.biomedcentral.com/articles/10.1186/s12982-017-0062-6), but I am not sure how one can check their model for a general ...
2
votes
1answer
370 views

Using regression for imputing missing data

I have been reading about regression models for missing data imputation and I'm quite confused regarding the following: if I can perfectly predict the value of feature f2 using feature f1, why would I ...
1
vote
0answers
46 views

Multiple Imputation query

I am using multiple imputation to deal with around 49 missing observations for my outcome variable from my 324 observation panel dataset. I used Stata to perform 10 imputations for this, using ...
3
votes
1answer
12k views

K-Nearest Neighbor imputation explanation

I have a dataframe with some missing data in it. I need to deal with those missing data before trying anything. I've seen that knnImputation in R is a good ...
1
vote
2answers
404 views

How to check if the imputed value from caret is reliable? (predicting missing values)

I have a dataset which has some missing values and I tried to predict them by using caret. My data set looks like this ...
8
votes
1answer
961 views

Guassian Process for Data Imputation

I recently came across Gaussian Processes in Gelman et al. (2013), and I am trying to learn more about their potential application for use in imputing time series data. The data of interest is a ...
1
vote
1answer
802 views

How do I deal with data with 50% missing value for a specific dimension?

I have a dataset with 50 thousand records. 50% percent of the records don't have gender and birth dates. I wish to analyze age-group/gender item purchase preference. Values are missing completely ...
2
votes
1answer
268 views

Imputation in production

I have a question regarding imputation I was not able to find an answer to. Any help would be greatly appreciated. Let's suppose I have a dataset, impute missing values using the median, train a ...
0
votes
1answer
78 views

How to perform OLS when available independent variables differ between data points?

I would like to investigate the relationship between high school grades for different subjects and university credits earned in the first year. I plan to use multiple linear regression (OLS). I have ...
2
votes
1answer
32 views

Comparing significance of several imputation methods

I’m trying to find the best method to fill gaps in precipitation dataset – which aren’t normally distributed – comparing several methods, from basic methods (simple averages) to complex ones (ANN). I’...
0
votes
1answer
151 views

Imputation of qualitative variables

I am working on a data set were few qualitative features are having missing values. Qualitative columns having null values(nan): ...
1
vote
0answers
1k views

Deciding between Multiple Imputation and Hot-Deck Imputation

I am in the data preparation stages of conducting a multiple regression analysis of US health survey data. The first task I have decided to do is impute missing values from the dataset of 8 variables (...
4
votes
1answer
3k views

When imputing missing values in a test set, should the new values come from the training set or be recalculated from the test set?

Both answers to this question on imputing missing values note that, when imputing missing values in a test set for model evaluation, the replacement values should be the ones calculated and used in ...
3
votes
0answers
1k views

How to use pooled results from multiple imputation?

I've been reading some posts about data imputation using multiple imputation, specifically the MICE R package. I get the main idea of creating multiple datasets with imputed data. The part that is not ...
1
vote
1answer
101 views

Can I use imputed data for imputating another variable?

I have a dataset with three columns A, B and C. The column A has some missing values and the column B too. Column C is complete. If I use a method like MICE for filling column A, can I then run the ...
4
votes
1answer
945 views

Handling null values in linear regression, which are suppose to be higher than the non-null values

I am currently doing a linear regression, where i try to predict the housing prices based on different variables that describe the house's spatial features (such as the distance to the closest city, ...
1
vote
0answers
971 views

KNN imputation for categorical variables

I am using preProcess in caret to knnImpute. As far as I understand, the imputation should include all the variables in the analysis and KNN imputation can only be done effectively if data is on the ...
0
votes
1answer
1k views

Missing values imputation of time series using na.kalman command

I have a time series for daily records of concentration of Total Phosphorus (TP) in a riiver and the water flow in the year of 2014. While I have complete data for the flow variable, I have missing ...
1
vote
1answer
522 views

Possible explanations for Imputation before train-test split?

I'm working on a real world data set containing missing information. I understand imputing missing values before data partitioning can lead to leakage of information. I'm using this R package MissMech ...
0
votes
1answer
55 views

Seeking regression modeling strategies for predicting prices based on categorical variables (one of which is ordered)

I have a question similar to this one, which never received an answer. Let's say I have widgets that have different quality ratings $q\in\{0,1,\dots,N_q\}$ and which are in different regions $R\in\{...
2
votes
1answer
44 views
3
votes
0answers
55 views

Bad data imputation: How to impute specific bad data and replace it with realistic ones?

I conducted an experiment with multiple human participants to analyze some air traffic scenarios. Some data has turned out to be very unrealistic. Take a look at the following pics which shows the ...
2
votes
0answers
406 views

stan - 2 approaches to missing value imputation; which is better and why?

So, me and a colleague have to impute some data, x, given a categorical variable. We arrived at two different approaches: a) as in the tutorial: split x into x_obs and x_mis, and treat x_mis as ...
0
votes
1answer
497 views

time series in R multiple companies yearly sales data

I have 2010-2017 annual sales and total assets data for 100 different companies from pharmaceutical, dyes, chemicals industries. I have following questions: kindly help with the best way to fill the ...
1
vote
0answers
290 views

Multivariate imputation in R

I am working with a dataset from the world bank, it's a relatively simple dataset with 11 variables for 211 countries from 1990-2015: ...
0
votes
0answers
180 views

Can the correlation under stochastic regression imputation exceed the correlation under regression imputation

The correlation of the imputed values under regression imputation is always equal to 1,since the first step in regression imputation involves building a model from the observed data,then predictions ...
1
vote
0answers
1k views

How many missing values are acceptable for random forest imputation (rfImpute) to be valid? (% vs. # remaining)

I'd like to use rfImpute to impute missing values in a data frame that looks something like this, but around 30 variables and much longer (~2000-3000 rows): ...
5
votes
1answer
4k views

Hot deck imputation, ''it preserves the distribution of the item values'', how can that be?

I read in this link, under section 2, first paragraph about hot deck that ''it preserves the distribution of item values''. I do not understand that, if one and the same donor is used for a lot of ...
2
votes
1answer
120 views

imputation with penalized (Cox) regression

I'm doing penalized (elastic net) Cox regression. But I also have missing data. Now, as I understand it, the reason we do multiple imputation is that once we do single imputation, our data points are ...
2
votes
0answers
313 views

Order in which Data Imputation, Outlier removal, Data Transformation should be done?

While preprocessing data we need to carry out the following steps: Missing value Imputation Outlier Detection Transformation of Data In what order should we perform these 3 steps while preprocessing ...
1
vote
0answers
57 views

More cases yield to a lower R-Squared in regression analysis. Is this due to variance?

I've fit a multiple linear regression model on a dataset with approximately 3000 cases. However, due to missing data, the original model only estimated on roughly 1200 cases. Using data imputation, I ...
1
vote
0answers
68 views

How can I fix incomplete data using ranking column?

Goal : I have to make my incomplete data into useful one. Here is example data ...
1
vote
0answers
185 views

Imputation of missing data of temperature

I am interested to show the historical temperatures in my city, but I have many missing values. Indeed, I have 4 diferents geographical points (stations) in the city with historical registred ...
16
votes
3answers
4k views

Methods to work around the problem of missing data in machine learning

Virtually any database we want to make predictions using machine learning algorithms will find missing values ​​for some of the characteristics. There are several approaches to address this problem, ...
1
vote
1answer
909 views

What does it mean to “condition on X”?

Could someone explain what this lingo means in regular English? Example sentences for context: "The single-partition hot deck based on a metric that conditions on $X$ has the problem that it fails to ...
0
votes
1answer
723 views

Multiple Imputation (mice) with interaction terms in model

Our model is a logistic regression. We have been adviced that in our case we should do multiple imputation (mice for exemple). However, in this particular model we have some interaction terms (like ...

1
3 4
5
6 7
12