Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Refers to a general class of methods used to "fill in" missing data. Methods used for doing this typically are related to interpolation (http://en.wikipedia.org/wiki/Interpolation) and require assumptions about why the data is missing (e.g. "missing at random")

0
votes
0answers
9 views

how imputation with extension works?

i was going through kaggle learning (https://www.kaggle.com/dansbecker/handling-missing-values), in handling the missing data section ,i found use of imputation with extension (i.e) adding extra a ...
1
vote
1answer
78 views

Predicting spendings overall and spendings for subcategories

I have a Dataset containing information about spendings of customers in various shops. There are 10 spending variables related to some categories (like spendings on clothing, spendings on hardware, ...
0
votes
0answers
29 views

Aggregate and interpolate overlapping time-series data

I'm trying to aggregate counter data from two different types of measurements. The first type of measure gives an exact value of the counter on a given day. ...
0
votes
1answer
63 views

Regression with missing Y’s

I use publicly available EU-Silc data to estimate the market price of social dwellings (subsidized dwellings). However my X variables are almost perfectly available,...
3
votes
0answers
47 views

Imputation and nested cross-validation

I am planning to do a nested cross-validation analysis using regularized regression. The inner loop will be used for model tuning and the outer loop for model assessment (test set). Because some data ...
0
votes
0answers
16 views

Multiple Imputation of time series data

I have many groups with a different number of members (learners). The members of each group came together in different time intervals whereupon not all members took part in each of their group ...
0
votes
1answer
24 views

Can I correct for randomly missing data where missingness is has a known relationship to the error term?

Suppose I have a population of observations I want to model as being drawn from some distributional family, which I believe adequately represents the true distribution. My goal is to estimate the ...
1
vote
0answers
12 views

Imputed values into Shapefile / fortified shapefile

I currently have a shape file with approximately 20 numeric variables. Several of these variables have missing values. As this is a shapefile I do not think using the median or mean as a form of ...
0
votes
0answers
15 views

Single imputation to multiple imputation: Can this be done properly with 'missForest' package?

I am currently working on devising a proper multiple imputation scheme utilizing the missForest package in R (see https://academic.oup.com/bioinformatics/article/28/...
2
votes
0answers
35 views

How to use MICE in R to fill missing values in test set?

It seems that MICE does not have a "predict" function which allows to use a fitted mids object to predict the missing values in test data set. I can certainly ...
0
votes
0answers
32 views

Imputating data without changing the Mahalanobis distance

I have a multi-variable dataset (rows/observations are independent). I want to remove outliers in the data based on Mahalanobis distance (MD). In base R there is already a function which calculates it ...
0
votes
1answer
25 views

Imputing nested time series data with R

Does anyone know what is the superior algorithm to impute data in time series? I had strong dropouts over time because it was free to participants how many times to participate in my study (otherwise ...
0
votes
0answers
10 views

Is it possible to store KNN data imputation as a model so the dataset doesn't have to be used?

I want to create a program that will use KNN method of data imputation to fill in missing values. I don't want to ship a dataset with the program just so it can use it to impute the values (data ...
0
votes
1answer
31 views

Proper Imputation and bias-correction on degrading signal with Kalman Filtering?

A signal degrades in its quality. Some signals are far more robust to degradation while others are not. We will simulate degradation by randomly removing values from a function and then applying ...
2
votes
0answers
110 views

Missing data imputation that can handle large data

I am looking for a reasonably scaling missing data imputation approach for big data (e.g. a well-scaling version of kNN - the standard versions we tried so far just ran out of memory) that fulfills ...
0
votes
0answers
21 views

can I fill missing values by using target variable?

I have a 3 column data with 2 features and 1 target variable. But the first features (numeric) have a large number of missing values. If I use kNN to fill in the missing values, I am wondering can I ...
0
votes
1answer
37 views

How does Excel interpolate / imputate missing values in time-series when fitting a line to a plot?

I have a scatter plot in Excel (upper part of the screenshot) of time-series data. In-between the values that I plot (to the left), are some missings. I fit a (linear) line to those values and display ...
0
votes
0answers
14 views

What is a good reference on the philosophy of data imputation?

I would like to read something interesting that addresses why and when data imputation is advisable. I have only been able to find technical stuff about particular imputation methods but I want to ...
1
vote
0answers
20 views

Shouldn't we consider larger standard errors for effect measures or outcomes that are converted in meta-analysis?

There are methods to convert effect measures in meta-analysis (pdf). There are also methods to convert outcomes; at least, I am aware of the conversion described in Furukawa et al. (2005) from ...
0
votes
0answers
12 views

what to do if the missing data in one column is based on some value/condition in another column in r?

I have a dataset with 20,000 observations and 19 variables. To start off with I have a gender column which has three levels namely 'M', 'F' and 'U' where U can be taken as not disclosed. Whenever ...
1
vote
0answers
21 views

How do I impute clustered data that is not time-series data?

The goal of my research is to understand whether MRI imaging characteristics can predict tumor pathology. The data consists of resected tumor samples, with multiple samples per patient. On the MRI, we ...
1
vote
2answers
38 views

Why are missing values MNAR harder to impute than MCAR or MAR?

Reading papers related to the imputation of missing values related to the -omics field, systematically imputation algorithms were less accurate when imputing MNAR compared to imputing MCAR. My ...
0
votes
1answer
25 views

Imputing values with linear regression, valid strategy or creating biases?

I am practicing on the titanic competition from kaggle. In the dataset the Age variable has a number of missing values and I am now left with the choice of what to ...
0
votes
0answers
19 views

Measurements to deterministic value

I have a number of measurements of two variables: the number of products, the weight. Sometimes the weight is missing and sometimes the number of products is missing. I want to use the given data to ...
0
votes
0answers
18 views

Imputing missing data with MICE where each observation has different levels

I have a set of observations that each consists of different levels. For example, I ask a $P$ individuals $N$ questions, each question with a possible $k_n$ discrete responses. This produces a table ...
0
votes
1answer
17 views

Imputing binary variable when no 0s, only 1s are available

I'm trying to impute missing values for a binary variable (values 0 and 1) with some challenging data (of about 1 million observations). The data can be divided into two groups: in group 1, we know ...
0
votes
1answer
93 views

Predictive Mean Matching as Single Imputation?

Multiple imputation is known to be advantageous compared to single imputation. However, in practice there are often non-statistical reasons why multiple imputation can not be used (e.g. the data ...
3
votes
2answers
152 views

data imputation of missing values in non-normally distributed explanatory variables

I have been told that mean imputation of missing values is inappropriate when the variables underlying distribution is non-normal. my variable is contiunous (but bound at 100) and most observations ...
0
votes
0answers
11 views

While performing label encoding or imputation, what should i do to the column with mostly 0's as values which is irrelevant to what column is about?

My DataFrame consists of 2919 rows. Now ,For example I have this column "2ndFlrSF" 2ndFlrSF: Second floor's Area in square feet and these are the values in it after i run my Pandas command ...
0
votes
0answers
29 views

Imputation in irregularly spaced time series data

I have irregular time series data containing missing values (called A). What I need is a regular time series with imputed values (called B). A spans roughly 3 years. Some days have multiple ...
0
votes
0answers
28 views

Imputing missing values in time series with Arima

I am using Arima and Kalman smoothing to impute missing values in univariate time series (similar way to this post: https://stats.stackexchange.com/questions/104565/how-to-use-auto-arima-to-impute-...
3
votes
2answers
62 views

Handling missing data for participants who have not completed any standardised measures and have only provided demographic answers

When managing missing data, how many questions should participant have completed, at a minimum, before imputing the remainder of their missing data? For example, a number of my participants only ...
0
votes
1answer
16 views

Imputation to Result in Known Total

I am using R and Amelia to impute missing data for the number of homeless children in several locations. There is information ...
0
votes
0answers
69 views

Extracting Factor Scores of Latent Variables after CFA in AMOS

I plan to extract the latent variables' factor scores after conducting Confirmatory Factor Analysis (CFA). I will use these factor scores as explanatory variables for my next statistical procedure - ...
0
votes
1answer
36 views

Predicting the probabilities of sales opportunities

I want to predict the probabilities of sales opportunities using a binary classification algorithmn. However after using logistic regression my results do not seem realistic. This could be due to ...
0
votes
1answer
48 views

Impute binary outcome variable for GLM using Stan in R

My outcome variable is a series of Bernoulli trials where some values are missing y $\in$ {0, 1, NA} How do you impute NA values for an outcome variable in rstan in the context of a GLM, assuming ...
1
vote
1answer
60 views

Imputing missing outcome data

I saw the other link (Multiple imputation for outcome variables) discussing missing outcome data imputation for complete case analysis. However, I have missing outcome data as well as missing ...
0
votes
0answers
13 views

Is it possible to preserve a imputation solution after adding a new variable with missing data?

I have imputed a dataset with missing values using MICE in R--all analyses have been completed. After the fact, I wanted to keep everything unchanged (so as to not repeat the analyses) but still add a ...
1
vote
0answers
35 views

Is there a way to estimate regression coefficients?

I'm currently working on a simulation study (based on empirical data) and for this simulation I created a model with multiple interaction terms. The interaction terms are between categorical variables,...
1
vote
0answers
54 views

Space-Time Principal Component Analysis with Missing Lat/Long Data

Thank you for your help, I am looking to run a space-time Principal Component Analysis on Shotspotter data from Brockton, MA: http://justicetechlab.org/data/. Shotspotter sensors record the timing, ...
2
votes
0answers
124 views

Overcome NA's in Random Forest and SVM?

I am using clinical data for prediction purposes with SVM and RF. Two of my columns are as following: ...
0
votes
0answers
36 views

Theoretical question about missing data in time series for prediction

I work with an ARIMA model with external regressors on a data set with data of two years at daily level. A central and important variable (let's call this variable "X") I use as an external regressor (...
2
votes
1answer
74 views

Forecast (impute) missing discrete values in multiple time series

I'm looking to forecast (impute) missing discrete values in multiple time series in order to reach a target volume in a consolidated time serie. The context: I have salesmen that are selling ...
1
vote
1answer
38 views

Drop data Vs fill data. Which one least hurts the integrity of the data?

I have a dilemma for an analysis I'm currently on. I doing some GARCH modelling of bitcoin and a fiat currency. There are some null values with the fiat datasets in comparison with bitcoin data as ...
0
votes
1answer
81 views

How does imputation work? I'm struggling to understand it

I have a short question. I am implementing Scikit-Learn in Typescript and currently blocked at understanding & implementing imputer (mean and regression strategies). Based on the example given ...
1
vote
0answers
54 views

Troubles with weekly data: reconciling weeks with months and years

I am working with weekly data and I am running into troubles when analyzing them because of their inconsistent nature. By inconsistent nature, I mean that not every month is made of the same number of ...
1
vote
0answers
17 views

Modeling when missingness will go away?

Are there any guidelines for how to create a model using data whose features have some missing values that will predict well when data quality improves and there is no longer missing values? I have ...
1
vote
0answers
36 views

Mice: partial imputation using where argument failing [closed]

I encounter a problem with the use of the mice function to do multiple imputation. I want to do imputation only on part of the missing data, what looking at the help seems possible and straightworward....
0
votes
0answers
16 views

Handle missing features [duplicate]

I am participating in a Kaggle competition and I would like to know what is the best way to handle missing attribute values in test data set. For example, if the train data set contains the attributes ...
0
votes
0answers
7 views

Repeated imputed values effect on distribution

In comparing two samples, one of which (red line) shows a consistent number of repeated imputed values creating a separate peak, would it make sense to use a parametric test? What are potential ...