Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

When the data present lack of information (gaps), i.e., are not complete. Hence, it is important to consider this feature when performing a analysis or test.

0
votes
0answers
14 views

How to treat “not applicable” value?

Even though there already exist questions with a similar topic, I still have not found an answer to my question. I am working on e-transparency of nonprofit organizations. I try to build an index to ...
0
votes
1answer
25 views

Two separate linear models

I am trying to fit the following linear model: ...
1
vote
0answers
21 views

Complex neural network design - 250K+ rows / Lot of missing variables

For the past years I have been developing different type of Neural Networks with great success. For the past few weeks I have been working on a (big) project I am kind of struggling with. I hope by ...
0
votes
0answers
15 views

Trajectory modeling with small dataset and missingness

I am struggling with a small dataset (260 records) for trajectory modeling. At the 3rd time point, there are 101 cases that are missing critical data points due to attrition. Does it still make sense ...
3
votes
0answers
28 views

Regression on Numerical Variable where NA is Present Using R [closed]

I am working on building a multivariable regression using a dataset in R. That dataset contains a variable that is generally an integer, but at times can have NA values. Because the NA values make ...
0
votes
1answer
31 views

Calculate the implied correlation for missing cells in a correlation matrix in R

I have a correlation matrix in R. Many of the correlations are specified, but there are some that are "NA". eg, A __ B __ C A 100% NA 25% B NA 100% 50% C 25% 50% ...
2
votes
0answers
33 views

Why don't people impute missing exposure data in database studies?

Investigators doing studies in large databases (e.g., EMR) in which there is often a lot of missing data usually (in my experience) want to exclude all subjects missing the exposure or outcome of ...
0
votes
0answers
11 views

what to do if the missing data in one column is based on some value/condition in another column in r?

I have a dataset with 20,000 observations and 19 variables. To start off with I have a gender column which has three levels namely 'M', 'F' and 'U' where U can be taken as not disclosed. Whenever ...
-1
votes
1answer
16 views

K-means in R: complete case analysis followed by nearest-neighbor assignment for partial data

I have a dataset of 3K observations with only 162 being a complete case. I have read here that it is possible to run knn on the complete cases and then conduct a nearest neighbour assignment for ...
1
vote
0answers
12 views

Is BSTS (Bayesian Structural Time Series) robust to missing values?

How do bats deal with missing values occurred in time series? Do I need to impute the missing values by myself or the bets algorithm with handle it automatically?
0
votes
0answers
21 views

Appropriate multiple imputation method for longitudinal data (R package mice)

I'm analyzing a dataset from a longitudinal study aimed at finding if a set of predictors is associated with the trajectories of an outcome, which is measured each day for seven days. The dataset is ...
0
votes
0answers
58 views

Proof: comparison between the statistical efficiency of two MLEs

Suppose that we have a simple random sample of size $n$. Based on this sample, we construct two log-likelihood functions. The first one is, \begin{eqnarray*} l_1 = \sum_{i=1}^r{\bigg[\ln f_\beta(y_i|...
2
votes
1answer
45 views

What type of missingness is this?

I have some missing data for a particular item on a 5-item measure, which is called Attitudes Towards Ageing. Several participants have declined to respond to the item because of the wording of the ...
1
vote
0answers
28 views

How to predict with label missing

I have be given a set of data where most label data (dependent variable) is missing (NaN). Precisely on 100000 rows only 1412 contain a 1 or 0 (the others contain NaN). I was first thinking "let's ...
1
vote
1answer
104 views

Mixed models in R : Compare measurements over several time points with missing data in 3 populations [closed]

I have data that look like the example below. There are 3 different groups (g1, g2 and ...
1
vote
0answers
17 views

Missing measurements in nonlinear chemical processes

I am using an imputation method to handle missing measurements;TSR.The prediction model used is LW-PLS. Based on the my results, the RMSE increases when the percentage of missing measurements ...
0
votes
1answer
26 views

How to best code the N/A response of the likert-type rating scale?

Say I have a dataset of people's opinion/"rating" on something, and they have to choose 1 out of 5 possible answers for each question - Very happy, ...
1
vote
0answers
29 views

How to apply a model built using Multiple Imputation to predict on dataset with missing data?

I understand that Professor Harrell recommends using the target variable in Multiple Imputation. An example using aregImpute of the rms package is in his lecture notes: http://hbiostat.org/doc/rms....
0
votes
0answers
16 views

SVM prediction with historical data that starts at different times

I am working with an SVM prediction model that uses historical data starting in 2016. I now have new data that I want to use with the SVM model but it does not start until 2017. How can I use the two ...
0
votes
0answers
12 views

How to model a field whose feature values can be all 0 in regression model

I want to predict price of products. For each product, I use one-hot encoding to model their features. These features come from a limited set of fields (i.e., product attributes). For example, a field ...
1
vote
0answers
13 views

Why are missing values MNAR harder to impute than MCAR or MAR?

Reading papers related to the imputation of missing values related to the -omics field, systematically imputation algorithms were less accurate when imputing MNAR compared to imputing MCAR. My ...
0
votes
1answer
23 views

Imputing values with linear regression, valid strategy or creating biases?

I am practicing on the titanic competition from kaggle. In the dataset the Age variable has a number of missing values and I am now left with the choice of what to ...
0
votes
0answers
19 views

Measurements to deterministic value

I have a number of measurements of two variables: the number of products, the weight. Sometimes the weight is missing and sometimes the number of products is missing. I want to use the given data to ...
0
votes
0answers
16 views

XGBoost single predictor MNAR

I'm developing a survival classification model (target variable is mortality before X years) with 10,000+ observations using XGBoost. The data is reasonably complete except for one predictor, I'll ...
3
votes
0answers
36 views

Dealing with many NA's in very large datasets for Lasso

I have a few very large and quite "dirty" (survey) datasets. Primarily, there are lots of NA's. These NA's are mostly the result of different questions being asked in different waves. It is perfectly ...
0
votes
0answers
24 views

Signed Rank Wilcoxon

I try to find some answers on forums but I'm not totally confident with my analyses (I'm not a statistician at all) and I didn't find clear answers. So, I perform Wilcoxon Signed Rank test because I ...
1
vote
0answers
10 views

Choosing Among Multiply Imputed Datasets

I am using multiple imputation to estimate treatment effects in a dataset that contains missing data. In some of my imputed datasets, the algorithm used in the analysis fails to converge; it's not ...
1
vote
0answers
31 views

When to use Multiple Imputation over Maximum Likelihood for Missing Data and vice-versa?

I've seen these being called the best techniques for dealing with missing data. But I'm wondering when to use one over the other and why? Edit: Why is this getting downvotes? I'm legitimately ...
1
vote
0answers
26 views

Replace missing values with calculated values in R [closed]

I'd like to be able to replace missing values for a specific variable with calculated values using other variables in R. For example, I have means, standard errors, and upper and lower bounds for ...
0
votes
1answer
16 views

Imputing binary variable when no 0s, only 1s are available

I'm trying to impute missing values for a binary variable (values 0 and 1) with some challenging data (of about 1 million observations). The data can be divided into two groups: in group 1, we know ...
1
vote
0answers
29 views

Specify how you are intending to use the data after cleaning

I have dataset about real estate data sets and there's one columns is "Garage Year Built". In this column contains 'NAN' values that means no garage in the house. So when I'm cleaning data I need to ...
4
votes
1answer
32 views

Special values in continuous numerical variables/features in Random Forest

I have a binary response variable I am seeking to predict using Random Forest. I have a sizable dataset of 150k rows, I have about 200 independent variables or features to use to model the outcome. ...
0
votes
0answers
27 views

Missing data in Stata

I have a GARCH time-series dataset of 5217 observations, spanning roughly 20 years. One of the variables, the trade-weighted US dollar index, misses 108 values, and as far as I can tell completely ...
1
vote
0answers
64 views

SEM with skipped questions in survey

I have conducted a survey on social media. One of the questions asks if you ever use a chatforum. If they answer "no" they skip question 2-5. Thus, I will have missing data despite that they are not ...
1
vote
1answer
20 views

Per protocol or Imputation when missing is small (<5%)

if ~2% of my data is missing on the outcome (continuous scale), out of a total of 200, two in control and three in intervention group, do I need to impute? Or can I make a case that with such small ...
0
votes
1answer
62 views

Predictive Mean Matching as Single Imputation?

Multiple imputation is known to be advantageous compared to single imputation. However, in practice there are often non-statistical reasons why multiple imputation can not be used (e.g. the data ...
0
votes
0answers
11 views

Higher order CFA in lavaan- weird error message about missing value

I conducted a CFA with lavaan, which worked fine. However, when I tried to do the exact same model, only with a second higher order latent factor that is explained by all other factors, I got a very ...
0
votes
0answers
16 views

Group comparision in repeated measures data with missing values?

I have data from blood samples collected from 20 patients before, during (at 30 min, 1 h, 2 h, 3 h, 4 h) and after a medical treatment (using the slightly different treatment protocols A and B). The ...
3
votes
2answers
91 views

data imputation of missing values in non-normally distributed explanatory variables

I have been told that mean imputation of missing values is inappropriate when the variables underlying distribution is non-normal. my variable is contiunous (but bound at 100) and most observations ...
0
votes
2answers
27 views

When is mean imputation appropriate to deal with missing variables?

Of my 407 observations, 28 do not have data for a variable. Would it be appropriate to use the mean of the observations I do have to substitute? What are the pros and cons of this method?
0
votes
1answer
11 views

How to account for sub-questions in regression

I have a dummy explanatory variable that indicates whether a subject responded "yes" to a question and then the sub-question responce (on a continuous scale of 1 to 5) for those that said yes. ...
0
votes
1answer
23 views

imputation method to deal with missing data (in explanatory variables)

I have a large data set of 700 ebay auctions and want to examine seller reputation effects on auction revenue. some sellers have "detailed ratings" (about 45%), these ratings are out of 5 stars across ...
0
votes
1answer
22 views

How to apply pairwise deletion for missing values

This method is always explained with listwise deletion which is very easy to understand. But I can't understand pairwise. Google didn't help as everywhere there is only definiton of it. Please ...
4
votes
3answers
92 views

Handling NAs in a regression ?? Data Flags?

I am right now working with a big data set with about 30 different variables. Almost all of my rows have a missing value in at least one of the rows. I would like to run a regression with several of ...
0
votes
0answers
7 views

Missing Data Analysis-Univariate dummy coding and pairwise correlation to detect bias

I'm working with survey data as independent variables (Income (3 categories),education (4 categories),depression (y/n), social support scores (continuous)). There is some missing data and I'm trying ...
3
votes
2answers
55 views

Handling missing data for participants who have not completed any standardised measures and have only provided demographic answers

When managing missing data, how many questions should participant have completed, at a minimum, before imputing the remainder of their missing data? For example, a number of my participants only ...
0
votes
1answer
28 views

Father/Mother education and working status for single parent household

My question is pratically the same as in: Missing data due to absent parent However I could not find a definitive solution. I face the same problem, meaning that I do not have observation for ...
0
votes
2answers
33 views

How to account for missing observations in multivariate regression?

My research concerns the eBay feedback mechanism. users have a "feedback score" (total positive reviews less negative reviews) and a "feedback percentage " $\frac{positives}{positives+negatives}$. ...
1
vote
0answers
30 views

Why MLE is not biased in presence of incomplete data when parameters distinctness does not hold?

When estimating MLE in presence of missing data, the missing data process can be ignored if the data are MAR and parameters distinctness assumption (generating vs missing data processes) does hold. I ...
0
votes
0answers
24 views

Multiple Imputation of Multilevel data

I am using Mice package in R for multiple imputation of a multilevel data where repeated measures are nested within individuals. But there is a bug in mice for which we need to convert the group ...