When the data present lack of information (gaps), i.e., are not complete. Hence, it is important to consider this feature when performing a analysis or test.

learn more… | top users | synonyms (2)

3
votes
2answers
36 views

Survey analysis with missing data by design

I have a survey with 400 responses looking at the satisfaction of customers with a company's service overall, as well as on various specific aspects (website, account manager, invoicing, etc.). ...
0
votes
0answers
6 views

What are publicly-available datasets that are good for learning about issues in missing data? [closed]

I'm working on a project on missing data in Bayesian data analysis and I'm having trouble finding good learning resources and example datasets on which to try different imputation techniques, etc. ...
0
votes
0answers
17 views

Imputing missing values of predictor for use in Regression Models

I have a panel data set that extends from January 2013 to July 2014. The response variable is complete for the entire period, however all of the predictor variables have values only up to June 2014. ...
0
votes
0answers
28 views

R MICE imputation failing

I am really baffled about why my imputation is failing in R's Mice 2.22 package. I am attempting a very simple operation with the following data frame: ...
0
votes
1answer
21 views

Principled way of combining time series with different spans and granularity into an econometric model

I want to forecast the price of something given various time series as inputs. The problem is that they are of different frequency (annual, quarterly, monthly, daily) and time periods (the more ...
1
vote
0answers
20 views

Find Weibull parameters from incomplete sample

I have samples with known number of missing elements (expressed as NaN's) that range from 10%-90% missing. The same function, Weibull, works for all cases. My question is then, how can I ...
1
vote
1answer
28 views

Dealing with missing values where the question was not asked

I have a question about missing values. We used 3 versions of a questionnaire were the possible answers were numeric (0-10) with 0 = no pain and 10 = worst possible pain. We had 4 pain questions ...
0
votes
0answers
53 views

fitting a model for time series data

Folks, I am working on time series traffic data where the waiting times are indexed over time, with 288 observations for 24 hour time period (interval of 5 minutes). I am trying to cleanse the data, ...
3
votes
1answer
33 views

Is there any rule of thumb to delete a variable in a large data set?

I'm working with a large set as a project for the business analytic course with $10^5$ observations and 170+ variables, some of which come with a missing value proportion of larger than 20%, even more ...
0
votes
1answer
6 views

GLM - X.intercept equal to NA [duplicate]

What does it mean X.Intercept equal to NA as a result of glm summary ? Thanks. Coefficients: (1 not defined because of singularities) ...
0
votes
1answer
18 views

identify nature of missingness for categorical variables

could you please give me some hints for identifying the nature of missingness for categorical variables' missing value? I mean, I gave a fast search on google scholar but I didn't find anything ...
0
votes
0answers
30 views

Handling Sparse Data Frames - algorithm selection

I am new to machine learning/statistical modelling. I am trying to run a classification on a highly sparse dataset with 100 features, most of which are categorical (TRUE/FALSE) with the remaining ...
0
votes
0answers
18 views

Missing Values in Dependent Variable and the Question on Tobit model (Panel Data)

I would appreciate some help and guidance on this issue. I have a panel data that looks like this. What I want to do is use the FDIflow variable as a dependent ...
0
votes
1answer
19 views

How do I deal with missing data for a repeated measure collected over time?

I have a real-life data set, with only one "measured" variable (i.e., patient waiting time). This single variable is collected weekly, in different clinics, for different providers, across time. I am ...
2
votes
0answers
46 views

Missing data not at random - Advice needed on method

I have been developing a logistic regression model based on retrospective data from a national trauma database of head injury in the UK. The key outcome is 30 day mortality (denoted as "Survive" ...
1
vote
1answer
57 views

How to handle missing data in a small $n$ large $k$ machine learning scenario?

I have a sample size $N=130$ and $1000$ variables. I am using machine learning techniques (SVM) for analysing the data. Some variables in the dataset have values that are so huge that they must be ...
0
votes
0answers
12 views

Incomplete inhibition curve fitting with Graphpad Prism6

One of my data set shows incomplete inhibition (for high concentrations points, inhibition response goes up to 45%-50% (of control). I am fitting my data using log(inhibitor) vs response -Variable ...
0
votes
0answers
16 views

Expectation maximisation for right-censored iid data from Normal

This is the data (which are length of ropes), $\textrm{Data}=\{99, 70, o ,89, 88, o, 88,70, o ,o\}$, where $o$ are censored data with value above $100$. Assume that data are from $\textrm{iid} \sim ...
0
votes
1answer
64 views

Single EM imputation with R (using Amelia or other packages)

I am trying to impute missing values with R. I would like to use the EM algorithm for that. As it seems this algorithm is implemented in the ...
0
votes
0answers
18 views

Estimating NA values in WinBUGS model

I am trying to fit a WinBUGS model with missing (NA) values as part of the input data. I'd like to use WinBUGS to generate missing estimates for the NA values from the full conditional distributions ...
0
votes
0answers
23 views

Imputation for small cells?

I have a question about the data requirements for imputation. I am trying to calculate the price elasticity of demand for service X where there are four possible provider types for that services. In ...
6
votes
1answer
130 views

Should additional crime reports about someone change our level of doubt about an initial crime report?

Edit: Note that this question is not about multiple unreliable witnesses to the same incident, but rather multiple incidents with only one witness each. Should the accumulation of separate alleged ...
0
votes
1answer
30 views

Factor analysis with categorical reponses and missing data

I factor analyzing a measure with 55 categorical items (3 categories each). I am use CFA to test a 7 factor model. I have a very large sample (>10,000), but approximately 20% of the sample is missing ...
0
votes
0answers
12 views

Conditional Distribution of an Independent Variable for missing data

Let $X=[X_1 X_2 X_3 ... X_p] $be a matrix of p independent variables where $X_i=[x_{i1} ... x_{in}]'$ is a nx1 vector. Let W be a nxn weight matrix based upon queen contiguity (so zero's along the ...
0
votes
0answers
39 views

Obtain factor scores in data set with missing values

I would like to obtain factor scores after factor analyzing data that contain missing values. I'm using Stata 13 to run the analysis. Here is the basic code (borrowed from the UCLA site): ...
1
vote
0answers
61 views

What are the pros and cons of using median imputation to handle missing value?

I have to choose between median or mean imputation to handle missing values. I feel median imputation will work better because it is a number that is already present in the data set and is less ...
1
vote
1answer
36 views

Imputation of a binary variable by Bayesian logistic regression

In the book "Flexible Imputation of Missing Data" by Van Buuren, the following algorithm is presented I think I understand the algorithm as given, but I would like to know what is the "more ...
1
vote
1answer
67 views

Missing factor levels after logistic regression glm()

I am quite new in the R universe, so please excuse me if the question is too simple.. I would like to perform a logistic regression on a marketing data set (only categorical variables), of the form ...
0
votes
0answers
21 views

MAR Vs. MNAR: attribute's missingness determined by the value of another attribute

Suppose I have a training data set that predicts the whether a person is unemployed or not using a decision tree. I have a certain set of attributes that I use to predict this. But for all samples ...
6
votes
3answers
79 views

Managing 'prefer not to says' in sensitive questionnaires

Consider a questionnaire where we ask someone about their sexuality. The five options, for simplicity, are: Heterosexual Homosexual Bisexual Other 'Prefer not to say' Assume we ask the population. ...
3
votes
3answers
213 views

How do I handle nonexistent or missing data?

I tried a forecasting method and want to check if my method is correct or not. My study is comparing different kinds of mutual funds. I want to use the GCC index as a benchmark for one of them but ...
0
votes
0answers
24 views

(binary) Matrix completion with less known data

Recently, I meet such problems, I call it matrix completion problem. For example, the row denotes the users and the column denotes items. And If one user like the ...
2
votes
0answers
45 views

Imputation of missing values for doing PCA in R [duplicate]

I have a dataset with approximately 4000 rows and 150 columns. I want to predict the values of a single column (= target). The data is on cities (demography, social, economic, ... indicators). A lot ...
0
votes
0answers
68 views

Identifying multivariate outliers in a large sample with missing data, using SPSS

I'm a psychology PhD student doing analysis on a relatively large set of data, obtained via online surveys. The purpose of the study is largely to determine normative data for a population of adults, ...
0
votes
0answers
15 views

How to aggregate daily data, with a large amount of missing data?

I have a conceptual problem in a rather large study of the following design: technical details of a telephone connection are monitored daily, and at the end of the month a questionnaire is filled out ...
1
vote
2answers
45 views

Handle missing values in factor variable

I have a huge dataset for a binary classification problem (about 1.5 million rows), and the feature space is quite large (145 dimension). Some of these features are factors (YES, NO), but there is ...
0
votes
1answer
85 views

Prediction when survey subsets create dramatically smaller Ns

Suppose you want to predict an outcome using a sample whose N is... 10,000 based on most demographic variables 9,000 based on Survey Question 1 3,000 who answered "Yes" to Question 1 and thus were ...
1
vote
0answers
15 views

Determine if missing values are informative(due to treatment) or just noise.

I'm not sure if this is those appropriate way to phrase this question. I have two populations of measurements. In my control population I have a quantitative measure of a proteins abundance. I expect ...
2
votes
1answer
123 views

Impute missing values using aregImpute

I have a data frame with 61 columns. Some data is missing. I read in Steyerberg's book about aregImpute in Hmisc. I used it with ...
0
votes
0answers
29 views

Joint model for ordinal repeated measures data and MNAR dropout using R

I have a dataset consisting of repeated measures data of graded toxicity scores (0-4) in a large number of patients being treated with a anti-cancer drug. We would like to identify predictors for ...
2
votes
1answer
189 views

Multiple imputation introduces negative values; dataset still valid?

After some detective work in my data sources, I realized the reasons for my previously reported 98% of missing data ratio. After implementing some data collection code fixes, the current missing data ...
0
votes
2answers
86 views

Error using mice() package in R for handling missing data [closed]

I am doing regression with a data with Y as target variable and 16 feature variables. I had two date feature variables which where as factor. I converted them to date format as shown below: ...
0
votes
1answer
51 views

NA produced in linear regression model

I have read similar posts to this but my problem is not resolved by the answers given. I want to do a v simple linear regression to see if bite incidence is related to district, zone (vacc or control) ...
0
votes
1answer
29 views

Missing data on Extreme Value Analysis

I am analyzing (extreme value analysis) the dataset which contain daily rainfall over 100 years of a single location. However there are around 500 missing values on the whole dataset. In this case the ...
1
vote
0answers
114 views

Ensuring exploratory study's validity with pseudo-simple random sampling

The context of my questions is as follows. I'm performing a cross-sectional secondary research study, involving open source software (OSS) projects. I collect data (information about the projects) ...
1
vote
1answer
37 views

Missing data analysis software

Does any standard statistical software like R, SAS or SPSS have procedures or codes to analyze log-linear models for missing data in contingency tables using maximum likelihood estimation (or EM ...
0
votes
0answers
18 views

Imputing values that depend on each other, such as percentiles or proportions

I have a data set with school level measures including test score percentiles. These percentiles are central to my analysis. For example, one measure is the test score of the 25th and 75th percentile ...
0
votes
0answers
23 views

GBM package: Why there is a missing node?

Why there is a missingNode as 3 as there are no missing values? I have the data in the following form: ...
0
votes
0answers
32 views

How to handle systematically missing values?

In my situation, one of two sources is not invoked if the confidence reported by the first source is higher than a threshold and hence it is missing in some examples. How can account for such missing ...
1
vote
0answers
15 views

Proper way to stack models when some models aren't always applicable?

Suppose you have two (or more) models that you want to ensemble together. However, some of the models are trained specifically on very specialized subsets of your data. If you do the stacking with a ...