When the data present lack of information (gaps), i.e., are not complete. Hence, it is important to consider this feature when performing a analysis or test.

learn more… | top users | synonyms (2)

1
vote
0answers
29 views

Imputation of missing values for doing PCA in R

I have a dataset with approximately 4000 rows and 150 columns. I want to predict the values of a single column (= target). The data is on cities (demography, social, economic, ... indicators). A lot ...
0
votes
0answers
22 views

Identifying multivariate outliers in a large sample with missing data, using SPSS

I'm a psychology PhD student doing analysis on a relatively large set of data, obtained via online surveys. The purpose of the study is largely to determine normative data for a population of adults, ...
0
votes
0answers
8 views

How to aggregate daily data, with a large amount of missing data?

I have a conceptual problem in a rather large study of the following design: technical details of a telephone connection are monitored daily, and at the end of the month a questionnaire is filled out ...
1
vote
2answers
37 views

Handle missing values in factor variable

I have a huge dataset for a binary classification problem (about 1.5 million rows), and the feature space is quite large (145 dimension). Some of these features are factors (YES, NO), but there is ...
0
votes
1answer
78 views

Prediction when survey subsets create dramatically smaller Ns

Suppose you want to predict an outcome using a sample whose N is... 10,000 based on most demographic variables 9,000 based on Survey Question 1 3,000 who answered "Yes" to Question 1 and thus were ...
1
vote
0answers
14 views

Determine if missing values are informative(due to treatment) or just noise.

I'm not sure if this is those appropriate way to phrase this question. I have two populations of measurements. In my control population I have a quantitative measure of a proteins abundance. I expect ...
0
votes
0answers
14 views

Basic Help Needed in Coding Data Using Stata [migrated]

id Year Lat Long 1 1990 10 11 3 1994 2 8 . 1998 2 8 1 1993 . . 1 1991 10 . 3 1996 . 8 I recently started ...
1
vote
0answers
12 views

In R: Replace values of a row if missing with values of another row [migrated]

I am relatively new to R and probably the solution to this problem is rather simple. I have a dataframe that looks like this: ...
2
votes
1answer
74 views

Impute missing values using aregImpute

I have a data frame with 61 columns. Some data is missing. I read in Steyerberg's book about aregImpute in Hmisc. I used it with ...
0
votes
0answers
12 views

Joint model for ordinal repeated measures data and MNAR dropout using R

I have a dataset consisting of repeated measures data of graded toxicity scores (0-4) in a large number of patients being treated with a anti-cancer drug. We would like to identify predictors for ...
2
votes
1answer
145 views

Multiple imputation introduces negative values; dataset still valid?

After some detective work in my data sources, I realized the reasons for my previously reported 98% of missing data ratio. After implementing some data collection code fixes, the current missing data ...
0
votes
2answers
61 views

Error using mice() package in R for handling missing data [closed]

I am doing regression with a data with Y as target variable and 16 feature variables. I had two date feature variables which where as factor. I converted them to date format as shown below: ...
0
votes
1answer
44 views

NA produced in linear regression model

I have read similar posts to this but my problem is not resolved by the answers given. I want to do a v simple linear regression to see if bite incidence is related to district, zone (vacc or control) ...
0
votes
1answer
24 views

Missing data on Extreme Value Analysis

I am analyzing (extreme value analysis) the dataset which contain daily rainfall over 100 years of a single location. However there are around 500 missing values on the whole dataset. In this case the ...
1
vote
0answers
104 views

Ensuring exploratory study's validity with pseudo-simple random sampling

The context of my questions is as follows. I'm performing a cross-sectional secondary research study, involving open source software (OSS) projects. I collect data (information about the projects) ...
1
vote
0answers
7 views

Plotting fraction of NAs of a data frame [migrated]

Does anyone know how to plot the graphs of figure 23.1 of the example chapter of Steyerberg's book? The R-function is called "na.plot2" and Displays for example the fraction of missing values in data ...
1
vote
1answer
34 views

Missing data analysis software

Does any standard statistical software like R, SAS or SPSS have procedures or codes to analyze log-linear models for missing data in contingency tables using maximum likelihood estimation (or EM ...
0
votes
0answers
15 views

Imputing values that depend on each other, such as percentiles or proportions

I have a data set with school level measures including test score percentiles. These percentiles are central to my analysis. For example, one measure is the test score of the 25th and 75th percentile ...
0
votes
0answers
20 views

GBM package: Why there is a missing node?

Why there is a missingNode as 3 as there are no missing values? I have the data in the following form: ...
0
votes
0answers
30 views

How to handle systematically missing values?

In my situation, one of two sources is not invoked if the confidence reported by the first source is higher than a threshold and hence it is missing in some examples. How can account for such missing ...
1
vote
0answers
14 views

Proper way to stack models when some models aren't always applicable?

Suppose you have two (or more) models that you want to ensemble together. However, some of the models are trained specifically on very specialized subsets of your data. If you do the stacking with a ...
3
votes
1answer
68 views

Missing data which simply cannot exist

I have read 80% of missing data in a single variable and understand the approach for dealing with missing data which simply cannot exist for 1 variable. I am trying to generalise this up to 2 or ...
0
votes
1answer
62 views

lasso with missing values and categorical variables

I have a dataset with a lot of missing values and mix of continues and categorical variables. I want to use something like group lasso to do features selection. Probably the output is binary 0,1 and ...
1
vote
1answer
48 views

How to determine whether a set of data are qualitative or quantitative?

Assume a data set with multiple columns, where the categorical data are coded. What is the best rule(s) or rule of thumb to determine whether each column contains qualitative data or quantitative ...
0
votes
0answers
25 views

how to handle the imbalanced data in regression analysis

The problem here is very similar to the problem asked by someben in 2012 (link:Sampling for Imbalanced Data in Regression). It involves the linear regression analysis using an unbalanced dataset. Say, ...
1
vote
0answers
43 views

Mean imputation/estimation of missing data

Could someone please refer me to papers that have imputed the mean to missing values of a continuous variable? (i.e. papers that have used mean imputation) I have imputed my missing IMD values using ...
1
vote
0answers
26 views

How to handle uninterested missing (unobserved) data?

Suppose I have $N$ items $\{x_n\}_{n=1}^N$ (also denote the feature vector as $x_n$) and I have observed $M$ pairwise preferences $\{x_{i_m} \succ x_{j_m}\}_{m=1}^M$ from a user. Now I want to predict ...
0
votes
0answers
11 views

rebuild model based on what columns a record has

I've built a model in R using glm, and in the new dataset that I need to predict, there exist some new levels for the columns that are non-numeric. I know there are so many approaches to deal with ...
3
votes
1answer
196 views

Determining probability distribution for datasets with missing values

As a part of my exploratory data analysis (EDA) prior to further analysis, I'm trying to determine a probability distribution of my pilot dataset's variables. A particular feature of this dataset is a ...
1
vote
2answers
44 views

Missing values in SPSS for Chi-Square Tests

I'm a postgraduate student, doing a accent perception study, this is my first time using SPSS! I'm conducting the above tests on my data, where my participants used Likert scales to analyse the ...
0
votes
0answers
9 views

Missing values on clinical scoring system in SPSS

I have a SPSS database with 747 patients with sickness symptoms and lab results as variables. With those symptoms and results I have generate two scoring systems. I sum the scores of all the variables ...
0
votes
1answer
42 views

Measures of goodness-of-fit using multiply imputed data in Zelig

I am running a logistic regression model in R using multiply imputed data created using Amelia II, which I am then analyzing using Zelig. I would like to be able to report some measures of ...
0
votes
0answers
15 views

Categorical Missing Value Replacement

I want to ask how I could replace categorical missing values with reference please?
2
votes
0answers
15 views

Multiple Imputation Variance

I have been reading up on multiple imputation, and I am interested in the between-imputation variance. However, not in the estimation of the parameters, but in the imputed values themselves. (1) ...
1
vote
2answers
38 views

How to replace only SOME missing data

I have two types of missing data (non-responses and "Don't know or does not apply") and I want to replace only the non-responses (with mean or by using EM or by using Multiple Imputation) and to see ...
1
vote
0answers
46 views

Multiple imputation constraining for not applicable values

I want to do a multiple imputation of missing data using SPSS. I have a nominal (categorical) variable X with missing values codified as '9' and not applicable values codified as '8'. How can I ...
1
vote
1answer
44 views

Covariance matrix estimation in presence of missing data

I want to estimate a covariance matrix from data with some missing values. Ideally I'd like an R package but python could be ok. R has some built in ways of doing this. You can use ...
0
votes
1answer
12 views

Data Analysis of No-Opinion Options

In my questionnaire, I had a "no-opinion" option for several items on a 5-point-likert scale. I know, there is a lot of discussion about sense and non sense of those, but the fact is, i've got a data ...
0
votes
0answers
39 views

Estimate linear regression using items randomly selected from an item pool

I am asking this question against the background of a linear regression with single predicted variable $Y$ and multiple predictors $X$. $X$ comes from a survey using an "item pool" which suggests that ...
-1
votes
1answer
34 views

Marginal probability density of x (obs) obtained by integrating x(missing)

I am reading a text book on missing data, and a sentence below is slightly challenging for me to understand. The marginal probability density of $ \left ( x_{obs}\ \right)$ is obtained by ...
1
vote
1answer
69 views

LASSO or other regularized regression with censored (missing) data

Here is my problem. I am looking at various time series curves. Let's call them total spend aggregated over all customers on various products versus time. At any given time, I want to predict the ...
1
vote
2answers
71 views

How can I partition a distribution into two sub-populations with fixed bias? (simulation)

I am trying to simulate a selection model for a variable $Y$ dependent on covariate vector $X$, so that two groups/sub-sets $S=(0,1)$ of observations on $Y$ are created, which have a fixed difference ...
0
votes
0answers
18 views

How to replace missing values with unsupervised random forest?

from http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#unsup : Formulating it as a two class problem has a number of payoffs. Missing values can be replaced effectively. Outliers ...
1
vote
0answers
26 views

Modeling Counts With Small Observations

I am new to Cross Validated SE so I am going to try and formulate my question to the best of my ability. I have a large data set that contains $5$ different fields. The fields are ...
1
vote
1answer
32 views

Missing values - for cases of “I do not know” or “Does not apply”

If I have a questionnaire with items on a Likert scale (1 to 5) and a sixth option "I do not know/Does not apply", is it okay to consider a "I do not know/Does not apply" response as a missing value ...
3
votes
1answer
52 views

How to generating MAR data with fixed proportion of missing?

To generate data with missing at random (MAR) mechanism, usually we can first generate a complete data set then model the missing probability for variable Y, i.e. $Pr(Y=missing|{\bf X})$, using ...
0
votes
0answers
18 views

Missing Value in Commodities Prices

I am trying to model the prices of four energetic commodities with ARIMA models in R. Unfortunately the price series is not regular, as for some days, like Christmas, no price is given. My series is ...
0
votes
0answers
9 views

Missing value replacement when 3 out of four scale items are missing

I have a subscale with four items and 316 participants. Five participants missed all four items, two participants missed three of the four items and five participants missed only one item (three ...
0
votes
1answer
28 views

Overfitting on Missing Value Imputations

When performing Missing Value imputations, should we be concerned about overfitting the data? Why or why not? For example: If I impute a variables missing value using a CART regression tree, should ...
1
vote
0answers
40 views

J48 Handling Missing Value with Tree based Imputation

Aloha, currently i have some trouble and question zu implement some kind of special missing value handling in WEKA J48 algorithm using WEKA JAVA API. I want to test the performance of SHAPIRO ...