# Tag Info

18

If you're importing your data with a command like, say, read.table('yourfile.txt', header=TRUE, ...) you can indicate what values are to be considered as "null" or NA values, by specifying na.strings = "999999999". We can also consider different values for indicating NA values. Consider the following file (fake.txt) where we want to treat "." and ...

12

Barring the fact that it's not necessary to shoot mosquitoes with a cannon (i.e. if you have one missing value in a million data points, just drop it), using the mean could be suboptimal to say the least: the result can be biased, and you should at least correct the result for the uncertainty. There are some other options, but the one easiest to explain is ...

11

Well, you should also considered that "don't know" is at least some kind of answer, whereas non-response is a purely missing value. Now, we often allow for "don't know" response in survey just to avoid forcing people to provide a response anyway (which might bias the results). For example, in the National Health and Nutrition Examination Survey, they are ...

11

You did not tell us very much about the nature of your missing data. Did you check for MCAR (Missing Completely at Random)? Given that you cannot assume MCAR, mean substitution can lead to biased estimators. As a non-mathematical starting point, I can recommend the following two references: Graham, Hohn W. (2009): Missing Data Analysis: Making It Work in ...

9

The first thing to say is that, for me, method 1 (sampling) seems to be without much merit - it is discarding the benefits of multiple imputation, and reduces to single imputation for each observation, as mentioned by Stas. I can't see any advantage in using it. There is an excellent discussion of the issues surrounding propensity score analysis with ...

8

Create a scatterplot to check whether it makes any sense to suppose that a single correlation coefficient is an adequate description of the association between the variables. For example, in these (simulated) data the correlation for ages 6-20 is 90%, for ages 50+ it's -70%, and overall it's 15%. In such a situation reporting a single correlation ...

8

Are the data "missing" in the sense of being unknown or does it just mean there is no loan (so the loan amount is zero)? It sounds like the latter, in which case you need an additional binary dummy to indicate whether there is a loan. No transformation of the loan amount is needed (apart, perhaps, from a continuous re-expression, such as a root or started ...

8

As @Dirk Eddelbuettel already mentioned, your questions is not very clear. In fact, I think you are asking two questions. The first question is related to your M(C)AR assumption. The second question is about (an) appropriate R package(s). (1) "Testing" for MAR To test if age has an effect on the missingness of your score variable, you could run a simple ...

8

There might be a clash of two paradigms. Multiple imputation is a heavily model-based Bayesian solution: the concept of the proper imputation essentially states that you need to sample from the well-defined posterior distribution of the data, otherwise you are screwed. Propensity score matching, on the other hand, is a semi-parametric procedure: once you ...

7

For a logistic regression fitted by maximum likelihood, as long as you have both (1) and (2) in the model, then no matter what "default" value that you give new runners for (2), the estimate for (1) will adjust accordingly. For example, let $X_1$ be the indicator variable for "is a new runner", and $X_2$ be the variable "previous laptime in seconds". Then ...

7

I can't really speak to the theoretical aspects of the question, but I'll give my experience using PS/IPTW models and multiple imputation. I've never heard of someone using multiply imputed data sets and random sampling to build a single data set. That doesn't necessarily mean it's wrong but it's a strange approach to use. The data set also isn't big ...

6

I was just wondering about exactly the same question when analyzing the latest National Hospital Discharge Survey data. Several variables have substantial missing values, such as marital status and type of procedure. This issue came to my attention because these categories showed up with strong (and significant) effects in most logistic regression analyses ...

6

This is a quick partial response to outline some options and correct some errors. You are implicitly seeking a method of moments estimator. Under your assumptions, letting $f$ be the failure rate and $n$ be the fleet size, the expectations of the $S_i$ (which are governed by a multinomial distribution) are \eqalign{ \mathbb{E}_{f;n}[S_0] = &f^2 n ... 6 This looks like a good problem for machine learning, so I'll concentrate on this group of methods. First and the most obvious idea is the kNN algorithm. There you first calculate the similarity among viewers and then predict the missing votes with the average vote on this picture cast by similar users. For details see Wikipedia. Another idea is to grow ... 6 As far as I understand your question, you want to investigate if missing values in your data appear due to some pattern. In this case, you don't need any "missing value analysis" -- this is the same problem as checking whether the score is bigger than 0.7 or whatever. Just convert your dataset into two-class factor (missing, not-missing) and look for ... 6 The general effect of replacing missing values with means or medians is to give you wrong results and neither is generally recommended. Better methods are things like Multiple Imputation or the EM algorythm (or both) to estimate the covariance matrix and take into account the uncertainty due to the missing information. But before using either of those ... 6 Factor analysis or principal components analysis may indeed yield solutions whose answers are rotated or mirrored versions of each other, so averaging the person scores is not a good idea. Several authors have explored the use of Procrustes analysis to correct for the rotational indetermination, so try searching on 'multiple imputation' and 'procrustes'. 5 Instead of assigning special value for non-existent first time runner previous lap time, simply use interaction term for previous lap time with the inverse of first time runner dummy:Y_i=\beta_0+\beta_1 FTR_i+\beta_2 (NFTR_i)\times PLT_i+...$$here Y_i is your input variable, ... is your other variables, FTR_i is dummy for the first time ... 5 I like the partial identification approach to missing data of Manski. The basic idea is to ask: given all possible values the missing data could have, what is the set of values that the estimated parameters could take? This set might be very large, in which case you could consider restricting the distribution of the missing data. Manski has a bunch of papers ... 5 Some years ago, I thought it might be a good idea to apply person-mean imputation (person-mean substitution or case-mean imputation) in case of item non-response. Nowadays, however, it seems obvious to me that this approach assumes that all scale items share similar characteristics (similar variance, standard deviation, item difficulty, etc.). In other ... 5 Unless there is some specific reason for people being NA, and unless you are interested in that reason, then I would say to not include people who are missing. You don't need an exact test here; all the cell sizes are reasonable. However 1) Don't you want some form of regression instead? and 2) Why is Albumin dichotomized into low and high? Dichotomizing ... 5 Missing at random (MAR) means that the missingness can be explained by variables on which you have full information. It's not a testable assumption, but there are cases where it is reasonable vs. not. For example, take political opinion polls. Many people refuse to answer. If you assume that the reasons people refuse to answer are entirely based on ... 5 I couldn't comment here due to low reputation score but please post your sample data how it looks like, what you want along with your question. Just words is too much confusing... Also this question should belong to stackoverflow. Edit: Use Dwin's method: f=function(x){ x<-as.numeric(as.character(x)) #first convert each column into numeric if it is ... 5 (Some moderator must have a warped sense of what is R and what is statistics. This is a coding question if I ever saw one.) Since the columns are of necessity "character" the values will be "character". new <- lapply( dfrm, function(x) x[x=="Hi"] <- median(as.numeric(as.character(x)), na.rm=TRUE) ) If they need to be numeric you can do this ... 5 My suggestion depends on how much data is missing and why it is missing. But this has nothing to do with PCA, really. If there is very little data missing, then it won't much matter what you do. Replacing with the median isn't ideal, but if there is not much missing, it won't be much different from a better solution. You could try doing PCA with both median ... 5 There is in fact a well documented way to deal with gappy matrices - you can decompose a covariance matrix \textbf{C} contructed from of your data \textbf{X}, which is scaled by the number of shared values n:$$ \textbf{C}=\frac{1}{n} \textbf{X} ^ {\text{T}} \textbf{X},~~~~~~~~~~~~~~~~ C_{jl} = \overline{X_{.j}Y_{.l}}  and then expand the principal ...

5

Substituting by the mean value is problematic and can lead to poor results. A principled way to tackle this problem is described in this paper. The idea is to formulate the problem in a probabilistic model which allows treating the missing components as hidden variables, and use the EM algorithm to estimate them. The paper also explains why is not ...

4

The basic way to see if your data is Weibull is to plot the log of cumulative hazards versus log of times and see if a straight line might be a good fit. The cumulative hazard can be found using the non-parametric Nelson-Aalen estimator. There are similar graphical diagnostics for Weibull regression if you fit your data with covariates and some references ...

4

If you want to import the data to R leave the cells blank. If file is saved as csv and imported to R, blank cells will be represented as NA automatically. If you want to do some analysis in OpenOffice, I think you will find that @Andy W advice useful. Built-in OpenOffice functions may behave weirdly if you use some custom NA declaration. Finally as ...

Only top voted, non community-wiki answers of a minimum length are eligible