New answers tagged

1

The problem isn’t specifically that it reduces the variance, but that it changes the variance of the dataset, making it a less accurate estimate for the variance of the actual population. More generally, it will make the dataset a less accurate reflection of the population, in many ways. It’s helpful to consider alternatives. Why would using 0 (or any ...


0

Yes, I like to idea of sampling from a distribution, when one has many missing values, to get a replacement value for missing value k. My choice, however, is a distribution centered at the sample median (not mean) and with variance given here https://www.jstor.org/stable/30037287?seq=1 . Perhaps sample from a truncated normal based upon the above ...


4

Another possible disadvantage with using the mean for missing values is that the reason the values are missing in the first place could be dependent on the missing values themselves. (This is called missing not at random.) For example, on a health questionnaire, heavier respondents may be less willing to disclose their weight. The mean of the observed ...


0

"Why is this variance reduction considered as a bad thing?" As an oversimplified example: imagine, for a moment, that you have an extremely small economy on an island somewhere, with just 5 people. Their Annual Incomes are as follows: Person 1: ♦10,000 Person 2: ♦10,000 Person 3: ♦12,000 Person 4: ♦13,000 Person 5: ♦25,000 A car company seeking to "break ...


24

Example with normal data. Suppose the real data are a random sample of size $n=200$ from $\mathsf{Norm}(\mu=100, \sigma=15),$ but you don't know $\mu$ or $\sigma$ and seek to estimate them. In the example below I'd estimate $\mu$ by $\bar X = 100.21$ and $\sigma$ by $S = 14.5,$ Both estimates are pretty good. (Simulation and computations in R.) set....


10

Using the mean for missing values is not ALWAYS a bad thing. In econometrics, this is a recommended course of action in some cases provided you understand what the consequences may be and in what cases it is helpful. As you have read, replacing missing values with the mean can reduce the variance. Here is what that can cause. Note that for regression ...


0

There is not a unique answer to your question. Because I tend to use Bayesian methods, I would split it into two variables. The first variable would be present/absent. The second would be the distance given that a car is present. Because there could exist blind spots, you also would need to decide the uncertainty present in the reading of present/absent. ...


0

(Unable to comment yet - sorry! I would have like to have commented on Joel's response.) I want to point out that, I believe, the quality of the imputation algorithm has bearing on the amount of data that may be validly imputed. If the imputation method is poor (i.e., it predicts missing values in a biased manner), then it doesn't matter if only 5% or 10% ...


1

If more information is available, you may wish to incorporate it into assigning a missing data point. For example, a study of players historical rankings over time, including those who were unranked for a period, might suggest a more meaningful value (derived from their average rank) than using just a team average value. Losing data is not usually the ...


0

You are correct to question if we should impute or not. We should not "fill" a time-series where the gaps or dips are real observations rather than failures or corrupted data. The days that there was no purchasing reflect the true underlying purchasing pattern. Most buyers do not go out and buy things every day; imputing these variable would create an ...


1

Sorry to hear about your glitch. I'm not sure about the answer, but I suppose it would depend on a few questions. A. What is this questionnaire going to be used for? Diagnosis? Research analysis? Medical treatment? I would definitely not use it for any diagnostic criteria that had actionable impact on your participants. For a peer-reviewed paper, that ...


0

You could express the complete log-likelihood as $$\ell(\theta) = -n\log\sigma_1-\frac{1}{2}\sum_{i=1}^n \sigma_1^{-2}(z_i-\mu_1)^2 -n\log\sigma^2-\frac{1}{2}\sum_{i=1}^n \sigma^{-2}(y_i-\beta_1-\beta_2 z_i)^2$$ to derive $$\mathbb E_{\theta^\text{old}}[\ell(\theta)|X^\text{obs}]$$as \begin{align}&-n\log\sigma_1-\frac{1}{2}\sum_{i=1}^m \sigma_1^{-2}(z_i-\...


Top 50 recent answers are included