Hot answers tagged error
22
@DikranMarsupial is exactly right, of course, but it occurred to me that it might be nice to illustrate his point, especially since this concern seems to come up frequently. Specifically, the residuals of a regression model should be normally distributed for the p-values to be correct. However, even if the residuals are normally distributed, that doesn't ...
12
Clock synchronization issues could indeed cause the peak to be shifted to the right. The following simulation in R shows this phenomenon. I used exponential times and normal clock differences to get a shape that roughly resembles your picture:
The distribution to the left (the actual differences, measured without error) has its peak at 0, whereas the ...
8
The question looks simple, but your reflection around it shows that it is not that simple.
Actually, p-values are a relatively late addition to the theory of statistics. Computing a p-value without a computer is very tedious; this is why the only way to perform a statistical test until recently was to use tables of statistical tests, as I explain in this ...
7
The question in parentheses is incorrectly stated. It is not the distribution of response that can be said to be within x% of the population distribution with some probability, it is the estimate from the random sample that can be claimed to cover with an interval x% wide the true parameter value with a specified probability. The statement in bold in the ...
7
The concepts are indeed intimately linked to each other.
The significance level is the probability of a type I error, or rather, the presumed probability of such an event. ${\rm P}({\rm type~I~error})= \alpha$ can generally only be obtained when working with continuous distributions, so in classic test theory a test is said to have significance level ...
6
Measures of forecast accuracy were a big topic in the forecasting community some years back, and they still pop up now and then. One very good article to look at is this one.
The problem with the MSE is that the square puts a very high weight on large deviations, so the MSE-optimal forecast will have fewer large errors but may have much more small errors ...
6
You are getting good answers here from @MansT & @gui11aume (+1 to each). Let me see if I can get more explicitly at something in both of their answers.
When working with discrete data, there are only certain p-values possible, and the problem is worse with fewer possibilities / smaller data sets. For example, imagine flipping a coin $n$ times. The ...
6
Because nobody has yet answered the final question--namely, to quantify the differences between the two formulas--let's take care of that.
For many reasons, it is appropriate to compare standard deviations in terms of their ratios rather than their differences. The ratio is
$$s_n / s = \sqrt{\frac{N-1}{N}} = \sqrt{1 - \frac{1}{N}} \approx 1 - ...
5
The pool of error metrics you can choose from is different between classification and regression. In the latter you try to predict one continuous value, and with classification you predict discrete classes such as "healthy" or "not healthy". From the examples you mentioned, root mean square error would be applicable for regression and AUC for classification ...
4
Surely any decision, made objectively or subjectively, would strongly depend on what you are measuring, and how precise your instrument of measurement is. The latter is just one part of the observed variation, and not always easy to discern or find existing evidence for. Thus I strongly suspect there is no objective, universally-applicable decision. You just ...
4
Let's say you have a distribution of observations
1 2 3 4 5 6 7 8
To figure out the mean you would sum the values and divide by the number of values
1 + 2 + 3 + 4 + 5 + 6 + 7 + 8
_____________________________ = 4.5
8
Consider one data point
3
How much does 3 deviate from the mean?
The answer is
4.5 - 3 = 1.5
Next, consider the ...
3
If you show the confidence interval as well as the value of the statistic, then there is no problem with giving as many significant figures as you wish, as in that case a large number of significant figures does not imply spurious precision as the confidence interval gives an indication of the likely actual precision (a credible interval would be better). ...
3
@whuber makes excellent point regarding the goal of this endeavour, but here's a idea of how to proceed. The idea is to add each cell a corresponding amount of generated noise.
> my.data <- matrix(1:9, nrow = 3)
> my.data
[,1] [,2] [,3]
[1,] 1 4 7
[2,] 2 5 8
[3,] 3 6 9
> random.stuff <- ...
3
Least squares is an optimal property when the error distribution is Gaussian. But when the error distribution is heavytailed or there are outliers it is a bad criterion because it gives too much weight to the observations with large errors. So when robust estimation is more appropriate MAE is a better criterion than RMSE. This applies regardless of whether ...
3
I have used them both, and have a few points to make.
Rmse is useful because it is simple to explain. Everybody knows what it is.
Rmse does not show relative values. If $rmse=0.2$, you must specifically know the range $\alpha <y_x< \beta$. If $\alpha=1, \beta=1000$, then 0.2 is a good value. If $\alpha=0, \beta=1$, it does not seem not so good ...
2
Based on your confusion matrix, you've got 5,908 data points and the vast, vast majority of them are of type 0 ('employee stayed'). The classifier can therefore get away with being "lazy" and picking the majority class unless it's absolutely certain that an example belongs to the other class. Note that your overall error rate is ~7%, which is quite close to ...
2
Edit:
Now that you edited your question, I should add that the procedure of Tonry & Davis that I describe below is pretty general. In fact, appart from reducing the squared difference between the convoluted, shifted and scaled template and the observed spectrum (which can be thought of as a Maximum Likelihood estimator of the parameters with equal ...
2
As mentioned in the comments above, quantile regression uses an asymmetric loss function ( linear but with different slopes for positive and negative errors). The quadratic (squared loss) analog of quantile regression is expectile regression.
You can google quantile regression for the references. For expectile regression see the R package expectreg and the ...
2
I took out the nested loop to make it clearer for me and stuck the prior for delta in its own loop, giving this for mu: delta[i] -> delta[id[r]]
note the explicit id vector for municipalities that is as long as the data which I've labelled 'id'
I think that's the biggest change I made? Model below (also some extras in case you want to include them).
...
2
The terminology isn't standardized enough to be able to answer these questions definitively. Tentatively ...
(1) I haven't heard of 'componentwise', but would assume it means the same as 'comparisonwise'. If so, the answer is 'yes' when you append 'under the null hypothesis'.
(2) I think 'experimentwise' is a synonym of 'familywise'. So 'yes' again.
(3) ...
2
Many classifiers can predict continuous scores. Often, continuous scores are intermediate results that are only converted to class labels (usually by threshold) as the very last step of the classification. In other cases, e.g. posterior probabilities for the class membership can be calculated (e.g. discriminant analysis, logistic regression).
You can ...
2
The Guide to Uncertainty in Measurement (GUM) recommends that the uncertainty be reported with no more than 2 digits and that the result be reported with the number of significant digits needed to make it consistent with the uncertainty. See
http://www.bipm.org/utils/common/documents/jcgm/JCGM_100_2008_E.pdf
The following code was my attempt to implement ...
2
The RMSE for your training and your test sets should be very similar if you have built a good model. If the RMSE for the test set is much higher than that of the training set, it is likely that you've badly over fit the data, i.e. you've created a model that tests well in sample, but has little predictive value when tested out of sample.
2
I think you have two different types os question there. One thing is what you ask in the title: "what are good RMSE values?" and another thing is how to compare models with different datasets using RMSE.
For the first, i.e the question in the title, it is important to recall that RMSE has the same unit as the dependent variable (DV). It means that there is ...
2
The only place I can think of where "error bars" (better to use confidence limits and specify the confidence level) are out of control is where they should have been shown on the log scale but weren't. For example, if one is estimating hazard ratios, odds ratios, risk ratios, or fold-change, it is more appropriate to use a log scale when presenting the ...
2
Options I'd consider
Show the error bars fully
Don't show error bars at all; show error some other way or not at all
As suggested above, use log scale, if appropriate
Delete the last time point (and explain why in the text)
I would not truncate the error bar. To me, that does distort the data (unintentionally, but still). The fact that that last time ...
2
Similar to Stijn's comment, there isn't sufficient information to make a conclusive judgment. There could be fraud (in either of the papers, not just the first), or there could have been a Type I / Type II error in one of the papers.
Out of interest if the two studies were identical and independent, so had identical power functions which factored, would it ...
2
The information provided does not indicate fraud by either party. Assuming 95% confidence as the acceptance point then 2 in 20 tests could be expected to produce a type 1 or 2 error. This is a major reason for doing meta analyses.
Furthermore, to check for fraud, one would want to review the record keeping for the two experiments and perhaps corroborative ...
1
Why not compare $RMSE = \sqrt{MSE}$ and $MAE = MAD$?
Actually,
$MAE \leq RMSE \leq \sqrt{n} MAE$ for regression models
($MAE \leq RMSE \leq \sqrt{MAE}$ for classification with partial class memberships $y_i$ and/or $\hat y_i$ are $\in [0, 1]$ -- i.e. they can actually take values in between 0 and 1).
If the RMSE is close the MAE, you have many small ...
1
I am not sure I understand the question. The '+' function in R does this already.
> a<-matrix(1:12,nr=3)
> b<-rnorm(12) # b is a vector
> a+b # R adds b to a as a matrix
[,1] [,2] [,3] [,4]
[1,] 1.144552 4.946283 8.026290 11.40905
[2,] 1.038299 5.752317 7.544441 10.78278
[3,] 3.173348 5.574810 9.805634 13.65378
...
Only top voted, non community-wiki answers of a minimum length are eligible
