# Tag Info

Certainly the residuals are some sort of estimators of $\epsilon$ (to be clear, the definition of the residual is the estimator, the observed residual is an estimate). If the model is correct, then they may sometimes be a fairly good estimate. Indeed $e = y - \hat y = X\beta + \epsilon - X(X'X)^{-1} X'(X\beta + \epsilon) = (I - H)\epsilon$, where $H = ... 6 Because nobody has yet answered the final question--namely, to quantify the differences between the two formulas--let's take care of that. For many reasons, it is appropriate to compare standard deviations in terms of their ratios rather than their differences. The ratio is$$s_n / s = \sqrt{\frac{N-1}{N}} = \sqrt{1 - \frac{1}{N}} \approx 1 - ... 6 Measures of forecast accuracy were a big topic in the forecasting community some years back, and they still pop up now and then. One very good article to look at is this one. The problem with the MSE is that the square puts a very high weight on large deviations, so the MSE-optimal forecast will have fewer large errors but may have much more small errors ... 5 I'll go out on a limb and disagree with @whuber here. I don't think there's anything wrong with putting bands around pdfs, as long as you understand what they are: pointwise errors. It's like the similar confidence bands around smooth curves for additive regression models. If the bands are wide enough, they make it look like the curve could be far away from ... 4 Surely any decision, made objectively or subjectively, would strongly depend on what you are measuring, and how precise your instrument of measurement is. The latter is just one part of the observed variation, and not always easy to discern or find existing evidence for. Thus I strongly suspect there is no objective, universally-applicable decision. You just ... 4 Let's say you have a distribution of observations 1 2 3 4 5 6 7 8 To figure out the mean you would sum the values and divide by the number of values 1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 _____________________________ = 4.5 8 Consider one data point 3 How much does 3 deviate from the mean? The answer is 4.5 - 3 = 1.5 Next, consider the ... 4 MSE has several advantages over MAE, but also some disadvantages. Just list some of them, include but not limited to: Decomposition of MSE into Variance and Bias square is one of the most famous advantages. This property helps us to understand the logic behind error, especially MSE, while MAE has no such mathematical meaning. MAE with absolute value ... 3 I think you have two different types os question there. One thing is what you ask in the title: "what are good RMSE values?" and another thing is how to compare models with different datasets using RMSE. For the first, i.e the question in the title, it is important to recall that RMSE has the same unit as the dependent variable (DV). It means that there is ... 3 The RMSE for your training and your test sets should be very similar if you have built a good model. If the RMSE for the test set is much higher than that of the training set, it is likely that you've badly over fit the data, i.e. you've created a model that tests well in sample, but has little predictive value when tested out of sample. 3 If you show the confidence interval as well as the value of the statistic, then there is no problem with giving as many significant figures as you wish, as in that case a large number of significant figures does not imply spurious precision as the confidence interval gives an indication of the likely actual precision (a credible interval would be better). ... 3 @whuber makes excellent point regarding the goal of this endeavour, but here's a idea of how to proceed. The idea is to add each cell a corresponding amount of generated noise. > my.data <- matrix(1:9, nrow = 3) > my.data [,1] [,2] [,3] [1,] 1 4 7 [2,] 2 5 8 [3,] 3 6 9 > random.stuff <- ... 3 You need to divide by$nto get the Brier score, and there should be no 100 in its formula. It is just the mean squared error. The "error rate" is not meaningful and will give misleading results, so I would ignore it. AIC is a measure of predictive discrimination whereas the Brier score is a combined measure of discrimination + calibration. models can ... 3 As Peter has said, there are a great many things beyond just what you've written that determine whether or not you can "trust" the outcome - that the type of regression you're using is appropriate, that the relationship between the ad size and visitors is correctly specified, and that you haven't missed any variables (for example, what if its not the size of ... 2 Many classifiers can predict continuous scores. Often, continuous scores are intermediate results that are only converted to class labels (usually by threshold) as the very last step of the classification. In other cases, e.g. posterior probabilities for the class membership can be calculated (e.g. discriminant analysis, logistic regression). You can ... 2 If you include the intercept/constant term in the model (which is included implicitly by default), then what you are essentially doing here is trying to fit 4 group means with 5 parameters: 4 contrasts plus the intercept. Obviously this isn't going to work, because there are more "unknowns" (parameters) than there are "equations" (group means). To get the ... 2 The Guide to Uncertainty in Measurement (GUM) recommends that the uncertainty be reported with no more than 2 digits and that the result be reported with the number of significant digits needed to make it consistent with the uncertainty. See http://www.bipm.org/utils/common/documents/jcgm/JCGM_100_2008_E.pdf The following code was my attempt to implement ... 2 Your intuition seems right, but for clarity, I would rewrite your equations as follows, \begin{align} \text{MSE}(\hat{y}) &= E\left[(y-\hat{y})^2\right] \\ &= \underbrace{E\left[(\hat{y}-E[\hat{y}])^2\right]}_{\text{Var}[\hat{y}]} + \underbrace{(y-E[\hat{y}])^2}_{\text{Bias}^2} \end{align} Herey$is the actual value of whatever we are ... 2 The information provided does not indicate fraud by either party. Assuming 95% confidence as the acceptance point then 2 in 20 tests could be expected to produce a type 1 or 2 error. This is a major reason for doing meta analyses. Furthermore, to check for fraud, one would want to review the record keeping for the two experiments and perhaps corroborative ... 2 Similar to Stijn's comment, there isn't sufficient information to make a conclusive judgment. There could be fraud (in either of the papers, not just the first), or there could have been a Type I / Type II error in one of the papers. Out of interest if the two studies were identical and independent, so had identical power functions which factored, would it ... 2 Options I'd consider Show the error bars fully Don't show error bars at all; show error some other way or not at all As suggested above, use log scale, if appropriate Delete the last time point (and explain why in the text) I would not truncate the error bar. To me, that does distort the data (unintentionally, but still). The fact that that last time ... 2 The only place I can think of where "error bars" (better to use confidence limits and specify the confidence level) are out of control is where they should have been shown on the log scale but weren't. For example, if one is estimating hazard ratios, odds ratios, risk ratios, or fold-change, it is more appropriate to use a log scale when presenting the ... 2 in the Bayesian framework there are no such things as true values. Although this is sort of true because basically everything is a random variable in the bayesian paradigm, I think it's incorrect to disallow the notion of a "true distribution" (i.e. true sampling function) in the bayesian paradigm. If this is the case, error could be understood as the ... 2 I would suggest two approaches. First compare relative differences in all measures, and then use linear regression. Compute relative differences, e.g.$(longitude_{B}-longitude_{A})/longitude_{A}$for all variables at all time points (assuming the devices measured all variables at the same time points). Then look at the distribution and change over time of ... 2 There are several issues here. First, as your response is a proportion, and you are trying to fit a logistic regression (glm with family=Binomial) you need to specify sample size that each of those proportions refer to. See How to do logistic regression in R when outcome is fractional?. Your procedure only makes sense of underlying those proportions ... 2 Since$X$and$\epsilon$are supposed to be independent and so uncorrelated, you can subtract the means, square and average to give$\sigma^2_Y = b^2 \sigma^2_X + \sigma^2_\epsilon$. But$b^2 \sigma^2_X = \rho_{X,Y}^2 \sigma^2_Y$so you can substitute and rearrange to get your result. 2 I agree that there is really not enough information in the question to give a definitive answer, but there is enough information to point to a few areas of concern. The first thing you should do is try to understand (by talking to the relevant people, ideally) how decisions are made about when, where, and how big to run the ads. This will help you hunt ... 2 Check whether your leave-one-out procedure leaves out a single sample each time or a triplet of samples (one for each class). If it leaves only a single sample, then your training set is always biased against the test sample. That can explain below chance performance. For example, consider a classifier that instead of looking at the data, simply outputs the ... 1 With linear regression we are fitting a model$Y = \beta*X +\varepsilon$.$Y$is the dependent variable, the$X$'s are the predictor (explanatory) variables. We use the data provided to us (the training set or the sample) to estimate the population$\beta$'s. The$X$'s are not considered random variables. The$Y$'s are random because of the error component. 1 First, wilcoxon test in scipy.stats does NOT use$W$as the test statics, it instead uses$T$as defined in Siegel's popular book: Non-parametric statistics for the behavioral sciences. And yes, as @whuber correctly pointed out, once you know$T$and sample size,$W\$ is also defined (@whuber, strictly speaking, not quite, one also need to know how 0 ...