Hot answers tagged

89

[...] where the top-5 error rate is the fraction of test images for which the correct label is not among the five labels considered most probable by the mode. First, you make a prediction using the CNN and obtain the predicted class multinomial distribution ($\sum p_{class} = 1$). Now, in the case of the top-1 score, you check if the top class (the one with ...


84

The tf.train.AdamOptimizer uses Kingma and Ba's Adam algorithm to control the learning rate. Adam offers several advantages over the simple tf.train.GradientDescentOptimizer. Foremost is that it uses moving averages of the parameters (momentum); Bengio discusses the reasons for why this is beneficial in Section 3.1.1 of this paper. Simply put, this enables ...


62

Minimizing square errors (MSE) is definitely not the same as minimizing absolute deviations (MAD) of errors. MSE provides the mean response of $y$ conditioned on $x$, while MAD provides the median response of $y$ conditioned on $x$. Historically, Laplace originally considered the maximum observed error as a measure of the correctness of a model. He soon ...


61

Let's denote the true value of interest as $\theta$ and the value estimated using some algorithm as $\hat{\theta}$. Correlation tells you how much $\theta$ and $\hat{\theta}$ are related. It gives values between $-1$ and $1$, where $0$ is no relation, $1$ is very strong, linear relation and $-1$ is an inverse linear relation (i.e. bigger values of $\theta$ ...


51

The residuals are always actual minus predicted. The models are: $$y=f(x;\beta)+\varepsilon$$ Hence, the residuals $\hat\varepsilon$, which are estimates of errors $\varepsilon$: $$\hat\varepsilon=y-\hat y\\\hat y=f(x;\hat\beta)$$ I agree with @whuber that the sign doesn't really matter mathematically. It's just good to have a convention though. And the ...


49

There are many alternatives, depending on the purpose. A common one is the "Relative Percent Difference," or RPD, used in laboratory quality control procedures. Although you can find many seemingly different formulas, they all come down to comparing the difference of two values to their average magnitude: $$d_1(x,y) = \frac{x - y}{(|x| + |y|)/2} = 2\frac{...


37

It wouldn't make sense if you were talking about known probabilities, e.g. with fair coin the probability of throwing heads is 0.5 by definition. However, unless you are talking about textbook example, the exact probability is never known, we only know it approximately. The different story is when you estimate the probabilities from the data, e.g. you ...


35

As an alternative explanation, consider the following intuition: When minimizing an error, we must decide how to penalize these errors. Indeed, the most straightforward approach to penalizing errors would be to use a linearly proportional penalty function. With such a function, each deviation from the mean is given a proportional corresponding error. Twice ...


34

The multiple R-squared that R reports is the coefficient of determination, which is given by the formula $$ R^2 = 1 - \frac{SS_{\text{res}}}{SS_{\text{tot}}}.$$ The sum of squared errors is given (thanks to a previous answer) by sum(sm$residuals^2). The mean squared error is given by mean(sm$residuals^2). You could write a function to calculate this, e.g....


34

Errors pertain to the true data generating process (DGP), whereas residuals are what is left over after having estimated your model. In truth, assumptions like normality, homoscedasticity, and independence apply to the errors of the DGP, not your model's residuals. (For example, having fit $p+1$ parameters in your model, only $N-(p+1)$ residuals can be ...


34

What happens if the residuals are not homoscedastic? If the residuals show an increasing or decreasing pattern in Residuals vs. Fitted plot. If the error term is not homoscedastic (we use the residuals as a proxy for the unobservable error term), the OLS estimator is still consistent and unbiased but is no longer the most efficient in the class of linear ...


33

Your classifier gives you a probability for each class. Lets say we had only "cat", "dog", "house", "mouse" as classes (in this order). Then the classifier gives somehting like 0.1; 0.2; 0.0; 0.7 as a result. The Top-1 class is "mouse". The top-2 classes are {mouse, dog}. If the correct class was "dog", it would be counted as "correct" for the Top-2 ...


32

The mean squared error as you have written it for OLS is hiding something: $$\frac{\sum_{i}^{n}(y_i - \hat{y}_i) ^2}{n-2} = \frac{\sum_{i}^{n}\left[y_i - \left(\hat{\beta}_{0} + \hat{\beta}_{x}x_{i}\right)\right] ^2}{n-2}$$ Notice that the numerator sums over a function of both $y$ and $x$, so you lose a degree of freedom for each variable (or for each ...


30

We do choose other error distributions. You can in many cases do so fairly easily; if you are using maximum likelihood estimation, this will change the loss function. This is certainly done in practice. Laplace (double exponential errors) correspond to least absolute deviations regression/$L_1$ regression (which numerous posts on site discuss). Regressions ...


29

\begin{align*} EPE(f) &= \int [y - f(x)]^2 Pr(dx, dy) \\ &= \int [y - f(x)]^2p(x,y)dxdy \\ &= \int_x \int_y [y - f(x)]^2p(x,y)dxdy \\ &= \int_x \int_y [y - f(x)]^2p(x)p(y|x)dxdy \\ &= \int_x\left( \int_y [y - f(x)]^2p(y|x)dy \right)p(x)dx \\ &= \int_x \left( E_{Y|X}([Y - f(X)]^2|X = x) \right) p(x)dx\\ &= E_{X}E_{Y|X}([Y - f(X)]^...


28

An error is the difference between the observed value and the true value (very often unobserved, generated by the DGP). A residual is the difference between the observed value and the predicted value (by the model).


28

No, in this case, it does not make sense to draw error bars using SDs. Take a step back. Why do we draw error bars with SDs? As you write, it's to show where "much" of the data lies. This makes sense if your data come from a normal distribution: 68% of your data will lie within $\pm 1$ SD from the mean, so showing the mean with an error bar of $\pm ...


25

I just came across a compelling reason for one answer to be the correct one. Regression (and most statistical models of any sort) concern how the conditional distributions of a response depend on explanatory variables. An important element of the characterization of those distributions is some measure usually called "skewness" (even though various and ...


24

A most relevant illustration from xkcd: with associated caption: ...an effect size of 1.68 (95% CI: 1.56 (95% CI: 1.52 (95% CI: 1.504 (95% CI: 1.494 (95% CI: 1.488 (95% CI: 1.485 (95% CI: 1.482 (95% CI: 1.481 (95% CI: 1.4799 (95% CI: 1.4791 (95% CI: 1.4784...


23

You should ask yourself what were you trying to achieve with your modeling approach. As you correctly said "how far from true solution am I" is a good starting point (notice this is also true for classification, we only get into specifics when we run into dichotomization, usually in more CS oriented machine learning, such as trees or SVMs). So, let's ...


19

Short Answer The probability density of a multivariate Gaussian distributed variable $x=(x_1, x_2,...,x_n)$, with mean $\mu=(\mu_1,\mu_2,...,\mu_n)$ is related to the square of the euclidean distance between the mean and the variable ($\vert \mu-x \vert_2^2$), or in other words the sum of squares. Long Answer If you multiply multiple Gaussian distributions ...


18

The equation (2.11) is a consequence of the following little equality. For any two random variables $Z_1$ and $Z_2$, and any function $g$ $$ E_{Z_1, Z_2} (g(Z_1, Z_2)) = E_{Z_2}(E_{Z_1 \mid Z_2}(g(Z_1, Z_2) \mid Z_2)) $$ The notation $E_{Z_1, Z_2}$ is the expectation over the joint distribution. The notation $E_{Z_1 \mid Z_2}$ essentially says "integrate ...


16

As far as I understand your question, the difference between the "Monte Carlo" approach and the bootstrap approach is essentially the difference between parametric and non-parametric statistics. In the parametric framework, one knows exactly how the data $x_1,\ldots,x_N$ is generated, that is, given the parameters of the model ($A$, $\sigma_A$, &tc. in ...


15

Which model is appropriate depends on how variation around the mean comes into the observations. It may well come in multiplicatively or additively ... or in some other way. There can even be several sources of this variation, some which may enter multiplicatively and some which enter additively and some in ways that can't really be characterized as either. ...


13

I find some parts in this book express in a way that is difficult to understand, especially for those who do not have a strong background in statistics. I will try to make it simple and hope that you can get rid of confusion. Claim 1 (Smoothing) $E(X) = E(E(X|Y)),\forall X,Y$ Proof: Notice that E(Y) is a constant but E(Y|X) is a random variable ...


13

The RSS is the sum of the square of the errors (difference between calculation and measurement, or estimated and real values): $ RSS = \sum{(\hat Y_i-Y_i)^2} $ The MSE is the mean of that sum of the square of the errors: $ MSE = \frac{1}{n}\sum{(\hat Y_i-Y_i)^2}$ The RMSE is the square root of the MSE: $ RMSE = \sqrt{MSE} $ A bit of math shows: $ RMSE ...


13

Why do you need a reference? This is a simple calculus problem: For the problem as you have formulated it to make sense, we must assume that all $x_i > 0$. Then define the function $$ f(z) = \sum_{i=1}^n \frac{(x_i - z)^2}{x_i} $$ Then calculate the derivative with respect to $z$: $$ f'(z) = -2\cdot \sum_{i=1}^n (1-\frac{z}{x_i}) $$ then solving the ...


12

No matter what Errror measurement you give, consider giving your complete result vector in an appendix. People who like to compare against your method but prefer another error measurement can derive such value from your table. $R^2$: Does not reflect systematic errors. Imagine you measure diameters instead of radii of circular objects. You have an expected ...


12

A common decomposition of the error incurred when forming a predictive model is into three pieces. 1) Bayes Error: Even the best predictor will sometimes be wrong. Imagine predicting height based on gender. If you had the best predictor available you would still incur error because height does not depend solely on gender. The best predictor is ...


12

The Random Change in your Monte Carlo Model is represented by a bell curve and the computation probably assumes normally distributed "error" or "Change". At least, your computer needs some assumption about the distribution from which to draw the "change". Bootstrapping does not necessarily make such assumptions. It takes observations as observations and if ...


Only top voted, non community-wiki answers of a minimum length are eligible