89

[...] where the top-5 error rate is the fraction of test images for which the correct label is not among the five labels considered most probable by the mode. First, you make a prediction using the CNN and obtain the predicted class multinomial distribution ($\sum p_{class} = 1$). Now, in the case of the top-1 score, you check if the top class (the one with ...


49

There are many alternatives, depending on the purpose. A common one is the "Relative Percent Difference," or RPD, used in laboratory quality control procedures. Although you can find many seemingly different formulas, they all come down to comparing the difference of two values to their average magnitude: $$d_1(x,y) = \frac{x - y}{(|x| + |y|)/2} = 2\frac{...


39

It depends on what exactly your weights apply to. Row weights Let $\mathbf{X}$ be the data matrix with variables in columns and $n$ observations $\mathbf x_i$ in rows. If each observation has an associated weight $w_i$, then it is indeed straightforward to incorporate these weights into PCA. First, one needs to compute the weighted mean $\boldsymbol \mu = \...


33

Your classifier gives you a probability for each class. Lets say we had only "cat", "dog", "house", "mouse" as classes (in this order). Then the classifier gives somehting like 0.1; 0.2; 0.0; 0.7 as a result. The Top-1 class is "mouse". The top-2 classes are {mouse, dog}. If the correct class was "dog", it would be counted as "correct" for the Top-2 ...


28

Let's start with an example. Say Alice is a track coach and wants to pick an athlete to represent the team in an upcoming sporting event, a 200m sprint. Naturally she wants to pick the fastest runner. A strictly proper scoring rule would be to nominate the fastest runner of the team over the 200m distance. This maximizes exactly what coach Alice wants in ...


20

I don't know if there is a straightforward generic interpretation, even analysing a particular case. For example, you may be interested in evaluating what would be the error if you predict all the cases with the mean value and compare it to your approach. Anyway, I believe RMSLE is usually used when you don't want to penalize huge differences in the ...


19

If the random variable is restricted to $[a,b]$ and we know the mean $\mu=E[X]$, the variance is bounded by $(b-\mu)(\mu-a)$. Let us first consider the case $a=0, b=1$. Note that for all $x\in [0,1]$, $x^2\leq x$, wherefore also $E[X^2]\leq E[X]$. Using this result, \begin{equation} \sigma^2 = E[X^2] - (E[X]^2) = E[X^2] - \mu^2 \leq \mu - \mu^2 = \mu(1-\...


19

Precision can be estimated directly from your data points, but accuracy is related to the experimental design. Suppose I want to find the average height of American males. Given a sample of heights, I can estimate my precision. If my sample is taken from all basketball players, however, my estimate will be biased and inaccurate, and this inaccuracy cannot be ...


16

Provided the measurement errors are independent and identically Normally distributed for each instrument, the solution is to match the two sets of measurements in sorted order. Although this is intuitively obvious (comments posted shortly after the question was posted state this solution), it remains to prove it. To this end, let the first set of ...


15

Which model is appropriate depends on how variation around the mean comes into the observations. It may well come in multiplicatively or additively ... or in some other way. There can even be several sources of this variation, some which may enter multiplicatively and some which enter additively and some in ways that can't really be characterized as either. ...


13

The Kappa coefficient is a chance-adjusted index of agreement. In machine learning it can be used to quantify the amount of agreement between an algorithm's predictions and some trusted labels of the same objects. Kappa starts with accuracy - the proportion of all objects that both the algorithm and the trusted labels assigned to the same category or class. ...


11

First, note that you typically take the absolute value in computing the relative error. A common solution to the problem is to compute $$\text{relative error}=\frac{\left| x_{\text{true}}- x_{\text{test}} \right|}{1+\left|x_{\text{true}} \right|} .$$


9

You can see what it means by studying the formula: $$ \alpha = \frac{K}{K-1}\left(1-\frac{\sum \sigma^2_{x_i}}{\sigma^2_T}\right) $$ where $T=x_1 + x_2 + ... x_K$. $T$ is the total score of a test with $K$ items, each scores $x_i$, respectively. Unpack the formula, using what we know about the covariance of a sum of RV's. If the test items are ...


8

According to mathworld: de Moivre developed the normal distribution as an approximation to the binomial distribution, and it was subsequently used by Laplace in 1783 to study measurement errors and by Gauss in 1809 in the analysis of astronomical data (Havil 2003, p. 157). Some extensions to the normal distribution have been developed in psychometry, ...


8

Presumably this nomenclature is chosen simply to draw an analogy between the GLM and linear regression. You are correct that this term is not strictly accurate, since the family of distributions chosen for the GLM is not actually the distribution of an "error" quantity in the model. One can construct quantities in the GLM that are essentially measures of ...


7

Yes, assuming by "gross sampling error" you mean mean-squared error or the $\epsilon$ term in a model like $Y=AX + \epsilon$ The error component of a model includes all sources of variability that are not explicitly included in the model. This includes sampling errors (uncertainty due to measuring only a subset of the population), measurement errors (...


7

Dichotomizing predictor variables actually reduces power to detect relationships between a continuous predictor and the response variable. Royston (2006) is one of many articles citing this as a reason why dichotomizing is a bad idea. You can see @gung's answer to this question highlighting even more problems, such as hiding potential nonlinear ...


7

Thank you very much amoeba for the insight regarding row weights. I know that this is not stackoverflow, but I had some difficulties to find an implementation of row-weighted PCA with explanation and, since this is one of the first results when googling for weighted PCA, I thought it would be good to attach my solution, maybe it can help others in the same ...


6

Two standard methods are Consult the "instrument maker's specifications," as indicated in the quotation. This is usually a crude fall-back to be used when no other information is available, because (a) what the instrument maker really means by "accuracy" and "precision" is often indeterminate and (b) how the instrument responded when new in a test lab was ...


6

What happens to the standard deviation? The most general way to do this is to transform the variable and compute the standard deviation of the transformed variable. For example, if $X\sim N(\mu,\sigma^2)$, then $e^X\sim\text{logN}(\mu,\sigma^2)$, which has s.d. $e^{\mu+\sigma^2/2}.\sqrt{e^{\sigma^2}-1}$. However, rather than having to do the ...


6

RMSE arises from what is probably the most important model in statistics, linear regression. A linear regression model is fit with least squares, which means minimizing the mean square error (MSE) for the sample. Take the square root of MSE, so that it's on the same scale as the data and hence easier to interpret, and you get RMSE. If that just makes you ...


6

Here's some intuition: The bivariate distribution of the errors has its maximum at 0. However, the distribution of the distance from the center does not, since the only point contributing density there is the one at the center. As you move out a little the bivariate density has decreased only a little but you get a little "circle" of contributions to the ...


6

One of the more interesting choices in R is rstan, where you could code this up yourself in the Stan modeling language (which tends to be amazing in that it can produce inference for models that we used to be unable to do for a long time). However, getting started can be a little challenging and it sounds like you'd like a higher level interface. That could ...


5

What matters is the magnitude of an error multiplied by its likelihood. When a noisy continuous variable is dichotomized, the magnitude of an error is huge because the error is to put someone in the wrong category - a 100% error. http://biostat.mc.vanderbilt.edu/wiki/pub/Main/BioMod/catgNoise.r is a script that can be run in RStudio (it requires the R ...


5

Say that I am building a linear regression model p for predicting some values $y_1,…,y_n$. If the data contains a few extreme outliers in the response - or even just one - the MSE fitted equation can be pulled arbitrarily far away from the MAE one. Consider the simplest regression model (just an intercept, $\alpha$), and following data: 0.0003 0.0001 0....


5

If the mutual information is zero then the variables are independent. The closer the mutual information gets to zero the closer the variables are to being independent. If the MI is small then knowing X tells you little about X - Y. Correlation is a linear operator. It is a weaker condition than independence. The error can be uncorrelated but nowhere close ...


5

We can model the experiment as $$x_i=x_i^*+\tilde u_i$$ $$y_i=y_i^*+\tilde v_i$$ $$\tilde u_i=\bar u + v_i$$ $$\tilde v_i=\bar v + u_i$$ where $x_i^*, y_i^*$ denote true values, $\tilde u_i,\tilde v_i $ are measurement errors, $\bar u,\bar v $ are their "fixed" components independent from observation (which could arise from wrong calibration of the sensors)...


5

When you combine questions you increase the reliability of the measure, and with a more reliable measure, you have more power. The increase in reliability that you get from more questions is estimated using the Spearman Brown prophecy formula, here's the Wikipedia page: https://en.wikipedia.org/wiki/Spearman%E2%80%93Brown_prediction_formula (Oh, it calls ...


5

The cumulative error (also referred to as system error) - It's a single direction error. e.g, - If you are to measure 10 km run & your stopwatch is running 2 sec faster every minute. So at the end of the experiment to calculate error you will add 2 secs for each minute. Additive errors (also referred as multiplicative) - Error term which can go either ...


5

It sounds like you want to calculate a standard error for the unobserved count (i.e. counts of values without the error) in each bin. For each bin you can calculate the probability that a given observation ($x_i^\text{obs}$ with associated standard deviation $\sigma_i$) could have come from any given bin. So the number of observations actually in some ...


Only top voted, non community-wiki answers of a minimum length are eligible