5

Noise does not have negative connotations in statistics or machine learning. In both cases, we are dealing with random variables, so if $\mathbf{x}$ is a random variable, then any function of it $f(\mathbf{x})$ will be a random variable as well. In such a case, the output of the function would be random, non-deterministic, or “noisy”. It is not the loss ...


4

No, it is not safe to assume that all loss functions are quadratics. One of the most common cost functions is the binomial cross-entropy $$ L(p) = y\log p +(1-y)\log(1-p) $$ where $0 \le p \le 1$ and $y \in \{0,1\}$. The function $L$ is not a quadratic because it is not a polynomial of degree 2.


4

There are many wrong or slightly off assumptions in this question, I will try to work through them. Minor point: Neural networks may or may not be statistical models (many of them are not). I would say that you need a likelihood function or at least a generative model of the data for a mathematical/computational model to be called a statistical model. ...


3

In my opinion, it makes the most sense to first consider which functional of the (unknown, and typically only implicitly considered) future distribution we want to elicit: the mean, the median, a particular quantile, or some more exotic functional. Only once we know that can we decide on a loss function, and we should choose a loss function that is optimized ...


2

What I think is a good solution so far, Improving the Prediction of Asset Returns With Machine Learning by Using a Custom Loss Function Dr. Dessain had the same question and answered in his paper. "Dessain (2021) offers arguably the most comprehensive overview to date, with 190 articles reviewed over the period 2010 – June 2021, but with a narrow focus ...


2

The best explanation is given in A Tutorial on Energy-Based Learning by LeCun et al, concretely in section 5. Also, Learning a Similarity Metric Discriminatively, with Application to Face Verification by Chopra et al. provides a detailed analysis for the case of face verification. The motivation for introducing the margin is to avoid a collapsed solution (...


2

"Because it is not convex, we cannot optimize a model." It is not true that you can't optimize a non-convex function, it just means there is no guarantee of a unique global minimum and that there may be local minima. This isn't actually as much of a problem as often we don't want to find the global minimum (especially for an unregularised neural ...


2

Even your simple suggestion to evaluate a function at random points is often used in cases where a full search is computationally infeasible and the function is not smooth. A famous example is RANSAC for estimating shape parameters from point clouds (here the function is the argmax of an accumulator array in a rasterized parameter space). There are much more ...


2

Five reasons come to mind quickly. Square loss brutally punishes bad misses. If you miss by $1$, your square loss is $1$, but if you miss by $2$, your square loss is $4$. This helps keep a model from making gigantic errors. Square loss is related to the variance of an error term, if you’re willing to assume that variance to be constant. Minimizing square ...


2

Because they aren’t convex. This is an example from actual empirical research I used in my other answer that tries to visualize loss landscape of an actual neural network: (source: https://www.cs.umd.edu/~tomg/projects/landscapes/ and arXiv:1712.09913)


1

No it is not convex. Note that $-ReLU(x)$ is the composition of a ReLU with a ($1\times 1$) linear layer.


1

For future readers: I clarified my understanding of the question in the comments. EDIT: This answer is not specific to LSTM or neural networks, it is true for any predictive algorithm. Response: In general, you probably can tell overfitting/underfitting from a single plot of true values (all, train and test) + training data predictions + testing data ...


1

Random search means that you explore the potential hyperparameter values by picking the random combinations of hyperparameters. Marsaglia (1972) invented and algorithm for sampling points uniformly at random in a sphere, this may or may not be how you would like to sample the hyperparameters. There are many different algorithms for generating pseudo-random ...


1

If you look at the Loss it is composed of two parts, 1st the loss when they are similar and second the loss when they are dissimilar. If model works very well, observation which are similar should have very small distance. Using Sqaure distance in first ensure that model is penalized if its gives high distance for similar observation (classes) For ...


1

Thanks everybody for the contributions! I am late to the party, but I wanted to add one more point regarding the log-link function, which was to me still unclear. I take the formula for the Gamma distribution from the bottom of gazza89's answer: $$ \frac{1}{\Gamma(k)(\frac{\mu}{k})^k}x^{k-1}e^{-\frac{xk}{\mu}} $$ Using the logarithm as link function amounts ...


Only top voted, non community-wiki answers of a minimum length are eligible