# Tag Info

19

This answer focuses entirely on mode estimation from a sample, with emphasis on one particular method. If there is any strong sense in which you already know the density, analytically or numerically, then the preferred answer is, in brief, to look for the single maximum or multiple maxima directly, as in the answer from @Glen_b. "Half-sample modes" may be ...

19

This is an example of overfitting on the Coursera course on ML by Andrew Ng in the case of a classification model with two features $(x_1, x_2)$, in which the true values are symbolized by $\color{red}{\large \times}$ and $\color{blue}{\large\circ},$ and the decision boundary is precisely tailored to the training set through the use of high order polynomial ...

18

The absolute value of this area is $$\int_{x=-\infty}^\infty \lvert F(x) - G(x)\rvert \,\mathrm{d}x,$$ which note – at least for continuous distributions – is exactly equal to $$\int_{x=-\infty}^\infty \lvert F^{-1}(x) - G^{-1}(x)\rvert \,\mathrm{d}x.$$ In one dimension, the latter is the 1-Wasserstein distance, the 1-Kantorovich distance, or the "earth-...

15

Saying "the mode" implies that the distribution has one and only one. In general a distribution may have many modes, or (arguably) none. If there's more than one mode you need to specify if you want all of them or just the global mode (if there is exactly one). Assuming we restrict ourselves to unimodal distributions*, so we can speak of "the" mode, they'...

11

In this case the NaN (not a number) is returned because the calculation of the exponential overflows in double precision arithmetic. An algebraically equivalent expression, expanded in a MacLaurin series around $0$, is $$\frac{\exp(x)}{1+\exp(x)} = \frac{1}{1+\exp(-x)} = 1 - \exp(-x) + \exp(-2x) - \cdots.$$ Because this is an alternating series, the error ...

8

There are two great recent articles on some of the geometric properties of deep neural networks with piecewise linear nonlinearities (which would include the ReLU activation): On the Number of Linear Regions of Deep Neural Networks by Montufar, Pascanu, Cho and Bengio. On the number of response regions of deep feed forward networks with piece-wise linear ...

8

Not quite. The setting is a probability space $(\Omega,\mathfrak{F},\mathbb{P})$ and a measurable function $X$ whose domain is $\Omega$ and whose codomain usually is $\mathbb{R}$ with its Borel sigma-algebra $\mathfrak{B}$ (but generally could be any measurable space). $X$ induces a probability distribution $\mathbb{P}_X$ as the push-forward of $\mathbb{P}$...

7

Roughness penalty methods seem like a good fit. Here you fit a curve $f$ to a set of data points that minimizes the sum of squared errors with a roughness penalty: $$\sum_i \left( y_i - f(x_i) \right)^2 + \lambda \int \left( f''(x) \right)^2 dx$$ The penalty term has nice intuitive appeal, it penalizes the roughness for your curve, the total amount that ...

6

Your argument is correct, though it would be worth specifying an increasing invertible function, otherwise the inequality would flip. To get the density function, you only need to differentiate the cdf. One thing interesting to note, if you grant me the use of a little measure theory, the pdf exists whether or on not the function is invertible. All that is ...

6

Consider the set of $x$, call it $S$, where $x<a$. You seek for the probability, $P(S)$. Any expression that lead to $S$ produces the exact same probability, $b$, irrespective of its decleration. If $f(x)$ is a monotonic (strictly) increasing function, $x<a$ directly implies $f(x)<f(a)$ and vice versa, i.e. if $f(x)<f(a)$, then $x<a$.

6

A simple and efficient method for this problem uses a variation of the well-known Box-Mueller transform, which connects the normal distribution to the uniform distribution on a ball. If we generate a random vector $\mathbf{Z} = (Z_1,...,Z_m)$ composed of IID standard normal random variables and a random variable $U \sim \text{U}(0,1)$ (independent of the ...

5

If you have samples from the distribution in a vector "x", I would do: mymode <- function(x){ d<-density(x) return(d$x[which(d$y==max(d$y)[1])]) } You should tune the density function so it is smooth enough on the top ;-). If you have only the density of the distribution, I would use an optimiser to find the mode (REML, LBFGS, simplex, etc.).... 5 This might be complicated to describe rigorously, using elementary notions, but the underlying concept is simple: almost all real numbers between$0$and$1$, when written in binary, have equally many zeros and ones. Because$Z$assigns zero probability to such numbers--it favors an imbalance of zeros and ones when$p\ne 1-p$--it must have a singular ... 5 Think about what a CDF represents in terms of probability. Let the variables on the x-axis be referred to as$x$and y-axis values be referred to as$y$. By definition the cumulative distribution function is showing the probability that a variable is less than or equal to$x$. More specifically, if you look at$x=0$for each curve the CDF is telling you:$P_{...

5

You aren't updating the "new" prior with the posterior! And it's much harder to do with functions, unless you're good with R functional programming, than with discretized values for $p$. I'm constructing an example similar to what you were doing, but not exactly the same. Here I assume the data follows a Binomial($n=10$, $p=0.5$) distribution,that $n=10$ ...

4

I understand you want to estimate the coefficient of the logarithm of a time trend, and predict the variable 335 days ahead after this. Your variable is $Y$ and the time trend is given by t<-seq(1,30), estimate Y.lm<-lm(Y~log(t)) Y.predict<-predict(Y.lm, nstep=335) is this what you want to do? EDIT: You can re-estimate the regression every time ...

4

This question, and all the responses by the OP, seems not to jibe in various ways. $X\sim \mathcal{N}(0,\sigma^2+1)$. $Z$, a gaussian with mean $X$, I assume this means that the conditional distribution of $Z$, given that $X = x$, is Gaussian with mean $x$, that is, $E[Z\mid X = x] = x$. As is known, by $E[Z\mid X]$ we mean a random variable that ...

4

To view argmax as a function, other than the trivial "constant" function, you would need to make it a function of one or more arguments, which could be, for instance, input data specifying a particular instance of a class of optimization problems. In your case, y could be the input data, i.e., the argument of the argmax function. But the optimization ...

4

Small corrections and notes: y should be $\frac{1}{\sqrt{\pi}}exp(-x^2)$ to be a density function. This density, as you say, represents a Gaussian distribution with specific mean/variance. Distribution function is referred as CDF, which is the integral of the density function. Now, let's clear the ambiguity. Sampling from a function means calculating a ...

4

Assume you want to apply diff to a vector $(x_1, \dots, x_n)$ of length $n$. The result will be the vector $(d_1, \dots, d_{n-1})$ of length $n-1$ with entries $$d_i = x_{i+1}-x_i.$$ Some people will use the notation $x_{[i]}$ or similar to indicate the vector $x$ with the $i$th component left out, i.e., $x_{[i]}=(x_1, \dots, x_{i-1}, x_{i+1}, \dots, x_n)... 4 This question is interesting because it concerns the general problem of regression in the sense of characterizing (or estimating based on data) the conditional distribution of one variable,$X_1,$based on values of another variable$X_2.$Their lack of statistical independence implies there is something to be gained from this. We may take our cues from ... 4 It simplifies the notation to work with $$g(\mathbf{y}) = f(\mathbf{y}+\mathbf{x}) - f(\mathbf{x})$$ because (as you can readily compute) $$g(\mathbf{0}) = 0;\ \nabla g(\mathbf{0}) = \nabla f(\mathbf{x});$$ and$f$is convex if and only if$g$is. In particular, note that for any$0\le h\le 1,$the convexity of$g$means$$g(h\mathbf{y}) = g((1-h)\... 4 The simplest and least error-prone approach - for low dimensions (see below!) - would still be rejection sampling: pick uniformly distributed points from the$m$-dimensional hypercube circumscribing the sphere, then reject all that fall outside the ball. runifball <- function(n, centre = 0, center = centre, radius = 1) { #Check inputs if (!missing(... 4 It is basically$\chi^2(P,Q)+1$, where the chi-squared divergence between two distributions is defined as$\chi^2(P,Q)=\sum_x {(P(x)-Q(x))^2\over Q(x)}=\sum_x {P^2(x)\over Q(x)}-1$. Note it is not zero if$P=Q$, so can't quite call it a distance unlike KL (using log 1=0). 3 I think your first expression should be$|\frac{\bar y}{y}-1|$, because negative values are meaningless in terms of error. Note that$|\frac{\bar y}{y}-1|=|\frac{y-\bar y}{y}|$. So the difference between the two formula above is only$\cdot 100$. It is common practice to use the short form, which provides the real information, "$\cdot100\$" is only there to ...

3

Your code does exactly what you have required of it. Function my.function returns only one value, and you asked yo put it in everywhere based on condition. I suggest to convert matrix to a vector, do the substitution and return back to matrix. For example: v<- as.vector(my.matrix) v[v>0] <- fun(v[v>0]) new.matrix <- matrix(v,ncol=ncol(my....

3

This is a good question. I think it involve theoretical mathematical proof. I have been working with Deep Learning (basically neural network) for a while (about a year), and based on my knowledge from all the papers I read, I have not seen proof about this yet. However, in term of experimental proof, I think I can provide a feedback. Let's consider this ...

Only top voted, non community-wiki answers of a minimum length are eligible