microhaus
  • Member for 1 year, 5 months
  • Last seen this week
Relationship between L1 penalty and margin in SVM
2 votes

Addressing only the following, as an extended comment: The documentation claims (and it is in line with what we know about LASSO) that $l_1$ regularisation leads to a sparse vector of coefficients. ...

View answer
Interpretation of $\mathcal{D}$ and difference between the accuracy parameter and training error
Accepted answer
1 votes

In my view, the assumption of a deterministic relationship between labels and inputs made in chapter 2 of that book can be interpreted, but it's a bit artificial because it's not probabilistic machine ...

View answer
confused about unbiasedness of sample mean
3 votes

Suppose one has a sample of $n$ chi-squared random variables. Then $\mathbb{E}[X_i]=1$. So, $\mathbb{E}[\overline{X}_n] = \frac{n}{n} = 1$ Now the sample has a chi-squared($n$) distribution so it's ...

View answer
How do you go from paper to code?
8 votes

To supplement Dikran Marsupial's answer, the following is an articulation of a process I use personally. This is from the perspective of coding machine learning algorithms for research, rather than ...

View answer
Gradient of the log likelihood for energy based models
Accepted answer
6 votes

The issue emerges in the evaluation of the second term in line $(3)$ and $(4)$ of your derivation. Note that $$\nabla_{\theta} Z(\theta)^{-1} = \nabla_{\theta} \frac{1}{\int_x \exp(-E_{\theta}(x))\, ...

View answer
Why does the exponential moving average equation divide with 1+(1-⍺)+...?
1 votes

The right hand side of the equality you have quoted is a geometric series. Identifying the first term $a=1$ and the common ratio $r=1-\alpha$, the series converges to $a/(1-r) = 1 / \alpha$ if and ...

View answer
What does "Expectation with respect to true unknown parameter" mean?
Accepted answer
1 votes

In the context of asymptotic results involving the maximum likelihood estimator, the use of the subscript $\theta_0$ in the expectations operator means the following, $$\mathbb{E}_{\theta_0}[l(\theta; ...

View answer
Why is $k = \sqrt{N}$ a good solution of the number of neighbors to consider?
Accepted answer
6 votes

There are a number of quantitative finite-sample results, and also asymptotic arguments, in support of using the heuristic $k = \sqrt{n}$, where $n$ is the sample size. However, in practice, it would ...

View answer
Smoothed CDF to calculate asymptotic normality
0 votes

Fix $z \in \mathbb{R}$ to be some value. Now for illustrative purposes, define corresponding i.i.d. random variables $Y_i = g(Z_i) = 1\{Z_i \leq z\}$. The following prompts should hopefully assist in ...

View answer
Looking to identify book by Michael I. Jordan from excerpts
Accepted answer
1 votes

The chapters of the monograph that are being listed are from a draft book by Michael Jordan that to the best of my knowledge was never published, but continues to be used in many probabilistic ...

View answer
Bishop equation 3.67
0 votes

\begin{align} p(t \vert \mathbf{x}, \mathcal{D}) &= \sum^L_{i=1}p(t, \mathcal{M}_i \vert \mathbf{x}, \mathcal{D}) \tag{marginalisation} \\ &= \sum^L_{i=1} p(t \vert \mathbf{x}, \mathcal{M_i}, \...

View answer
Conditional Expectation : How is E[E[xy|x]]=E[xE[y|x]]?
2 votes

To supplement the other answers by Taylor and mhdadk, I find it has never led me astray to have absolute clarity on the probability distributions with respect to which one is computing expectations. ...

View answer
Posterior of factors in factor analysis
Accepted answer
1 votes

The derivation relies on a result known as the matrix inversion lemma, or Woodbury matrix identity. From wikipedia: $$(A + UCV)^{-1} = A^{-1} - A^{-1}U(C^{-1} + VA^{-1}U)^{-1}VA^{-1}$$ Identifying $...

View answer
How to adapt a linear time Newton-Raphson numerical method for an optimisation problem with positivity constraints?
1 votes

The class of methods devised to solve this generic problem are known in the optimisation literature as projected Newton methods. The following is extracted from Bertsekas and Gafni (1983), with some ...

View answer
Derivation of Hessian for multinomial logistic regression in Böhning (1992)
Accepted answer
2 votes

The source of the issue in my view comes from a confusion concerning dimensionality, and because the Hessian departs from the usual context in that there is sub-partitioning going on. Other than a ...

View answer
Gradient of Log Normalizing Constant - Does it have a name and do we know of any properties?
Accepted answer
1 votes

This answer assumes that you are interested in computing $\nabla_{\theta} \log Z$ rather than $\nabla_{x} \log Z$. Without foreclosing the possibility that there may exist other results relevant to ...

View answer
Relationship between variance of gradient and SGD convergence
Accepted answer
1 votes

Only in response to: I'm looking for a reference that talks about the relationship between the variance of the gradient and convergence of SGD. Whilst the Robbins-Monro stochastic approximation ...

View answer
Confusion about maximum likelihood estimation notation
Accepted answer
4 votes

I'm trying to learn some machine learning theory, in particular maximum likelihood estimation. At risk of being pedantic, but for the avoidance of doubt, I have found it useful to be clear on which ...

View answer
Computing the Bayes estimator under weighted squared error loss - interchanging derivatives and integrals
Accepted answer
0 votes

Solution. Using the results in here linked by StubbornAtom. The loss function you have specified can be interpreted as a generalisation of squared error loss, known as weighted squared error loss. ...

View answer
Latent Dirichlet Allocation and topic distributions
Accepted answer
1 votes

I think it would be helpful in this instance to draw a distinction between: 1. How the hyperparametrisation encodes prior beliefs about the topic vectors $\phi_k$ sampled from the Dirichlet prior $p(\...

View answer
Kelly Criterion for game with payoff equal to normal distribution
1 votes

Some references. To supplement the comments of @Dave Harris, here are a cluster of references you might consider using to start further formalising what you've done already. Depending on your ...

View answer
Showing that $f_\varphi(x)$ is a member of the one-parameter exponential family and $\sum_{i = 1}^n - \log(X_i)$ is sufficient for $\varphi$
0 votes

Here is my attempt. 1.Show that $f(x; \varphi)$ is in the one-parameter exponential family. Assume that $\varphi > 0$. Rewriting $f(x; \varphi)$, we have that \begin{align*} f(x; \varphi) &= \...

View answer
Why does this theorem for minimal sufficient from the "All of Statistics" textbook by Wasserman have these exponents of $n$?
Accepted answer
2 votes

In that book, $n$ random variables $X_1, \dots, X_n$, constituting the "data set", are represented as $X^n$. Whereas realisations of those $n$ random variables $x_1, x_2, \dots , x_n$ are ...

View answer
How do we conclude that a statistic is sufficient but not minimal sufficient?
Accepted answer
4 votes

Proof strategy. Show that $T = \sum Y^2_i$ is minimal sufficient. Show that $U = (\sum Y_i, \sum Y^2_i)$ is sufficient. Use the following theorem to show that $T = g(U)$ for some function $g$. ...

View answer
Likelihood $L(\theta; \mathbf{y})$: Is $\theta$ a vector of parameters or is it a single parameter?
Accepted answer
0 votes

It depends on the nature of the assumptions on your $n$ random variables $Y_1, ..., Y_n$, which in turn will influence the kind of parametric statistical model that is specified. Here is a coin ...

View answer
Is my calculation for the MLE correct? How do I check whether it's biased?
Accepted answer
1 votes

In response to: So how do I find $E(x^2_i)$? Assuming there are no issues with the steps in the derivation up to that point, you can use the following standard result relating expectation, variance ...

View answer
Relationship between Bayes Rule and Bayesian Networks
0 votes

In response to I did not fully understand why we are multiplying the intermediate conditionals i.e.: $...Pr(Burglar|Storm=T)xPr(Cat|Storm=T)...$ and Is there a logical explanation for the ...

View answer
How does one define the sum of N random variables in Python?
Accepted answer
2 votes

The scipy-stats package is useful if you are using Python. Here is a code-snippet to get you started - it generates one realisation of the random variable $Y$ using $n = 50$ iid copies of $X$. Have ...

View answer
Meaning of θj in equation for partial derivative of MSE
Accepted answer
0 votes

In response to question and also following comment: Sorry, but could I ask a small follow-up question? If θj = 5 then should the jth element in θ also be 5? E.g - If on the left hand side θj= θ3 = 5 ...

View answer
Meaning of a varaible for calculating the partial derviative of MSE cost function
Accepted answer
1 votes

From the context you have provided, my reading is that $x^{(i)}_j$ is the $j$-th element of the $i$-th input vector $\mathbf{x}^{(i)}$, where there are $i = 1,..., m$ training instances. Addressing ...

View answer