microhaus
  • Member for 1 year, 5 months
  • Last seen this week
How do you go from paper to code?
8 votes

To supplement Dikran Marsupial's answer, the following is an articulation of a process I use personally. This is from the perspective of coding machine learning algorithms for research, rather than ...

View answer
SGD for Gaussian Process estimation
8 votes

This conference paper from NeurIPs 2020 may contain what you are looking for - it contains some theoretical guarantees on using mini-batch stochastic gradient descent in context of Gaussian processes.

View answer
Gradient of the log likelihood for energy based models
Accepted answer
6 votes

The issue emerges in the evaluation of the second term in line $(3)$ and $(4)$ of your derivation. Note that $$\nabla_{\theta} Z(\theta)^{-1} = \nabla_{\theta} \frac{1}{\int_x \exp(-E_{\theta}(x))\, ...

View answer
Why is $k = \sqrt{N}$ a good solution of the number of neighbors to consider?
Accepted answer
6 votes

There are a number of quantitative finite-sample results, and also asymptotic arguments, in support of using the heuristic $k = \sqrt{n}$, where $n$ is the sample size. However, in practice, it would ...

View answer
Subscript notation in expectations (variational autoencoder)
Accepted answer
6 votes

It means expectation with respect to $q_{\phi}(\mathbf{z} | \mathbf{x}^{(i)})$. So: $$\mathbb{E}_{q_{\phi}(\mathbf{z} | \mathbf{x}^{(i)})}[\log p_{\theta}(\mathbf{x}^{(i)} | \mathbf{z})] = \int_{\...

View answer
Deriving posterior update equation in a Variational Bayes inference
Accepted answer
5 votes

Joint distribution. Using the graphical model you provided, we get the following joint distribution over all variables of interest, conditioning on model parameters. $$p(\Theta, \mathbf{v} | a_0, b_0, ...

View answer
SVM loss function
Accepted answer
5 votes

There are some minor issues with your understanding, and I think it would help to clarify exactly what is being maximised, together with what is not being maximised. Loss function. I am going through ...

View answer
Confusion about maximum likelihood estimation notation
Accepted answer
4 votes

I'm trying to learn some machine learning theory, in particular maximum likelihood estimation. At risk of being pedantic, but for the avoidance of doubt, I have found it useful to be clear on which ...

View answer
How do we conclude that a statistic is sufficient but not minimal sufficient?
Accepted answer
4 votes

Proof strategy. Show that $T = \sum Y^2_i$ is minimal sufficient. Show that $U = (\sum Y_i, \sum Y^2_i)$ is sufficient. Use the following theorem to show that $T = g(U)$ for some function $g$. ...

View answer
confused about unbiasedness of sample mean
3 votes

Suppose one has a sample of $n$ chi-squared random variables. Then $\mathbb{E}[X_i]=1$. So, $\mathbb{E}[\overline{X}_n] = \frac{n}{n} = 1$ Now the sample has a chi-squared($n$) distribution so it's ...

View answer
Clarifying the applications of Bayesian networks
Accepted answer
3 votes

You are correct in stating that a Bayesian network allows us to answer probability queries. However, the way you have framed the question suggests as if somehow you expected them to provide you with ...

View answer
what's the difference between hypothesis function outputed by algorithm A and polynomial function in the book,"foundations of machine learning"
Accepted answer
3 votes

To be clear, the learning algorithm $\mathcal{A}$ refers to the procedure of selecting a hypothesis $h$ from the restricted hypothesis class $\mathcal{H}$, so if you are selecting $h$ by say ...

View answer
Overview of the linear model $Y = X \cdot \beta + \varepsilon$
3 votes

Here is why further prompts on your question have been made by commenters. It is to avoid the following vague answer, which I suspect is not what you want: Which is the 'known data', $X$ or $\beta$? ...

View answer
Portfolio theory: confusion about variance-covariance matrix
Accepted answer
3 votes

I think what is going on here is the matrix-vector product $\Sigma^{-1} \mathbf{1}$ and quadratic form $\mathbf{1}^T\Sigma^{-1} \mathbf{1}$ are being used as a convenient way of specifying summation. ...

View answer
About the derivation of EM for mixture of Gaussians
Accepted answer
3 votes

What I understand Andrew Ng is saying is the following: We want to derive estimates of the parameters by solving for $\phi, \mu, \Sigma$. We do this by maximising $l(\phi, \mu, \Sigma)$ i.e. the log ...

View answer
Operator notation - numerically stable forward recursions in linear-chain conditional random fields (CRFs)
Accepted answer
3 votes

The correct expansion of $\oplus_{i \in S}(\log \Psi_t(j, i, x_t) + \log \alpha_{t-1}(i))$ you are looking for is $(A)$. Taking the logs of the forward recursion in equation (4.6), we have: $$\begin{...

View answer
Relationship between L1 penalty and margin in SVM
2 votes

Addressing only the following, as an extended comment: The documentation claims (and it is in line with what we know about LASSO) that $l_1$ regularisation leads to a sparse vector of coefficients. ...

View answer
Conditional Expectation : How is E[E[xy|x]]=E[xE[y|x]]?
2 votes

To supplement the other answers by Taylor and mhdadk, I find it has never led me astray to have absolute clarity on the probability distributions with respect to which one is computing expectations. ...

View answer
Derivation of Hessian for multinomial logistic regression in Böhning (1992)
Accepted answer
2 votes

The source of the issue in my view comes from a confusion concerning dimensionality, and because the Hessian departs from the usual context in that there is sub-partitioning going on. Other than a ...

View answer
Why does this theorem for minimal sufficient from the "All of Statistics" textbook by Wasserman have these exponents of $n$?
Accepted answer
2 votes

In that book, $n$ random variables $X_1, \dots, X_n$, constituting the "data set", are represented as $X^n$. Whereas realisations of those $n$ random variables $x_1, x_2, \dots , x_n$ are ...

View answer
How does one define the sum of N random variables in Python?
Accepted answer
2 votes

The scipy-stats package is useful if you are using Python. Here is a code-snippet to get you started - it generates one realisation of the random variable $Y$ using $n = 50$ iid copies of $X$. Have ...

View answer
How to eliminate graph cycles?
Accepted answer
2 votes

At the most basic level, here is what the issue is when you have a DAG with a cycle. Let $X$ and $Y$ be random variables, that is, nodes in a DAG. Case 1. Consider the factorised joint distribution ...

View answer
Comparison of Bayesian and Classical estimates
2 votes

From what I understand, your query concerns whether it is "correct" to compare the Bayes estimators, belonging to a "Bayesian paradigm", with other frequentist estimators such as ...

View answer
What is Type II maximum likelihood?
2 votes

Empirical Bayes is a means of using the observed data to compute point estimates of the hyperparameters parametrising your priors. Which only makes sense in context of a hierarchical Bayesian model, ...

View answer
How do I check my own Baum Welch Algorithm
2 votes

A surprisingly powerful check on whether there is an issue with your mathematical derivation of the Baum-Welch algorithm; or whether there is a bug in your implementation, is whether the log-...

View answer
Understanding the set of latent variables $Z$ in variational inference
Accepted answer
2 votes

Your question raises some important points which I also struggled with during self-study. I am by no means an expert, but I will attempt to clarify the various usages of the term "parameter" ...

View answer
Interpretation of $\mathcal{D}$ and difference between the accuracy parameter and training error
Accepted answer
1 votes

In my view, the assumption of a deterministic relationship between labels and inputs made in chapter 2 of that book can be interpreted, but it's a bit artificial because it's not probabilistic machine ...

View answer
Why does the exponential moving average equation divide with 1+(1-⍺)+...?
1 votes

The right hand side of the equality you have quoted is a geometric series. Identifying the first term $a=1$ and the common ratio $r=1-\alpha$, the series converges to $a/(1-r) = 1 / \alpha$ if and ...

View answer
What does "Expectation with respect to true unknown parameter" mean?
Accepted answer
1 votes

In the context of asymptotic results involving the maximum likelihood estimator, the use of the subscript $\theta_0$ in the expectations operator means the following, $$\mathbb{E}_{\theta_0}[l(\theta; ...

View answer
Looking to identify book by Michael I. Jordan from excerpts
Accepted answer
1 votes

The chapters of the monograph that are being listed are from a draft book by Michael Jordan that to the best of my knowledge was never published, but continues to be used in many probabilistic ...

View answer