8
votes
Stochastic gradient descent: why randomise training set
Generally, in case your data is ordered (see e.g. Mnist data set) SGD will have problems. Also, in case you run through it multiple times (so called epoches) having the same order on each run through ...
- 6,724
7
votes
Accepted
Root-finding via Robbins-Monro method: A real and simple example
It seems to be performing just fine.
Let's do a small study. The plots below document 10,000 steps of the process of finding the root $\theta$ for four values of $b$ (indicated in their titles) and ...
- 334k
2
votes
Difference between Stochastic Approximation (SA) and Stochastic Gradient Descent (SGD)
According to (Kushner & Yin, sec. 1.1.3), stochastic approximation and stochastic gradient descent are the same thing (emphasis mine):
...stochastic approximation form of linearized least squares:...
- 342
2
votes
Accepted
Variational inference with discrete variational parameters
Well, in general this is an instance of a discrete optimization problem, and in general there are no methods more efficient than brute-force search over all possible values of these parameters. In ...
- 3,029
2
votes
Accepted
Variational Inference with intractable score function
The only reason Variational Inference (VI) requires knowing the density $q(z|\lambda)$ is if you seek to maximize the ELBO w.r.t. $\lambda$, which is equivalent to minimizing $\text{KL}(q(z|\lambda) \...
- 3,029
2
votes
Root-finding via Robbins-Monro method: A real and simple example
To add to @whuber's excellent answer, and to further address OP's question "How to make the RM perform better?", the condition presented where b=0.25 or ...
- 3,073
1
vote
Question about the Robbins-Monro algorithm for sequential maximum likelihood estimation
I think I found the source of my confusion. In this setting, the distribution over $x$ is unrelated to the model distribution $p(x\mid \theta)$, which does not even have to be of the same type as the ...
- 111
1
vote
How to model spatial data using SPDE and finite element method
What you're trying to do has been published last year in this paper (10.1016/j.cma.2020.113533 "The statistical finite element method (statFEM) for coherent synthesis of observation data and ...
1
vote
Stochastic Models (probability)- simple symmetric random walk question
The final position is $X_n=\sum_{i=1}^n X_i$, with mean $0$ and variance $n$ (and approximately normal). So, $$P(-3\sqrt{n}<X_n<2\sqrt{n})=P(-3<Z<2)=\Phi(2)-\Phi(-3)$$
where you can find $\...
- 58.2k
1
vote
Variational inference with discrete variational parameters
It is correct that typical variational inference methods model the surrogate distribution $q_{\phi}(z \vert x)$ as a continuous latent space. A challenge of the following approach is dealing with ...
- 376
1
vote
Accepted
Is it possible to combine SPSA and Adam?
See Robust and efficient algorithms for high-dimensional black-box quantum optimization.
I'll remark from what I understand that the idea of combining SPSA and Adam doesn't seem very good. The issue ...
1
vote
Best estimate for Stochastic difference equation
You have $x(t_n)+\Delta x(t_n)=x(t_n)+x(t_n)\Delta t+f(t_n)\Delta t$, so that:
$$\frac{\Delta x(t_n)}{\Delta t}=x(t_n)+f(t_n).$$
Usually Weiner processes are defined to have mean 0, so that the left ...
- 14.1k
1
vote
Simulation of Secretary problem: optimal pool size given k=2?
I reran this at a million per row, and left it to go all weekend.
It look a long time, but the interior minimum went away.
It is a feature of the random number generation and not the physics.
- 9,853
1
vote
Accepted
Can you perform stochastic learning followed by batch learning in neural networks?
Yes you can perform stochastic learning followed by batch learning in neural networks. In fact the very same paper discusses it:
Another method to remove noise is to use “mini-batches”, that is, ...
- 47.6k
Only top scored, non community-wiki answers of a minimum length are eligible