bayerj
  • Member for 11 years
  • Last seen this week
  • Munich
What're the differences between PCA and autoencoder?
Accepted answer
75 votes

PCA is restricted to a linear map, while auto encoders can have nonlinear enoder/decoders. A single layer auto encoder with linear transfer function is nearly equivalent to PCA, where nearly means ...

View answer
How and why do normalization and feature scaling work?
54 votes

It is true that preprocessing in machine learning is somewhat a very black art. It is not written down in papers a lot why several preprocessing steps are essential to make it work. I am also not sure ...

View answer
Proper way of using recurrent neural network for time series analysis
Accepted answer
51 votes

What you describe is in fact a "sliding time window" approach and is different to recurrent networks. You can use this technique with any regression algorithm. There is a huge limitation to this ...

View answer
Data normalization and standardization in neural networks
Accepted answer
44 votes

A standard approach is to scale the inputs to have mean 0 and a variance of 1. Also linear decorrelation/whitening/pca helps a lot. If you are interested in the tricks of the trade, I can recommend ...

View answer
What are the differences between sparse coding and autoencoder?
Accepted answer
36 votes

Finding the differences can be done by looking at the models. Let's look at sparse coding first. Sparse coding Sparse coding minimizes the objective $$ \mathcal{L}_{\text{sc}} = \underbrace{||WH - ...

View answer
How does LSTM prevent the vanishing gradient problem?
Accepted answer
32 votes

The vanishing gradient is best explained in the one-dimensional case. The multi-dimensional is more complicated but essentially analogous. You can review it in this excellent paper [1]. Assume we ...

View answer
How to calculate a Gaussian kernel effectively in numpy
27 votes

I think the main problem is to get the pairwise distances efficiently. Once you have that the rest is element wise. To do this, you probably want to use scipy. The function scipy.spatial.distance....

View answer
What does "permutation invariant" mean in the context of neural networks doing image recognition?
Accepted answer
25 votes

In this context this refers to the fact that the model does not assume any spatial relationships between the features. E.g. for multilayer perceptron, you can permute the pixels and the performance ...

View answer
What is the difference between ZCA whitening and PCA whitening?
25 votes

Given an Eigendecomposition of a covariance matrix $$ \bar{X}\bar{X}^T = LDL^T $$ where $D = \text{diag}(\lambda_1, \lambda_2, \dots, \lambda_n)$ is the diagonal matrix of Eigenvalues, ordinary ...

View answer
Calculating likelihood from RMSE
Accepted answer
21 votes

The root mean squared error and the likelihood are actually closely related. Say you have a dataset of $\lbrace x_i, z_i \rbrace$ pairs and you want to model their relationship using the model $f$. ...

View answer
Neural networks vs support vector machines: are the second definitely superior?
19 votes

I am using neural networks for most problem. The point is that it's in most cases more about the experience of the user than about the model. Here are some reasons why I like NNs. They are flexible. ...

View answer
What's the difference between feed-forward and recurrent neural networks?
17 votes

What George Dontas writes is correct, however the use of RNNs in practice today is restricted to a simpler class of problems: time series / sequential tasks. While feedforward networks are used to ...

View answer
What are the "hot algorithms" for machine learning?
Accepted answer
15 votes

Deep Learning got a lot of focus since 2006. It's basically an approach to train deep neural networks and is leading to really impressive results on very hard datasets (like document clustering or ...

View answer
How to generate uniformly distributed points in the 3-d unit ball?
15 votes

The easiest way is to sample points uniformly in the corresponding hypercube and discard those that do not lie within the sphere. In 3D, this should not happen that often, about 50% of the time. (...

View answer
How to select kernel for SVM?
13 votes

I always have the feeling that any hyper parameter selection for SVMs is done via cross validation in combination with grid search.

View answer
How to visualize/understand what a neural network is doing?
Accepted answer
12 votes

Neural networks are sometimes called "differentiable function approximators". So what you can do is to differentiate any unit with respect to any other unit to see what their relationshsip is. You ...

View answer
Why convolutional neural networks belong to deep learning?
11 votes

First, mind that deep learning is a buzz term. There is not even a consensus of a formal definition in the research community. A discussion of the term does not lead anywhere, really. It's just a ...

View answer
How to recode categorical variable into numerical variable when using SVM or Neural Network
11 votes

In NLP, where words are typically encoded as 1-of-k, the use of word embeddings has emerged recently. The wikipedia page with its references is a good start. The general idea is to learn a vectorial ...

View answer
When would one want to use AdaBoost?
11 votes

Adaboost can use multiple instances of the same classifier with different parameters. Thus, a previously linear classifier can be combined into nonlinear classifiers. Or, as the AdaBoost people like ...

View answer
What's a component in gaussian mixture model?
Accepted answer
10 votes

A mixture of Gaussians is defined as a linear combination of multiple Gaussian distributions. Thus it has multiple modes. The dimension refers to the data (e.g. the color, length, width, height and ...

View answer
Gradient of multivariate Gaussian log-likelihood
Accepted answer
9 votes

What about optimization? Let's see if I understand you correctly. You have a model $p(y|x, \theta)$ conditioned on some observation $x$ and a set of parameters $\theta$ and a prior $p(\theta)$ leading ...

View answer
Machine Learning using Python
9 votes

Python has a wide range of ML libraries (check out mloss.org as well). However, I always have the feeling that it's more of use for ml researchers than for ml practitioners. Numpy/SciPy and ...

View answer
Are there useful applications of SVD that use only the smallest singular values?
8 votes

Slow feature analysis (SFA) uses the smalles Eigenvalues of the covariance matrix of temporal differences to find the slowest features in a time series, Minor component analysis (MCA) uses the ...

View answer
What are speed differences beetwen ML implementations in different languages?
Accepted answer
8 votes

It depends heavily on the algorithm. There are several things for which writing code in C won't give you any benefit: matrix operations (dot products, element wise multiplications/applications of ...

View answer
Semi-supervised learning, active learning and deep learning for classification
Accepted answer
8 votes

It seems as if deep learning might be very interesting for you. This is a very recent field of deep connectionist models which are pretrained in an unsupervised way and fine tuned afterwards with ...

View answer
Dimensionality reduction technique to maximize separation of known clusters?
8 votes

There is two methods that come to my mind which you might be interested in. The first is making use of known clusters and is called 'Neighbourhood components analysis' by Goldberger et al . The idea ...

View answer
Why the infrequent use of machine learning techniques in translational biomedicine?
Accepted answer
7 votes

Machine learning techniques often lack interpretability. Also, they tend to be rather crude from a statistical point of view --- e.g. neural networks make no assumptions about the input data. I have ...

View answer
Estimate the Kullback–Leibler (KL) divergence with Monte Carlo
7 votes

Here I assume that you can only sample from the models; an unnormalized density function is not available. You write that $$D_{KL}(f || g) = \int_{-\infty}^{\infty} f(x) \log\left(\underbrace{\...

View answer
Bayesian recurrent neural network with keras and pymc3/edward
Accepted answer
7 votes

From a pure implementation perspective, it should be straightforward: take your model code, replace every trainable Variable creation with ed.Normal(...) or sth similar, establish variational ...

View answer
Weight Decay in Neural Neural Networks Weight Update and Convergence
Accepted answer
7 votes

It is not surprising that weight decay will hurt performance of your neural network at some point. Let the prediction loss of your net be $\mathcal{L}$ and the weight decay loss $\mathcal{R}$. Given a ...

View answer
1
2 3 4 5 6