PCA is restricted to a linear map, while auto encoders can have nonlinear enoder/decoders. A single layer auto encoder with linear transfer function is nearly equivalent to PCA, where nearly means ...

It is true that preprocessing in machine learning is somewhat a very black art. It is not written down in papers a lot why several preprocessing steps are essential to make it work. I am also not sure ...

What you describe is in fact a "sliding time window" approach and is different to recurrent networks. You can use this technique with any regression algorithm. There is a huge limitation to this ...

A standard approach is to scale the inputs to have mean 0 and a variance of 1. Also linear decorrelation/whitening/pca helps a lot. If you are interested in the tricks of the trade, I can recommend ...

Finding the differences can be done by looking at the models. Let's look at sparse coding first. Sparse coding Sparse coding minimizes the objective $$\mathcal{L}_{\text{sc}} = \underbrace{||WH - ... View answer Accepted answer 32 votes The vanishing gradient is best explained in the one-dimensional case. The multi-dimensional is more complicated but essentially analogous. You can review it in this excellent paper . Assume we ... View answer 27 votes I think the main problem is to get the pairwise distances efficiently. Once you have that the rest is element wise. To do this, you probably want to use scipy. The function scipy.spatial.distance.... View answer Accepted answer 25 votes In this context this refers to the fact that the model does not assume any spatial relationships between the features. E.g. for multilayer perceptron, you can permute the pixels and the performance ... View answer 25 votes Given an Eigendecomposition of a covariance matrix$$ \bar{X}\bar{X}^T = LDL^T $$where D = \text{diag}(\lambda_1, \lambda_2, \dots, \lambda_n) is the diagonal matrix of Eigenvalues, ordinary ... View answer Accepted answer 21 votes The root mean squared error and the likelihood are actually closely related. Say you have a dataset of \lbrace x_i, z_i \rbrace pairs and you want to model their relationship using the model f. ... View answer 19 votes I am using neural networks for most problem. The point is that it's in most cases more about the experience of the user than about the model. Here are some reasons why I like NNs. They are flexible. ... View answer 17 votes What George Dontas writes is correct, however the use of RNNs in practice today is restricted to a simpler class of problems: time series / sequential tasks. While feedforward networks are used to ... View answer Accepted answer 15 votes Deep Learning got a lot of focus since 2006. It's basically an approach to train deep neural networks and is leading to really impressive results on very hard datasets (like document clustering or ... View answer 15 votes The easiest way is to sample points uniformly in the corresponding hypercube and discard those that do not lie within the sphere. In 3D, this should not happen that often, about 50% of the time. (... View answer 13 votes I always have the feeling that any hyper parameter selection for SVMs is done via cross validation in combination with grid search. View answer Accepted answer 12 votes Neural networks are sometimes called "differentiable function approximators". So what you can do is to differentiate any unit with respect to any other unit to see what their relationshsip is. You ... View answer 11 votes First, mind that deep learning is a buzz term. There is not even a consensus of a formal definition in the research community. A discussion of the term does not lead anywhere, really. It's just a ... View answer 11 votes In NLP, where words are typically encoded as 1-of-k, the use of word embeddings has emerged recently. The wikipedia page with its references is a good start. The general idea is to learn a vectorial ... View answer 11 votes Adaboost can use multiple instances of the same classifier with different parameters. Thus, a previously linear classifier can be combined into nonlinear classifiers. Or, as the AdaBoost people like ... View answer Accepted answer 10 votes A mixture of Gaussians is defined as a linear combination of multiple Gaussian distributions. Thus it has multiple modes. The dimension refers to the data (e.g. the color, length, width, height and ... View answer Accepted answer 9 votes What about optimization? Let's see if I understand you correctly. You have a model p(y|x, \theta) conditioned on some observation x and a set of parameters \theta and a prior p(\theta) leading ... View answer 9 votes Python has a wide range of ML libraries (check out mloss.org as well). However, I always have the feeling that it's more of use for ml researchers than for ml practitioners. Numpy/SciPy and ... View answer 8 votes Slow feature analysis (SFA) uses the smalles Eigenvalues of the covariance matrix of temporal differences to find the slowest features in a time series, Minor component analysis (MCA) uses the ... View answer Accepted answer 8 votes It depends heavily on the algorithm. There are several things for which writing code in C won't give you any benefit: matrix operations (dot products, element wise multiplications/applications of ... View answer Accepted answer 8 votes It seems as if deep learning might be very interesting for you. This is a very recent field of deep connectionist models which are pretrained in an unsupervised way and fine tuned afterwards with ... View answer 8 votes There is two methods that come to my mind which you might be interested in. The first is making use of known clusters and is called 'Neighbourhood components analysis' by Goldberger et al . The idea ... View answer Accepted answer 7 votes Machine learning techniques often lack interpretability. Also, they tend to be rather crude from a statistical point of view --- e.g. neural networks make no assumptions about the input data. I have ... View answer 7 votes Here I assume that you can only sample from the models; an unnormalized density function is not available. You write that$$D_{KL}(f || g) = \int_{-\infty}^{\infty} f(x) \log\left(\underbrace{\...

It is not surprising that weight decay will hurt performance of your neural network at some point. Let the prediction loss of your net be $\mathcal{L}$ and the weight decay loss $\mathcal{R}$. Given a ...