Questions tagged [machine-learning]

Machine learning algorithms build a model of the training data. The term "machine learning" is vaguely defined; it includes what is also called statistical learning, reinforcement learning, unsupervised learning, etc. ALWAYS ADD A MORE SPECIFIC TAG.

2
votes
1answer
538 views

Gaussian is conjugate of Gaussian?

Someone told me that, Gaussian distribution is conjugate to distribution because a Gaussian times a Gaussian would still be Gaussian distribution ? Why is that ? Say the following situation: $X\sim N(...
1
vote
1answer
2k views

Better accuracy with validation set than test set

I trained a model with some algorithms like random forest, logistic regression and so on. My dataset was split into 80% CV train data (so actually 60% of data to train the model and 20 % for testing ...
8
votes
1answer
2k views

CART: Selection of best predictor for splitting when gains in impurity decrease are equal?

My question deals with Classification trees. Consider the following example from the Iris data set: I want to manually select the best predictor for the first split. According to the CART algorithm, ...
8
votes
3answers
1k views

How to make the randomforest trees vote decimals but not binary

My question is about binary classification, say separating good customers from bad customers, but not regression or non-binary classification. In this context, a random forest is an ensemble of ...
7
votes
1answer
5k views

How to Identify Overfitting in Convolutional Neural network?

I understand that dropout is used to reduce over fitting in the network. This is a generalization technique. In convolutional neural network how can I identify overfitting? One situation that I can ...
3
votes
2answers
1k views

Trouble applying hidden Markov models

Edit: I updated the question to hopefully make it more easy to understand. I think it was overly complex. I’m having a problem applying hidden Markov models to a game I’m building to learn about ...
3
votes
1answer
997 views

Ranking two models based on ROC-AUC and PR-AUC

I have two methods/classifiers (completely different models) that I need to decide which one is better. The dataset is imbalanced. I trained both classifiers on the same dataset and then I computed ...
1
vote
1answer
407 views

Does changing the parameter search space after nested CV introduce optimistic bias?

Suppose I am fitting a Ridge and I decide to search a parameter space for c:[1,2,3]. I perform nested CV on my whole dataset and find the performance not so great. I therefore expand my search space ...
1
vote
1answer
402 views

Graphical path Coordinate Descent in case of semi-differentiable functions such as Lasso

I am trying to understand how the graphical solution path to the optimum would look in the case of Lasso Regression. I can find only Pictures for the differentiable or non differentiable case. The ...
186
votes
4answers
161k views

What does the hidden layer in a neural network compute?

I'm sure many people will respond with links to 'let me google that for you', so I want to say that I've tried to figure this out so please forgive my lack of understanding here, but I cannot figure ...
86
votes
2answers
63k views

What is an embedding layer in a neural network?

In many neural network libraries, there are 'embedding layers', like in Keras or Lasagne. I am not sure I understand its function, despite reading the documentation. For example, in the Keras ...
40
votes
4answers
74k views

Recall and precision in classification

I read some definitions of recall and precision, though it is every time in the context of information retrieval. I was wondering if someone could explain this a bit more in a classification context ...
47
votes
7answers
4k views

Where to start with statistics for an experienced developer

During the first half of 2015 I did the coursera course of Machine Learning (by Andrew Ng, GREAT course). And learned the basics of machine learning (linear regression, logistic regression, SVM, ...
45
votes
3answers
43k views

Clustering with K-Means and EM: how are they related?

I have studied algorithms for clustering data (unsupervised learning): EM, and k-means. I keep reading the following : k-means is a variant of EM, with the assumptions that clusters are ...
45
votes
2answers
34k views

Recurrent vs Recursive Neural Networks: Which is better for NLP?

There are Recurrent Neural Networks and Recursive Neural Networks. Both are usually denoted by the same acronym: RNN. According to Wikipedia, Recurrent NN are in fact Recursive NN, but I don't really ...
44
votes
4answers
23k views

How are kernels applied to feature maps to produce other feature maps?

I am trying to understand the convolution part of convolutional neural networks. Looking at the following figure: I have no problems understanding the first convolution layer where we have 4 ...
52
votes
4answers
23k views

What're the differences between PCA and autoencoder?

Both PCA and autoencoder can do demension reduction, so what are the difference between them? In what situation I should use one over another?
30
votes
4answers
11k views

Optimising for Precision-Recall curves under class imbalance

I have a classification task where I have a number of predictors (one of which is the most informative), and I am using the MARS model to construct my classifier (I am interested in any simple model, ...
23
votes
3answers
3k views

Sites for predictive modeling competitions

I participate in predictive modeling competitions on Kaggle, TunedIt, and CrowdAnalytix. I find that these sites are a good way to "work-out" for statistics/machine learning. Are there any other ...
24
votes
4answers
28k views

Number of features vs. number of observations

Are there any papers/books/ideas about the relationship between the number of features and the number of observations one needs to have to train a "robust" classifier? For example, assume I have 1000 ...
33
votes
3answers
15k views

Pre-training in deep convolutional neural network?

Have anyone seen any literature on pre-training in deep convolutional neural network? I have only seen unsupervised pre-training in autoencoder or restricted boltzman machines.
24
votes
3answers
26k views

Neural network with skip-layer connections

I am interested in regression with neural networks. Neural networks with zero hidden nodes + skip-layer connections are linear models. What about the same neural nets but with hidden nodes ? I am ...
23
votes
6answers
6k views

Alternatives to classification trees, with better predictive (e.g: CV) performance?

I am looking for an alternative to Classification Trees which might yield better predictive power. The data I am dealing with has factors for both the explanatory and the explained variables. I ...
17
votes
2answers
19k views

What are the differences between the Baum-Welch algorithm and Viterbi training?

I am currently using Viterbi training for an image segmentation problem. I wanted to know what the advantages/disadvantages are of using the Baum-Welch algorithm instead of Viterbi training.
17
votes
2answers
3k views

Backpropagation algorithm

I got a slight confusion on the backpropagation algorithm used in multilayer perceptron (MLP). The error is adjusted by the cost function. In backpropagation, we are trying to adjust the weight of ...
27
votes
4answers
18k views

When should I balance classes in a training data set?

I had an online course, where I learned, that unbalanced classes in the training data might lead to problems, because classification algorithms go for the majority rule, as it gives good results if ...
26
votes
2answers
6k views

Why is mean squared error the cross-entropy between the empirical distribution and a Gaussian model?

In 5.5, Deep Learning (by Ian Goodfellow, Yoshua Bengio and Aaron Courville), it states that Any loss consisting of a negative log-likelihood is a cross-entropy between the empirical distribution ...
21
votes
2answers
6k views

When should we discretize/bin continuous independent variables/features and when should not?

When should we discretize/bin independent variables/features and when should not? My attempts to answer the question: In general, we should not bin, because binning will lose information. Binning is ...
19
votes
2answers
15k views

Computing the decision boundary of a linear SVM model

Given the support vectors of a linear SVM, how can I compute the equation of the decision boundary?
22
votes
2answers
10k views

Restricted Boltzmann machines vs multilayer neural networks

I've been wanting to experiment with a neural network for a classification problem that I'm facing. I ran into papers that talk of RBMs. But from what I can understand, they are no different from ...
21
votes
2answers
1k views

How to choose between learning algorithms

I need to implement a program that will classify records into 2 categories (true/false) based on some training data, and I was wondering at which algorithm/methodology I should be looking at. There ...
19
votes
2answers
2k views

When is “Nearest Neighbor” meaningful, today?

In 1999, Beyer et al. asked, When is "Nearest Neighbor" meaningful? Are there better ways of analyzing and visualizing the effect of distance flatness on NN search since 1999? Does [a given] data ...
29
votes
3answers
39k views

Why are bias nodes used in neural networks?

Why are bias nodes used in neural networks? How many you should use? In which layers you should use them: all hidden layers and the output layer?
17
votes
2answers
19k views

Estimating the most important features in a k-means cluster partition

Is there a way to determine which features / variables of the dataset are the most important / dominant within a k-means cluster solution?
15
votes
1answer
5k views

Derivation of change of variables of a probability density function?

In the book pattern recognition and machine learning (formula 1.27), it gives $$p_y(y)=p_x(x) \left | \frac{d x}{d y} \right |=p_x(g(y)) | g'(y) |$$ where $x=g(y)$, $p_x(x)$ is the pdf that ...
14
votes
1answer
5k views

When over/under-sampling unbalanced classes, does maximizing accuracy differ from minimizing misclassification costs?

First of all, I would like to describe some common layouts that Data Mining books use explaining how to deal with Unbalanced Datasets. Usually the main section is named Unbalanced Datasets and they ...
28
votes
2answers
28k views

Is it essential to do normalization for SVM and Random Forest?

My features' every dimension has different range of value. I want to know if it is essential to normalize this dataset.
25
votes
5answers
23k views

What is the difference between convolutional neural networks and deep learning?

I want to use deep learning in my project. I went through a couple of papers and a question occurred to me: is there any difference between convolution neural network and deep learning? Are these ...
22
votes
5answers
15k views

Should you ever standardise binary variables?

I have a data set with a set of features. Some of them are binary $(1=$ active or fired, $0=$ inactive or dormant), and the rest are real valued, e.g. $4564.342$. I want to feed this data to a ...
14
votes
3answers
9k views

Minimum number of layers in a deep neural network

At what point do we start classifying multi layered neural networks as deep neural networks or to put it in another way 'What is the minimum number of layers in a deep neural network?'
24
votes
9answers
6k views

What do statisticians do that can't be automated?

Will software eventually make statisticians obsolete? What is done that can't be programmed into a computer?
19
votes
3answers
14k views

Test for linear separability

Is there a way to test linear separability of a two-class dataset in high dimensions? My feature vectors are 40-long. I know I can always run logistic regression experiments and determine hitrate vs ...
15
votes
1answer
10k views

Clustering: Should I use the Jensen-Shannon Divergence or its square?

I am clustering probability distributions using the Affinity Propagation algorithm, and I plan to use Jensen-Shannon Divergence as my distance metric. Is it correct to use JSD itself as the distance, ...
14
votes
2answers
3k views

Why is optimizing a mixture of Gaussian directly computationally hard?

Consider the log likelihood of a mixture of Gaussians: $$l(S_n; \theta) = \sum^n_{t=1}\log f(x^{(t)}|\theta) = \sum^n_{t=1}\log\left\{\sum^k_{i=1}p_i f(x^{(t)}|\mu^{(i)}, \sigma^2_i)\right\}$$ I was ...
15
votes
1answer
774 views

Thesaurus for statistics and machine learning terms

Does there exist any reference thesaurus for statistics and machine learning terms? I know that Wikipedia articles often contain synonyms, but I would like to have a mere thesaurus that I could go ...
14
votes
3answers
4k views

Advantages of ROC curves

What is the advantages of the ROC curves? For example I am classifying some images which is a binary classification problem. I extracted about 500 features and applied a features selection algorithm ...
12
votes
3answers
3k views

What's the maximum value of Kullback-Leibler (KL) divergence

I am going to use KL divergence in my python code and I got this tutorial. On that tutorial, to implement KL divergence is quite simple. ...
12
votes
1answer
2k views

Understanding no free lunch theorem in Duda et al's Pattern Classification

I have some questions about the notations used in Section 9.2 Lack of Inherent Superiority of Any Classifier in Duda, Hart and Stork's Pattern Classification. First let me quote some relevant text ...
10
votes
2answers
2k views

What is the advantage of reducing dimensionality of predictors for the purposes of regression?

What are the applications or advantages of dimension reduction regression (DRR) or supervised dimensionality reduction (SDR) techniques over traditional regression techniques (without any ...
7
votes
2answers
2k views

Building background for machine learning for CS student

I am a CS graduate student and I am starting to get really interested in Machine Learning (and Predictive Analytics). I have started working on a text classification project with a professor to learn ...