Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [machine-learning]

Machine learning algorithms build a model of the training data. The term "machine learning" is vaguely defined; it includes what is also called statistical learning, reinforcement learning, unsupervised learning, etc. ALWAYS ADD A MORE SPECIFIC TAG.

Filter by
Sorted by
Tagged with
95
votes
4answers
68k views

How to select kernel for SVM?

When using SVM, we need to select a kernel. I wonder how to select a kernel. Any criteria on kernel selection?
45
votes
2answers
62k views

Linear kernel and non-linear kernel for support vector machine?

When using support vector machine, are there any guidelines on choosing linear kernel vs. nonlinear kernel, like RBF? I once heard that non-linear kernel tends not to perform well once the number of ...
37
votes
3answers
3k views

Variance of $K$-fold cross-validation estimates as $f(K)$: what is the role of “stability”?

TL,DR: It appears that, contrary to oft-repeated advice, leave-one-out cross validation (LOO-CV) -- that is, $K$-fold CV with $K$ (the number of folds) equal to $N$ (the number of training ...
25
votes
1answer
6k views

Is there any algorithm combining classification and regression?

I'm wondering if there's any algorithm could do classification and regression at the same time. For example, I'd like to let the algorithm learn a classifier, and at the same time within each label, ...
27
votes
3answers
12k views

What are the impacts of choosing different loss functions in classification to approximate 0-1 loss

We know that some objective functions are easier to optimize and some are hard. And there are many loss functions that we want to use but hard to use, for example 0-1 loss. So we find some proxy loss ...
34
votes
5answers
22k views

What exactly is a Bayesian model?

Can I call a model wherein Bayes' Theorem is used a "Bayesian model"? I am afraid such a definition might be too broad. So what exactly is a Bayesian model?
15
votes
1answer
8k views

How could stochastic gradient descent save time comparing to standard gradient descent?

Standard Gradient Descent would compute gradient for the entire training dataset. ...
9
votes
2answers
7k views

How does a uniform prior lead to the same estimates from maximum likelihood and mode of posterior?

I am studying different point estimate methods and read that when using MAP vs ML estimates, when we use a "uniform prior", the estimates are identical. Can somebody explain what a "uniform" prior is ...
159
votes
3answers
65k views

ROC vs precision-and-recall curves

I understand the formal differences between them, what I want to know is when it is more relevant to use one vs. the other. Do they always provide complementary insight about the performance of a ...
67
votes
3answers
45k views

Proper way of using recurrent neural network for time series analysis

Recurrent neural networks differ from "regular" ones by the fact that they have a "memory" layer. Due to this layer, recurrent NN's are supposed to be useful in time series modelling. However, I'm not ...
61
votes
2answers
32k views

Why only three partitions? (training, validation, test)

When you are trying to fit models to a large dataset, the common advice is to partition the data into three parts: the training, validation, and test dataset. This is because the models usually have ...
53
votes
5answers
14k views

On the importance of the i.i.d. assumption in statistical learning

In statistical learning, implicitly or explicitly, one always assumes that the training set $\mathcal{D} = \{ \bf {X}, \bf{y} \}$ is composed of $N$ input/response tuples $({\bf{X}}_i,y_i)$ that are ...
72
votes
8answers
83k views

How to compute precision/recall for multiclass-multilabel classification?

I'm wondering how to calculate precision and recall measures for multiclass multilabel classification, i.e. classification where there are more than two labels, and where each instance can have ...
62
votes
4answers
82k views

Softmax vs Sigmoid function in Logistic classifier?

What decides the choice of function ( Softmax vs Sigmoid ) in a Logistic classifier ? Suppose there are 4 output classes . Each of the above function gives the probabilities of each class being the ...
60
votes
8answers
11k views

How can I help ensure testing data does not leak into training data?

Suppose we have someone building a predictive model, but that someone is not necessarily well-versed in proper statistical or machine learning principles. Maybe we are helping that person as they are ...
31
votes
2answers
42k views

Relative importance of a set of predictors in a random forests classification in R

I'd like to determine the relative importance of sets of variables toward a randomForest classification model in R. The ...
14
votes
5answers
11k views

Kernel SVM: I want an intuitive understanding of mapping to a higher-dimensional feature space, and how this makes linear separation possible

I am trying to understand the intuition behind kernel SVM's. Now, I understand how linear SVM's work, whereby a decision line is made which splits the data as best it can. I also understand the ...
16
votes
1answer
1k views

How to build the final model and tune probability threshold after nested cross-validation?

Firstly, apologies for posting a question that has already been discussed at length here, here, here, here, here, and for reheating an old topic. I know @DikranMarsupial has written about this topic ...
245
votes
7answers
176k views

Bagging, boosting and stacking in machine learning

What's the similarities and differences between these 3 methods: Bagging, Boosting, Stacking? Which is the best one? And why? Can you give me an example for each?
43
votes
6answers
24k views

Why is multicollinearity not checked in modern statistics/machine learning

In traditional statistics, while building a model, we check for multicollinearity using methods such as estimates of the variance inflation factor (VIF), but in machine learning, we instead use ...
72
votes
4answers
38k views

Why are neural networks becoming deeper, but not wider?

In recent years, convolutional neural networks (or perhaps deep neural networks in general) have become deeper and deeper, with state-of-the-art networks going from 7 layers (AlexNet) to 1000 layers (...
71
votes
2answers
42k views

Solving for regression parameters in closed-form vs gradient descent

In Andrew Ng's machine learning course, he introduces linear regression and logistic regression, and shows how to fit the model parameters using gradient descent and Newton's method. I know gradient ...
46
votes
3answers
9k views

Understanding Naive Bayes

From StatSoft, Inc. (2013), Electronic Statistics Textbook, "Naive Bayes Classifier": To demonstrate the concept of Naïve Bayes Classification, consider the example displayed in the ...
38
votes
7answers
79k views

Data normalization and standardization in neural networks

I am trying to predict the outcome of a complex system using neural networks (ANN's). The outcome (dependent) values range between 0 and 10,000. The different input variables have different ranges. ...
35
votes
5answers
24k views

Cost function of neural network is non-convex?

The cost function of neural network is $J(W,b)$, and it is claimed to be non-convex. I don't quite understand why it's that way, since as I see that it's quite similar to the cost function of logistic ...
16
votes
1answer
2k views

Which Theories of Causality Should I know?

Which theoretical approaches to causality should I know as an applied statistician/econometrician? I know the (a very little bit) Neyman–Rubin causal model (and Roy, Haavelmo etc.) Pearl's Work on ...
21
votes
3answers
6k views

What happens when you apply SVD to a collaborative filtering problem? What is the difference between the two?

In Collaborative filtering, we have values that are not filled in. Suppose a user did not watch a movie then we have to put an 'na' in there. If I am going to take an SVD of this matrix, then I have ...
18
votes
1answer
12k views

Opinions about Oversampling in general, and the SMOTE algorithm in particular [closed]

What is your opinion about oversampling in classification in general, and the SMOTE algorithm in particular? Why would we not just apply a cost/penalty to adjust for imbalance in class data and any ...
24
votes
2answers
6k views

Intuition behind logistic regression

Recently I began studying machine learning, however I failed to grasp the intuition behind logistic regression. The following are the facts about logistic regression that I understand. As the basis ...
21
votes
2answers
9k views

Comparing clusterings: Rand Index vs Variation of Information

I was wondering if anybody had any insight or intuition behind the difference between the Variation of Information and the Rand Index for comparing clusterings. I have read the paper "Comparing ...
21
votes
2answers
3k views

Do all machine learning algorithms separate data linearly?

I am an enthusiast of programming and machine learning. Only a few months back I started learning about machine learning programming. Like many who don't have a quantitative science background I also ...
9
votes
1answer
4k views

How does linear base learner works in boosting? And how does it works in the xgboost library?

I know how to implement linear objective function and linear boosts in XGBoost. My concrete question is: when the algorithm it fits the residual (or the negative gradient) is it using one feature at ...
4
votes
1answer
4k views

How to calculate decision boundary from support vectors?

I want to obtain decision boundary of SVM using OpenCV 2.4.11, but it seems that it's not returning it explicitly, but only support vectors. How we can calculate decision boundary from support ...
5
votes
2answers
424 views

How to predict the next number in a series while having additional series of data that might affect it?

Let's say we want to predict the price of Big Mac for the year 2020. We have 2 indexes that we think might make an influence to Big Mac price determination. ...
154
votes
2answers
65k views

Generative vs. discriminative

I know that generative means "based on $P(x,y)$" and discriminative means "based on $P(y|x)$," but I'm confused on several points: Wikipedia (+ many other hits on the web) classify things like SVMs ...
142
votes
6answers
124k views

What are the advantages of ReLU over sigmoid function in deep neural networks?

The state of the art of non-linearity is to use rectified linear units (ReLU) instead of sigmoid function in deep neural network. What are the advantages? I know that training a network when ReLU is ...
92
votes
3answers
131k views

How do you calculate precision and recall for multiclass classification using confusion matrix?

I wonder how to compute precision and recall using a confusion matrix for a multi-class classification problem. Specifically, an observation can only be assigned to its most probable class / label. I ...
26
votes
2answers
11k views

Variance estimates in k-fold cross-validation

K-fold cross-validation can be used to estimate the generalization capability of a given classifier. Can I (or should I) also compute a pooled variance from all validation runs in order to obtain a ...
34
votes
5answers
4k views

Can you overfit by training machine learning algorithms using CV/Bootstrap?

This question may well be too open-ended to get a definitive answer, but hopefully not. Machine learning algorithms, such as SVM, GBM, Random Forest etc, generally have some free parameters that, ...
24
votes
1answer
3k views

Computation of the marginal likelihood from MCMC samples

This is a recurring question (see this post, this post and this post), but I have a different spin. Suppose I have a bunch of samples from a generic MCMC sampler. For each sample $\theta$, I know the ...
14
votes
2answers
9k views

What measure of training error to report for Random Forests?

I'm currently fitting random forests for a classification problem using the randomForest package in R, and am unsure about how to report training error for these ...
11
votes
2answers
2k views

Compare classifiers based on AUROC or accuracy?

I have a binary classification problem and I experiment different classifiers on it: I want to compare the classifiers. which one is a better measure AUC or accuracy? And why? ...
9
votes
2answers
12k views

Why does PCA maximize total variance of the projection?

Christopher Bishop writes in his book Pattern Recognition and Machine Learning a proof, that each consecutive principal component maximizes the variance of the projection to one dimension, after the ...
28
votes
3answers
9k views

In boosting, why are the learners “weak”?

See also a similar question on stats.SE. In boosting algorithms such as AdaBoost and LPBoost it is known that the "weak" learners to be combined only have to perform better than chance to be useful, ...
18
votes
3answers
18k views

Why does gap statistic for k-means suggest one cluster, even though there are obviously two of them?

I am using K-means to cluster my data and was looking for a way to suggest an "optimal" cluster number. Gap statistics seems to be a common way to find a good cluster number. For some reason it ...
10
votes
1answer
505 views

What justifies this calculation of the derivative of a matrix function?

In Andrew Ng's machine learning course, he uses this formula: $\nabla_A tr(ABA^TC) = CAB + C^TAB^T$ and he does a quick proof which is shown below: $\nabla_A tr(ABA^TC) \\ = \nabla_A tr(f(A)A^TC) \\...
10
votes
0answers
1k views

Machine learning self-learning book? [duplicate]

Possible Duplicate: Machine learning cookbook / reference card / cheatsheet? I wonder if there is a good self-learning textbook for machine learning? I am particularly looking for those in ...
7
votes
1answer
5k views

What is probabilistic inference?

I am reading Chris Bishop's Pattern Recognition and Machine Learning textbook. I came across the term probabilistic inference several times. I have a couple of questions. Is probabilistic inference ...
12
votes
1answer
977 views

What is the meaning of the axes in t-SNE?

I'm currently trying to wrap my head around the t-SNE math. Unfortunately, there is still one question I can't answer satisfactorily: What is the actual meaning of the axes in a t-SNE graph? If I were ...
5
votes
2answers
533 views

Graphical interpretation of LASSO

I've a question regarding to the graphical intuition of the LASSO. I'm understanding why the lasso produce zero coefficient in case of intersecting a corner of the diamond. But I don't understand the ...