Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [machine-learning]

Machine learning algorithms build a model of the training data. The term "machine learning" is vaguely defined; it includes what is also called statistical learning, reinforcement learning, unsupervised learning, etc. ALWAYS ADD A MORE SPECIFIC TAG.

91
votes
7answers
27k views

Why is accuracy not the best measure for assessing classification models?

This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference. ...
195
votes
3answers
18k views

How to know that your machine learning problem is hopeless?

Imagine a standard machine-learning scenario: You are confronted with a large multivariate dataset and you have a pretty blurry understanding of it. What you need to do is to make predictions ...
44
votes
5answers
8k views

When is unbalanced data really a problem in Machine Learning?

We already had multiple questions about unbalanced data when using logistic regression, SVM, decision trees, bagging and a number of other similar questions, what makes it a very popular topic! ...
407
votes
11answers
461k views

What is the difference between test set and validation set?

I found this confusing when I use the neural network toolbox in Matlab. It divided the raw data set into three parts: training set validation set test set I notice in many training or learning ...
68
votes
2answers
15k views

Why isn't Logistic Regression called Logistic Classification?

Since Logistic Regression is a statistical classification model dealing with categorical dependent variables, why isn't it called Logistic Classification? Shouldn't the "Regression" name be reserved ...
349
votes
5answers
107k views

How to understand the drawbacks of K-means

K-means is a widely used method in cluster analysis. In my understanding, this method does NOT require ANY assumptions, i.e., give me a dataset and a pre-specified number of clusters, k, and I just ...
130
votes
5answers
38k views

Training with the full dataset after cross-validation?

Is it always a good idea to train with the full dataset after cross-validation? Put it another way, is it ok to train with all the samples in my dataset and not being able to check if this particular ...
36
votes
4answers
22k views

Classification probability threshold

I have a question regarding classification in general. Let f be a classifier, which outputs a set of probabilities given some data D. Normally, one would say: well, if P(c|D) > 0.5, we will assign a ...
48
votes
3answers
18k views

Can a random forest be used for feature selection in multiple linear regression?

Since RF can handle non-linearity but can't provide coefficients, would it be wise to use random forest to gather the most important features and then plug those features into a multiple linear ...
75
votes
7answers
29k views

Bias and variance in leave-one-out vs K-fold cross validation

How do different cross-validation methods compare in terms of model variance and bias? My question is partly motivated by this thread: Optimal number of folds in $K$-fold cross-validation: is leave-...
414
votes
20answers
144k views

The Two Cultures: statistics vs. machine learning?

Last year, I read a blog post from Brendan O'Connor entitled "Statistics vs. Machine Learning, fight!" that discussed some of the differences between the two fields. Andrew Gelman responded favorably ...
134
votes
4answers
83k views

Choice of K in K-fold cross-validation

I've been using the $K$-fold cross-validation a few times now to evaluate performance of some learning algorithms, but I've always been puzzled as to how I should choose the value of $K$. I've often ...
55
votes
17answers
10k views

Machine learning cookbook / reference card / cheatsheet?

I find resources like the Probability and Statistics Cookbook and The R Reference Card for Data Mining incredibly useful. They obviously serve well as references but also help me to organize my ...
48
votes
4answers
21k views

Class imbalance in Supervised Machine Learning

This is a question in general, not specific to any method or data set. How do we deal with a class imbalance problem in Supervised Machine learning where the number of 0 is around 90% and number of 1 ...
226
votes
8answers
66k views

Why is Euclidean distance not a good metric in high dimensions?

I read that 'Euclidean distance is not a good distance in high dimensions'. I guess this statement has something to do with the curse of dimensionality, but what exactly? Besides, what is 'high ...
34
votes
3answers
16k views

PCA and the train/test split

I have a dataset for which I have multiple sets of binary labels. For each set of labels, I train a classifier, evaluating it by cross-validation. I want to reduce dimensionality using principal ...
121
votes
9answers
48k views

Obtaining knowledge from a random forest

Random forests are considered to be black boxes, but recently I was thinking what knowledge can be obtained from a random forest? The most obvious thing is the importance of the variables, in the ...
73
votes
8answers
27k views

Feature selection for “final” model when performing cross-validation in machine learning

I am getting a bit confused about feature selection and machine learning and I was wondering if you could help me out. I have a microarray dataset that is classified into two groups and has 1000s of ...
104
votes
5answers
53k views

How does a Support Vector Machine (SVM) work?

How does a Support Vector Machine (SVM) work, and what differentiates it from other linear classifiers, such as the Linear Perceptron, Linear Discriminant Analysis, or Logistic Regression? * (* I'm ...
49
votes
10answers
60k views

Hold-out validation vs. cross-validation

To me, it seems that hold-out validation is useless. That is, splitting the original dataset into two-parts (training and testing) and using the testing score as a generalization measure, is somewhat ...
35
votes
1answer
45k views

How to interpret error measures?

I am running the classify in Weka for a certain dataset and I've noticed that if I'm trying to predict a nominal value the output specifically shows the correctly and incorrectly predicted values. ...
12
votes
1answer
7k views

How to know if a learning curve from SVM model suffers from bias or variance?

I created this learning curve and I want to know if my SVM model suffers from bias or variance? How can I conclude that from this graph?
89
votes
6answers
31k views

Is it possible to train a neural network without backpropagation?

Many neural network books and tutorials spend a lot of time on the backpropagation algorithm, which is essentially a tool to compute the gradient. Let's assume we are building a model with ~10K ...
27
votes
2answers
7k views

Do we need gradient descent to find the coefficients of a linear regression model?

I was trying to learn machine learning using the Coursera material. In this lecture, Andrew Ng uses gradient descent algorithm to find the coefficients of the linear regression model that will ...
165
votes
9answers
32k views

Why the sudden fascination with tensors?

I've noticed lately that a lot of people are developing tensor equivalents of many methods (tensor factorization, tensor kernels, tensors for topic modeling, etc) I'm wondering, why is the world ...
117
votes
8answers
46k views

Why is Newton's method not widely used in machine learning?

This is something that has been bugging me for a while, and I couldn't find any satisfactory answers online, so here goes: After reviewing a set of lectures on convex optimization, Newton's method ...
66
votes
2answers
17k views

Why is ridge regression called “ridge”, why is it needed, and what happens when $\lambda$ goes to infinity?

Ridge regression coefficient estimate $\hat{\beta}^R$ are the values that minimize the $$ \text{RSS} + \lambda \sum_{j=1}^p\beta_j^2. $$ My questions are: If $\lambda = 0$, then we see that the ...
26
votes
3answers
24k views

Why is AUC higher for a classifier that is less accurate than for one that is more accurate?

I have two classifiers A: naive Bayesian network B: tree (singly-connected) Bayesian network In terms of accuracy and other measures, A performs comparatively worse than B. However, when I use the R ...
18
votes
2answers
6k views

How does linear discriminant analysis reduce the dimensions?

There are words from "The Elements of Statistical Learning" on page 91: The K centroids in p-dimensional input space span at most K-1 dimensional subspace, and if p is much larger than K, this ...
13
votes
1answer
7k views

Dropping one of the columns when using one-hot encoding

My understanding is that in machine learning it can be a problem if your dataset has highly correlated features, as they effectively encode the same information. Recently someone pointed out that ...
107
votes
5answers
17k views

What skills are required to perform large scale statistical analyses?

Many statistical jobs ask for experience with large scale data. What are the sorts of statistical and computational skills that would be need for working with large data sets. For example, how about ...
76
votes
1answer
7k views

Help me understand Support Vector Machines

I understand the basics of what a Support Vector Machines' aim is in terms of classifying an input set into several different classes, but what I don't understand is some of the nitty-gritty details. ...
12
votes
3answers
16k views

What algorithms need feature scaling, beside from SVM?

I am working with many algorithms: RandomForest, DecisionTrees, NaiveBayes, SVM (kernel=linear and rbf), KNN, LDA and XGBoost. All of them were pretty fast except for SVM. That is when I got to know ...
16
votes
1answer
2k views

What problem does oversampling, undersampling, and SMOTE solve?

In a recent, well recieved, question, Tim asks when is unbalanced data really a problem in Machine Learning? The premise of the question is that there is a lot of machine learning literature ...
124
votes
7answers
178k views

What is the influence of C in SVMs with linear kernel?

I am currently using an SVM with a linear kernel to classify my data. There is no error on the training set. I tried several values for the parameter $C$ ($10^{-5}, \dots, 10^2$). This did not ...
22
votes
3answers
15k views

Cross-validation or bootstrapping to evaluate classification performance?

What is the most appropriate sampling method to evaluate the performance of a classifier on a particular data set and compare it with other classifiers? Cross-validation seems to be standard practice, ...
13
votes
4answers
6k views

Are there any non-distance based clustering algorithms?

It seems that for K-means and other related algorithms, clustering is based off calculating distance between points. Is there one that works without it?
65
votes
6answers
8k views

Variable selection for predictive modeling really needed in 2016?

This question has been asked on CV some yrs ago, it seems worth a repost in light of 1) order of magnitude better computing technology (e.g. parallel computing, HPC etc) and 2) newer techniques, e.g. [...
48
votes
8answers
20k views

Book for reading before Elements of Statistical Learning?

Based on this post, I want to digest Elements of Statistical Learning. Fortunately it is available for free and I started reading it. I don't have enough knowledge to understand it. Can you recommend ...
50
votes
6answers
21k views

Binary classification with strongly unbalanced classes

I have a data set in the form of (features, binary output 0 or 1), but 1 happens pretty rarely, so just by always predicting 0, I get accuracy between 70% and 90% (depending on the particular data I ...
40
votes
7answers
65k views

Is it a good practice to always scale/normalize data for machine learning? [duplicate]

My understanding is that when some features have different ranges in their values (for example, imagine one feature being the age of a person and another one being their salary in USD) will affect ...
41
votes
2answers
59k views

Linear kernel and non-linear kernel for support vector machine?

When using support vector machine, are there any guidelines on choosing linear kernel vs. nonlinear kernel, like RBF? I once heard that non-linear kernel tends not to perform well once the number of ...
36
votes
3answers
2k views

Variance of $K$-fold cross-validation estimates as $f(K)$: what is the role of “stability”?

TL,DR: It appears that, contrary to oft-repeated advice, leave-one-out cross validation (LOO-CV) -- that is, $K$-fold CV with $K$ (the number of folds) equal to $N$ (the number of training ...
29
votes
2answers
40k views

Relative importance of a set of predictors in a random forests classification in R

I'd like to determine the relative importance of sets of variables toward a randomForest classification model in R. The ...
34
votes
5answers
17k views

Practical hyperparameter optimization: Random vs. grid search

I'm currently going through Bengio's and Bergsta's Random Search for Hyper-Parameter Optimization [1] where the authors claim random search is more efficient than grid search in achieving ...
25
votes
3answers
11k views

What are the impacts of choosing different loss functions in classification to approximate 0-1 loss

We know that some objective functions are easier to optimize and some are hard. And there are many loss functions that we want to use but hard to use, for example 0-1 loss. So we find some proxy loss ...
38
votes
2answers
53k views

Measures of variable importance in random forests

I've been playing around with random forests for regression and am having difficulty working out exactly what the two measures of importance mean, and how they should be interpreted. The ...
33
votes
5answers
15k views

What exactly is a Bayesian model?

Can I call a model wherein Bayes' Theorem is used a "Bayesian model"? I am afraid such a definition might be too broad. So what exactly is a Bayesian model?
11
votes
3answers
948 views

Statistical significance (p-value) for comparing two classifiers with respect to (mean) ROC AUC, sensitivity and specificity

I have a test set of 100 cases and two classifiers. I generated predictions and computed ROC AUC, sensitivity and specificity for both classifiers. Question 1: How can I compute p-value to check if ...
6
votes
2answers
2k views

What is the justification for unsupervised discretization of continuous variables?

A number of sources suggest that there are many negative consequences of the discretization (categorization) of continuous variables prior to statistical analysis (sample of references [1]-[4] below). ...