Questions tagged [machine-learning]
Machine learning algorithms build a model of the training data. The term "machine learning" is vaguely defined; it includes what is also called statistical learning, reinforcement learning, unsupervised learning, etc. ALWAYS ADD A MORE SPECIFIC TAG.
19,989
questions
560
votes
11
answers
650k
views
What is the difference between test set and validation set?
I found this confusing when I use the neural network toolbox in Matlab.
It divided the raw data set into three parts:
training set
validation set
test set
I notice in many training or learning ...
- 7,149
491
votes
20
answers
174k
views
The Two Cultures: statistics vs. machine learning?
Last year, I read a blog post from Brendan O'Connor entitled "Statistics vs. Machine Learning, fight!" that discussed some of the differences between the two fields. Andrew Gelman responded favorably ...
441
votes
5
answers
172k
views
How to understand the drawbacks of K-means
K-means is a widely used method in cluster analysis. In my understanding, this method does NOT require ANY assumptions, i.e., give me a dataset and a pre-specified number of clusters, k, and I just ...
- 6,799
348
votes
8
answers
149k
views
Why is Euclidean distance not a good metric in high dimensions?
I read that 'Euclidean distance is not a good distance in high dimensions'. I guess this statement has something to do with the curse of dimensionality, but what exactly? Besides, what is 'high ...
- 3,697
299
votes
8
answers
212k
views
Bagging, boosting and stacking in machine learning
What's the similarities and differences between these 3 methods:
Bagging,
Boosting,
Stacking?
Which is the best one? And why?
Can you give me an example for each?
- 3,099
277
votes
3
answers
30k
views
How to know that your machine learning problem is hopeless?
Imagine a standard machine-learning scenario:
You are confronted with a large multivariate dataset and you have a
pretty blurry understanding of it. What you need to do is to make
predictions ...
Tim♦
- 134k
253
votes
5
answers
121k
views
ROC vs precision-and-recall curves
I understand the formal differences between them, what I want to know is when it is more relevant to use one vs. the other.
Do they always provide complementary insight about the performance of a ...
- 18.9k
248
votes
9
answers
122k
views
Why is Newton's method not widely used in machine learning?
This is something that has been bugging me for a while, and I couldn't find any satisfactory answers online, so here goes:
After reviewing a set of lectures on convex optimization, Newton's method ...
- 2,471
228
votes
10
answers
125k
views
Why is accuracy not the best measure for assessing classification models?
This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference.
...
Tim♦
- 134k
227
votes
8
answers
273k
views
What are the advantages of ReLU over sigmoid function in deep neural networks?
The state of the art of non-linearity is to use rectified linear units (ReLU) instead of sigmoid function in deep neural network. What are the advantages?
I know that training a network when ReLU is ...
- 12.5k
222
votes
13
answers
202k
views
What is the difference between data mining, statistics, machine learning and AI?
What is the difference between data mining, statistics, machine learning and AI?
Would it be accurate to say that they are 4 fields attempting to solve very similar problems but with different ...
210
votes
3
answers
100k
views
Generative vs. discriminative
I know that generative means "based on $P(x,y)$" and discriminative means "based on $P(y|x)$," but I'm confused on several points:
Wikipedia (+ many other hits on the web) classify things like SVMs ...
- 3,131
209
votes
4
answers
199k
views
What does the hidden layer in a neural network compute?
I'm sure many people will respond with links to 'let me google that for you', so I want to say that I've tried to figure this out so please forgive my lack of understanding here, but I cannot figure ...
- 2,227
197
votes
6
answers
71k
views
Training on the full dataset after cross-validation?
TL:DR: Is it ever a good idea to train an ML model on all the data available before shipping it to production? Put another way, is it ever ok to train on all data available and not check if the model ...
- 18.9k
193
votes
10
answers
46k
views
Why the sudden fascination with tensors?
I've noticed lately that a lot of people are developing tensor equivalents of many methods (tensor factorization, tensor kernels, tensors for topic modeling, etc) I'm wondering, why is the world ...
191
votes
8
answers
388k
views
What is the influence of C in SVMs with linear kernel?
I am currently using an SVM with a linear kernel to classify my data. There is
no error on the training set. I tried several values for the parameter $C$
($10^{-5}, \dots, 10^2$). This did not ...
- 2,655
181
votes
4
answers
173k
views
Choice of K in K-fold cross-validation
I've been using the $K$-fold cross-validation a few times now to evaluate performance of some learning algorithms, but I've always been puzzled as to how I should choose the value of $K$.
I've often ...
- 2,357
181
votes
2
answers
223k
views
A list of cost functions used in neural networks, alongside applications
What are common cost functions used in evaluating the performance of neural networks?
Details
(feel free to skip the rest of this question, my intent here is simply to provide clarification on ...
180
votes
7
answers
129k
views
How to intuitively explain what a kernel is?
Many machine learning classifiers (e.g. support vector machines) allow one to specify a kernel. What would be an intuitive way of explaining what a kernel is?
One aspect I have been thinking of is ...
- 1,801
172
votes
11
answers
191k
views
What is the difference between off-policy and on-policy learning?
Artificial intelligence website defines off-policy and on-policy learning as follows:
"An off-policy learner learns the value of the optimal policy independently of the agent's actions. Q-learning ...
- 8,647
168
votes
3
answers
179k
views
Gradient Boosting Tree vs Random Forest
Gradient tree boosting as proposed by Friedman uses decision trees as base learners. I'm wondering if we should make the base decision tree as complex as possible (fully grown) or simpler? Is there ...
- 2,003
166
votes
9
answers
140k
views
Objective function, cost function, loss function: are they the same thing?
In machine learning, people talk about objective function, cost function, loss function. Are they just different names of the same thing? When to use them? If they are not always refer to the same ...
- 1,789
148
votes
6
answers
84k
views
Why are neural networks becoming deeper, but not wider?
In recent years, convolutional neural networks (or perhaps deep neural networks in general) have become deeper and deeper, with state-of-the-art networks going from 7 layers (AlexNet) to 1000 layers (...
- 6,749
141
votes
9
answers
59k
views
Obtaining knowledge from a random forest
Random forests are considered to be black boxes, but recently I was thinking what knowledge can be obtained from a random forest?
The most obvious thing is the importance of the variables, in the ...
- 3,994
134
votes
10
answers
83k
views
Bias and variance in leave-one-out vs K-fold cross validation
How do different cross-validation methods compare in terms of model variance and bias?
My question is partly motivated by this thread: Optimal number of folds in $K$-fold cross-validation: is leave-...
- 18.9k
133
votes
5
answers
78k
views
How does a Support Vector Machine (SVM) work?
How does a Support Vector Machine (SVM) work, and what differentiates it from other linear classifiers, such as the Linear Perceptron, Linear Discriminant Analysis, or Logistic Regression? *
(* I'm ...
- 7,559
133
votes
8
answers
157k
views
Is it necessary to scale the target value in addition to scaling features for regression analysis?
I'm building regression models. As a preprocessing step, I scale my feature values to have mean 0 and standard deviation 1. Is it necessary to normalize the target values also?
- 2,653
131
votes
4
answers
157k
views
What does a "closed-form solution" mean?
I have come across the term "closed-form solution" quite often. What does a closed-form solution mean? How does one determine if a close-form solution exists for a given problem? Searching online, I ...
- 2,613
130
votes
5
answers
158k
views
What are the main differences between K-means and K-nearest neighbours?
I know that k-means is unsupervised and is used for clustering etc and that k-NN is supervised. But I wanted to know concrete differences between the two?
- 1,647
127
votes
4
answers
249k
views
Softmax vs Sigmoid function in Logistic classifier?
What decides the choice of function ( Softmax vs Sigmoid ) in a Logistic classifier ?
Suppose there are 4 output classes . Each of the above function gives the probabilities of each class being the ...
- 1,765
124
votes
7
answers
89k
views
Why use gradient descent for linear regression, when a closed-form math solution is available?
I am taking the Machine Learning courses online and learnt about Gradient Descent for calculating the optimal values in the hypothesis.
h(x) = B0 + B1X
why we ...
- 1,343
122
votes
3
answers
152k
views
tanh activation function vs sigmoid activation function
The tanh activation function is:
$$tanh \left( x \right) = 2 \cdot \sigma \left( 2 x \right) - 1$$
Where $\sigma(x)$, the sigmoid function, is defined as:
$$\sigma(x) = \frac{e^x}{1 + e^x}$$.
...
- 1,373
120
votes
1
answer
81k
views
What is an ablation study? And is there a systematic way to perform it?
What is an ablation study? And is there a systematic way to perform it? For example, I have $n$ predictors in a linear regression which I will call as my model.
How will I perform an ablation study ...
- 8,647
120
votes
6
answers
51k
views
Is it possible to train a neural network without backpropagation?
Many neural network books and tutorials spend a lot of time on the backpropagation algorithm, which is essentially a tool to compute the gradient.
Let's assume we are building a model with ~10K ...
- 35.8k
118
votes
4
answers
48k
views
Why isn't Logistic Regression called Logistic Classification?
Since Logistic Regression is a statistical classification model dealing with categorical dependent variables, why isn't it called Logistic Classification? Shouldn't the "Regression" name be reserved ...
- 2,166
118
votes
5
answers
231k
views
How do you calculate precision and recall for multiclass classification using confusion matrix?
I wonder how to compute precision and recall using a confusion matrix for a multi-class classification problem. Specifically, an observation can only be assigned to its most probable class / label. I ...
- 1,291
117
votes
5
answers
21k
views
What skills are required to perform large scale statistical analyses?
Many statistical jobs ask for experience with large scale data. What are the sorts of statistical and computational skills that would be need for working with large data sets. For example, how about ...
- 2,817
116
votes
12
answers
63k
views
When should linear regression be called "machine learning"?
In a recent colloquium, the speaker's abstract claimed they were using machine learning. During the talk, the only thing related to machine learning was that they perform linear regression on their ...
- 1,509
115
votes
10
answers
203k
views
Validation Error less than training error?
I found two questions here and here about this issue but there is no obvious answer or explanation yet.I enforce the same problem where the validation error is less than training error in my ...
- 1,253
114
votes
2
answers
109k
views
What is an embedding layer in a neural network?
In many neural network libraries, there are 'embedding layers', like in Keras or Lasagne.
I am not sure I understand its function, despite reading the documentation. For example, in the Keras ...
- 1,243
109
votes
12
answers
39k
views
Explain "Curse of dimensionality" to a child
I heard many times about curse of dimensionality, but somehow I'm still unable to grasp the idea, it's all foggy.
Can anyone explain this in the most intuitive way, as you would explain it to a child,...
- 2,827
109
votes
7
answers
15k
views
Detecting a given face in a database of facial images
I'm working on a little project involving the faces of twitter users via their profile pictures.
A problem I've encountered is that after I filter out all but the images that are clear portrait ...
- 1,117
108
votes
4
answers
99k
views
How to select kernel for SVM?
When using SVM, we need to select a kernel.
I wonder how to select a kernel. Any criteria on kernel selection?
- 7,149
108
votes
1
answer
69k
views
Conditional inference trees vs traditional decision trees
Can anyone explain the primary differences between conditional inference trees (ctree from party package in R) compared to the ...
- 8,070
108
votes
6
answers
47k
views
On the importance of the i.i.d. assumption in statistical learning
In statistical learning, implicitly or explicitly, one always assumes that the training set $\mathcal{D} = \{ \bf {X}, \bf{y} \}$ is composed of $N$ input/response tuples $({\bf{X}}_i,y_i)$ that are ...
- 1,496
102
votes
8
answers
37k
views
When is unbalanced data really a problem in Machine Learning?
We already had multiple questions about unbalanced data when using logistic regression, SVM, decision trees, bagging and a number of other similar questions, what makes it a very popular topic! ...
Tim♦
- 134k
102
votes
2
answers
84k
views
Solving for regression parameters in closed-form vs gradient descent
In Andrew Ng's machine learning course, he introduces linear regression and logistic regression, and shows how to fit the model parameters using gradient descent and Newton's method.
I know gradient ...
- 3,615
97
votes
9
answers
133k
views
How to compute precision/recall for multiclass-multilabel classification?
I'm wondering how to calculate precision and recall measures for multiclass multilabel classification, i.e. classification where there are more than two labels, and where each instance can have ...
- 1,305
93
votes
6
answers
50k
views
Feature selection for "final" model when performing cross-validation in machine learning
I am getting a bit confused about feature selection and machine learning
and I was wondering if you could help me out. I have a microarray dataset that is
classified into two groups and has 1000s of ...
- 2,485