Questions tagged [machine-learning]

Machine learning algorithms build a model of the training data. The term "machine learning" is vaguely defined; it includes what is also called statistical learning, reinforcement learning, unsupervised learning, etc. ALWAYS ADD A MORE SPECIFIC TAG.

28
votes
2answers
25k views

How to statistically compare the performance of machine learning classifiers?

Based on estimated classification accuracy, I want to test whether one classifier is statistically better on a base set than another classifier . For each classifier, I select a training and testing ...
20
votes
2answers
6k views

Cross Validation (error generalization) after model selection

Note: Case is n>>p I am reading Elements of Statistical Learning and there are various mentions about the "right" way to do cross validation( e.g. page 60, page 245). Specifically, my question is how ...
40
votes
1answer
54k views

Performance metrics to evaluate unsupervised learning

With respect to the unsupervised learning (like clustering), are there any metrics to evaluate performance?
21
votes
3answers
13k views

Feature map for the Gaussian kernel

In SVM, the Gaussian kernel is defined as: $$K(x,y)=\exp\left({-\frac{\|x-y\|_2^2}{2\sigma^2}}\right)=\phi(x)^T\phi(y)$$ where $x, y\in \mathbb{R^n}$. I do not know the explicit equation of $\phi$. I ...
13
votes
2answers
6k views

Use of nested cross-validation

Scikit Learn's page on Model Selection mentions the use of nested cross-validation: ...
13
votes
1answer
5k views

Including Interaction Terms in Random Forest

Suppose we have a response Y and predictors X1,....,Xn. If we were to try to fit Y via a linear model of X1,....,Xn, and it just so happened that the true relationship between Y and X1,...,Xn wasn't ...
12
votes
2answers
8k views

What measure of training error to report for Random Forests?

I'm currently fitting random forests for a classification problem using the randomForest package in R, and am unsure about how to report training error for these ...
20
votes
2answers
25k views

10-fold Cross-validation vs leave-one-out cross-validation

I'm doing nested cross-validation. I have read that leave-one-out cross-validation can be biased (don't remember why). Is it better to use 10-fold cross-validation or leave-one-out cross-validation ...
15
votes
4answers
6k views

Is KNN a discriminative learning algorithm?

It seems that KNN is a discriminative learning algorithm but I can't seem to find any online sources confirming this. Is KNN a discriminative learning algorithm?
28
votes
3answers
9k views

In boosting, why are the learners “weak”?

See also a similar question on stats.SE. In boosting algorithms such as AdaBoost and LPBoost it is known that the "weak" learners to be combined only have to perform better than chance to be useful, ...
11
votes
3answers
6k views

Perform linear regression, but force solution to go through some particular data points

I know how to perform a linear regression on a set of points. That is, I know how to fit a polynomial of my choice, to a given data set, (in the LSE sense). However, what I do not know, is how to ...
11
votes
2answers
6k views

What is the difference between 'regular' linear regression and deep learning linear regression?

I want to know the difference between linear regression in a regular machine learning analysis and linear regression in "deep learning" setting. What algorithms are used for linear regression in deep ...
9
votes
3answers
2k views

Expected best performance possible on a data set

Say I have a simple machine learning problem like a classification. With some benchmarks in vision or audio recognition, I, as a human, are a very good classifier. I therefore have an intuition on how ...
2
votes
2answers
2k views

How to bootstrap panel data?

I'm fitting some machine learning algorithms (e.g. SVM) on my panel data. It's taking too long for my entire dataset, so I'm considering generating smaller samples from bootstrapping then fit the SVM ...
6
votes
1answer
2k views

How does gradient boosting calculate probability estimates?

I have been trying to understand gradient boosting reading various blogs, websites and trying to find my answer by looking through for example the XGBoost source code. However, I cannot seem to find ...
1
vote
1answer
4k views

Decision boundaries and Gaussian density functions

This is for my hw, and if anyone can solve the first part of the question it will be great. Here is the question: Assume a two-class problem with equal a priori class probabilities and Gaussian class-...
89
votes
11answers
29k views

Explain “Curse of dimensionality” to a child

I heard many times about curse of dimensionality, but somehow I'm still unable to grasp the idea, it's all foggy. Can anyone explain this in the most intuitive way, as you would explain it to a child,...
63
votes
4answers
34k views

What makes the Gaussian kernel so magical for PCA, and also in general?

I was reading about kernel PCA (1, 2, 3) with Gaussian and polynomial kernels. How does the Gaussian kernel separate seemingly any sort of nonlinear data exceptionally well? Please give an intuitive ...
46
votes
7answers
80k views

Validation Error less than training error?

I found two questions here and here about this issue but there is no obvious answer or explanation yet.I enforce the same problem where the validation error is less than training error in my ...
72
votes
11answers
44k views

Having a job in data-mining without a PhD

I've been very interested in data-mining and machine-learning for a while, partly because I majored in that area at school, but also because I am truly much more excited trying to solve problems that ...
49
votes
5answers
39k views

Is random forest a boosting algorithm?

Short definition of boosting: Can a set of weak learners create a single strong learner? A weak learner is defined to be a classifier which is only slightly correlated with the true ...
36
votes
4answers
29k views

How does rectilinear activation function solve the vanishing gradient problem in neural networks?

I found rectified linear unit (ReLU) praised at several places as a solution to the vanishing gradient problem for neural networks. That is, one uses max(0,x) as activation function. When the ...
30
votes
8answers
39k views

What math subjects would you suggest to prepare for data mining and machine learning?

I'm trying to put together a self-directed math curriculum to prepare for learning data mining and machine learning. This is motivated by starting Andrew Ng's machine learning class on Coursera and ...
25
votes
7answers
63k views

In Naive Bayes, why bother with Laplace smoothing when we have unknown words in the test set?

I was reading over Naive Bayes Classification today. I read, under the heading of Parameter Estimation with add 1 smoothing: Let $c$ refer to a class (such as Positive or Negative), and let $w$ ...
37
votes
4answers
26k views

Is a strong background in maths a total requisite for ML?

I'm starting to want to advance my own skillset and I've always been fascinated by machine learning. However, six years ago instead of pursuing this I decided to take a completely unrelated degree to ...
26
votes
2answers
11k views

Variance estimates in k-fold cross-validation

K-fold cross-validation can be used to estimate the generalization capability of a given classifier. Can I (or should I) also compute a pooled variance from all validation runs in order to obtain a ...
27
votes
3answers
6k views

Cross-validation including training, validation, and testing. Why do we need three subsets?

I have a question regarding the Cross-validation process. I am in the middle of a course of the Machine Learning on the Cursera. One of the topic is about the Cross-validation. I found it slightly ...
20
votes
8answers
16k views

Perform K-means (or its close kin) clustering with only a distance matrix, not points-by-features data

I want to perform K-means clustering on objects I have, but the objects aren't described as points in space, i.e. by objects x features dataset. However, I am able ...
34
votes
2answers
11k views

Gradient Boosting for Linear Regression - why does it not work?

While learning about Gradient Boosting, I haven't heard about any constraints regarding the properties of a "weak classifier" that the method uses to build and ensemble model. However, I could not ...
17
votes
2answers
14k views

Boosting: why is the learning rate called a regularization parameter?

The learning rate parameter ($\nu \in [0,1]$) in Gradient Boosting shrinks the contribution of each new base model -typically a shallow tree- that is added in the series. It was shown to dramatically ...
14
votes
3answers
11k views

Weighting more recent data in Random Forest model

I'm training a classification model with Random Forest to discriminate between 6 categories. My transactional data has approximately 60k+ observations and 35 variables. Here's an example of how it ...
24
votes
4answers
17k views

When to avoid Random Forest?

Random forests are well known to perform fairly well on a variety of tasks and have been referred to as the leatherman of learning methods. Are there any types of problems or specific conditions in ...
13
votes
2answers
3k views

Supervised learning with “rare” events, when rarity is due to the large number of counter-factual events

Suppose you get to observe "matches" between buyers and sellers in a market. You also get to observe characteristics of both buyers and sellers which you would like to use to predict future matches &...
11
votes
4answers
1k views

Good examples/books/resources to learn about applied machine learning (not just ML itself)

I've taken an ML course previously, but now that I am working with ML related projects at my job, I am struggling quite a bit to actually apply it. I'm sure the stuff I'm doing has been researched/...
25
votes
2answers
21k views

Neural Network: For Binary Classification use 1 or 2 output neurons?

Assume I want to do binary classification (something belongs to class A or class B). There are some possibilities to do this in the output layer of a neural network: Use 1 output node. Output 0 (<...
14
votes
3answers
8k views

For linear classifiers, do larger coefficients imply more important features?

I'm a software engineer working on machine learning. From my understanding, linear regression (such as OLS) and linear classification (such as logistic regression and SVM) make a prediction based on ...
15
votes
4answers
15k views

Comparing two classifier accuracy results for statistical significance with t-test

I want to compare the accuracy of two classifiers for statistical significance. Both classifiers are run on the same data set. This leads me to believe I should be using a one sample t-test from what ...
17
votes
2answers
16k views

Best way to perform multiclass SVM

I know that the SVM is a binary classifier. I would like to extend it to multi-class SVM. Which is the best, and maybe the easiest, way to perform it? code: in MATLAB ...
22
votes
5answers
11k views

Clustering procedure where each cluster has an equal number of points?

I have some points $X=\{x_1,...,x_n\}$ in $R^p$, and I want to cluster the points so that: Each cluster contains an equal number of elements of $X$. (Assume that the number of clusters divides $n$.) ...
10
votes
1answer
3k views

How to fit weights into Q-values with linear function approximation

In reinforcement learning, linear function approximation is often used when large state spaces are present. (When look up tables become unfeasible.) The form of the $Q-$value with linear function ...
12
votes
2answers
24k views

Why scaling is important for the linear SVM classification?

When performing the linear SVM classification, it is often helpful to normalize the training data, for example by subtracting the mean and dividing by the standard deviation, and afterwards scale ...
10
votes
2answers
11k views

How to make Random Forests more interpretable?

Are there any methods that one could utilize to make Random Forest more interpretable? Random Forest performs much better than CART but it is a lot less interpretable.
10
votes
2answers
514 views

What is meant by the variance of *functions* in *Introduction to Statistical Learning*?

On pg. 34 of Introduction to Statistical Learning: $\newcommand{\Var}{{\rm Var}}$ Though the mathematical proof is beyond the scope of this book, it is possible to show that the expected test MSE, ...
7
votes
3answers
2k views

Is a lower training accuracy possible in overfitting (one class SVM)

I am using the heart_scale data from LibSVM. The original data includes 13 features, but I only used 2 of them in order to plot the distributions in a figure. Instead of training the binary classifier,...
7
votes
3answers
9k views

What balancing method can I apply to a imbalanced data set?

I'm trying to solve one classification problem from the UCI database repository. Unfortunately (or fortunately), I've noticed that my dataset is imbalanced. I've structured the data as 5 classes, ...
6
votes
1answer
658 views

What do “real values” refer to in supervised classification?

I'm using supervised classification algorithms from mlpy to classify things into two groups for a question-answering system. I don't really know how these algorithms work, but they seem to be doing ...
5
votes
2answers
384 views

Graphical interpretation of LASSO

I've a question regarding to the graphical intuition of the LASSO. I'm understanding why the lasso produce zero coefficient in case of intersecting a corner of the diamond. But I don't understand the ...
4
votes
2answers
2k views

Detecting Bimodal Distribution

I have histograms of audio signals where they have bimodal "normal" distribution. What I want to do is to detect these subpopulations inorder to have a threshold, this is meant to divide the values ...
3
votes
1answer
788 views

How to do a bias-variance analysis on a machine learning modelling process

I searched on topics of the bias and variance trade-off and got back lots of questions with different levels of response. The information is scattering too much and unsystematic to answer my own ...
4
votes
2answers
2k views

Why is my high degree polynomial regression model suddenly unfit for the data?

I'm building a ridge regression model in scikit-learn and trying to find the optimal degree polynomial to use. The data I'm working with is a fairly predictable time series of hourly traffic volumes, ...