Skip to main content
63 votes
Accepted

How is softmax_cross_entropy_with_logits different from softmax_cross_entropy_with_logits_v2?

You have every reason to be confused, because in supervised learning one doesn't need to backpropagate to labels. They are considered fixed ground truth and only the weights need to be adjusted to ...
Maxim's user avatar
  • 3,319
36 votes
Accepted

Gradient descent doesn't find solution to ordinary least squares on this dataset?

The short answer is that your step size is too big. Instead of descending the canyon wall, your step is so big that you're jumping across from one side to higher up on the other! Cost function below: ...
Matthew Gunn's user avatar
  • 22.5k
34 votes
Accepted

Is there any supervised-learning problem that (deep) neural networks obviously couldn't outperform any other methods?

Here is one theoretical and two practical reasons why someone might rationally prefer a non-DNN approach. The No Free Lunch Theorem from Wolpert and Macready says We have dubbed the associated ...
Matt Krause's user avatar
  • 21.3k
33 votes

Apply word embeddings to entire document, to get a feature vector

One simple technique that seems to work reasonably well for short texts (e.g., a sentence or a tweet) is to compute the vector for each word in the document, and then aggregate them using the ...
D.W.'s user avatar
  • 6,688
32 votes
Accepted

Most interpretable classification models

1) I would argue that decision trees are not as interpretable as people make them out to be. They look interpretable, since each node is a simple, binary decision. The problem is that as you go down ...
Wayne's user avatar
  • 21.3k
30 votes
Accepted

My machine learning model has precision of 30%. Can this model be useful?

As Dave argues, if "false negatives have no associated costs", then your best course of action would be not to classify anything as positive, i.e., as 1. You inspect nothing at all, you ...
Stephan Kolassa's user avatar
29 votes
Accepted

Is supervised learning a subset of reinforcement learning?

It's true that any supervised learning problem can be cast as an equivalent reinforcement learning problem: Let states correspond to the input data. Let actions correspond to predictions of the output....
user20160's user avatar
  • 32.8k
28 votes

Is there any supervised-learning problem that (deep) neural networks obviously couldn't outperform any other methods?

Somewhere on this playlist of lectures by Geoff Hinton (from his Coursera course on neural networks), there's a segment where he talks about two classes of problems: Problems where noise is the key ...
Ben Ogorek's user avatar
  • 5,397
27 votes

How can you account for COVID-19 in your models?

We do forecasting for retail: supermarkets, drugstores etc. We add predictors to explain our sales time series, specifically different predictors for different phases of the lockdowns. On the one hand,...
Stephan Kolassa's user avatar
23 votes
Accepted

(Why) Is absolute loss not a proper scoring rule?

Let's first make sure we agree on definitions. Consider a binary random variable $Y \sim \text{Ber}(p)$, and consider a loss function $L(y_i|s)$, where $s$ is an estimate of $p$ given the data. In ...
doubled's user avatar
  • 4,947
21 votes
Accepted

Training error in KNN classifier when K=1

Training error here is the error you'll have when you input your training set to your KNN as test set. When K = 1, you'll choose the closest training sample to your test sample. Since your test sample ...
gunes's user avatar
  • 57.6k
21 votes

Why do we use Linear Models when tree based models often work better than linear models?

Many excellent answers already. I would add a few more aspects. You do not say what you mean by "outperform", as in "tree models often outperform linear models". You presumably ...
Stephan Kolassa's user avatar
20 votes

What makes a classifier misclassify data?

Let's assume you are talking about mis-classification on training data, i.e., difficult to minimize the loss on training data set, no testing data over-fitting problem involved. You are correct that, ...
Haitao Du's user avatar
  • 37.1k
18 votes

Apply word embeddings to entire document, to get a feature vector

You can use doc2vec similar to word2vec and use a pre-trained model from a large corpus. Then use something like .infer_vector() in gensim to construct a document ...
tokestermw's user avatar
18 votes
Accepted

Is Random Forest a good option for unbalanced data Classification?

Note: This post is fairly old, and might not be correct. Use it only as a starting point, not an authoritative answer. The random forest model is built on decision trees, and decision trees are ...
shadowtalker's user avatar
  • 12.7k
18 votes
Accepted

Distinguishing between two groups in statistics and machine learning: hypothesis test vs. classification vs. clustering

Great question. Anything can be good or bad, useful or not, based on what your goals are (and perhaps on the nature of your situation). For the most part, these methods are designed to satisfy ...
gung - Reinstate Monica's user avatar
18 votes

My machine learning model has precision of 30%. Can this model be useful?

I would say neither group is entirely correct. The question is what do you want to do with the model, and what will happen for positive or negative model predictions? There are screening tests used ...
Thomas Lumley's user avatar
16 votes

Measures of ordinal classification error for ordinal regression

Gaudette and Japkowicz 2009 compared various metrics for ordinal classification accuracy and they showed that, as a single statistic, the RMSE (root mean squared error) or MSE (mean squared error) ...
Tripartio's user avatar
  • 2,206
16 votes

Why do we use Linear Models when tree based models often work better than linear models?

For most problems it is easy to beat a tree model with a regression model. That's because tree models allow for all possible interactions among predictors and these are seldom needed. Interactions ...
Frank Harrell's user avatar
15 votes

Is there any supervised-learning problem that (deep) neural networks obviously couldn't outperform any other methods?

Two linearly perfected correlated variables. Can deep-network with 1 million hidden layers and 2 trillion neutrons beat a simple linear regression? EDITED In my experience, sample collection is more ...
SmallChess's user avatar
  • 7,291
15 votes
Accepted

Why don't we use a symmetric cross-entropy loss?

Consider a classification context like you mentioned, where $q(y \mid x)$ is the model distribution over classes, given input $x$. $p(y \mid x)$ is the 'true' distribution, defined as a delta function ...
user20160's user avatar
  • 32.8k
14 votes

Why does regularization wreck orthogonality of predictions and residuals in linear regression?

I wrote a comprehensive explanation on this question in my site. It might be useful for readers. I'll talk about the ridge regularization here because it can be shown to neatly use the same equations ...
Firebug's user avatar
  • 19.4k
14 votes

Why do we use Linear Models when tree based models often work better than linear models?

Linear regression is also existent outside machine learning where it requires much less data. Why does Machine Learning need a lot of data while one can do statistical inference with a small set of ...
Sextus Empiricus's user avatar
13 votes
Accepted

Are all Machine Learning algorithms divided into Classification and Regression, not just supervised learning?

All unsupervised algorithms, e.g. clustering, dimension reduction (PCA, t-sne, autoencoder,...), missing value imputation, outlier detection, ... Some of them might internally use regression or ...
Michael M's user avatar
  • 11.8k
13 votes

Why do we use Linear Models when tree based models often work better than linear models?

In addition to the excellent answers given already, a regression model yields parameter estimates, which tree models do not. If all you are interested in is prediction (which, as I understand it, is ...
Peter Flom's user avatar
  • 123k
12 votes

Distinguishing between two groups in statistics and machine learning: hypothesis test vs. classification vs. clustering

Not going to address clustering because it's been addressed in other answers, but: In general, the problem of testing whether two samples are meaningfully different is known as two-sample testing. ...
Danica's user avatar
  • 25k
12 votes
Accepted

Why does regularization wreck orthogonality of predictions and residuals in linear regression?

An image might help. In this image, we see a geometric view of the fitting. Least squares finds a solution in a plane that has the closest distance to the observation. (more general a higher ...
Sextus Empiricus's user avatar
11 votes
Accepted

Supervised learning with uncertain data?

As a numerical quality you ascribe to your data, I think this "certainty" could surely be used as a weight. Higher "certainty" scores increase the weight a datum has on the decision function, which ...
Firebug's user avatar
  • 19.4k
11 votes

What makes a classifier misclassify data?

In addition to @hxd1011 (+1). Class imbalance in relative terms or absolute terms. In both cases we build an inadequate representation of the class of interest. Usually the later is more difficult to ...
usεr11852's user avatar
  • 44.7k
11 votes
Accepted

Detecting manipulation (e.g, photo copy-pasting) in images

In general, it's hard to detect tampering and it's a whole field of research in digital image forensics. I'll try to summarise some of the key approaches to this problem. What you're talking about is ...
MachineEpsilon's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible