Are loss functions what define the identity of each supervised machine learning algorithm?

Question

For supervised machine learning algorithms (ie: regularized logistic regression, SVM, decision trees, etc), are their specific loss functions the main/only reason they differ from one another?

Answer 1

No. You can use different algorithms with same loss function, for example you can minimize squared errors using linear regression, neural network, random forest etc. Each of those algorithms would give different results, will achieve them in different ways and will differ in performance. So nonetheless that they used same loss function, you will get different results.

More formally, usually we think of machine learning in terms of finding best set of parameters $\theta$ of some function $f$ that approximates your target variable $y$ by minimizing the loss function $J$

$$ \underset{\theta}{\operatorname{arg\,min}}\; J(y, f(X; \theta)) $$

So there are two main components of machine learning model:

1. The function $f$ that approximates $y$. You can have many different functions, think of linear regression, regression tree, neural network, $k$-nearest neighbors regression, they all will produce very different kinds of functions.
2. The loss function $J$. It (only!) tells us how bad is our approximation of $y$.

Additionally, there is also the algorithm for minimizing the loss function, it is the procedure that we apply to find the optimal parameters. In many cases it is just a matter of choosing an optimizer that we use, but in other cases th algorithm defines the model (e.g. $k$NN)

Answer 2

There are different scenarios for which certain algorithms may be more appropriate. Decision trees are often used when it is necessary for a human to be able to understand the decision (for example, in medical diagnoses). However, it's easy to overfit single decision trees, which can be mitigated by using ensembles of decision trees, or random forests. Another example is that convolutional neural networks are widely used in current research, especially image recognition, because the convolutional kernels can capture many aspects of images, especially invariance to translation. In many cases, the specific domain of your problem has a large effect on the best learning algorithm to use.