Is it true that the type of ML model used is irrelevant? I am training a model on a dataset and all types of relevant algorithms I have used converge close to the same accuracy score, meaning that no one is significantly better performing than the other. For example, if you're training a random forest and a neural network on MNIST, you'll observe an accuracy score of around 98%. Why is this the case, that bottlenecks in performance seem to be dictated by input data rather than the choice of the algorithm?
 A: The idea that "ml model doesn't matter" is brushing a few concepts under the rug.
First and most to the point, there's a fundamental question of how well can you predict your outcome given your features. If your income is "income" and the only feature you have is "subject's height", it doesn't matter what model you use, you just can't predict the outcome well. If you have a high resolution picture and the variable you want to predict is "is the main subject of this picture a cat?", this can be known with very high accuracy, however the relation between inputs and outputs is extremely complex. In the second case, ML model definitely matters.
The idea that "ML model doesn't matter" is an observation that in many (but definitely not all) real world applications of ML, the best way to improve predictions is to get more relevant data, and there's only a minimal amount of improvement you can get from switching the model you are using. So if you have an application in which your ML model is not performing up to requirements, your time is often better spent trying to find more relevant features rather than comparing outcomes of 30 different model choices.
EDIT: It's also worth noting that in your example with MNIST, 98% is actually not very good. Convolutional NN have an error rate of ~0.25%, meaning 1/8 as many errors the Random Forrests you tried. While 98% may seem "good enough", note that for certain applications, such as recognizing whether an object is moving or not for autonomous driving, using a model with a 98% accuracy could lead to hundreds of thousands of deaths if uses by millions of drivers. So the difference between 98% accurate, 99%, 99.9%, etc are all very meaningful differences.
A: The answer to "Is it true that the type of ML model used is irrelevant?" is most definitely "No". While other answers make a reasonable points e.g. about value of collecting new data vs. selecting between models, I feel there is one point missing. Namely, your observation that "model is irrelevant" is suffering from observation bias: you are training models on MNIST that are known to perform well on this type of data. This may give false impression that the type of model doesn't matter. There will be a large number of algorithms/models that do very poorly on MNIST, try for instance a very simple linear regression and compare that to random forest and neural nets. My intuition (no rigorous evidence) is that certain algorithms also do similarly on certain datasets; I have often observed simple neural nets (MLPs) and Random forests / xgboost to be a case in point. But of course if we had more complex image data then all these methods would do much worse than convolutional NN's, and so on.
Edit: I would advise against adopting the general notion that most problems can just be solved/made progress on by collecting more data. There is a plethora of applied ML problems where improvements in model architecture, optimization etc. can yield large benefits.
A: There's a lot of truth in what you say and that's certainly the argument in what some people have branded data centric AI. For a start, a lot of academic research looks at optimizing some measure (e.g. accuracy) on a fixed given dataset (e.g. ImageNet), which kind of makes sense to measure progress in algorithms. However, in practice, instead of tinkering with minute improvements in algorithms it is often better to just get more data (or label in different ways). Similarly, in Kaggle competitions there will often be pretty small differences between well-tuned XGBoost, LightGBM, Random Forrest and certain Neural Network architectures on tabular data (plus you can often squeeze out a bit more by ensembling them), but in practice you might be pretty happy with just using of these (never mind that you could be better by a few decimal points that for many applications might be irrelevant, or at least less important than the model running fast and cheaply).
On the other hand, it is clear that some algorithms are just much better at certain tasks than others. E.g. look at the spread in performance on ImageNet, results got better year by year and e.g. the error rate got halved from 2011 to 2012 when a convolutional neural network got used. You even see a big spread in neural network performance when assessed on a newly created similar test set ranging from below 70% to over 95%. That certainly is a huge difference in performance. Or, if you get a new image classification task and have just 50 to 100 images of some reasonable size (i.e. 100 or more pixels or so in each dimension) from each class, your first thought should really be transfer learning with some kind of neural network (e.g. convolutional NN or some vision transformer) picked based on trading off good performance on ImageNet with feasible size. In contrast, it's pretty unlikely that training a RF, XGBoost, or a neural network from scratch would come anywhere near that approach in performance.
Additionally, let's not forget that often a lot is to be gained by creating the right features (especially in tabular data) or by representing the data in a good way (e.g. it turns out that you can turn audio data into spectrograms and then use neural networks for images on that, and that works pretty well). While, if one misses creating the right features or represents the data in a poor way, even a theoretically good model will struggle.
