Convolutional neural networks (CNNs) and/or vision transformers (both neural networks) with transfer learning would have been my first thought, too. However, there are situations, where that might not work so well. Note that in the work you cite, it seems like they had "2042 training images and 284 unseen (test) images divided into 68 categories of gemstones". Just 2042 training images for 68 categories is at best 30 images per category (if all categories are the same size). In such low data setting with the images possibly very different than what pre-training (usually on ImageNet, the photos of which are probably not so much like images of precious stones taken in a professional setting, I'd guess - pre-training on ImageNet is often of limited value for e.g. X-rays or satellite images, too) could be one of those settings. In such a setting, it could also be the case that good feature engineering can create really good features for traditional tabular data algorithms (like random forest, XGBoost and the like). The training approach taken for the neural networks and the image resizing also sound like they may have been optimal. E.g.
- The only data augmentation was "random horizontal or vertical flip", when lots of augmentations would likely be useful starting with rotations.
- Not much was set on how hyperparameters were set. A good validation approach (e.g. cross-validation) helps with that and also helps with combining different algorithms.
- Nowadays, one would probably use cosine-decay for transfer learning, possibly with differential learning rates across layers instead of the stepwise learning rate decay they used.
- Ideally, one uses images as large as possible, and several architectures don't enforce a fixed size.
- Just a center crop for the test set is likely inferior to test-time-augmentation.
- ResNet50 (and esp. ResNet18) are relatively old pre-trained models. If you look at popular repositories like timm you can see a lot more recent architectures. With popular deep learning Python libraries like the easy to use fastai one you can directly use models from that repository (see also their book and free course).
I'd usually think that with a lot of data, neural networks should become better and better vs. feature engineering + RF. Of course, if the two are still competitive with a lot of data, then combining them might be an option (and cross-validation would help you figure out how to do this, the keyword here would be "model stacking").
Kaggle competitions are usually quite a good source on what is current good practice and there's the occasional panel interview where Kaggle GMs tell you what they think is a good way to do things for e.g. vision tasks.
Another thing to always keep in mind is whether the data is good / matches what you intend to do (e.g. photos taken by professionals in a well lit shop vs. via smartphone, or even differences in cameras used by different sources), whether there are data leaks (e.g. the correct classification is given by the text next to the gem, or more expensive gems being displayed in nicer settings) and whether there are ways to trick the model (people would of course potentially have a strong incentive, if they could somehow get away with selling glass-beads as expensive gems, so make sure you have a representative sample of fake stones in all sorts of colors and shapes including those used for precious stones, too).
