Which Deep Learning Method to use for the classification of precious stones (diamonds, sapphire, ruby etc) based on digital photo images and data set? m trying to build Machine Learning Model for the classification of precious stones (like diamonds, sapphire, ruby) based on digital images. So with time the it was performed 150,000 gemstone certifications and the characteristics of each stone are recorded in a structured manner including its digital photo-image. I can use this data set to augment the human work, and with help of Deep Learning to do the stone classification
So based on my experience and research CNN are most suitable for this work as they ar handling spatial data (images). But according to this paper "Bona Hiu Yan Chow, ‘Automatic Gemstone Classification Using Computer Vision’Gemstone classificationthe Random Forest algorithm provides the best accuracy and outperformed the CNN ( ResNet-18 and ResNet-50 ) which I dont really understand. Any explanation why is the case? Is the state-of-the-art YOLO v8. or v7. more accurate as it works better for smaller objects such as stones? So for my case when have a bid data set (over 100 000 images) should I stick to CNN or still use Random Forest? Any help, as I have to choose the right model with best accuracy
Thanks
 A: Convolutional neural networks (CNNs) and/or vision transformers (both neural networks) with transfer learning would have been my first thought, too. However, there are situations, where that might not work so well. Note that in the work you cite, it seems like they had "2042 training images and 284 unseen (test) images divided into 68 categories of gemstones". Just 2042 training images for 68 categories is at best 30 images per category (if all categories are the same size). In such low data setting with the images possibly very different than what pre-training (usually on ImageNet, the photos of which are probably not so much like images of precious stones taken in a professional setting, I'd guess - pre-training on ImageNet is often of limited value for e.g. X-rays or satellite images, too) could be one of those settings. In such a setting, it could also be the case that good feature engineering can create really good features for traditional tabular data algorithms (like random forest, XGBoost and the like). The training approach taken for the neural networks and the image resizing also sound like they may have been optimal. E.g.

*

*The only data augmentation was "random horizontal or vertical flip", when lots of augmentations would likely be useful starting with rotations.

*Not much was set on how hyperparameters were set. A good validation approach (e.g. cross-validation) helps with that and also helps with combining different algorithms.

*Nowadays, one would probably use cosine-decay for transfer learning, possibly with differential learning rates across layers instead of the stepwise learning rate decay they used.

*Ideally, one uses images as large as possible, and several architectures don't enforce a fixed size.

*Just a center crop for the test set is likely inferior to test-time-augmentation.

*ResNet50 (and esp. ResNet18) are relatively old pre-trained models. If you look at popular repositories like timm you can see a lot more recent architectures. With popular deep learning Python libraries like the easy to use fastai one you can directly use models from that repository (see also their book and free course).

I'd usually think that with a lot of data, neural networks should become better and better vs. feature engineering + RF. Of course, if the two are still competitive with a lot of data, then combining them might be an option (and cross-validation would help you figure out how to do this, the keyword here would be "model stacking").
Kaggle competitions are usually quite a good source on what is current good practice and there's the occasional panel interview where Kaggle GMs tell you what they think is a good way to do things for e.g. vision tasks.
Another thing to always keep in mind is whether the data is good / matches what you intend to do (e.g. photos taken by professionals in a well lit shop vs. via smartphone, or even differences in cameras used by different sources), whether there are data leaks (e.g. the correct classification is given by the text next to the gem, or more expensive gems being displayed in nicer settings) and whether there are ways to trick the model (people would of course potentially have a strong incentive, if they could somehow get away with selling glass-beads as expensive gems, so make sure you have a representative sample of fake stones in all sorts of colors and shapes including those used for precious stones, too).
