m trying to build Machine Learning Model for the classification of precious stones (like diamonds, sapphire, ruby) based on digital images. So with time the it was performed 150,000 gemstone certifications and the characteristics of each stone are recorded in a structured manner including its digital photo-image. I can use this data set to augment the human work, and with help of Deep Learning to do the stone classification

So based on my experience and research CNN are most suitable for this work as they ar handling spatial data (images). But according to this paper "Bona Hiu Yan Chow, ‘Automatic Gemstone Classification Using Computer Vision’Gemstone classificationthe Random Forest algorithm provides the best accuracy and outperformed the CNN ( ResNet-18 and ResNet-50 ) which I dont really understand. Any explanation why is the case? Is the state-of-the-art YOLO v8. or v7. more accurate as it works better for smaller objects such as stones? So for my case when have a bid data set (over 100 000 images) should I stick to CNN or still use Random Forest? Any help, as I have to choose the right model with best accuracy


  • 2
    $\begingroup$ "Random forest" doesn't say all. Those are still images we want to classify. What features are used as input to the random forest algorithm? My guess would be... features computed by convolutional kernels. $\endgroup$
    – Stef
    Feb 6, 2023 at 9:34

1 Answer 1


Convolutional neural networks (CNNs) and/or vision transformers (both neural networks) with transfer learning would have been my first thought, too. However, there are situations, where that might not work so well. Note that in the work you cite, it seems like they had "2042 training images and 284 unseen (test) images divided into 68 categories of gemstones". Just 2042 training images for 68 categories is at best 30 images per category (if all categories are the same size). In such low data setting with the images possibly very different than what pre-training (usually on ImageNet, the photos of which are probably not so much like images of precious stones taken in a professional setting, I'd guess - pre-training on ImageNet is often of limited value for e.g. X-rays or satellite images, too) could be one of those settings. In such a setting, it could also be the case that good feature engineering can create really good features for traditional tabular data algorithms (like random forest, XGBoost and the like). The training approach taken for the neural networks and the image resizing also sound like they may have been optimal. E.g.

  • The only data augmentation was "random horizontal or vertical flip", when lots of augmentations would likely be useful starting with rotations.
  • Not much was set on how hyperparameters were set. A good validation approach (e.g. cross-validation) helps with that and also helps with combining different algorithms.
  • Nowadays, one would probably use cosine-decay for transfer learning, possibly with differential learning rates across layers instead of the stepwise learning rate decay they used.
  • Ideally, one uses images as large as possible, and several architectures don't enforce a fixed size.
  • Just a center crop for the test set is likely inferior to test-time-augmentation.
  • ResNet50 (and esp. ResNet18) are relatively old pre-trained models. If you look at popular repositories like timm you can see a lot more recent architectures. With popular deep learning Python libraries like the easy to use fastai one you can directly use models from that repository (see also their book and free course).

I'd usually think that with a lot of data, neural networks should become better and better vs. feature engineering + RF. Of course, if the two are still competitive with a lot of data, then combining them might be an option (and cross-validation would help you figure out how to do this, the keyword here would be "model stacking").

Kaggle competitions are usually quite a good source on what is current good practice and there's the occasional panel interview where Kaggle GMs tell you what they think is a good way to do things for e.g. vision tasks.

Another thing to always keep in mind is whether the data is good / matches what you intend to do (e.g. photos taken by professionals in a well lit shop vs. via smartphone, or even differences in cameras used by different sources), whether there are data leaks (e.g. the correct classification is given by the text next to the gem, or more expensive gems being displayed in nicer settings) and whether there are ways to trick the model (people would of course potentially have a strong incentive, if they could somehow get away with selling glass-beads as expensive gems, so make sure you have a representative sample of fake stones in all sorts of colors and shapes including those used for precious stones, too).

  • $\begingroup$ great thanks for the answer. I have another question. How to choose the CNN model in the case when my data set is much bigger ( it is 150 000 images) and need around 20 classes? $\endgroup$
    – Bob9710
    Feb 6, 2023 at 8:35
  • $\begingroup$ I mean there is this link medium.com/ai-techsystems/… and also this kaggle.com/code/lsind18/gemstones-multiclass-classification-cnn but my question is how to change the CNN model when have 150 000 images and around 20 classes? $\endgroup$
    – Bob9710
    Feb 6, 2023 at 9:08
  • 1
    $\begingroup$ Largely, the size of the CNN to when you use to do transfer learning may not change too much when you have more data, what would change is how much you need to regularize what you do to avoid overfitting (with a lot of data from all classes, overfitting may become less of a concern and less strong regularization may be needed). Generally, you'd think larger images and better performing models on ImageNet might work better in a new setting, too, but you'd want to trade that off vs. what you can $\endgroup$
    – Björn
    Feb 6, 2023 at 13:34
  • 1
    $\begingroup$ actually train on your system that you have available (GPU memory & not too small batch size are limitations, the param_count column gives you some hint in that respect). Additionally, if pre-training is of limited help, a very large model might need a lot of re-training, which might again be time consuming. You may also want to not immediately start with the largest model you could possibly do and do initial experiments with something smaller. Something like some version of tf_efficientnet_b2 to tf_efficientnet_b4 in size might be a plausible starting point depending on your system. $\endgroup$
    – Björn
    Feb 6, 2023 at 13:38
  • 1
    $\begingroup$ How you regularize (e.g. how much weight decay, how much dropout, how many layers on top of the base network etc.) you'd want to determine by some suitable cross-validation (see fast.ai/posts/2017-11-13-validation-sets.html for what one should amongst other things consider, this depends a lot on the specific situation). $\endgroup$
    – Björn
    Feb 6, 2023 at 13:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.