I am working on finding 3 specific minerals from satellite information.

For each pixel of the map of a 700x700 pixel area, specific number of sensor readings are provided (e.g. 8 readings for 8 frequencies). The quantity of labeled data I have is a bit small i.e. 310 points (these points are actually visited, and the mineral label is determined by a geologist).

Main problem: We cannot say a predicted mine map will be good just because the classification results on test data (i.e. %20 of the 310 data points) have been good (e.g. above %98 accuracy).

I am facing a few issues:

  • While performance on 20% test data set is very good (%98.0 +) with most classifiers (10+ model types like kNN, DT, NB, NNet, SVM also several ensemble models like Bagging, RF, Adaboost, ...) the actual performance on satellite images (i.e. the mines it discovers by classifying pixel of satellite information) is horrible for some of the minerals. The map it provides either does not discover some minerals, or suggests that mineral is all over the place (which is not).

  • The classes are very much unbalanced. One type of mineral has perhaps 30 times coverage than the other type and one other type is very rare in the area.

I am trying to understand the fundamental problems I have. I have identified these:

  1. Data is possibly not enough (sampling theory suggests that for the huge area of 0.5 million pixels, I must have specific number of labeled examples). I must possibly have more labeled data.
  2. Data is unbalanced. I must possibly select balanced number of data from each class.
  3. Some classifiers consider a very small acceptable variance for features (i.e. they will only select a class if the unseen data features are very similar to the labeled samples they have seen, while some others will accept features with larger variance). Some classifiers allow adjusting meta parameters to solve this problem while some others do not have much meta parameters for that.

Could you please:

  • Identify other fundamental problems you see in this project?
  • Suggest solutions
  • Or help me to move into a direction (e.g. devise rules for data gathering) that eventually allow us to test a classifier with a test data set and be sure that the map we get will be mostly correct... (because now we get a very different map with each classifier or meta parameter set, most of which have very good accuracy on test data).

Thank you for your time.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.