1
$\begingroup$

I'm currently developing neural networks to classify medical images. One dataset contains 2 classes of images (Images which can be used and cannot used). So my model will check whether given image can be used or not.This is a binary classification.

Likewise, I have 12 datasets (Each has 2 class images) to classify. All these 12 datasets are similar but not identical. So I thought to make 12 neural networks to perform classifications.

But one model might be 2GB in size and having 12 models will arise a storage problem. So I need to find out minimum number of models should I train to evaluate all these 12 datasets.

What I did

I already trained one neural network for one dataset and use that model to evaluate rest (11 datasets). So, I got 95% of prediction accuracy for 5 datasets and 6 datasets with lower than 90% accuracy.

Since I'm dealing with medical images, accuracy of classification should be high like 98%. So I'd like to know the statistical approach to find out the best possible minimum number of models for this? Can I use methods like ROC ?

Improve this question
$\endgroup$
7
  • $\begingroup$ What's different about the datasets that you would fit a separate model for each? Also, if your goal is to classify images from new patients, it seems you would ultimately need a way to generate a single prediction. $\endgroup$
    – user20160
    Commented Aug 3, 2018 at 2:29
  • $\begingroup$ @user20160 It's 12 Leads ECG images. So these 12 sets are not identical but kind of similar. I need to classify these leads data separately. That's why I need 12 classifiers of less than 12. After 12 leads evaluation, single prediction will be generated. $\endgroup$
    – Samitha Nanayakkara
    Commented Aug 3, 2018 at 8:48
  • $\begingroup$ I'm not very familiar with ECG; isn't it true that the 12 channels are recorded simultaneously? If so, why not treat everything as a 12-dimensional time series, for which you can fit a single classifier, rather than a separate classifier for each channel? $\endgroup$
    – user20160
    Commented Aug 3, 2018 at 9:00
  • $\begingroup$ @user20160 Ah yeah they are. But I have to make several classifiers in this situation since I have images. So making several classifiers is a must. $\endgroup$
    – Samitha Nanayakkara
    Commented Aug 3, 2018 at 9:08
  • $\begingroup$ Could you say more about what an 'image' is in this context and how they're related (I thought each channel was voltage over time)? Something like a time-frequency transform? If the channels are simultaneously recorded, it seems the images would represent different pieces of information about a common, underlying physical process. If that's the case, then perhaps it's possible to use all images as simultaneous inputs to a single classifier that, in principle, could be more powerful than separate classifiers by exploiting dependencies between images. $\endgroup$
    – user20160
    Commented Aug 3, 2018 at 9:15

2 Answers 2

Reset to default
0
$\begingroup$

How about you attack this problem in a hierarchical fashion. Say you train one network on the images. It gets around 95%, but you want atleast 98%. Then, what I would try to do, is to take that 5% that was misclassified, and then create a network that is trained on that data alone. Hopefully, this will further cut down on the number of misclassifications (although probably not as good at 95% accuracy). Now, we need a way to tell if the image belongs in the former class or the latter class. There are a few ways to do this, but as a starting point, you could train a third nerual net to find the best allocation of images to maximize accuracy. Note I don't mean to train all 3 simultaneously, as that would probably be very expensive, but to train them sequentially. Or if you can find common features between the classes, you could build something like a bayesian net which determines the best allocation.

Improve this answer
$\endgroup$
3
  • $\begingroup$ This method doesn’t seem like will solve my problem. I have 12 datasets and I need n number of classifers to classify data inside those 12 datasets rather than training 12 classifiers. n<12 $\endgroup$
    – Samitha Nanayakkara
    Commented Aug 3, 2018 at 6:49
  • $\begingroup$ So throw all your images into one data set, (or take some percentage of randomly chosen images if training takes too long), train a net on that. If that hits 98% then good. If not, take all that didnt classify correctly, and then train a net just on those, and then add a net to choose which classifier an image goes into, and keep adding nets till you have reached your desired accuracy. In the case above, I have made 2 (3 if you count the hieractical one) classifiers. Thats less than 12. $\endgroup$
    – Anonymous Emu
    Commented Aug 3, 2018 at 20:25
  • $\begingroup$ I guess what I meant by this, is like what people are saying above, the fewest classifiers you need isn't a well defined concept. The closest we have is the fewest I had to use to get the level of accuracy I wanted. $\endgroup$
    – Anonymous Emu
    Commented Aug 3, 2018 at 20:37
0
$\begingroup$

There isn't really anything to prove as having "best possible minimum number of division" is a matter taste. It mostly depending on the assumptions you are willing to make.

There is a tradeoff between the amount of models and individual model accuracy. This is due to the approximation-estimation error tradeoff discussed later.

Practically:

  • Decide ahead of time on possible divisions of the datasets.
  • Choose between the divisions by cross validation.
  • A model for a division will be composed of submodels, one for each group in the division. Each submodel is trained using all training data in all the datasets of the group.

Approximation-Estimation Error Tradeoff

In general, since you are mostly interested in a specific model, I think the right way to consider your performance is by the standard decomposition of the generalization error into approximation and estimation: $$ Err_{gen} = Err_{app} + Err_{est} $$ Where $Err_{app}$, or approximation error, is the lower error possible in the model, as no one knows if your model is able to actually represent the data. The estimation error second error is how close can you get to the best model.

You can lower $Err_{app}$ by choosing a better model. Your intuition did take this error into account, as you tried to make the model able to capture all the data.

However, dividing the data into smaller dataset, each for a different algorithm, will cause each algorithm to be trained by less data. Usually, the probability to achieve a bad classifier, with estimation error bigger than a fixed value, shrinks exponentially with the amount of data.

So we have a tradeoff here - the more models, the less approximation error, but bigger estimation error (as we train on less data). While we could calculate the VC dimension of the different models, we can't know ${Err}_{app}$, unless we have infinite data. Thus, it seems we can't calculate which division minimizes ${Err}_{gen}$.

However, we can empirically decide which amount seems to gives the best classifier by cross validation. The possible divisions should be thought of ahead of time, as adaptive data analysis is dangerous. Finally, note that if you plan to compare many divisions, we should ensure we have enough data - either by using bootstrap or by gather more data (better).

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you for your reply. I have 30,000 data from each set. Isn’t that enough? Also I’d like to have more than 95% of accuracy for all models. So what kind of assumptions I have to make? Also since I want to make minimum number of models (possible) how can I statistically prove x% of accuracy is enough? Cut off accuracy? $\endgroup$
    – Samitha Nanayakkara
    Commented Aug 3, 2018 at 6:32
  • $\begingroup$ Look, in order to "prove" what is the best combination, you need to know what will minimize ${Err}_{app}$. In order to do so, you need to assume something about the distribution from which the data is generated (like smoothness of density function, or a parametric family). However, you realy have enough samples so your empirical estimates will be close enough to the true probability. So the right way to compare the models would be to calculate their error using cross validation. $\endgroup$
    – tmrlvi
    Commented Aug 3, 2018 at 9:26
  • $\begingroup$ Also, always remember that you may need a different model altogether. Maybe your neural network architecture is not the right architecture. $\endgroup$
    – tmrlvi
    Commented Aug 3, 2018 at 9:27
  • $\begingroup$ I’m not quite familiar with that solution. Can youprovide little bit more information on how can I do that with those data? Thank you. $\endgroup$
    – Samitha Nanayakkara
    Commented Aug 3, 2018 at 9:28
  • $\begingroup$ In cross validation, you would divide each of the dataset to $K$ groups. Now, you train a model (containing whatever submodels you want) on data composed of $K-1$ groups from all the datasets. You measure the performance of the model on the group left out. You do it $K$ times (each time leaving out a different set), and report back the average of the performance. You can find more information about the procedure here $\endgroup$
    – tmrlvi
    Commented Aug 3, 2018 at 9:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.