0
$\begingroup$

Context: A machine learning model will be trained using multiple datasets, and the aim is to see if any dataset is better (more balanced, more representative etc) for training. Further context thanks to @dipetkov's suggestion: we have a costly training process. We are trying to understand if further or mixed training dataset B with dataset A can help, based on high-level summaries (plots or statistics). My assumption here is more data evenly divided across different classes will result in better downstream performance.

Assume I have 5 datasets, and possibly thousands of classes. For example:

datasetA, class1=10 samples,...class 5000=12777 samples
datasetD, class1=5 samples,...class 5000=61 samples

Importantly: sample sizes between classes as well as across datasets for the same class can vary drastically. I take the log of data to deal with this, also open to other suggestions!

I am trying to compare which dataset is superior based on how many samples (more the better) it has. Some questions arise: 1) do we compare raw counts or percentage in dataset, 2) which statistic is reasonable to use? for example using mean per dataset can be very skewed due to outlier classes, and when we use median, it may come up zero because since the number of classes are huge, it is expected most of them to be zero.

I tried below idea: I simply plot log counts across different classes for different datasets in each row (columns are class ids). The problem is, it's very cluttered and not very informative. The best you can make out is saying "first row seems to be the brightest across the board, so let's choose that" etc. Plot:

https://i.sstatic.net/46T7d.jpg

I'm open to any suggestions, using multiple plots, summary statistics, etc as long as I can justify it on a statistical ground :)

Thank you!

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ "More balanced" is not better, especially not if there is a question of a dataset being representative of a population. "More representative" begs the question "more representative of what?, so you would need to compare your datasets to some ground truth or some dataset of known "quality". $\endgroup$
    – Stephan Kolassa
    Commented Oct 26, 2022 at 11:08
  • $\begingroup$ @StephanKolassa we are enforcing balance to remove/mitigate biases present in dataset. the aim is to construct an artificial dataset. we don't have a ground truth reference, since the task is generative and there are no "labels" of truth. if my goal was to identify high number of samples per class better, while penalizing sparsity amongst classes, what would be a good canonical way of visualizing this? $\endgroup$
    – user1696420
    Commented Oct 26, 2022 at 11:41
  • $\begingroup$ Why not combine the datasets into one? $\endgroup$
    – dipetkov
    Commented Oct 26, 2022 at 12:32
  • $\begingroup$ @dipetkov some datasets are not public domain and we are trying to see if adding dataset B adds anything to A (and training is very costly so would like to have some proxy way of at least guessing about it) $\endgroup$
    – user1696420
    Commented Oct 26, 2022 at 14:04
  • $\begingroup$ So you want to pre-determine whether the additional datasets bring something or not, just based on high-level summaries. I think this is an important detail omitted from the body of your question. I would add it if I were you. $\endgroup$
    – dipetkov
    Commented Oct 26, 2022 at 14:18

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.