I have 100 datasets. All of them have varying number of features. There are around 20,000 samples in each of them. Every $i$-th sample in the 100 datasets has the same label ($0/1$). The data is highly imbalanced, so there are much fewer positive labels than 0 labels.
I want to give a score/weight to each dataset defining how successful it is in representing the labels. How can I do it?
Build a model on each dataset and report your resampled evaluation metric.
Build it properly, using nested resampling, with an inner tuning loop and an outer testing loop.
In the case of the imbalance, you have to make some choices regarding resampling strategy parameters and stratification (i.e. at each resample at least one $1$ must be tested), evaluation metric (e.g. $AUC$ is more informative than accuracy in this case), etc.
In your case, I would very much like to try a method like The group lasso for logistic regression: collate your 100 data sets into one big data set X of size $20 000 \times p$, where $p$ is the total number of variables. Then apply the group lasso logistic regression on $(X,y)$, $y$ being the vector of classes (0s and 1s), and the groups being the 100 groups of variables.
A posteriori, you could sort the groups according to their importance in the model by considering the $\ell_2$-norm of their coefficients (normalized if the groups have very different sizes).
