For my dataset of ~19K data points to cluster, I want to use a criterion to choose the number of clusters. BIC (Bayesian Information Criterion) gives too few clusters (~180) while AIC (Akaike Information Criterion) gives too many (~1400). Intuitively, I feel that ~500 clusters would be optimal putting ~40 data points in each cluster on average. But apparently, I need to have a statistical explanation for choosing ~500. Is there a way to combine AIC and BIC such that we have neither too few nor too many clusters?
I am not asking about when choosing one of AIC or BIC over the other. I already know that BIC penalizes the number of free parameters much more than AIC, but based on prior information about the data I have, I want to have a penalty which is not as high as BIC's and not as low as AIC's.
I can just select 500 clusters and go ahead, but the reviewers of the submitted papers always need some statistical reason for choosing cluster count, that's actually why I need that.
Here are the formulas that I use for BIC and AIC:
BIC: $-2 \times ln(L) + ln(p) \times k\times n $
AIC: $-2 \times ln(L) + 2\times k\times n$
where
p = the number of data points to cluster
k = the number of clusters
n = the number of dimensions of each data point
L = the likelihood.
