I am trying to evaluate a novel dimensionality reduction technique. Specifically, we start with a data set with around 1,000 features/covariates per observation. My technique maps this down to 12. Doing this causes a small drop-off in model (regularized logistic regression) performance.

My informal hypothesis is that my technique is capturing "most of the information" in the original data. I'd like to quantify this rigorously so I can test it. Unlike PCA or similar things I don't have access to a rigorous mathematical theory to show this so I'm looking for an empirical measure.

At present, I have two ideas. The first is a ratio of the Bayesian Information Criterion values. Specifically, let $f$ be the model trained on the data with the original observations, and let $\tilde{f}$ be that trained on the dimensionally reduced data. Then I'd look at $$\frac{BIC(\tilde{f})}{BIC(f)}.$$

The glaring issue I can see here is that the value of $k$, corresponding to the number of features (see the link above), differs across models so these aren't directly comparable.

The second idea is to compare the change in log likelihood over the increase in parameters. Suppose the full data set has $n$ features per observation and the reduced one has $m<<n$ features. Let $L_n$, $L_m$ be the log likelihoods of the models fit on these respective datasets. I was considering looking at: $$ \frac{L_n-L_m}{n-m} $$

As my background is more machine learning and biology than statistics proper, I'm sorry if this is a well known topic.


1 Answer 1


It seems that both your suggestions boil down to the same idea: assessing whether the loss of informativeness (as measured by the likelihood of a model) is compensated by the increased simplicity (as measured by the number of features).

As mentioned in a previous answer, the BIC contains two terms:

  1. A likelihood term, which measures the ability of the model to explain the observed data;
  2. A regularization term, which penalizes the complexity of the model (i.e. its number of free parameters).

The point of the latter term is to avoid overfitting and to allow that the selected model will generalize well to unobserved data, i.e. to avoid that the more complicated model will be necessarily selected. This can be interpreted as a form of Occam's razor: if two models explain the data equally well, the simpler one should be favored.

The point of the BIC is thus precisely to be able to compare models having different numbers of features. The likelihood of $f$ will be lower than the likelihood for $\tilde{f}$ (since you said that your method causes a small drop-off in model performance); but the penalty term $k \log T$ (where $k$ is the number of features and $T$ the number of samples) will be equal to $n \log T$ for $f$ and $m \log T$ for $\tilde{f}$. You can thus perfectly compare $BIC(f)$ and $BIC(\tilde{f})$ (and different rules of thumb for doing so exist).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.