# How can I rigorously quantify the increase performance due to additional parameters?

I am trying to evaluate a novel dimensionality reduction technique. Specifically, we start with a data set with around 1,000 features/covariates per observation. My technique maps this down to 12. Doing this causes a small drop-off in model (regularized logistic regression) performance.

My informal hypothesis is that my technique is capturing "most of the information" in the original data. I'd like to quantify this rigorously so I can test it. Unlike PCA or similar things I don't have access to a rigorous mathematical theory to show this so I'm looking for an empirical measure.

At present, I have two ideas. The first is a ratio of the Bayesian Information Criterion values. Specifically, let $$f$$ be the model trained on the data with the original observations, and let $$\tilde{f}$$ be that trained on the dimensionally reduced data. Then I'd look at $$\frac{BIC(\tilde{f})}{BIC(f)}.$$

The glaring issue I can see here is that the value of $$k$$, corresponding to the number of features (see the link above), differs across models so these aren't directly comparable.

The second idea is to compare the change in log likelihood over the increase in parameters. Suppose the full data set has $$n$$ features per observation and the reduced one has $$m< features. Let $$L_n$$, $$L_m$$ be the log likelihoods of the models fit on these respective datasets. I was considering looking at: $$\frac{L_n-L_m}{n-m}$$

As my background is more machine learning and biology than statistics proper, I'm sorry if this is a well known topic.

The point of the BIC is thus precisely to be able to compare models having different numbers of features. The likelihood of $$f$$ will be lower than the likelihood for $$\tilde{f}$$ (since you said that your method causes a small drop-off in model performance); but the penalty term $$k \log T$$ (where $$k$$ is the number of features and $$T$$ the number of samples) will be equal to $$n \log T$$ for $$f$$ and $$m \log T$$ for $$\tilde{f}$$. You can thus perfectly compare $$BIC(f)$$ and $$BIC(\tilde{f})$$ (and different rules of thumb for doing so exist).