Original problem formulation - 12/04/2019:

I have a data set of 14 observations with six features each and an output variable of two classes. I applied a logistic regression model with the following workflow:

  1. Do not specify any hyperparameters, thus have a first degree decision boundary and a regularization parameter of zero.

  2. Split the data set into a training set (8 observations) and a test set (6 observations).

  3. Fit a model on the training set and calculate the test error from test set (by averaging over +-1000 random lists)
  4. Plot the learning curves and observe that we have a high variance/low bias issue
  5. Conclude that the model complexity cannot be reduced since the decision boundary is already of the first degree and thus I can only examine the effect of removing certain features or try to get a larger data set (which is not possible in the foreseeable future)

The questions if have are the following:

  • Can I conclude 5. from my previous points?
  • Is it correct to say that I am calculating the test error (and not approximating it) as I did not have the need to finetune any hyperparameters and thus my test set still qualifies as truly unseen?
  • Is it strange to have a model of such low complexity and then still have a high variance/low bias error type?
  • Assuming the second bullet point is correct, can I apply LOOCV (leave one out cross validation) to calculate the test error (or am I then approximating the test error again)?

The following was added after reading the answer by EdM - 14/04/2019

Model adjustments made: Following the comments by EdM and by Reading up on "An introduction to statistical learning". I realized that I need to incorperate some sort of feature shrinkage/selection, as I am working with a high-dimensionality problem. This was done by applying ridge-regression in combination with the Leave-One-Out Cross-Validation method (feature standardization was also applied). The graph below indicates thus the error on the cross validation set for different k-order polynomial decision boundaries and different values for the tuning parameter. The higher order-polynomials are given as an informative feature, as I don't really want to use these because it would only enlarge my high-dimensionality problem (as also indicated by EdM above).

note: I am not trying to convince a skeptical reader with my model. The goal is more to provide a proof-of-concept of application of Machine Learning in my research field.

Question: Ridge aregression still indicates that with a tuning parameter (lambda) of zero and a first degree factorization (k), the lowest error on the CV set is obtaind. This would thus effectively reduce the problem again to a least squares fit. Which, as I understood from the literature, cannot be used due to the high dimensionality. Am I missing something here or should I just do what the graph indicates and stick with the least squares fit?

enter image description here


Although you might think that a model with only 6 features is low in complexity, it's too complex for the data at hand. You do not have enough data to fit a logistic regression model reliably with 6 unpenalized features.

The high variance/low bias behavior you observe on multiple random train/test splits is what is expected with an overfitted model: too many parameter values estimated for the number of cases available. The usual rule of thumb for avoiding overfitting in unpenalized logistic regression is to have 10-20 cases of the least prevalent class per feature considered. With only 14 cases total you have no more than 7 in the least prevalent class, so you would be hard pressed to evaluate even a single unpenalized feature reliably.

If you were nevertheless to go on to investigate transformations of predictors (e.g., splines or other polynomial fits) to try to improve performance, you would be compounding your problems. In applying that rule of thumb for avoiding overfitting, you need to count each additional estimated parameter as an additional feature. So without additional precautions you would be heading even further down the overfitting path. As John von Neumann is alleged to have said: "With four parameters I can fit an elephant, and with five I can make him wiggle his trunk."

One possible way around this problem would be to use a penalized approach like ridge regression or LASSO. These approaches deliberately trade off increased bias for reduced variance by lowering the magnitudes of regression coefficients or (in LASSO) removing some features completely. That could, in principle, be combined with evaluation of polynomial or other transformations of features. With only 14 cases total, however, it would be difficult to convince a skeptical reader that your conclusions were correct.

With respect to hyperparameter choices, remember that standard unpenalized logistic regression has no hyperparameters to choose. It provides a model of class-membership probabilities. You only produce a decision boundary when you impose a probability cutoff for predicting class membership, based on some decision about relative costs and benefits of different misclassifications. That choice of cutoff might be considered a hyperparameter, but that comes after the logistic regression per se.

You do have hyperparameter choices to make if you choose the complexity of feature transformations by cross-validation or if you use penalization. In either case, however, the predictor function provided by the logistic regression model will still be linear in the coefficients of the features, however the features were transformed and however the coefficients were penalized. So the nature of the ultimate decision boundary remains the same as for standard logistic regression even if its location, the dimensions of its feature space, or its direction in feature space are different. Is that still a "first degree decision boundary" if some of the features have been non-linearly transformed from their originally observed values? I wouldn't worry too much about that terminology.

See section 4.2 of Harrell's Regression Modeling Strategies or of his associated class notes for guidance on cases per feature in different types of regressions. Chapters 6 and 7 of An Introduction to Statistical Learning provide one introduction to penalized methods. See Chapter 7 of The Elements of Statistical Learning for extensive discussion of what types of error you can estimate with these approaches even if you had more data. An important and difficult question is how well your "test error" would generalize to a new sample from the same underlying population.

Where to go from here

You want to "provide a proof-of-concept of application of Machine Learning in my research field." So you have to set up tests that will fairly represent how machine learning might be applied.

Note that with 14 cases and 6 features it is possible to include all predictors in an unpenalized linear model. The theoretical impossibility of an unpenalized linear model arises when the number of features exceeds the number of cases. So your model including all features is not violating a strict "high dimensionality" cutoff, although the unpenalized model will probably be overfit.

Also, there are 3 potential problems in the way you are performing cross validation.

First, leave-one-out cross validation is not always a good choice, as discussed in detail on this page.

Second you seem to be using classification error as the measure of model quality. If so, that is probably not wise as misclassification error depends on your particular choice of a cutoff for the probabilities returned by your logistic regression model. A proper scoring rule like the Brier score is probably a better choice. It's best to first optimize the probability predictions, then apply whatever cutoff makes sense for classification (if classification is necessary). Although the Brier score is the mean-square error between predicted probabilities from the logistic model and the actual (0,1) values, note that no logistic regression is a "least squares fit," as a logistic model has no closed-form solution and the fit is based on maximum likelihood.

Third, it's not clear whether you standardized your continuous features before the ridge regression analysis. As the ridge penalty is applied to the sum of squares of all coefficients, without standardization you get different results if a predictor based on length, say, is measured in miles versus millimeters. (Some software does this automatically for you, then transforms coefficients back to the original scales.)

When studies in your field involve such a low ratio of cases to predictors, then there is not much hope of doing substantially better than what you already have done. As you might expect, the more complex models tend to require more penalization and it's not really clear that any of them will outperform the standard unpenalized (and likely overfit) model even if you use a proper scoring rule like the Brier score.

You could, however, consider using your knowledge of the subject matter to design simulations of data sets (with appropriate choices of coefficients and error terms) to evaluate how machine learning techniques might be of value in larger-scale studies.

  • $\begingroup$ Hi EdM, thanks for the thorough response. a) Do you have a reference for that rule of thumb I can read up on? b) Just hypothetically, If i had enough observations. Would the conclusion I make in point 5 be correct (that researching the effect of polynomial features and a regularization parameter does not make sense when a high variance error is estblished in combination with a first degree decision boundary and regularization parameter equal to zero). c) does a regularization parameter (different from zero) has any effect in combination with first a degree decision boundary? $\endgroup$ – Arjan Dexters Apr 12 '19 at 9:20
  • $\begingroup$ @ArjanDexters I have edited my answer to address those issues. $\endgroup$ – EdM Apr 12 '19 at 15:41
  • $\begingroup$ Thanks again, I have edited my question... $\endgroup$ – Arjan Dexters Apr 14 '19 at 8:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.