Averaging model Coefficients in binary logistic regression

Question

I am running a binary logistic regression to estimate the probability for frost occurrences of a frost event given a set of explanatory variables. I intent to carry an accuracy assessment and therefore have divided my data into two proportions. 75% of it for training the model, and 25% for testing it. The thing is, every time I select a random sample of 75% to train the model, I get different estimates of coefficients. I therefore decided to run several iterations and average the coefficients to come up with final coefficient estimates. My question is; Is this kind of averaging acceptable in statistics? and what else might I do? any direction to relevant literature of these is appreciated.

Answer 1

Firstly, each time you randomly sample 75% of your data, you are selecting a different set of observations. You are therefore training a model on a different dataset each time, which is why your model coefficients are different each time. This is normal. They will vary more greatly if you have a small dataset to begin with. If you have a very large dataset, there will be very little variation between random samples.

The purpose of splitting the model is not directly to estimate model coefficients, but to examine the ability of the specified model trained on the training data to accurately predict frost events (or estimate the probability of frost events) in the testing data. This is most commonly used to ensure your specified model is not over-fitting the data. You would typically use the AUC statistic to check for this in logistic regression (see What does AUC stand for and what is it?). By 'specified model', I mean the choice of predictor variables, interactions, non-linear forms, etc.

If you split only once, an implicit choice has been made to ignore the variation induced by randomly splitting 75/25. If you don't want to ignore this, then you can split repeatedly using cross-validation - see https://en.wikipedia.org/wiki/Cross-validation_(statistics). Cross-validation can be done a number of ways. "Repeated random sub-sampling validation" is the method that you have already intuitively considered! You would then want to average the values of the AUC (or whichever model-checking statistic you are using, but NOT the values of the coefficients) over these repeated splits to get your overall estimate of model validity.

Incidentally, you'll find that if you repeat this random sampling process enough times, the average of the model coefficients will approach the values you get if you train a model using the whole dataset. This suggests that if you are a-priori happy with your specified model (eg you are happy there is no over-fitting), then you should just use the complete dataset in the first place without splitting.

Answer 2

I would not recommend to average the coefficient cross different models.

Every time we build a model, we are optimizing the objective (for example, classification accuracy) on a given specific data set. If data set changes, the model changes. Averaging across different models will make the optimization on invalid. In fact, there are some "averaging method", and I will discuss it later. But averaging coefficient on a "high bias model" like logistic regression (linear model usually has high bias and low variance in terms of bias variance trade off) may not be a good idea..

Note, even adding one data point to training data set may cause the coefficients change. But the idea is, the 75% of the data is representative enough, that the coefficients will "not change too much".

For example, think about a toy example, that we want to use a person's weight to classify it is a male or female. Day one, you select 75% of the data and get the coefficient is $0.123$, and Day two you get another set of training data, and the coefficient is $0.1232$, there are some differences, but not a dramatic changes. The key is we are capturing the physical relationship between weight and gender. A very small variations on the coefficient is normal. Using any of the coefficient is good and will not make too much difference.

---

On the other hand, there are some cases, averaging models (not averaging coefficients but doing something like majority vote) will improve the model. Those methods called "ensemble methods", and the idea is trying to averaging many over-fitted models to reduce the "variance" of the model. But logistic regression is less likely to over-fitting, comparing to neural network or decision tree with many splits.

Answer 3

I don't think averaging the coefficients make any sense.

But why would you want to do it? If you want to build multiple logistic models, you can just build them, use the predicted probabilities to make classification and then average the classified decision.

If you are talking about cross-validation, each individual model is expected to be different (otherwise CV is pointless). You would use your cross-validation results to estimate your out-of-sample errors. Again, there is no need to average the coefficients.