I have a vector of information entropy values that range between 0 and 1 which I want to explain with some explanatory variables.
I realized that the distribution of the entropy values in my dataset is not really suitable for ordinary linear regression, as the distribution is highly negatively skewed, the majority of observations are close to 1:
It's also a problem that the range is limited [0,1].
So, I found out that I have a couple of options how to deal with these issues:
- (1) Logit transform and run linear regression
- (2) Binomial/logit GLM, or
- (3) Beta regression.
My question pertains to the first option:
My entropy value was calculated from a binary case, thus it reduces to: $$ H(x) = -p(x_1)\log_2p(x_1)+(1-p(x_1))\log_2(1-p(x_1)) $$ It seems somehow redundant to apply a logit transformation on it which would result in $\log(H(x)/(1-H(x)))$.
I still ran my linear model with this logit transformed entropy variable, but I feel like there is a relation between these two (entropy and logit) so that I can just take a shortcut instead of having the double log, but I'm struggling to figure out how to further simplify it.