1
$\begingroup$

I have a vector of information entropy values that range between 0 and 1 which I want to explain with some explanatory variables.

I realized that the distribution of the entropy values in my dataset is not really suitable for ordinary linear regression, as the distribution is highly negatively skewed, the majority of observations are close to 1:

enter image description here

It's also a problem that the range is limited [0,1].

So, I found out that I have a couple of options how to deal with these issues:

  • (1) Logit transform and run linear regression
  • (2) Binomial/logit GLM, or
  • (3) Beta regression.

My question pertains to the first option:

My entropy value was calculated from a binary case, thus it reduces to: $$ H(x) = -p(x_1)\log_2p(x_1)+(1-p(x_1))\log_2(1-p(x_1)) $$ It seems somehow redundant to apply a logit transformation on it which would result in $\log(H(x)/(1-H(x)))$.

I still ran my linear model with this logit transformed entropy variable, but I feel like there is a relation between these two (entropy and logit) so that I can just take a shortcut instead of having the double log, but I'm struggling to figure out how to further simplify it.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

I think the usual approach would be to predict p, the probability, using a logistic link, which is normal logistic regression. By calculating this entropy based on p, you end up with a transformed value which will probably indeed needs some additional transformations before it works well in a regression. Do you have any particular reason you want to apply this entropy transform first? Otherwise, I think logistic regression is the tried and logical approach here.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I have also tried a logistic GLM and then it made me think why I would even need to calculate entropy values to begin with if I could just use the raw count of successes (So say my p = successes/overall trials) which yielded significant results but I wanted an analysis that would incorporate and predict actual entropy values. I guess I can eventually take my predicted p's and calculate the entropy values in the very end of the glm.. So, I assume the logit transform approach is garbage and logistic GLM is most reasonale $\endgroup$
    – Maria
    Commented Jan 11, 2021 at 13:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.