Logistic regression models the relationship between a set of independent variables and the probability that a case is a member of one of the categories of the dependent variable. If the probability is greater than 0.5, the case is classified in the modeled category. If the probability is less than 0.50, the case is classified in the other category. The problem is that when I run the model with my dataset, the probabilities are far from 0.5, in fact it never gets to that value.
Here is part of My dataset:
sum_profit direction profit_cl1 10 up 0.00 0 Not_up -0.03 -5 Not_up 0.04 -5 Not_up -0.04
I want to find a relationship between the price of oil and the stock price of a Colombian oil company. So the variable 'sum_profit' is the sum of the change in the stock price in the next ten minutes. The variable 'profit_cl1' shows me the net change in the oil price in the last 10 minutes.
So what I want to know is that if the oil price changes in the last 10 minutes how would I expect the stock price direction to be in the following 10 minutes (Up or Down).
The problem is that my probabilities once I run the logistic regression are far from 0.5 even though the model is significant
glm.fit=glm(formula = direction ~ profit_cl1, family = binomial, data = datos) Deviance Residuals: Min 1Q Median 3Q Max -0.6786 -0.6786 -0.6131 -0.6131 1.8783 Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) -1.57612 0.01618 -97.394 <2e-16 *** profit_cl1 0.22485 0.02288 9.829 <2e-16 *** --- Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 48530 on 50309 degrees of freedom Residual deviance: 48434 on 50308 degrees of freedom AIC: 48438 Number of Fisher Scoring iterations: 4
The code to get the probabilities:
log.probs=predict(glm.fit, type="response") mean(log.probs)=0.1873
the 0.1873 is very far from 0.5.
Sorry but I did not know where else to look for help! I appreciate any suggestion!