I wanted to find possible correlations with the helpfulness score given by the users. For this I used scaled_HS as my dependent variable y. For my variable x I used 63 language features from the LIWC manual. The helpfulness score got scaled by the maximum number of views, to give ratings such as 500 thumbs up and 1000 views a bigger weight than 2 thumbs up and 4 views. Below is a small description of my dataframe:
I wanted to find possible correlations with the helpfulness score given by the users. For this I used scaled_HS as my dependent variable y. For my variable x I used 63 language features from the LIWC manual. The helpfulness score got scaled by the maximum number of views, to give ratings such as 500 thumbs up and 1000 views a bigger weight than 2 thumbs up and 4 views. Below is a small description of my:
I wanted to find possible correlations with the helpfulness score given by the users. For this I used scaled_HS as my dependent variable y. For my variable x I used 63 language features from the LIWC manual. The helpfulness score got scaled by the maximum number of views, to give ratings such as 500 thumbs up and 1000 views a bigger weight than 2 thumbs up and 4 views. Below is a small description of my dataframe:
P-value of negative binomial regression too high?
I am currently working with regression analysis in combination with language analysis on online reviews. I would like to give you a short description of my task and what I want to achieve with it, in order to maybe find a proper solution.
I wanted to find possible correlations with the helpfulness score given by the users. For this I used scaled_HS as my dependent variable y. For my variable x I used 63 language features from the LIWC manual. The helpfulness score got scaled by the maximum number of views, to give ratings such as 500 thumbs up and 1000 views a bigger weight than 2 thumbs up and 4 views. Below is a small description of my:
Rating Spoiler HS WC Tone WPS Dic function. ppron i ... swear netspeak assent nonflu filler QMark Exclam keywords mistakes scaled_HS
0 1 0 0.37 150 25.77 16.67 90.67 46.67 5.33 1.33 ... 0.00 0.00 0.00 0.0 0.0 0.00 4.00 0.000000 0.186667 0.017273
1 1 1 0.16 131 1.55 26.20 89.31 58.02 6.87 0.76 ... 0.00 0.00 0.76 0.0 0.0 3.05 0.00 0.000000 0.137405 0.026339
2 1 0 0.24 133 39.19 33.25 92.48 50.38 3.01 2.26 ... 0.00 0.00 0.00 0.0 0.0 0.00 0.00 0.000000 0.097744 0.044079
After applying simple linear regression with the Pearsonr function I found out that some features were significant but had very low coefficients, except HS but I forgot to delete the column, because I have the scaled HS anyways:
Coefficients p-value
Rating (-0.06201207484873128, 0.015435587631467379)
Spoiler (0.059553066433509855, 0.020030343178780382)
HS (0.4200389620235087, 3.095697207704496e-66)
Review_length (0.0611843321063948, 0.016866533916445086)
WC (0.0629356733799715, 0.013965934404277043)
Tone (-0.07394090951501098, 0.0038637998296509337)
shehe (0.07131791547159314, 0.005331048300074106)
conj (-0.05033621584191379, 0.04937650321109507)
compare (-0.06621954834327438, 0.009690693447463867)
number (0.05913351896038223, 0.02092332389500587)
quant (-0.06552319095849933, 0.010484851019228214)
posemo (-0.05931337090970264, 0.020536361646821617)
negemo (0.05131647584743782, 0.04510762970526194)
male (0.06826773827085715, 0.007656218956918288)
death (0.05074141927289752, 0.047572524702827156)
QMark (0.053348318103051344, 0.03724269305864121)
After checking the mean and std of my dependent variable y I found out that my data is dispersed:
scaled_HS
mean 0.01242697466467958
std 0.02743090432486186
So I thought that a negative binomial regression would fit much better than a linear regression model. After using the statsmodel package I got the following table:
Generalized Linear Model Regression Results
==============================================================================
Dep. Variable: scaled_HS No. Observations: 1525
Model: GLM Df Residuals: 1461
Model Family: NegativeBinomial Df Model: 63
Link Function: log Scale: 1.0000
Method: IRLS Log-Likelihood: -93.260
Date: Tue, 07 Feb 2023 Deviance: 21.239
Time: 07:03:00 Pearson chi2: 33.7
No. Iterations: 6
Covariance Type: nonrobust
===============================================================================
coef std err z P>|z| [0.025 0.975]
-------------------------------------------------------------------------------
Rating -0.0589 0.109 -0.540 0.589 -0.273 0.155
Spoiler -0.0572 0.593 -0.096 0.923 -1.220 1.106
HS 4.5688 1.316 3.471 0.001 1.989 7.149
WC 5.122e-05 0.001 0.036 0.972 -0.003 0.003
Tone -0.0018 0.016 -0.114 0.909 -0.033 0.029
WPS -0.0028 0.010 -0.272 0.785 -0.023 0.017
Dic -0.0205 0.075 -0.272 0.786 -0.169 0.127
function. -0.0671 0.101 -0.661 0.509 -0.266 0.132
ppron -0.0324 0.388 -0.084 0.933 -0.793 0.728
i 0.0527 0.393 0.134 0.893 -0.718 0.823
you 0.1103 0.466 0.237 0.813 -0.802 1.023
shehe 0.1647 0.491 0.335 0.737 -0.798 1.127
they 0.0642 0.457 0.141 0.888 -0.831 0.960
ipron 0.0665 0.123 0.540 0.590 -0.175 0.308
adverb 0.0802 0.124 0.647 0.518 -0.163 0.323
conj -0.0281 0.118 -0.238 0.812 -0.259 0.203
negate 0.0495 0.238 0.209 0.835 -0.416 0.515
adj 0.0353 0.150 0.235 0.814 -0.258 0.329
compare -0.0563 0.189 -0.298 0.766 -0.426 0.314
interrog -0.0231 0.253 -0.091 0.927 -0.518 0.472
number 0.0035 0.161 0.022 0.983 -0.312 0.319
quant -0.1101 0.176 -0.626 0.531 -0.455 0.235
affect -0.0286 1.014 -0.028 0.978 -2.015 1.958
posemo -0.0059 1.008 -0.006 0.995 -1.981 1.970
negemo 0.0553 1.046 0.053 0.958 -1.996 2.106
anx -0.0961 0.517 -0.186 0.853 -1.109 0.917
anger -0.0627 0.397 -0.158 0.875 -0.841 0.716
sad -0.2089 0.463 -0.451 0.652 -1.116 0.698
family 0.0265 0.468 0.057 0.955 -0.891 0.944
friend -0.3726 0.875 -0.426 0.670 -2.087 1.341
female -0.0805 0.347 -0.232 0.817 -0.761 0.600
male -0.0277 0.313 -0.088 0.930 -0.642 0.587
insight -0.0469 0.207 -0.226 0.821 -0.453 0.359
cause -0.1160 0.221 -0.525 0.600 -0.549 0.317
discrep -0.0251 0.241 -0.104 0.917 -0.497 0.447
tentat -0.0234 0.179 -0.131 0.896 -0.375 0.328
certain -0.0242 0.205 -0.118 0.906 -0.425 0.377
differ -0.0384 0.166 -0.231 0.817 -0.364 0.287
see -0.0170 0.188 -0.090 0.928 -0.386 0.352
feel -0.0078 0.325 -0.024 0.981 -0.645 0.630
bio -0.0860 0.293 -0.294 0.769 -0.660 0.488
sexual 0.1453 0.695 0.209 0.834 -1.217 1.507
affiliation 0.0709 0.265 0.267 0.789 -0.448 0.590
achieve 0.0106 0.205 0.052 0.959 -0.390 0.412
power -0.0958 0.175 -0.547 0.584 -0.439 0.247
reward 0.0172 0.176 0.098 0.922 -0.328 0.363
risk 0.0496 0.412 0.120 0.904 -0.757 0.857
focuspast 0.0047 0.117 0.040 0.968 -0.224 0.234
focusfuture 0.0510 0.201 0.254 0.800 -0.343 0.445
time -0.0120 0.107 -0.112 0.911 -0.221 0.197
home 0.0394 0.399 0.099 0.921 -0.742 0.820
money -0.0196 0.541 -0.036 0.971 -1.080 1.041
relig 0.0978 0.749 0.131 0.896 -1.369 1.565
death 0.0169 0.480 0.035 0.972 -0.924 0.958
informal 0.0847 1.507 0.056 0.955 -2.869 3.038
swear -0.1255 1.613 -0.078 0.938 -3.288 3.037
netspeak -0.0516 1.524 -0.034 0.973 -3.038 2.935
assent 0.0018 1.562 0.001 0.999 -3.060 3.064
nonflu -0.1514 1.590 -0.095 0.924 -3.268 2.965
filler -0.2321 1.896 -0.122 0.903 -3.948 3.484
QMark 0.0523 0.256 0.204 0.838 -0.449 0.554
Exclam 0.0518 0.149 0.349 0.727 -0.240 0.343
keywords -2.5026 26.795 -0.093 0.926 -55.019 50.014
mistakes -4.1954 7.054 -0.595 0.552 -18.020 9.629
My question would be: Why is this regression model showing poorer findings than the linear regression model? My y variable is in range of 0 and 1 and the std > mean. Thank you for your help!
lang-py