1
$\begingroup$

I am currently working with regression analysis in combination with language analysis on online reviews. I would like to give you a short description of my task and what I want to achieve with it, in order to maybe find a proper solution.

I wanted to find possible correlations with the helpfulness score given by the users. For this I used scaled_HS as my dependent variable y. For my variable x I used 63 language features from the LIWC manual. The helpfulness score got scaled by the maximum number of views, to give ratings such as 500 thumbs up and 1000 views a bigger weight than 2 thumbs up and 4 views. Below is a small description of my dataframe:

Rating  Spoiler     HS  WC  Tone    WPS     Dic     function.   ppron   i   ...     swear   netspeak    assent  nonflu  filler  QMark   Exclam  keywords    mistakes    scaled_HS
0   1   0   0.37    150     25.77   16.67   90.67   46.67   5.33    1.33    ...     0.00    0.00    0.00    0.0     0.0     0.00    4.00    0.000000    0.186667    0.017273
1   1   1   0.16    131     1.55    26.20   89.31   58.02   6.87    0.76    ...     0.00    0.00    0.76    0.0     0.0     3.05    0.00    0.000000    0.137405    0.026339
2   1   0   0.24    133     39.19   33.25   92.48   50.38   3.01    2.26    ...     0.00    0.00    0.00    0.0     0.0     0.00    0.00    0.000000    0.097744    0.044079

After applying simple linear regression with the Pearsonr function I found out that some features were significant but had very low coefficients, except HS but I forgot to delete the column, because I have the scaled HS anyways:

       Coefficients            p-value
Rating (-0.06201207484873128, 0.015435587631467379)
Spoiler (0.059553066433509855, 0.020030343178780382)
HS (0.4200389620235087, 3.095697207704496e-66)
Review_length (0.0611843321063948, 0.016866533916445086)
WC (0.0629356733799715, 0.013965934404277043)
Tone (-0.07394090951501098, 0.0038637998296509337)
shehe (0.07131791547159314, 0.005331048300074106)
conj (-0.05033621584191379, 0.04937650321109507)
compare (-0.06621954834327438, 0.009690693447463867)
number (0.05913351896038223, 0.02092332389500587)
quant (-0.06552319095849933, 0.010484851019228214)
posemo (-0.05931337090970264, 0.020536361646821617)
negemo (0.05131647584743782, 0.04510762970526194)
male (0.06826773827085715, 0.007656218956918288)
death (0.05074141927289752, 0.047572524702827156)
QMark (0.053348318103051344, 0.03724269305864121)

After checking the mean and std of my dependent variable y I found out that my data is dispersed:

    scaled_HS
mean 0.01242697466467958
std  0.02743090432486186

So I thought that a negative binomial regression would fit much better than a linear regression model. After using the statsmodel package I got the following table:

                 Generalized Linear Model Regression Results                  
==============================================================================
Dep. Variable:              scaled_HS   No. Observations:                 1525
Model:                            GLM   Df Residuals:                     1461
Model Family:        NegativeBinomial   Df Model:                           63
Link Function:                    log   Scale:                          1.0000
Method:                          IRLS   Log-Likelihood:                -93.260
Date:                Tue, 07 Feb 2023   Deviance:                       21.239
Time:                        07:03:00   Pearson chi2:                     33.7
No. Iterations:                     6                                         
Covariance Type:            nonrobust                                         
===============================================================================
                  coef    std err          z      P>|z|      [0.025      0.975]
-------------------------------------------------------------------------------
Rating         -0.0589      0.109     -0.540      0.589      -0.273       0.155
Spoiler        -0.0572      0.593     -0.096      0.923      -1.220       1.106
HS              4.5688      1.316      3.471      0.001       1.989       7.149
WC           5.122e-05      0.001      0.036      0.972      -0.003       0.003
Tone           -0.0018      0.016     -0.114      0.909      -0.033       0.029
WPS            -0.0028      0.010     -0.272      0.785      -0.023       0.017
Dic            -0.0205      0.075     -0.272      0.786      -0.169       0.127
function.      -0.0671      0.101     -0.661      0.509      -0.266       0.132
ppron          -0.0324      0.388     -0.084      0.933      -0.793       0.728
i               0.0527      0.393      0.134      0.893      -0.718       0.823
you             0.1103      0.466      0.237      0.813      -0.802       1.023
shehe           0.1647      0.491      0.335      0.737      -0.798       1.127
they            0.0642      0.457      0.141      0.888      -0.831       0.960
ipron           0.0665      0.123      0.540      0.590      -0.175       0.308
adverb          0.0802      0.124      0.647      0.518      -0.163       0.323
conj           -0.0281      0.118     -0.238      0.812      -0.259       0.203
negate          0.0495      0.238      0.209      0.835      -0.416       0.515
adj             0.0353      0.150      0.235      0.814      -0.258       0.329
compare        -0.0563      0.189     -0.298      0.766      -0.426       0.314
interrog       -0.0231      0.253     -0.091      0.927      -0.518       0.472
number          0.0035      0.161      0.022      0.983      -0.312       0.319
quant          -0.1101      0.176     -0.626      0.531      -0.455       0.235
affect         -0.0286      1.014     -0.028      0.978      -2.015       1.958
posemo         -0.0059      1.008     -0.006      0.995      -1.981       1.970
negemo          0.0553      1.046      0.053      0.958      -1.996       2.106
anx            -0.0961      0.517     -0.186      0.853      -1.109       0.917
anger          -0.0627      0.397     -0.158      0.875      -0.841       0.716
sad            -0.2089      0.463     -0.451      0.652      -1.116       0.698
family          0.0265      0.468      0.057      0.955      -0.891       0.944
friend         -0.3726      0.875     -0.426      0.670      -2.087       1.341
female         -0.0805      0.347     -0.232      0.817      -0.761       0.600
male           -0.0277      0.313     -0.088      0.930      -0.642       0.587
insight        -0.0469      0.207     -0.226      0.821      -0.453       0.359
cause          -0.1160      0.221     -0.525      0.600      -0.549       0.317
discrep        -0.0251      0.241     -0.104      0.917      -0.497       0.447
tentat         -0.0234      0.179     -0.131      0.896      -0.375       0.328
certain        -0.0242      0.205     -0.118      0.906      -0.425       0.377
differ         -0.0384      0.166     -0.231      0.817      -0.364       0.287
see            -0.0170      0.188     -0.090      0.928      -0.386       0.352
feel           -0.0078      0.325     -0.024      0.981      -0.645       0.630
bio            -0.0860      0.293     -0.294      0.769      -0.660       0.488
sexual          0.1453      0.695      0.209      0.834      -1.217       1.507
affiliation     0.0709      0.265      0.267      0.789      -0.448       0.590
achieve         0.0106      0.205      0.052      0.959      -0.390       0.412
power          -0.0958      0.175     -0.547      0.584      -0.439       0.247
reward          0.0172      0.176      0.098      0.922      -0.328       0.363
risk            0.0496      0.412      0.120      0.904      -0.757       0.857
focuspast       0.0047      0.117      0.040      0.968      -0.224       0.234
focusfuture     0.0510      0.201      0.254      0.800      -0.343       0.445
time           -0.0120      0.107     -0.112      0.911      -0.221       0.197
home            0.0394      0.399      0.099      0.921      -0.742       0.820
money          -0.0196      0.541     -0.036      0.971      -1.080       1.041
relig           0.0978      0.749      0.131      0.896      -1.369       1.565
death           0.0169      0.480      0.035      0.972      -0.924       0.958
informal        0.0847      1.507      0.056      0.955      -2.869       3.038
swear          -0.1255      1.613     -0.078      0.938      -3.288       3.037
netspeak       -0.0516      1.524     -0.034      0.973      -3.038       2.935
assent          0.0018      1.562      0.001      0.999      -3.060       3.064
nonflu         -0.1514      1.590     -0.095      0.924      -3.268       2.965
filler         -0.2321      1.896     -0.122      0.903      -3.948       3.484
QMark           0.0523      0.256      0.204      0.838      -0.449       0.554
Exclam          0.0518      0.149      0.349      0.727      -0.240       0.343
keywords       -2.5026     26.795     -0.093      0.926     -55.019      50.014
mistakes       -4.1954      7.054     -0.595      0.552     -18.020       9.629

My question would be: Why is this regression model showing poorer findings than the linear regression model? My y variable is in range of 0 and 1 and the std > mean. Thank you for your help!

Improve this question
$\endgroup$
4
  • $\begingroup$ you cannot use negative binomial, your response is continuous. linear regression is thus the right direction. $\endgroup$
    – utobi
    Commented Feb 7, 2023 at 8:56
  • $\begingroup$ @utobi thank you! Is there any reason why my coefficients are that low on linear regression? Do you think I should use the normal Helpfulness score and not the scaled one? $\endgroup$
    – PythonG
    Commented Feb 7, 2023 at 9:04
  • $\begingroup$ Yes, I'd use the original score since that's what you measure unless you have specific reason to standardise it. $\endgroup$
    – utobi
    Commented Feb 7, 2023 at 9:09
  • $\begingroup$ @utobi sounds good! I simply wanted to scale it to the max views, because otherwise review text that have 2 thumbs up and 4 views tend to have the same score as as reviews with 500 thumbs up and 1000 views, which cant be true as there needs to be something that differentiates these two reviews no? $\endgroup$
    – PythonG
    Commented Feb 7, 2023 at 9:14

2 Answers 2

Reset to default
0
$\begingroup$

If I understand correctly, your model has scaled_HS as the response (y) and HS as one of the predictors (x). If that is the case, you cannot meaningfully interpret the coefficients of the other predictors. Surely HS explains most of the variation in scaled_HS!

What happens to the coefficients when you take HS out?

Improve this answer
$\endgroup$
4
  • $\begingroup$ Yes I dropped the HS from the dependent variable. This is the negative binomial Regression without HS: The significance did not change in the slightest, I will try the approach of @utobi from the comments and use linear regression on the HS and not the scaled HS $\endgroup$
    – PythonG
    Commented Feb 7, 2023 at 11:21
  • $\begingroup$ How do you compute scaled_HS from HS? Also, could you please post the output of the original linear regression without HS as a predictor? $\endgroup$
    – Doctor Milt
    Commented Feb 7, 2023 at 11:37
  • $\begingroup$ Right, so: HS_scaled = HS * scaled_V = (Helpfully viewed / Views) * (Views / max[Views]) = Helpfully viewed / max[Views] Have I got that right? $\endgroup$
    – Doctor Milt
    Commented Feb 7, 2023 at 13:56
  • $\begingroup$ I found the error after rechecking my dataset and saw that Hs mean and std indicate that nb2 regression is not the right regression model! $\endgroup$
    – PythonG
    Commented Feb 9, 2023 at 18:36
0
$\begingroup$

In regard to the comment of @Doctor Milt:

Hs is the result of Helpfully viewed/ Views.

HS  Helpfully_viewed    Views
0   0.37    28  76
1   0.16    44  268
2   0.24    72  299
3   0.29    19  65
4   0.38    10  26
...     ...     ...     ...
1520    0.25    2   8
1521    0.25    2   8
1522    0.25    2   8
1523    0.33    3   9
1524    0.00    0   5

Furthermore I computed scaled views by dividing each value in Views with themax[Views] function.

    scaled_V
0   0.046683
1   0.164619
2   0.183661
3   0.039926
4   0.015971
...     ...
1520    0.004914
1521    0.004914
1522    0.004914
1523    0.005528
1524    0.003071

The Final HS scaled column is the product of HS *scaled_V. The original Linear regression without HS is as follows:

Rating (-0.06201207484873128, 0.015435587631467379)
Spoiler (0.059553066433509855, 0.020030343178780382)
Views (0.7880693559507788, 3e-323)
Review_length (0.0611843321063948, 0.016866533916445086)
WC (0.0629356733799715, 0.013965934404277043)
Tone (-0.07394090951501098, 0.0038637998296509337)
shehe (0.07131791547159314, 0.005331048300074106)
conj (-0.05033621584191379, 0.04937650321109507)
compare (-0.06621954834327438, 0.009690693447463867)
number (0.05913351896038223, 0.02092332389500587)
quant (-0.06552319095849933, 0.010484851019228214)
posemo (-0.05931337090970264, 0.020536361646821617)
negemo (0.05131647584743782, 0.04510762970526194)
male (0.06826773827085715, 0.007656218956918288)
death (0.05074141927289752, 0.047572524702827156)
QMark (0.053348318103051344, 0.03724269305864121)
Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Are you posting a self-answer? $\endgroup$
    – User1865345
    Commented Feb 7, 2023 at 12:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.