I need to predict the number of retweeets a tweet receives as a function of (1) whether there is a hyperlink within the tweet's text and (2) the position of the hyperlink within the tweet. If a tweet doesn't contain a hyperlink then the position of the hyperlink within the tweet is a null value. Example data for this situation would be:

numberOfRetweets     hyperlinkInTweet?     positionOfHyperlink
(dependent var)    (binary, 0=no, 1=yes)   (integer, 1 to 135)    
      5                      0                    null
     10                      1                     50
     35                      1                     12
      0                      0                    null
      0                      1                      5
      :                      :                      :

If a tweet doesn't contain a hyperlink (hyperlinkInTweet = 0) then it cannot have a position within the tweet text where the hyperlink begins (positions range from 1 to 135 inclusive). I've toyed with the idea of using zero in place of null, but I have reservations about doing this. Exploratory analysis of the data indicates that about half of the tweets contain a hyperlink.

I'm wondering if this is a nested regression problem, but would be grateful for a second opinion on the best course of action for this problem. Many thanks.

  • $\begingroup$ What type of model do you use? A Poisson-GLM? Your question is solved by using the hyperlink dummy and the product of the dummy with the position as regressors (like an interaction). $\endgroup$ – Michael M Jun 19 '14 at 9:38

Since you have a count variable, we think about Poisson regression. But other count variable models could be substituted, such as negative binomial regression. The most important aspect is that these models have a multiplicative expectation structure, which is important if you have complications, such as number of retweets over different time periodsa, when time period length can enter as a new variable.

I think you can substitute zero for your null values, in this context. Let us see what this means. The poisson regression model is $$ \DeclareMathOperator{\P}{Pr} \P(Y_i=y_i) = e^{-\lambda_i} \frac{\lambda_i^y}{y_i!} $$ with $\lambda_i >0$ the parameter. For each observation $i$, we have the model $\lambda_i = \exp(x_i^T \beta) = \exp(\eta_i)$ where $\eta_i$ is called the linear predictor. Here we have $\beta=(\beta_0, \beta_1, \beta_2)$ and $x=(1, x_1, x_2)$ where $$ \begin{align} x_1 &= \text{$1$ if no hyperlink, $0$ if hyperlink} \\ x_2 &= \text{$0$ if no hyperlink, else length of hyperlink (or some non-linear function of length)} \end{align} $$ Then, for your example data (line 1,2,3) we get $$ \begin{align} \eta_1 &= \beta_0 +\beta_1 \\ \eta_2 &= \beta_0 + 50\beta_2 \\ \eta_3 &= \beta_0 + 12\beta_2 \\ \end{align} $$ (Probably you will need some non-linear function of length, such as log. Some plot may tell you!). This looks to me as an eminently sensible and interpretable model. The parameter $\beta_1$ can be seen as a kind of "equivalent length of hyperlink" for the cases without a hyperlink.

  • $\begingroup$ I've used a negative binomial regression model due to the overdispersion. I'm also going to look at using a hurdle model to address the large number of zeros in the dependent variable. Many thanks for your suggestions. $\endgroup$ – TonyKarl Jun 20 '14 at 2:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.