-1
$\begingroup$

I have data from a content website where the primary goal is to have users spend more time reading content.

Here is a link to the data set (too large to paste) which can be downloaded and read into R as a csv.

eg_data <- read.csv("eg_data.csv", header = T, stringsAsFactors = F)
> glimpse(eg_data)
Observations: 35,326
Variables: 7
$ ID             <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, ...
$ pageType        <chr> "cats", "dogs", "bunnies", "cats", "cats", "cats", "cats", "cats", "cats", "dog...
$ word_count     <int> 1267, 1892, 345, 1565, 1941, 1267, 1271, 1565, 1941, 1892, 1271, 1941, 1892, 20...
$ deviceCategory <chr> "desktop", "desktop", "mobile", "mobile", "mobile", "mobile", "mobile", "mobile...
$ userType       <chr> "Returning Visitor", "Returning Visitor", "Returning Visitor", "New Visitor", "...
$ Medium         <chr> "carrots", "carrots", "oranges", "apples", "oranges", "oranges", "oranges", "or...
$ timeOnPage     <int> 3, 87, 16, 150, 318, 109, 29, 48, 35, 0, 149, 264, 98, 165, 69, 25, 0, 5, 132, ...

Target variable is timeOnPage.

Having explored the data I have found considerable variation within the variables. For example, device can be one of either of Tablet, Mobile or Desktop:

library(tidyverse)

# device category
eg_data %>% 
  group_by(deviceCategory) %>% 
  summarise(AvgTime = round(mean(timeOnPage), 0),
            countn = n()) %>% 
  arrange(-AvgTime)

# A tibble: 3 x 3
  deviceCategory AvgTime countn
           <chr>   <dbl>  <int>
1        desktop     192   3052
2         tablet     182   4791
3         mobile     155  27483

Clearly some variation, particularly between smaller screens (mobile phones) and either tablet or desktop.

Similar across source of traffic (Medium):

# medium
eg_data %>% 
  group_by(Medium) %>% 
  summarise(AvgTime = round(mean(timeOnPage), 0),
            countn = n()) %>% 
  arrange(-AvgTime)

# A tibble: 8 x 3
    Medium AvgTime countn
     <chr>   <dbl>  <int>
1 pinapple     280      2
2   apples     178    308
3  carrots     168     15
4  oranges     164    817
5 coconuts     163  32512
6    pears     150   1311
7 cabbages     147    360
8     peas      44      1

Here's the breakdown by pageType

> # pageType
> eg_data %>% 
+   group_by(pageType) %>% 
+   summarise(AvgTime = round(mean(timeOnPage), 0),
+             countn = n()) %>% 
+   arrange(-AvgTime)
# A tibble: 6 x 3
  pageType AvgTime countn
    <chr>   <dbl>  <int>
1    cats     175  19261
2    dogs     167   1623
3   birds     157   3435
4  horses     152   2601
5 bunnies     141   7738
6    fish      73    668

Because there is some variation among the predictors, I believe that my explanatory variables should at least be reasonably good predictors of target variable time spent.

Here's my attempt at building a simple linear model:

# tuning & parameters
set.seed(123)
train_control <- trainControl(
  method = "cv",
  number = 5,
  verboseIter = TRUE
)

# train a linear model
lm_mod <- train(
  timeOnPage ~ word_count + pageType + deviceCategory + userType + Medium,
  data = eg_data,
  method = "lm",
  trControl = train_control
)

After running this if I type lm_mod into the console I see the following statistics:

  RMSE      Rsquared  
  223.6664  0.01054297

Both of these numbers are very poor given the data. Given mean(eg_data$timeOnPage) = 162.059 and RMSE is over 200.

I have experimented by stripping some variables out but resulting changes to the evaluation metric are minimal.

Anyone familiar with web analytics will know that both device category and medium variables are typically important when analysing performance, be it conversion to transaction, sign up or in this case time reading.

I tried a decision tree and found similarly poor RMSE. I prefer not to use e.g. Random Forrest or any other method because I like the ability of lm to explain relationships using the coefficients (So hoping to stay within linear model framework).

Having reached a point of desperation I'm hoping that by posting a disguised copy of the data someone can verify that my data are indeed poor predictors of timeOnPage, or, hopefully, point out a way of improving this simple linear model wrt RMSE and $R^2$.

$\endgroup$
  • $\begingroup$ Any particular reason why the response and e.g. count are left untransformed? $\endgroup$ – Michael M Nov 5 '17 at 11:00
  • $\begingroup$ @MichaelM why would they be? Please expand if I have missed something or you have a suggestion? $\endgroup$ – Doug Fir Nov 5 '17 at 16:02
1
$\begingroup$

There is variation, but in each variable, there is one category that is very clearly the largest. That means that for a large part of your samples, you are just predicting the mean, and it doesn't really explain the difference between your records. The group mobile-coconut-cat, for example, is your largest category by far. Your predictions for this group are the mean, which is the best you can do if you want to minimize RMSE. The variation within this group is something that you cannot explain with the data, and if you don't have other predictors, there is no way to make those predictions better.

There are perhaps some small things that can be done to improve your model. First of all the relation with word_count is probably nonlinear. So adding it like this doesn't help, it only hurts the model. There might be a sweet spot of the number of words for an article, and you should use some transformation that allows the model to find this sweet spot. Second, the smallest categories are probably overfit, so for example the group apples has a high average time, but it's not a big group, so this could be just coincidence. In your model, it has its own coefficient, which probably overestimates the actual average time spent on those pages. For regularization, I would look at a mixed model with lme4, or something like LASSO from the glmnet package.

Conversely, as in, there your model might not be flexible enough, there are no interactions in it. But short articles may be more popular on mobile, while long articles are more popular on desktop. There is no freedom for the model to find such a pattern at the moment. I don't think adding this interaction (wordcount * channel) will immediately improve things, but combined with some nonlinearity in wordcount, and regularization, it could help a bit.

All this might lift your rsquared a bit, but I wouldn't count on getting up to 10%. Whether or not someone stays longer at the page has a lot to do with whether it's actually what he or she is looking for, whether nothing IRL happens that stops him or her from reading on, perhaps if it's a fast or a slow reader, probably patience has a role in it. None of that can be 'predicted' from your data.

$\endgroup$
  • $\begingroup$ Thanks a lot for the clear and easy to understand answer. Sounds like I have a large segment crowding out the model, for lack of a better phrase (mobile-coconut-cat). Thanks also for the suggestion on interaction terms, lasso and regularization. $\endgroup$ – Doug Fir Nov 5 '17 at 16:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.