1
$\begingroup$

We are developing a software product that monitors user interactions with some website. Each interaction is referred as a hit. One of the things we measure is how long it takes for the user to get the response from the website. We have a threshold that defines which hit is considered to have a latency breach, such that if the measured duration of the hit is longer than the threshold - this hit is considered as a breach. We collect these data and analyze them once every 5 minutes. One of the parameters we monitor is which web-browser produced each hit. For instance, we can tell if one hit came from Firefox and the other came from Chrome. One of the things we would like our analysis to realize is whether there is a problem related to a specific browser type. For example, if the data are as follows:

|Browser Type      |Hit count for last 5 minutes | Number of hits with breach
|------------------|-----------------------------|---------------------------
|Internet Explorer |2200                         |12
|Chrome            |1400                         |15
|Firefox           |1700                         |840

We would like to be able to tell it is very likely we have a specific problem with Firefox.

If I was to phrase this in a statistical way, I assume it would be something like: we would like to be able to tell when the abundance of a certain web-browser among all breaching hits is significantly higher than its abundance in the entire population of all hits. The meaning of "significantly" for something we can decide later, might be - a 99% confidence.

My question is, what is the right way to analyze this?

$\endgroup$
0

1 Answer 1

1
$\begingroup$

I might be tempted to fit a simple probit model (though I get similar results with a logit or linear probability model). Here's an example in Stata.

First, we make an N=5300 dataset that has a binary outcome variable breach.

. /* Make Data */
. clear

. set obs 5300
obs was 0, now 5300

. gen browser = "Internet Explorer"

. replace browser = "Chrome" in 2201/3600
(1400 real changes made)

. replace browser = "Firefox" in 3601/l
(1700 real changes made)

. sort browser

. gen breaches = 0

. replace breaches = 1 in 1/15 // Chrome
(15 real changes made)

. replace breaches = 1 in 1401/2240 // FF
(840 real changes made)

. replace breaches = 1 in 3101/3112 //IE
(12 real changes made)

. sencode browser, replace

The data now looks like this:

          browser   breaches  
Internet Explorer          0  
Internet Explorer          0  
           Chrome          0  
           Chrome          0  
Internet Explorer          0  
           Chrome          0  
Internet Explorer          0  
Internet Explorer          0  
          Firefox          1  
Internet Explorer          0  

The sample statistics match your table:

. /* Check Data */
. tab browser breach

                  |       breaches
          browser |         0          1 |     Total
------------------+----------------------+----------
           Chrome |     1,385         15 |     1,400 
          Firefox |       860        840 |     1,700 
Internet Explorer |     2,188         12 |     2,200 
------------------+----------------------+----------
            Total |     4,433        867 |     5,300 

The probit coefficients are not terribly meaningful, but the sign and significance are suggestive:

. /* Probit */
. probit breach i.browser


Iteration 0:   log likelihood = -2361.5048  
Iteration 1:   log likelihood = -1413.0229  
Iteration 2:   log likelihood = -1339.1382  
Iteration 3:   log likelihood = -1335.7079  
Iteration 4:   log likelihood = -1335.6975  
Iteration 5:   log likelihood = -1335.6975  

Probit regression                                 Number of obs   =       5300
                                                  LR chi2(2)      =    2051.61
                                                  Prob > chi2     =     0.0000
Log likelihood = -1335.6975                       Pseudo R2       =     0.4344

------------------------------------------------------------------------------------
          breaches |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------------+----------------------------------------------------------------
           browser |
          Firefox  |   2.285602   .1018549    22.44   0.000      2.08597    2.485233
Internet Explorer  |  -.2452505   .1398289    -1.75   0.079      -.51931    .0288091
                   |
             _cons |  -2.300347   .0972129   -23.66   0.000    -2.490881   -2.109813
------------------------------------------------------------------------------------

This is the predicted probability of breach for each browser type:

. margins browser

Adjusted predictions                              Number of obs   =       5300
Model VCE    : OIM

Expression   : Pr(breaches), predict()

------------------------------------------------------------------------------------
                   |            Delta-method
                   |     Margin   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------------+----------------------------------------------------------------
           browser |
           Chrome  |   .0107143   .0027516     3.89   0.000     .0053213    .0161072
          Firefox  |   .4941176   .0121259    40.75   0.000     .4703512    .5178841
Internet Explorer  |   .0054545   .0015703     3.47   0.001     .0023768    .0085323
------------------------------------------------------------------------------------

We can also do a formal hypothesis test that the predicted probabilities are the same, comparing Chrome to Firefox and IE to Firefox. We reject the individual and the joint hypotheses that differences in predicted breach are zero at any significance level. Compared to Firefox, the probability of breach is about 0.5 lower for the other two browsers.

. margins rb2.browser

Contrasts of adjusted predictions
Model VCE    : OIM

Expression   : Pr(breaches), predict()

-------------------------------------------------------------------
                                |         df        chi2     P>chi2
--------------------------------+----------------------------------
                        browser |
           (Chrome vs Firefox)  |          1     1511.41     0.0000
(Internet Explorer vs Firefox)  |          1     1597.22     0.0000
                         Joint  |          2     1598.01     0.0000
-------------------------------------------------------------------

---------------------------------------------------------------------------------
                                |            Delta-method
                                |   Contrast   Std. Err.     [95% Conf. Interval]
--------------------------------+------------------------------------------------
                        browser |
           (Chrome vs Firefox)  |  -.4834034   .0124342      -.507774   -.4590328
(Internet Explorer vs Firefox)  |  -.4886631   .0122272      -.512628   -.4646982
---------------------------------------------------------------------------------
$\endgroup$
5
  • $\begingroup$ Hello Dimitry, thanks for your answer. I must admit that I am not completely understand this approach. Can you please simplify it somehow for me? Which utility are you using for those calculations? $\endgroup$
    – odavid
    Jan 28, 2015 at 12:10
  • $\begingroup$ I am using a statistical package called Stata. The probit model can be implemented fairly easily in almost any modern software, like R. $\endgroup$
    – dimitriy
    Jan 28, 2015 at 21:40
  • $\begingroup$ Are you familiar with linear regression? $\endgroup$
    – dimitriy
    Jan 28, 2015 at 21:41
  • $\begingroup$ Not really. But as much as I understands, the linear regression is suppose to help me to predict a variable according to existing data. I'm not sure if that's the case since I don't need to predict anything. I have all the data and just need a formula which can tell me if there's an exception. $\endgroup$
    – odavid
    Jan 29, 2015 at 7:42
  • $\begingroup$ Linear regression allows you to estimate a formula for the probability that a breach happens for a particular type of browser and to distinguish whether any differences you see are real or just chance. It's not just for prediction. It sounds like you either need to take an intro stats class or to hire a statistician to help you. $\endgroup$
    – dimitriy
    Jan 29, 2015 at 9:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.