1
$\begingroup$

I have sample data that I expect to contain values from at least several Poisson distributions (set around various lambda values). Some of these lambda values are nicely spaced, leading to what are visually obvious distinct distributions. My goal is to set confidence intervals (bins) around the most believable of these distributions so I can parse the data in a defensible way.

So I'm thinking about using the conditional Chi-squared dispersion test once I have each lambda estimated... But I'm wondering what would be the best way to choose these values for lambda other than just simple visual inspection, which is where I'm stuck right now.

$\endgroup$
  • 1
    $\begingroup$ It's not completely clear to me what you're trying to do, nor why 'binning' helps to 'parse' data, nor how "using the conditional Chi-squared dispersion test" achieves whatever it is you are attempting. Can you clarify? $\endgroup$ – Glen_b Sep 11 '13 at 2:15
  • $\begingroup$ If you have some idea of how many types there are, you can try a finite mixture approach. $\endgroup$ – Dimitriy V. Masterov Sep 11 '13 at 3:35
  • $\begingroup$ Hi Glen: The goal is parse the data into their respective distributions. Because over-dispersion is so well studied and the conditional Chi-squared tests for that, I'm thought there would be a way to use that in a two step process: choose various values for lambda objectively based on the data and then test each of these with the dispersion test and the data to confirm variance of each bin adhers to Poisson assumptions. $\endgroup$ – Ryan Lynch Sep 11 '13 at 18:49
  • $\begingroup$ Perhaps that's not the way to do it, but again the goal is to parse mixed sample data that comes from various Poisson distributions into each individual lambda centered distribution. $\endgroup$ – Ryan Lynch Sep 11 '13 at 18:52
  • $\begingroup$ @RyanLynch Posting some sample data may help. $\endgroup$ – Dimitriy V. Masterov Sep 12 '13 at 16:16
2
$\begingroup$

Here's an example of using a finite mixture model of how to get the $\lambda$s where the outcome is the length of hospital stays (los) for 1495 Medicare patients. I am using Stata's user-written fmm and fmmlc, but this can certainly be done with other software. You can even add covariates if you like, though I avoided them for simplicity. You can also classify patients based on most likely latent class membership to parse the data. That sounds like a better way to parse, if I understood your aims.

Here are the summary stats for the outcome:

. webuse medpar, clear;

. sum los, detail;

                       Length of Stay
-------------------------------------------------------------
      Percentiles      Smallest
 1%            1              1
 5%            1              1
10%            2              1       Obs                1495
25%            4              1       Sum of Wgt.        1495

50%            8                      Mean           9.854181
                        Largest       Std. Dev.      8.832906
75%           13             70
90%           19             74       Variance       78.02022
95%           23             91       Skewness       3.651526
99%           46            116       Kurtosis       29.20186

I choose the number of components (or latent classes) based on the one with lowest BIC since I don't have a lot of theories about old folks. Here's what 6 components looks like:

6 component Poisson regression                    Number of obs   =       1495
                                                  Wald chi2(0)    =          .
Log likelihood = -4811.1664                       Prob > chi2     =          .

------------------------------------------------------------------------------
         los |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
component1   |
       _cons |   .8919962    .096325     9.26   0.000     .7032027     1.08079
-------------+----------------------------------------------------------------
component2   |
       _cons |   1.975354   .0590755    33.44   0.000     1.859568     2.09114
-------------+----------------------------------------------------------------
component3   |
       _cons |   2.666272   .0561255    47.51   0.000     2.556268    2.776276
-------------+----------------------------------------------------------------
component4   |
       _cons |   3.972175     .03505   113.33   0.000     3.903478    4.040871
-------------+----------------------------------------------------------------
component5   |
       _cons |   3.220256    .054852    58.71   0.000     3.112748    3.327764
-------------+----------------------------------------------------------------
component6   |
       _cons |   4.632102   .0777041    59.61   0.000     4.479805      4.7844
-------------+----------------------------------------------------------------
 /imlogitpi1 |   5.010191   .7229048     6.93   0.000     3.593324    6.427059
 /imlogitpi2 |   5.802065   .7129656     8.14   0.000     4.404678    7.199452
 /imlogitpi3 |   5.279303   .7182948     7.35   0.000     3.871471    6.687135
 /imlogitpi4 |   2.164426   .7513177     2.88   0.004     .6918707    3.636982
 /imlogitpi5 |   3.722884    .759756     4.90   0.000      2.23379    5.211979
------------------------------------------------------------------------------
         pi1 |    .205884   .0279458                      .1564441    .2660209
         pi2 |    .454495   .0294642                      .3976062    .5125971
         pi3 |   .2694614   .0295284                      .2156231    .3310685
         pi4 |   .0119598   .0028617                       .007474    .0190861
         pi5 |   .0568267   .0154425                      .0331196    .0958215
         pi6 |   .0013732   .0009736                      -.000535    .0032814
------------------------------------------------------------------------------

Exponentiating the _cons gives you the Poisson $\lambda$s (you can also use predict yhat1, equation(component1)). For example, the healthiest group has a mean expected stay of $\exp\{.8919962\}=2.44$, almost two and a half days. The pis are the mixing probabilities. You can see that the sickest people are fairly rare. As you will see below, there are only about 2 of them.

Here are some other helpful stats:

-----------------------------------------------------------------------------
Final class counts and proportions based on estimated posterior probabilities

   Component       Proportion        Count
---------------+---------------+---------------
       1             0.206          307.797
       2             0.454          679.470
       3             0.269          402.845
       4             0.012           17.880
       5             0.057           84.956
       6             0.001            2.053
---------------+---------------+---------------

-----------------------------------------------------------------------
Classification of subjects based on most likely latent class membership

   Component       Frequency        Percent
---------------+---------------+---------------
       1               272            18.19
       2               765            51.17
       3               374            25.02
       4                18             1.20
       5                64             4.28
       6                 2             0.13
---------------+---------------+---------------
     Total            1495

-------------------------------
Average posterior probabilities

    Mean             LC1       LC2       LC3       LC4       LC5       LC6 
------------|--------------------------------------------------------------
     p1     |       0.849     0.151     0.000     0.000     0.000     0.000
     p2     |       0.101     0.776     0.123     0.000     0.000     0.000
     p3     |       0.000     0.120     0.802     0.000     0.078     0.000
     p4     |       0.000     0.000     0.000     0.982     0.015     0.003
     p5     |       0.000     0.000     0.133     0.003     0.864     0.000
     p6     |       0.000     0.000     0.000     0.000     0.000     1.000
------------|--------------------------------------------------------------


---------------------------------
Distinctiveness of latent classes
---------------------------------
Entropy:                    0.741
---------------------------------


--------------------------------------
Information criteria
--------------------------------------
AIC:                          9644.333
BIC:                          9702.741
Sample size adjusted BIC:     9667.798

# free parameters:                  11
Log likelihood:              -4811.166
--------------------------------------

Here's a classification of patients based on most likely latent class membership looks like:

  los          _class_5  
      3   FMM Component 1  
      9   FMM Component 2  
     13   FMM Component 3  
     50   FMM Component 4
     29   FMM Component 5  
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.