2
$\begingroup$

The background

I have a dataset with some zeros - based on how I segment my data, it is either 50% of the observations or 80% of the observations. The data is not actually count data, but from what i have read, that doesn't really matter for a poisson regression.

The data can be interpreted as a "signal" or "offer" sent to a person and whether the person made some purchase. The output variable is a USD amount if person makes a purchase and 0 otherwise. therefore the data looks something like:

<columns describing the offer>, 
<columns describing the person>,
<money that the customer payed (0 when no purchase is made>

Now i would like to model the amount of money payed as a function of person characteristics and offer characteristics. For that i am using a poisson QML model (with some lasso shrinkage on top, as i have many variables and also to make it more fancy).

The question

Since i have in the dataset plenty of zeros - what will be the impact of those on my estimates from the poisson regression? Will my estimates be completely useless? Do zeros just make my estimated coefficient lower than what is true value?

In the end of the day, I want to find how characteristics of the sent offer impact the expected payment of the person of given characteristics. I.e if i send a specific offer to 100 people, 20 of them pay 1 dollars and 80 do not make a purchase i want to know that the expected payment of person in this group is 0.2 dollars.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The output variable is a USD amount if person makes a purchase and 0 otherwise.

Don't model this using a Poisson distribution, unless your purchases are on the order of 1, 2, ... USD.

The Poisson is not only characterized by many zeros, but also by low non-zero outcomes. If you have a high Poisson parameter, then you will get probability mass over high outcomes... but no mass for zero outcomes any more.

I would suggest you model your data in two stages. First model the number of purchases someone makes, but not (yet) the amount. This will often be zero, sometimes one, rarely more than one. (Depending on how you deal with repeat customers. But if you model the response to each "signal", this should hold.) The Poisson will likely be a reasonably good fit here. Second, model the amount your customer spends per purchase. A gamma or possibly a normal distribution would make sense here.

Improve this answer
$\endgroup$
8
  • $\begingroup$ The poisson regression is also used in empirical literature to model trade flows between countries, where the output variable can take values much higher than 1 or 2 USD though? $\endgroup$
    – ira
    Commented Aug 20, 2019 at 9:30
  • $\begingroup$ That surprises me very much. Can you point to literature online? $\endgroup$
    – Stephan Kolassa
    Commented Aug 20, 2019 at 9:34
  • $\begingroup$ An original paper by Silva and Tenreyro: personal.lse.ac.uk/tenreyro/jensen08k.pdf and then their followup on the topic: personal.lse.ac.uk/TENREYRO/ppml-fsr.pdf $\endgroup$
    – ira
    Commented Aug 20, 2019 at 9:50
  • $\begingroup$ Thank you. I don't have the time to go through that, but I assume there are no zeros in trade flow data? You can certainly use large parameter Poissons to model high values, though I have a hard time understanding why you wouldn't use OLS, but then you won't get any zeros. $\endgroup$
    – Stephan Kolassa
    Commented Aug 20, 2019 at 9:53
  • $\begingroup$ There are plenty of zeros in trade data $\endgroup$
    – ira
    Commented Aug 20, 2019 at 9:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.