I have a dataset of events: each row is an event, and each column is a feature. There are millions of events and several dozen features. The features are mostly numerical (a few are categorical and I plan to one-hot encode them). One of the features is time, and I think there is a time trend.

I know the events are not perfectly independent, but I think it's too hard to model that, so as a first approximation, I'm comfortable assuming that events are generated by Poisson process with the intensity that depends on the features. I want to predict the intensity, given the feature values.

To clarify with a simple example: suppose I have the log which contains, for each accident, its time, weather data, location data, type of road, traffic conditions, etc. - in total, say 50 variables. I want to predict the intensity of car accidents given specific values for all those variables.

I expect that the true relationship between the features and the intensity is relatively smooth but quite complicated (certainly non-linear and non-monotonic). I also expect that the intensity is never negligible (that is, for any feature values, the events might still happen once in a while).

What machine learning or statistical models should I consider?

I assume kernel density estimation and other local techniques are hopeless because there are too many dimensions.

I thought Gaussian Mixed Models might help, but I haven't seen any papers or links that describe their use in a similar situation (I'm concerned that the "thin tails" of gaussian distributions would not do well given that in my data the intensity never gets too close to zero).

  • $\begingroup$ I didn't follow what you meant by "too many dimensions": isn't this a Poisson process in time, with just one dimension? If it's not a temporal process, then please tell us in which space these events are located. $\endgroup$ – whuber May 11 '16 at 20:33
  • $\begingroup$ @whuber thanks - I just edited the question to add an example that I hope clarifies the problem. By the high number of dimensions, I mean there are too many variables that affect the intensity (50 in my example), and if I try to sort and group the data by time, the other variables jump around a lot even within a short time span. $\endgroup$ – max May 11 '16 at 21:58
  • $\begingroup$ But that's no problem for, say, computing a kernel density of the events in time. After doing that, you have a relatively simple multiple regression problem with millions of observations, only a few dozen features, and just one response (the density): that's routine. You have no apparent need to perform kernel density estimates in any larger space. $\endgroup$ – whuber May 11 '16 at 22:05
  • $\begingroup$ @whuber Let's simplify this, and say there's no time trend. Then, without conditioning on other variables, the events are generated by the Poisson process with a constant intensity. That intensity is unknown, but can be trivially estimated with a very high precision. So I'll get a constant. The problem is that I want to know the intensity conditional on other variables. I don't see how I can do that with just a multiple regression. $\endgroup$ – max May 12 '16 at 1:21
  • $\begingroup$ You will have identical problems in any situation where the response is a constant! $\endgroup$ – whuber May 12 '16 at 16:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.