1
$\begingroup$

I am analysing data set that contain information about one user file creation proces in a two year period. Data set contain date stamp, file size, owner etc. I am using Poisson distribution for number of files (since I monitor period from 08:00 AM - 04.00 PM and number of generated files can be quite large). Inter_arrival time should be exponentials distribution. My question is since my time stamp is arround to minute I have many records with same time stamp ( in this two year there is arround 55 mondays when user created file, in two mondeys that are not the same one I have creation of file in same time 09:26) what is best way to solve this should I analyse every monday and then make some sum or mean?

$\endgroup$
0
$\begingroup$

I guess your aim is to model the file creation process over the days of a week? If so you can model the time course of created files during the day/weak as to be an inhomogeneous Poisson process. Therefore you divide the time period into temporal windows, compute the number of events within each of the windows and take this average as the expectation value of the Poisson distribution during that time window. Note that this allows you to have multiple events (counts) within a window and two identical time stamps are no problem. Depending on the window size you change the expectations for the individual windows but the sum over all expectations in a day (i.e. Monday) will be equal to the average count on such a day for which you pooled the data (average across all Mondays). Let us assume that $c_t$ is the list of counts of file creations within time windows t of fixed size $\Delta$ (i.e. 60mins) pooled from all days of recording. The average count of file creations between 08:00 AM and 09:00 AM is then $\langle \textbf{c}_{t=0} \rangle = \frac{1}{N_{weekdays}}\sum_{d \in \{'Monday','Tuesday',...\}} c_{t=0,~day=d}$ which would be your Poisson parameter (the expectation value, typically denoted with $\mu$ or $\lambda$) used for modelling the distribution in time bin t. When you want to describe only the Mondays you restrict the data to Monday data. Then for every time bin t on Mondays you have a Poisson distribution with parameter $\langle \textbf{c}_t \rangle_{day = 'Monday'}$ and the property:

$\langle c_T\rangle_{day='Monday'} = \sum_t^T{\langle c_t \rangle}_{day='Monday'}$

The same is true for summing the expectations in a full weak:

$\sum_{d \in \{'Monday','Tuesday',...\}} \langle c_T\rangle_{day=d} = \sum_t^T{\langle c_t \rangle}_{day='Monday'}$

One nice feature of inhomogeneous Poisson processes is that the expectation of summed bins is equal to the sum of the expectations of that bins.

So the key answer to your question is to bin your data to the scales of interest and pool in a way that best allows to answer your specific questions. In that case assuming the Poisson assumption is valid (coefficient of variation and Fano factor close to 1) an inhomogeneous Poisson process is the correct way to model the time course of file creations.

Tip: In the literature the repeated measurements (like your Mondays) are often called trials. The existence of trials allows you to compute an expectation and variance within each time bin. For Poisson both should be equal. Two identical time stamps in different trials is no problem.

$\endgroup$
  • $\begingroup$ Did I understand your question right, could it help you in any way? Pls. give a short feedback. $\endgroup$ – Jojo Mar 1 '17 at 17:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.