4
$\begingroup$

I have a discrete data set showing number of customers entering a shop per day, with the following descriptives:

n=303 min=531 max=1695 mean=1100 Std: 193

As I want to run a monte-carlo simulation to show the customers per day, I want to find the best-fitting model to describe the data.

My problem is that the data fits a Weibull-distribution very well (p=0,9), but do not fit any discrete models at all. Is it possible to use the Weibull distribution to generate data (grouping into bins, being nearest integer), or would this be regarded bad scientific practice?

Ps. Long time reader, first time asker. :)

Improve this question
$\endgroup$
4
  • $\begingroup$ Yes. This is actually how simulation works. Create a interval of 0.5 to the right and left for each number. For example, for n = 35, consider interval 34.5 - 35.5. If simulated number is between this interval you may consider it 35. $\endgroup$
    – Neeraj
    Commented Nov 18, 2017 at 19:27
  • $\begingroup$ @Neeraj - Thank you for your reply. Though, my question was more concerning the scientific practice of fitting a continuous distribution to the specified discrete data, and then simulating a discrete output, rather than the practical method of doing so. :) $\endgroup$
    – johs32
    Commented Nov 18, 2017 at 19:59
  • 1
    $\begingroup$ @josh32 This approach is completely scientific. Just think about time. It is always measured on discrete scale (maximum upto 2 or 3 decimals after seconds). Just think about revenue. It might be possible that a shopkeeper never receive its revenue in decimal like 1.10101010... Still both the variables modelled using continuous distribution when all possible outcomes are countable. Also in binomial when "n" is very large, it can be approximated by normal distribution. $\endgroup$
    – Neeraj
    Commented Nov 18, 2017 at 20:13
  • 3
    $\begingroup$ @johs32 Is there any particular reason you need to have an explicit functional form rather than either sampling the empirical distribution you have or some smoothed estimate of the pmf? (e.g. a discrete kernel - based estimate)? $\endgroup$
    – Glen_b
    Commented Nov 19, 2017 at 1:14

2 Answers 2

Reset to default
4
$\begingroup$

There's generally no problem with discretizing a continuous density that gives a suitable approximation to a discrete empirical distribution. If you want to be a bit more formal about it, there's a corresponding discrete distribution (or several, depending on how you round -- whether you're rounding off, truncating, or rounding up). You can define whichever discretized distribution you like and then formally fit that discrete distribution if you wish but with large range and a high minimum, that will make little difference.

However, the act of discretizing will change both the mean and the variance slightly. (If that's a concern you can go back to working with the resulting discrete distribution but it probably won't be important for your purposes.)

Once you have your parameters (given whichever sort of rounding/truncation you're using) giving a suitable discrete approximation to what you need, then simulation is as simple as generating from the continuous distribution and rounding. (For some purposes even rounding may not be necessary.)

If you wish to perform inference (testing, confidence intervals for the mean, that kind of thing) on it there can be a few wrinkles with such distributions (they're not always mathematically as convenient once discretized). However, when the numbers are never very small (as here), to a first approximation you can often get by perfectly well by simply ignoring the discretization when it's convenient to do so; for most things it won't matter much.

It may be worth computing the impact of doing so (ignoring the effect of discretizing) for anything of consequence, so that you can see whether it's worth worry about; even if you can't do it algebraically, you would be able to use simulation to gauge how much it matters.

I'd warn you against casually interpreting a high p-value from a goodness of fit test as any suggestion that the approximation is sufficiently good for your purposes (and if your sample sizes are often large) the converse -- a low p-value won't necessarily imply that the approximation involved will have any substantial consequences for you. The question of "is this close enough to use" is not addressed by a p-value.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for your very fulfilling answer. As I am quite new to statistical modelling, I just was not sure whether this was okay. Talking to a friend of mine (currently taking his PhD in theoretical physics), did not lessen my concerns. In his world, one could never describe discrete data with a continuous distribution. Your input on inference is appreciated, I did not think of that. $\endgroup$
    – johs32
    Commented Nov 24, 2017 at 1:25
  • $\begingroup$ The reason for using a Weibull distribution is only that this (from my limited knowledge of statistical modeling) seemed like the best way to smoothly describe the data at hand, taking the fluctuations in input into account. I have never before heard about kernel-based estimates, but from I can see that might actually be a more sound way to go about it! $\endgroup$
    – johs32
    Commented Nov 24, 2017 at 1:41
  • 1
    $\begingroup$ I've definitely seen physicists do such things (continuous approximations to discrete phenomena, for example). But if that was an issue you just do it as a discretized version of the Weibull and then it's a discrete approximation of a discrete variable. As an example, the number of clicks on a Geiger counter or similar sensor is always discrete, but unless the click rate is fairly low, the model for it is usually continuous (decay described with a differential equation rather than a difference equation) $\endgroup$
    – Glen_b
    Commented Nov 24, 2017 at 1:42
2
$\begingroup$

My answer/comments here (way too long for a SE comment ) don't and are not meant to respond to the particular question BUT as BACON once opined "To ask the proper question is half of knowing" . I am suggesting that the OP didn't ask what I think is the larger/proper question i.e. how to simulate/forecast future day's activity but rather focused on the issue of continuous vs discrete , a problem/opportunity in itself. Please don't downvote/criticize my response as not being pertinent to the question but rather being oriented to a bigger issue.

If you have 303 days of data and you want to predict/simulate tomorrow , why would you use the observed history for all 303. The answer is ..iF you assume that tomorrow is like every other day then you are good to go. What we find is that if you actually model arrivals as it relates to day-of-the-week , you can then get a conditional distribution for tomorrow reflecting the day-of-the-week using the residuals from a simple (perhaps toooo simple !) model. The residuals reflect the conditional distribution around the expected value for tomorrow and can be harvested to provide a monte-carlo distribution for tomorrow.

Now just a little bit more realistic. If there are unusual values (the source of your fat-tailed distribution ) they can be identified along with ant time trends, level shifts , day-of-the-month effects and of course memory effects to effectively construct a conditional distribution reflecting unknown/un-identified sources of variation. With a "richer model" we will be able to get a better expectation for tomorrow and the uncertainty in that expectation.

Now the good news is that if anomalies ( one-time pulses) have been detected and remedied to make a prediction ( N.B. all predictions are simulations and all simulations are predictions ..they are synonyms ) it is now possible (and correct) to enable the possibility of anomalies occurring in the next period. If the next day is a Monday and no anomalies have been observed in the past on any Monday then pulses will b expected tomorrow BUT if previous Mondays have been effected then anomalies will be appropriately pro-rated.

I point you to to a reference that discusses the simulation/prediction of activity for the demand for daily cash cash.http://autobox.com/cms/index.php/afs-university/intro-to-forecasting/doc_download/53-capabilities-presentation slide 49 ..

In summary the statistical action is all about the residuals as they are equivalent to an adjusted observation incorporating factors that can reflect/explain identified variation e.g. holiday effects and even particular day-of-the-month effects

Incorporation of forecasting methods as a precursor to "simulation" is clearly on the horizon (so to speak !).

EDITED AFTER RECEIPT OF DATA:

At first glance your integer data looks very straight-forward BUT when you have time series data you need to have a complete set i.e. no missing dates so that calendar "features" can be detected. Please fill in all dates . The fact that your extraction didn't fill in 0's is typical of accounting systems. Secondly I suggest that you upvote my answer and accept it to close the current question and then open up a new question that is more honest and direct as compared to this question. Also request that forecasts need to be integers because only integers can arise.

enter image description here

EDITED AFTER RECEIPT OF DATA:

I took your first product (AR) and blank-filled the missing dates and obtained 443 daily historical values (1/4/16-3/21/17 ) enter image description here . Since the series is short I disabled Holiday effects detection along with Monthly Indicators (although there was some evidence of this) and introduced the data to AUTOBOX , my tool of choice. If one were to naively simulate simply based on the histogram then one would draw samples from here enter image description here essentially showing no discimination for the day being predicted/simualted. A more nuanced approach would be to model the data and partition historical variability to signal and noise with the noise being the conditional distribution as the basis for randomness/simulation enter image description here . This is the histogram of errors from a model which used DAILY EFFECTS as a predictor while isolating exceptional values and a level shift. A summary of the descriptive statistics by day is here enter image description here

Here is the Actual and Cleansed graph enter image description here and the Actual/Fit/Forecast graph enter image description here

The equation is here enter image description here with Forecasts here the next 21 days (444-464) enter image description here reflecting possible anomalies in the future.

To illustrate , this is the forecast distribution/simulation for day 444 ( 1 period out ) 3/22/17 a Monday enter image description here while this is for day 445 enter image description here. So simulating the future requires a prediction for the future as all days are possibly different in their expectations and an estimate of the uncertainty(randomness) around that prediction . Forecasts are made and then integerized because all of the history is reported as integers . Here is a pix of the output showing history and projections enter image description here

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you for you concern I can see now, that I have over simplified the question. From other forums I am used to simplifying my questions as much as possible. The data is not actually that of a shop, but rather pallets for a freight-terminal. The output is not used as a forecast, but rather as input for a larger model, showing how adjusting processes will impact the operations in the terminal. One of the problems the terminal is facing, is actually that it is almost imposible to identify all variables for a proper forecasting model. $\endgroup$
    – johs32
    Commented Nov 24, 2017 at 1:21
  • $\begingroup$ Whereas what you might say is true that it is almost impossible to EXCLICITELY identify the true causal variables it is possible to model their IIMPLICIT effect. A correctly formed ARIMA model allows you to use the history of the series ( which has been caused by exogenous variables) to provide an expectation and uncertainty for the future if the true predictor.causal series behave in a consistent manner i.e. paribus ceteris. $\endgroup$
    – IrishStat
    Commented Nov 24, 2017 at 1:30
  • $\begingroup$ Arima models are being used currently, with some success, to forecast the total amount of freight. (However the use of arima made for some problems this year. As spring came a month earlier than usual, suddenly tons and tons of soil was in demand.) What would really be benificial though, is to forecast more details on the type- and destination of the freight, which, due to the very large number and variety of variables, and their fluctuations, Arima models are simply not precise enough to provide valuable forecasting results on this level of detail. $\endgroup$
    – johs32
    Commented Nov 24, 2017 at 2:06
  • $\begingroup$ it all depends on the degree of granularity at the type & destination level and the quality of the ARIMA models/software that you are employing. If you are not identifying and incorporating features like level shifts , local time trends and such then perhaps you need better tools. In either case if you wish to contact me off line at my email , please do so and I will further evaluate your issues $\endgroup$
    – IrishStat
    Commented Nov 24, 2017 at 2:58
  • $\begingroup$ even better .. post one of your series here ,,,, I sense that you might be confronted with data that has low counts e.g. 0,1,0,0,0,0,1,0,0,2.1,0,0... and are trying to "fit a model" that will generate integer forecasts .But I am only surmising this .... $\endgroup$
    – IrishStat
    Commented Nov 24, 2017 at 11:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.