30

You have a discretized version of the negative log distribution, that is, the distribution whose support is $[0, 1]$ and whose pdf is $f(t) = - \log t$. To see this, I'm going to redefine your random variable to take values in the set $\{ 0, 1/N, 2/N, \ldots, 1 \}$ instead of $\{0, 1, 2, \ldots, N \}$ and call the resulting distribution $T$. Then, my claim ...


26

This depends on the models (and maybe even software) you want to use. With linear regression, or generalized linear models estimated by maximum likelihood (or least squares) (in R this means using functions lm or glm), you need to leave out one column. Otherwise you will get a message about some columns "left out because of singularities"$^\dagger$. But if ...


18

Firstly, feel free to ask questions like this on our users' list (http://mc-stan.org/mailing-lists.html) where we discuss not only issues related to Stan implementations/optimizations/etc but also practical statistical and modeling questions. As to your question, it's absolutely a fine approach. There are many ways to justify it more rigorously (for ...


17

Methods of fitting discrete distributions There are three main methods* used to fit (estimate the parameters of) discrete distributions. 1) Maximum Likelihood This finds the parameter values that give the best chance of supplying your sample (given the other assumptions, like independence, constant parameters, etc) 2) Method of moments This finds the ...


16

This is an answer to @jbrucks extension (but answers the original as well). One general test of whether 2 samples come from the same population/distribution or if there is a difference is the permutation test. Choose a statistic of interest, this could be the KS test statistic or the difference of means or the difference of medians or the ratio of ...


16

The proper terminology is Cumulative Distribution Function, (CDF). The CDF is defined as $$F_X(x) = \mathrm{P}\{X \leq x\}.$$ With this definition, the nature of the random variable $X$ is irrelevant: continuous, discrete, or hybrids all have the same definition. As you note, for a discrete random variable the CDF has a very different appearance than for a ...


15

So you've been told you need an appropriate distance measure. Here are some leads: Clustering mixed data A generalized Mahalanobis distance for mixed data Estimating the Mahalanobis distance from mixed continuous and discrete data Generalization of the Mahalanobis distance in the mixed case Distance functions for categorical and mixed variables ...


15

If that's what your course said, it's wrong. While discrete distributions can have a finite number of possible outcomes, they are not required to; you can have a discrete distribution that has an infinite number of possible outcomes - the number of elements should be no more than countable. A common example would be a geometric distribution; consider the ...


14

By definition your distribution is discrete, because you can obtain all the values by counting. Your confusion may stem from two sources. One is that often people assume that discrete also means finite. This is not true, e.g. the Poisson distribution is defined on the non-negative integers, which is an infinite countable set $[0,+\infty)$. Another source ...


11

Let's assume $r_i$, the rank of list element $i$, has a value in $\{0, 1, \ldots, n-1\}$ for a list with $n$ elements (ties can be broken randomly). Then we could define the probability of selecting $i$ to be: $$p_i = \frac{\alpha^{r_i}}{\sum_{k=1}^n \alpha^{r_k}}$$ This is basically just an appropriately normalized truncated geometric distribution, and it ...


10

Discrete Data can only take certain values. Example: the number of students in a class (you can't have half a student). Continuous Data is data that can take any value (within a range) Examples: A person's height: could be any value (within the range of human heights), not just certain fixed heights, Time in a race: you could even measure it to ...


10

There are two parts to your question - how to display discrete data (a data visualization issue) and how to do it in Python (a "what function do I call" issue). I will deal with the first one. With discrete distributions, there are a number of possible ways to display data. Leaving aside direct implementation issues for the present, I see three main ...


10

While @dsaxton's answer is correct, I think it makes it more difficult for beginners in statistics to grasp the concept of variance, so I'll offer another answer that helps you get a better "feel" for the what the variance is actually "doing." An equivalent expression for the variance in this case is: $Var(X)$ =$ \sum_{i=1}^6(X_i-\bar{X})^2\over{6}$. ...


10

Two common distributions for count data are the poisson or the negative-binomial one. If we fit these to your data, the NB works a lot better: library(MASS) # for fitdistr() xx <- 0:20 counts <- c(49, 36, 42, 26, 22, 22, 8, 12, 2, 4, 7, 0, 1, 1, 1, 1, 2, 1, 0, 1, 0) obs <- rep(xx,counts) poisson.density <- length(obs)*dpois(xx,mean(obs)) nb &...


9

As is often the case in statistics, it depends on what you mean. If you mean "I calculate my test statistic on a sample drawn from a discrete distribution and then look up the standard tables" then you'll get a true type I error rate lower than the one you chose (possibly a lot lower). How much depends on "how discrete" the distribution is. If the ...


9

In a sense, what you have done is characterize all nonnegative integer-valued distributions. Let's set aside the description of the random process for a moment and focus on the recursions in the question. If $f_n = p_n (1 - F_{n-1})$, then certainly $F_n = p_n + (1-p_n) F_{n-1}$. If we rewrite this second recursion in terms of the survival function $S_n =...


9

One option is to still use the KS test statistic, but instead of using the standard p-value from the KS test (which as you say is not appropriate when estimating from the data), calculate the p-value using a permutation test. The basic steps would be: Calculate the KS test statistic for the data as is (divided by the estimates). Now combine the 2 datasets ...


8

I've had to deal with this kind of problem in the past, and I think there could be 2 interesting approaches: Continuousification: transform symbolic attributes with a sequence of integers. There are several ways to do this, all of which described in this paper. You can try NBF, VDM and MDV algorithms. Discretization: transform continuous attributes into ...


8

In the case $p(n) = p < 1$, we have some known properties. We can solve the recurrence relation $$ F(n) = p + F(n-1)(1-p); \; F(0) = p $$ has the solution $$ F(n) = P(N \le n) = 1- (1-p)^{n+1} $$ which is the geometric distribution. It is well studied. The more general case of $p(n)$ can probably not be computed in closed form, and thus likely does ...


8

There are expressions you can write down, but I hope you realize how uninformative they are. Saying that the variables are not known to be indpendent, without saying anything else, gives no usable information. It's like saying that you have a friend whose name is not known to be Bob, then asking what you can say about your friend's height and age. So, here ...


8

Perhaps an example will help with the understanding. The following is some R code (I don't have easy access to Minitab and R is free): n <- 100 one <- rbinom(10000, n, 0.45) two <- rbinom(10000, n, 0.55) out <- sapply(seq(along=one), function(i) prop.test( c(one[i],two[i]), c(n,n) )$p.value ) plot(one,two, asp=1, pch=16, col=ifelse( out &...


8

I am gonna be a bit imprecise, but hopefully intuitive. Discrete and continuous probability distributions must be treated differently. For any value in a discrete distribution there is a finite probability. With a fair coin, the probability of heads is 0.5, with a fair six sided die, the probability of a 1 is one sixth, etc. However, the probability of any ...


8

As one of the authors of the methods you're using, I can say with some certainty that the answer to your Question 1 (can you apply the fitting and hypothesis-test methods to a dataset that contains all recorded events in a system) is "yes". In fact, in the 24 datasets that we analyzed in Clauset, Shalizi and Newman, "Power-law distributions in empirical data....


8

Your question raises small questions of terminology and more interesting questions of how to think about data. I will stick with your question and focus on discrete variables. Most of what I say carries over to continuous variables, but with some need for re-wording and/or some differences in procedures. First off, the mode is usually introduced and ...


8

It's wrong because - as the answer explained - there are discrete atoms at 0 and 2. By that cdf, you can wait exactly 0 time with positive probability (similarly with 2). Because of that, the waiting time is mixed, not continuous. Presumably you've been given definitions of all three. How are continuous r.v.s defined? If it's not immediately clear from ...


8

There are many possible ways to discretise a continuous variable: see [Garcia 2013] On page 739 I could see at least 5 methods based on chi-square. The optimality of the discretization is actually dependent on the task you want to use the discretised variable in. In your case logistic regression. And as discussed in Garcia2013, finding the optimal ...


8

There are sloppy ways and rigorous ways. The sloppy ways are shorthands, like "$X\in\{1,2,3\}$", that are either nonsensical or (in this example) just plain wrong when interpreted according to the correct conventional meanings of the symbols. (The second statement literally means $X$ is one of three specified integers--which aren't random variables at all.)...


8

There are two pieces to this: first you need to figure out what the individual probabilities are, then you need to plot them somehow. A binomial PMF is just a set of probabilities over a number of 'successes'. A bivariate binomial PMF will be a set of probabilities over a grid of possible combinations of 'successes'. In your case, you have $n_i = n_j = ...


7

As far as I know, and I've researched this issue deeply in the past, there are no predictive modeling techniques (beside trees, XgBoost, etc.) that are designed to handle both types of input at the same time without simply transforming the type of the features. Note that algorithms like Random Forest and XGBoost accept an input of mixed features, but they ...


7

Assume $K$ is the largest value of $k$ (i.e. the largest month/period observed in your data). Here is the hazard function with a fully discrete parametrization of time, and with a vector of parameters $\mathbf{B}$ a vector of conditioning variables $\mathbf{X}$: $h_{j,k} = \frac{e^{\alpha_{k} + \mathbf{BX}}}{1 + e^{\alpha_{k} + \mathbf{BX}}}$. The hazard ...


Only top voted, non community-wiki answers of a minimum length are eligible