New answers tagged

1

I'll give my two cents, though it's far from a "perfect" solution (if one exists). First, as for the issue of varying dataset sizes, you can normalize the token frequencies by the document count, and choose a threshold over the normalized frequency rather than the absolute one. Second, as you noticed, using a threshold in such cases is problematic, ...


0

Likelihood and entropy measure different things. The former tells us how likely it is to observe the dataset $X$ given the parameters $\theta$. The latter is supposed to tell us how much uncertainty is in the distribution. So, comparing these two metrics is like comparing weight and hair length of a student: can be done, but somewhat pointless. Let us ...


4

You have two questions so I'll answer them in turn (1) Why are the maxima Gumbel distributed? Essentially, this comes from the extremal types theorem which is a limiting distribution for block maxima. The most common limits distribution is the CLT which states $$ \frac{\bar{X} - \mu}{\sigma} \to N(0,1) \text{ as } n \to \infty$$. Now suppose we have data ...


2

One way to proceed is to transform to polar coordinates $(Z_1,Z_2) \mapsto (R,\Theta)$ such that $$Z_1=R\cos\Theta \quad, \quad Z_2=R\sin\Theta$$ Then, $$\frac{Z_1^2+Z_2^2}{Z_1+Z_2}=\frac{R}{\cos\Theta+\sin\Theta}=\frac{R}{\sqrt 2\sin\left(\Theta+\frac{\pi}4\right)}$$ Now $R$ and $\Theta$ are independently distributed with $R$ having a Rayleigh distribution ...


5

Let $X_1,...,X_m \sim \text{SRSWOR}(n)$ denote the sampled values. For simplicity, suppose we select an odd number of values $m=2k+1$ from the distribution, so the median is the order statistic $X_{(k+1)}$. To facilitate our analysis, let $R(x) = \sum_{i=1}^m \mathbb{I}(X_i \leqslant x)$ ​denote the number of sample values that are no greater than $x$. ...


1

If we assume that all bitstrings are equally likely then the problem is quite simple. Each matched pair of bits in the bitstrings will match with fixed probability $\tfrac{1}{2}$ and the number of matches has distribution: $$M \sim \text{Bin}(n, \tfrac{1}{2}).$$ Consequently, the probability of at least $n/2$ matches (for even $n$) is: $$\begin{align} \...


0

I achieved some conclusions. Is this reasoning correct? If it is not, I can conclude something about the comparison between Stable and t-skewed distribution with the output below? Yes it is. Indeed BIC criterion can be applied among non nested models and if the number of parameters is the same, BIC boil down in a comparison among LLF. Therefore, a fortiori,...


9

Regression analysis accommodates situations where the explanatory variables/covariates can have any values, including zero. There is no particular model needed for this --- it can be implemented in almost any model. If there are lots of zeros for a particular covariate, the only issue this creates is that it affects the leverage of the data points and one ...


3

Let's think about this for a moment. Continuous state space means that there are infinitely many states linked via transition probabilities such that they are all accessible by traversing a single graph (let's suppose). The expected values in such a case are asymptotic, meaning that they either have a cleverly constructed closed form (such as mean of a ...


1

A rather comprehensive guide I just found is the Field Guide to Continuous Probability Distributions by Gavin E. Crooks. At 210 pages it seems to contain a wealth of information, but as said in comments, for users a subject-specific guide might be more useful, and this is not. Convincingly, this document comes with a version number and a long revision ...


0

You would<most often be better off using a Weibull plot for assessing distribution fit, see Weibull plot to assess goodness of fit. If you rather want/need a formal goodness of fit test, see A goodness of fit test for the Weibull distribution, but the advice at Is normality testing 'essentially useless'? would equally apply for Weibull testing, ...


5

Sample from a Pareto distribution. If $Y\sim\mathsf{Exp}(\mathrm{rate}=\lambda),$ then $X = x_m\exp(Y)$ has a Pareto distribution with density function $f_X(x) = \frac{\lambda x_m^\lambda}{x^{\lambda+1}}$ and CDF $F_X(x) = 1-\left(\frac{x_m}{x}\right)^\lambda,$ for $x\ge x_m > 0.$ The minimum value $x_m > 0$ is necessary for the integral of the density ...


0

By using law of total expectation: $$\mathbb{E}[e^{tXY}] = \mathbb{E}[\mathbb{E}[e^{tXY}|Y] = \mathbb{E}[e^{tXY}|Y=0]P(Y=0) + \mathbb{E}[e^{tXY}|Y=1]P(Y=1) = \mathbb{E}[e^{0}]0.7 + \mathbb{E}[e^{tX}]0.3 = 0.7 + 0.3M_{X}(t)$$


0

You probably meant this to be a density function on the positive real line, I will assume that. Then, introduce some notation. Let $\lambda(x)$ be a positive function for $x\geq 0$, with integral $\Lambda(x)=\int_0^x \lambda(t) \; dt$. Then $\Lambda$ will be a positive, increasing function. In addition, assume that $\lim_{x\to\infty} \Lambda(x)=\infty$. ($\...


0

There is no good reason not to do such a generalization, which would unite the Weibull and inverse Weibull distributions. So reasons must be historical or accidental. Also see the comments thread.


1

Hints: You can fill in the details! The distribution of the differences $X_1 - X_3$ and $X_2 - X_4$ do not depend on $\mu$, so we can assume $\mu=0$. This reduces to ordinary exponential distributions. The distribution of the difference of two iid exponential distributions is a Laplace distribution. The absolute value of a Laplace distributed random ...


3

It depends on the estimation method you use, which you haven't really specified clearly. From what you have written here, I take it that you are concerned with the case where you estimate an unbounded distribution first and then truncate the result, instead of estimating directly from a truncated distribution. In that case, you might over- or under-...


13

First, note that the range of $\DeclareMathOperator{\P}{\mathbb{P}} Y$ is $(1, \infty)$. First find the cumulative distribution function of $Y$ in the usual way: $$\begin{align} F_Y(t) & = \P(Y \leq t) = \P(e^X \le t) \\ & = \P( X \leq \ln(t) ) \\ & = F_X( \ln(t) ) = 1-e^{-\lambda \ln(t)} \\ ...


2

If the threshold value is $1000,$ then it seems safe to use a Poisson distribution with mean $\lambda = 20,$ because $P(X > 1000) \approx 0,$ for $X\sim\mathsf{Pois}(\lambda = 20).$ [Computation in R.] 1 - ppois(1000, 20) [1] 0 A simulation of a million samples of size $n=200$ from $\mathsf{Pois}(20),$ with truncation of any observations above $1000.$. ...


1

I guess the problem assumes that you'll use the exponential distribution for the waiting time, P(next tram time > t) = 1/(average time) * exp(-t / average time). This is equivalent to saying that tram arrival times follow Poisson process. And then use a memory-less quality of the exponential distribution which is P(next tram time T > a + b | T > a) =...


4

The corresponding density function can be found to be $$ f(x)= (cx+ c-1) e^{-cx}, \quad x\ge 0 $$ which can be written as a mixture $$ cx e^{-cx} + (c-1) e^{-cx} $$ which is a mixture of two gamma distributions, assuming $c\ge 1$ (necessary to avoid negative mixture weights). But from the form of the cdf (cumulative distribution function) given in the post,...


1

Using the definition of the Beta function and that $n$ and $y$ are integers with $0 \le y \le n$ on the support $$\int_0^1 \binom{n}{y} x^y(1-x)^{n-y}dx \\ =\binom{n}{y}\int_0^1 x^y(1-x)^{n-y}dx \\ =\binom{n}{y}\operatorname{B}(y+1,n-y+1) \\= \binom{n}{y}\frac{\Gamma(y+1)\Gamma(n-y+1)}{\Gamma(n+2)} \\= \frac{n!}{y!(n-y)!}\frac{y!(n-y)!}{(n+1)!} \\= \frac1{n+...


1

It's perhaps a little more complex than it appears, requiring multiple steps. First, let's define the train interarrival time ($x$) cumulative distribution function as $F(x)$, where $\int xf(x)dx = 7.5$. Now, it should be intuitively clear that given, say, two interarrival times $x_1$ and $x_2$, the probability that our hapless student arrives during ...


4

Hint Pull out the binomial coefficient, the remaining integral is related to the Beta function. You already used the knowledge about $Y$ being conditionally binomial by writing down the integral.


0

The answer is mostly no. And the reason is because of the context of the covariates being currencies. Your covariates are not on the same scale across data points. Inflation rate fluctuates over time. \$10 today is different from \$10 ten years later. For some unstable countries, values change rapidly between days. The multi-modal distribution on the right ...


1

Assuming that the trams are independent on each other and runs the same way 24/7, their arrivals can be modeled as Poisson process, whose inter-arrival time follows exponential distribution with mean = 7.5 minutes. Student S entering the queue every day can be considered as a resampling of the process with a random shift t, which does not impact the ...


0

Tracking links to previous "similar" problems, one finds several over the last four years--some on this site and some on the physics site. Unfortunately, none of them is clearly stated and several mention more than one probability model. So this "Answer" of mine has to be speculative. Suppose the proportion $p_i$ of particles in state $i$ ...


1

This sounds like a queueing process. You could model two separate processes $-$ the arrival time of the student and the departure time of the tram $-$ and use these to construct a model for wait times (the difference between departure and arrival times). Alternatively, you could model the wait times directly. Typically processes of this nature are modeled ...


0

Here is a simple example using the text Ulysses. I'll use a simple bash script to acquire the type frequencies: cat ulysses.txt | tr 'A-Z' 'a-z' | tr -dc 'a-z ' | tr ' ' '\n' | sort | uniq -c | sort -k1,1nr | awk '{print $1}' > ulysses_freq.txt And then use R to fit model using mle. The normalized frequency of the element of rank k, $f(k;s,N)$, is ...


2

First, a little intuition. When $\tau = 1$, $X = S$. So $P(X = S) = p$. That means that once we're given $S = s$, $X = s$ with probability $p$. When $\tau = 0$, then knowing that $S = s$ doesn't tell us anything about $X$, and $X \sim U[0,1]$ with probability $1-p$. This mean that $X|S = s$ is a mixture of a uniform distribution and a degenerate distribution....


0

There is a difference. If we denote the probability vector $\pi$, then the first case of 10,000 repetitions of $X \sim \text{Mult}(1, \pi)$ retains information about the $X_i$ in each of the 10,000 repetitions. The second case of one repetition of $Y \sim \text{Mult}(10000, \pi)$ does not retain information about the $X_i$ in each of the 10,000 trials. Here ...


0

Log transformations are sometimes helpful in statistical analyses of how long it takes people to react to an event. For example, in Effects of category diversity on learning, memory, and generalization a single analysis involved several dependent variables (p. 295). The MANOVA test for significant predictors assumed that the dependent variables were ...


2

It shows that something is wrong. AIC is defined as $2k - 2\log \hat L$, so it would be infinite when the likelihood $\hat L$ is zero since $\log(0) = -\infty$. This would happen if the data has probability of zero or probability density of zero under the model. In such a case, your model is not appropriate for the data, since the data is considered to be ...


0

Let $X_1,\cdots ,X_{100}\overset{iid}{\sim} F_X(x)$ for some distribution $F_X(x)$ that has an expected value $\mu$. Since $X_1$ is distributed as $F_X(x)$, $\mathbb E[X_1]=\mu$. Since $X_2$ is distributed as $F_X(x)$,… In the absence of other information about the subject, the value that you expect is the population value.


0

Another idea is to use weights to weight out the outlier. Then use the resulting fitted model (say m1) as starting values for the distribution parameters, when fitting the model (say m2) with all the data points (i.e. without weights). This is done by using argument, e.g. start.from=m1, in the function m2 <- gamlss( ,start.from=m1)


4

Disclosure: I wrote the samc R package used in this answer This answer is more of a supplement to Stephan Kolassa's answer In it, he showed how to construct a transition matrix representing the problem: Image credit: Stephan Kolassa's answer Now, as indicated in comments/another answer, this matrix can be simplified a variety of different ways, but I'm ...


1

Seems very close to Shannon - related theorems. If you posit "HTH" as your "end of message" string, you want to estimate the chance that "HTH" shows up in random data. I suspect a little digging into his work will provide the equations/ formulas of interest. And because I can't resist, "HTH"


0

You can try Monte Carlo simulation: Generate a large number of datasets with the properties that you know about (initially normal, truncated 20%, anything else you know or can reasonably assume?) Use some goodness-of-fit measure to compare these datasets to the one given (you can start with simply plotting and eyeballing) Focus on the things you know about ...


1

linear or any n-dimensional hyperplane cuts the data so it minimizes the MSE (or any other metric used to measure distance). However if you have a probability distribution (including multi-dimensional distributions), it provides the location of the cluster and the likelihood of finding a sample in a specific region. If the data is clustered, then a ...


1

If you can safely assume that your underlying data is normally distributed, then as Otto Kässi writes, you have a truncated normal distribution. If you know where it was truncated, this is good (and with 800 data points below the point of truncation, simply using the maximum observation will likely be a sufficiently good estimate of it, and any uncertainty ...


1

The main challenge in this kind of question is whether you can identify a hard upper limit. If so, you can use a rescaled beta distribution (assuming there is also a hard lower limit), or possibly a flipped-and-shifted distribution like the gamma or the negative binomial (assuming there is no lower limit), all possibly with some zero inflation (which would ...


2

If possible I would show a plot of the distribution, rather than just some summary statistics, since it is more informative. Alternatively, if some summary statistics are required, I would use a five-number summary or similar, rather than trying to find a distribution which fits the data. Fitting a distribution might look like a more elegant solution, since ...


1

Mixture distributions make the most sense if we have grounds to suspect that the data belong to multiple different subpopulations, as per the textbook you cite. Usually, we don't know which of the subpopulation a given instance belongs to - and we may not even know how many such subpopulations exist. As such, we could even consider straightforward ANOVA as a ...


2

It's always hard to prove a negative. However, I have never seen such an interpretation. The naive idea that the weighted sum of quantiles would be the corresponding quantile of the mixture distribution is of course false. I would be interested in being proven wrong and look forward to downvotes (and actual such interpretations).


12

There is a fun way to answer this problem using martingales, and in particular using https://en.wikipedia.org/wiki/Optional_stopping_theorem. I first saw this trick in the book A First Look at Rigorous Probability Theory by Jeffrey S. Rosenthal, in the martingale chapter. (I don't have the book in front of me at the moment but I'll edit and add a page or ...


1

Final formula appears to be sum(i = 1 to length(pattern) : if (first i flips of pattern match the last i flips of pattern) then P(series of i flips matches first i flips of pattern)^-1 else 0) (For a fair coin, this reduces to Conway's algorithm; it can probably be proven by a similar method.)


31

At any given point in the game, you're $3$ or fewer "perfect flips" away from winning. For example, suppose you've flipped the following sequence so far: $$ HTTHHHTTTTTTH $$ You haven't won yet, but you could win in two more flips if those two flips are $TH$. In other words, your last flip was $H$ so you have made "one flip" worth of ...


17

First, you can refactor your R code to be (IMHO) a little more legible, also using pbapply::pbreplicate() to get a nice progress bar: n_sims <- 1e5 library(pbapply) results <- pbreplicate(n_sims,{ flips <- NULL while(length(flips)<3 || !identical(tail(flips,3),c("H","T","H"))){ flips <- c(flips,...


7

Here is a somewhat clumsy brute-force method to obtain the probabilities and order statistics. Getting the mean will take more work. So first just generate the possible sequences and associated probabilities where "HTH" are the last 3 flips (with that sequence not occurring previously). Then look for patterns. For integer patterns the go to ...


Top 50 recent answers are included