# Tag Info

1

While this reference is not to your ECDF-like estimator on $[0,1]$ I believe it is logically equivalent to the Kaplan-Meier estimator as used in Survival Analysis, even though that's applied to a time range $[0,\infty)$. Estimating the bias would be possible once you have a reasonable estimate of the distribution via kernel smoothing if it is well enough ...

2

If the process parameters are changing the process is out of (statistical) control, as you say. It is not necessarily producing bad product—yet. The idea is to preempt quality problems by investigating signs of process instability & fixing them as necessary, while sparing engineers' time by limiting the number of wild-goose chases they're sent on. ...

0

Don't run clustering (such as k-means) on 1-dimensional data. Why: 1-dimensional data can be sorted. Algorithms that exploit sorting are much more efficient than algorithms that do not exploit this. Look at classic statistics And forget about buzzwords such as "data mining" and "clustering"! For your task, I recommend you use kernel density estimation. ...

5

Here is the R source code for the internal pt function: http://svn.r-project.org/R/trunk/src/nmath/pt.c The relevant code snippet is if(!R_FINITE(n)) return pnorm(x, 0.0, 1.0, lower_tail, log_p); In other words, pt automatically detects an infinite degrees of freedom parameter and calls pnorm

2

Yes. In fact the implementation of the t-distribution in R detects this special case, and simply calls the relevant functions for the normal distribution.

1

The XMeans algorithm can be used to estimate the total number of clusters directly from the data, without human guidance. The Weka package has a Java implementation. An expectation maximization algorithm can also be used to automatically estimate the total number of clusters as well. There is a Weka implementation of that also. In addition, there is at ...

0

It sounds like what you're asking about would lead one to simply combine all the data from each location into a single large sample and use that distribution as the single worldwide distribution of weights. Imagine we only had three locations: A: 125 137 159 117 136 129 133 145 B: 144 137 119 C: 131 128 Then the combined sample of ...

1

I agree with what others have said -- namely that "variance" is probably the wrong word to use (seeing as the function you are considering isn't a probability distribution but a time-series). I think you may want to approach this problem from a different perspective -- just fit the two time series with LOWESS curves. You can calculate 95% confidence ...

0

In you survey, question 1 is a ordinal variable and question 2 is a binary variable. The first thing to do is making histograms and frequency tables of these variables. They should give you already some information. Secondly, you can perform a chi-square test in order to discover whether there are any meaningfull relationships between thes two variables, ...

2

You can proceed some distance with that assumptions, but you cannot recover a specific expression for the probability because the joint distribution can be anything. Using the restriction on the rv's we have $X_3 =1- X_2-X_1$. Using the restrictions on the constants, we have $a_3=1-a_2-a_1$. Then we search for the joint probability $$P( |X_1 - a_1| < ... 1 The skewness and kurtosis of particular distributions are known functions of the distribution. e.g. the Normal distribution has a skewness of 0 and a kurtosis of 3 (often given as an "excess kurtosis" of 0. The formulas for skewness and kurtosis are widely available on the web, e.g. Skewness and kurtosis For any given sample, skewness and kurtosis can be ... 0 I second the Wikipedia recommendation. Standard warnings about Wikipedia apply, but I find it much more readable for self-teaching than any book. Usually I look up whatever probability distribution I'm concerned with at the moment. Everything I learned about power laws I learned from reading Wikipedia, then following up the academic references given there on ... 0 I usually refer to NIST Engineering Statistics Handbook for the common ones (http://www.itl.nist.gov/div898/handbook/eda/section3/eda366.htm) and they recommend the following two books for detailed treatment of the subject: Johnson, Kotz, and Balakrishnan, (1994), Continuous Univariate Distributions, Volumes I and II, 2nd. Ed., John Wiley and Sons. Evans, ... 0 Ideally, you shouldn't be guessing the parameters. For DBSCAN, the epsilon parameter is a distance. Whenever you are using DBSCAN, you should first understand distance in your data set. Without having a working and reliable distance, DBSCAN results won't be convincing. And once you have understood your distances, it should no longer be hard to choose ... 0 If you take a sample, each sample having two (or more) observations, from a larger population (or from a probability distribution), the means (and other statistics) of those samples would have a distribution. This sampling distribution would not be the same distribution as the distribution of the original population. For example, if your original ... 1 An area in statistics where this problem arises naturally is Approximate Bayesian Computation. What you actually want to do is to summarise the whole sample into "informative" statistics that can later be compared using a suitable metric: this problem is not trivial at all. I would even say that it is actually one of the "hot topics" in statistics. It is ... 1 Let's say someone gives you a spreadsheet with a large set of numbers, say 10,000, and says "What distribution does this set of numbers follow (Gaussian, exponential, Poisson, etc.)"? How do you go about determining the answer? If its real data, the actual answer is generally 'none of the above, nor anything else specific you can put a name to'. You ... 0 I think you would formulate hypotheses about the distributions which you test on the data. The structure of data may give you hints on whether a discrete or a continuous distribution is at hand. A famous test is Kolmogorov Smirnov, which compares expected to observed cummulative distribution functions to test a hypothesized null distribution. See the ... 2 Just a very small addition to wolfies answer. Since the probability generating function of the sum is$$ G(t) = \prod_{i=1}^n \left( 1 - p_i +p_i t \right) \, , $$and P(S=k) is the coefficient of t^k, we have$$ P(S=k) = \sum_{\substack{A\subset\{1,\dots,n\}\\ |A|=k}} \left( \prod_{i\in A} p_i \right)\left(\prod_{j\in \{1,\dots,n\}\setminus A} ...

4

Let $X_i$ ~ $Bernoulli(p_i)$ with probability generating function (pgf): $$\text{pgf} = E[t^{X_i}] = 1 - p_i (1-t)$$ Let $S = \sum_{i=1}^n X_i$ denote the sum of $n$ such independent random variables. Then, the pgf for the sum $S$ of $n=16$ such variables is: \begin{align*}\displaystyle \text{pgfS} &= E[t^S] \\&= E[t^{X_1}] E[t^{X_2}] \dots ... 3 @wolfies comment, and my attempt at a response to it revealed an important problem with my other answer, which I will discuss later. Specific Case (n=16) There is a fairly efficient way to code up the full distribution by using the "trick" of using base 2 (binary) numbers in the calculation. It only requires 4 lines of R code to get the full distribution ... 5 (PS) First of all I think Glen_b is right in his above comments on the usefulness of such a test: real data are surely not exactly Pareto distributed, and for most practical applications the question would be "how good is the Pareto approximation?" – and the QQ plot is a good way to show the quality of such an approximation. Any way you can do your test ... 4 So X and Y are independent, with f_X(x) = \frac{e^{-(x-\mu)^2/2\sigma^2}}{\sqrt{2\pi}\sigma} \, , $$and$$ f_Y(y) = \frac{y^{k-1}e^{-y/\theta}}{\theta^k\Gamma(k)} I_{(0,\infty)}(y) \, . $$If Z=X+Y, then$$ f_Z(z)=\int_{-\infty}^\infty f_X(z-y) f_Y(y)\,dy \, . $$Completing the square in the exponent and looking for the terms involving y, ... 2 I think that your ratio is slash distributed. http://en.wikipedia.org/wiki/Slash_distribution It should be possible to derive the probability <1/M from there. Another option perhaps is bootstrapping, where you would take random samples from your data and evaluate the criterion. The proportion of cases in which it is true would be the probability of ... 1 1- Is one of these statistics (KDE's p-value, KS statistic or the two-tailed p-value) recommended for my needs? If so, why? Your needs as expressed do not seem to be sufficiently clearly defined as to differentiate between them. They both test for a difference in distribution. 2- What is the difference between the "KS statistic" and a "two-tailed ... 3 Although your question is very general, the rule is that there isn't a standard way of fitting a distribution to a vector of observations and several methods exist for it. You can start by creating a histogram of your data. In this way, you can immediately see if the shape of the histogram resembles any of the widely known and used statistical distributions ... 0 Maybe fitdistr()? require(MASS) hist(x, freq=F) fit<-fitdistr(x,"log-normal")estimate lines(dlnorm(0:max(x),fit[1],fit[2]), lwd=3) > fit meanlog sdlog 3.8181643 0.1871289 > dput(x) c(52.6866903145324, 39.7511298620398, 50.0577071855833, 33.8671245370402, 51.6325665911116, 41.1745418750494, 48.4259060939127, 67.0893697776377, ... 3 ... I've just noticed you have the 'regression' tag there. If you do have a regression problem you can't look at the univariate distribution of the response to assess the distributional shape, since it depends on the pattern of the x's. If you're asking about checking whether a response (y) variable in some kind of regression or GLM has a lognormal or a ... 2 This is a matter of model selection, of course, assuming that you just want to check whether your data comes from one model or the other and that your goal is not finding the right model among the infinite dimensional ocean of distributions. So, one option is to use AIC (which favours models with the lowest AIC value, and I will not attempt to describe ... 0 Given that the two-sample Kolmogorov-Smirnov test is a nonparametric procedure, you have to dig into the area of Bayesian nonparametrics to find Bayesian analogous tools. Here is a paper where you can start: http://arxiv.org/abs/0910.5060 I warn you that nonparametric Bayesian theory is not easy to digest. 1 OP: what is the distribution of the absolute value of the Skellam distribution Let X ~ SkellamDistribution(a,b), with pmf f(x):$$f(x) = e^{-a-b} \left(\frac{a}{b}\right)^{x/2} I_x\left(2 \sqrt{a b}\right)$$Then, the pmf of Y=|X| will be, say g(y):$$g(y) = \begin{cases}f(0) & y = 0 \\ f(y) + f(-y) & y \ge 1 \end{cases}$$All ... 3 Two quite different questions! Is the absolute value of the difference between two Poisson distributions a Poisson distribution? This one is easily answered: clearly no, since the relationship between the mean and variance doesn't hold. what is the distribution of the absolute value of the Skellam distribution. This second one is a little ... 1 I will denote \hat \theta the maximum likelihood estimator, while \theta^{\left(m+1\right)} and \theta^{\left(m\right)} are any two vectors. \theta_0 will denote the true value of the parameter vector. I am suppressing the appearance of the data. The (untruncated) 2nd-order Taylor expansion of the log-likelihood viewed as a function of ... 1 I would go with Monte Carlo simulation. Pretty straightforward and less chance of mistakes. There may be some slick math-stats solution too, but Monte Carlo is probably faster. 1 From Bayes' theorem, the posterior is proportional to the prior \times the likelihood. That is,$$P(a|E,d,b)\propto P(E|a) P(a|d,b)$$(which you should be able to show for yourself). When you take the product and collect like terms, you get something that will be a density for a, up to a multiplicative constant. If you simplify that result and can ... 2 Yes. By definition, the value of the CDF (call it F_\rho) at (s,t) is the chance that the first component is less than or equal to s and the second is less than or equal to t:$$F_\rho(s,t) = \frac{1}{2 \pi \sqrt{1-\rho ^2}}\int_{-\infty}^t \int_{-\infty}^s e^{-\frac{\frac{x^2}{2}-\rho x y+\frac{y^2}{2}}{1-\rho ^2}} dx dy.$$Performing the x ... 2 The answer would appear to be YES. I am not sure how this is usually proven (since the bivariate Normal cdf does not have a convenient closed form) ... but as a quick thought, I would appeal perhaps to graphical ideas. In particular, consider the contours of the zero mean bivariate Normal as \rho increases, as per: Choose any (x,y) point ... The cdf ... 1 Sounds like the streams you are interested in could be modelled as inhomogeneous Poisson processes. Not surprising that the time between items is not normal --- I'm guessing it is closer to exponential. You may be interested in this white paper, which discusses a simple stat for detecting a change in rate of this type of data. It might provide a ... 1 While I'd normally recommend checking exponentiality by use of diagnostic plots (such as Q-Q plots), I'll discuss tests, since people often want them: As Tomas suggests, the Kolmogorov-Smirnov test is not suitable for testing exponentiality with an unspecified parameter. However, if you adjust the tables for the parameter estimation, you get Lilliefors' ... 0 Under certain regularity conditions, the maximum likelihood estimates follow asymptotically a normal distribution with mean the true parameter values and covariance matrix the inverse of the Fisher information matrix also evaluated at the true parameter values. The Delta method is typically used to derive standard errors for a nonlinear function of the MLEs ... 3 First of all, I think that you should look at the seasonal distributions separately, since the bimodal distribution is likely to be the outcome of two fairly separate processes. The two distributions might be controlled by different mechanisms, so that e.g. winter distributions could be more sensitive to yearly climate. If you want to look at population ... 2 You can use a qq-plot, which is a graphical method for comparing two probability distributions by plotting their quantiles against each other. In R, there is no out-of-the-box qq-plot function for the exponential distribution specifically (at least among the base functions). However, you can use this: qqexp <- function(y, line=FALSE, ...) { y ... 3 I would do it by first estimating the only distribution parameter rate using fitdistr. This won't tell you if the distribution fits or not, so you must then use goodness of fit test. For this, you can use ks.test: require(vcd) require(MASS) # data generation ex <- rexp(10000, rate = 1.85) # generate some exponential distribution control <- ... 3 Are these distributions of something over time? Counts, perhaps? (If so then you might need something quite different from the discussions here so far) What you describe doesn't sound like it would be very well picked up as a difference in variance of the distributions. It sounds like you're describing something vaguely like this (ignore the numbers on the ... 0 (As promised, I am posting the answer to my own question. And there is no need for... convoluted mathematical statistics here, it is indeed straightforward, but it only was revealed to be so because of @Glen_b comments). The cdf of U can be written$$F_U(u) = P(U \le u) = P(U \le u\mid \tilde Z =c)\cdot P(\tilde Z =c) + P(U \le u\mid \tilde Z =0)\cdot ...

1

I have came across a paper where they stated that: "The product of independent log-normal quantities also follows a log-normal distribution", pp-345. It also have very rich understanding of the Lognormal Distribution. You can download the Article from here: http://stat.ethz.ch/~stahel/lognormal/bioscience.pdf As for the Second question, If I come across any ...

2

I know I am resurrecting an old thread but I had recently been looking for similar information and was unable to find it on the forum. I eventually managed to implement a solution for my study and I will add my experiences on using Monte Carlo simulation. Monte Carlo Simulations is a rather broad, all-encompassing term for any statistical simulation method ...

3

IMHO, the matricial notation $Y=X\beta+\epsilon$ complicates things. Pure vector space language is cleaner. The model can be written $\boxed{Y=\mu + \sigma G}$ where $G$ has the standard normal distributon on $\mathbb{R}^n$ and $\mu$ is assumed to belong to a vector subspace $W \subset \mathbb{R}^n$. Now the language of elementary geometry comes into play. ...

1

This can be done using the Law of Total Expectation. $$\text{E} (X) = \text{E}_Y [ \text{E}_{X | Y} ( X | Y)]$$ Now $\text{E}_{X|Y}(X|Y)] = N\cdot p_x = N\cdot Y/M$, so \begin{eqnarray} \text{E} (X) &=& \text{E}_Y [ N\cdot Y/M]\\ &=& \frac{N}{M}\cdot\text{E}_Y [Y]\\ &=& \frac{N}{M}\cdot (Mp)\\ ...

2

Let's assume a seller on some e-commerce web-site receives 500 ratings of which 400 are good and 100 are bad. We think of this as the result of a Bernoulli experiment of length 500 which led to 400 successes (1 = good) while the underlying probability $p$ is unknown. The naive quality in terms of ratings of the seller is 80% because 0.8 = 400 / 500. But ...

Top 50 recent answers are included