New answers tagged

0

Calling summary on fixef does not directly yield descriptive statistics as the commands together also report certain standard errors and p-values. First applying as.matrix seems to do the trick, see below. library(plm) data("Produc", package = "plm") zz <- plm(log(gsp) ~ log(pcap) + log(pc) + log(emp) + unemp, data = Produc, index = c("state","...


0

I have found this article that discusses an alternate method for encoding variables in the supervised case. Since it demonstrates the use of frequency encoding for supervised learning, the same encoding would also be feasible for the unsupervised case. The authors also present their own (superior) method to encoding variables.


1

As long as the frequency of the categories isn´t correlated with the target, it shouldn´t make any sense. The paper you mentioned is solely targeted for anomaly detection where it is obvious that the frequency and target variable (Outlier Y/N) may be connected. In an extreme example, you have a variable with two distinct values that have an equal amount ...


2

Goodfellow et al. 2016, p. 65 states: A Gaussian mixture model is a universal approximator of densities, in the sense that any smooth density can be approximated with any specific nonzero amount of error by a Gaussian mixture model with enough components. M. Carreira ascribes this property to kernel density estimation with reference to Scott 1992 ...


2

You can resort to the bootstrap version of Student's t-test. It works as follows: Compute the sample mean and standard deviation for each group and label the results $X_1$ and $s_1$ for group 1, and $X_2$ and $s_2$ for group 2. Set $d_1=\frac{s_1^2}{n_1}$ and $d_2=\frac{s_2^2}{n_2}$, where $n_1$ and $n_2$ are the sample sizes. Generate a bootstrap sample ...


0

The expected value of a non-central $t$ with non-centrality $\delta$ and d.f. $\nu$ is $$ E\left[t\right] = \delta \sqrt{\nu/2} \frac{\Gamma\left((\nu-1)/2\right)}{\Gamma\left(\nu/2\right)}. $$ Thus it is trivial to find an unbiased estimator of $\delta$: $$ \hat{\delta} = t \sqrt{2/\nu} \frac{\Gamma\left(\nu/2\right)}{\Gamma\left((\nu-1)/2\right)} $$ In R: ...


3

The density function of the Benini distribution is strictly quasi-concave, and is strictly decreasing if and only if $\beta \leqslant \alpha (1+\alpha)/2$. The mode of the distribution has the explicit form: $$\text{Mode} = \hat{x} = \sigma \cdot \exp \bigg( \bigg\{ \frac{\sqrt{1+8\beta} - (1+2\alpha)}{4 \beta} \bigg\} \bigg),$$ where the brackets $\{ \...


5

The series converges to a distribution function. It can be evaluated in closed form. Upon identifying the terms varying with $n,$ write your function in a simpler form as $$F_{U_i}(y)=\frac{2(R^{\alpha}y/\theta)^k}{\Gamma(k)}\sum_{n=0}^\infty \frac {(-R^\alpha y / \theta)^n}{n!(k+n)((k+n)\alpha+2)} = \frac{2x^k}{\Gamma(k)}\sum_{n=0}^\infty \frac{(-x)^n}{n!...


2

After having done some homework on the subject, I think I've got a better handle on the proof that I find in [1]. I wanted to take an opportunity to set down my understanding for pedagogic purposes. Scope: I'm going to limit this answer to the case of a strictly monotonic cumulative distribution function. Its my understanding that, in his answer to this ...


1

The old rule of thumb was that the sample size is too small when the expected count of one of the cells of the contingency table is lower than five. Recall that the expected count of a cell is the ratio $$\frac{row\_count \times column\_count}{total}.$$ In http://www.biostathandbook.com/small.html and http://www.biostathandbook.com/fishers.html, owing ...


0

What is the most robust statistical method (test) to compare each coin flip observation distribution with the other coin observation distribution to determine whether both of distributions are the same or different? This sounds a lot like a binomial test of proportions. Comparing binomial proportions has been analyzed several ways, including Frequentist ...


1

The result can be found in the 2012 paper Some results on the truncated multivariate $t$ distribution by Ho et al. First, let $T(\cdot\,|\,\nu)$ be the cdf of the (untruncated) standard $t$ distribution with $\nu$ degrees of freedom. In the following exposition, $a$ and $b$ are the lower and upper truncation limits, respectively. Before we can give the ...


0

At an intuitive level, if the standard deviation is 500, then assuming a normal distribution, 66% of the time the loan repayments will be within 900 of the mean. So a loan repayment of 5 above the mean is such a tiny fraction of 900 that it's unlikely that the probability would be so low. One can verify this by drawing out the curve and the probabilities ...


1

I was able to find an actual formulaic solution to my question in the paper here: https://www.research.manchester.ac.uk/portal/files/84031132/FULL_TEXT.PDF $$E[X] = G_\nu(1)(A_\nu^{-\frac{\nu-1}{2}}-B_\nu^{-\frac{\nu-1}{2}})$$ $$E[X^2] = \frac{\nu}{\nu-2}+G_\nu(1)(aA_\nu^{-\frac{\nu-1}{2}}-bB_\nu^{-\frac{\nu-1}{2}})$$ Where: $a$ is the lower bound $b$ ...


2

A mixture PDF is $$f_M(x)=\frac{1}{N}\sum_{i=1}^N f_i(x)$$ Integrating for finding the mean yields: $$\mu_M=\int_{-\infty}^\infty xf_M(x)dx={1\over N}\sum_{i=1}^N\int_{-\infty}^\infty xf_i(x)dx={1\over N}\sum_{i=1}^N\mu_i$$ For the variance, let's find the second moments first: $$E[M^2]=\int_{-\infty}^\infty x^2f_M(x)dx={1\over N}\sum_{i=1}^N\int_{-\infty}^\...


1

Logistic regression with a logit link function tries to find the expected (average) odds of Y given X. But since, we are assuming the dataset to be gaussian in the first place, we know that the mean and std_dev that gives maximum likelihood is nothing but the mean and std_dev of given dataset. Yes, the sample mean would give the Maximum Likelihood ...


2

There are errors in your second expression in particular terms involving $\sigma$. Assuming $X_1, X_2$ are independent normal with variance $\sigma^2$. Here $\Sigma = \begin{bmatrix} \sigma^2 & 0 \\ 0 & \sigma^2\ \end{bmatrix}=\sigma^2I_2$, then $|\Sigma|=\sigma^4$ and $|\Sigma|^\frac12=\sigma^2$ \begin{align} P(x_1, x_2) &= \prod_{i=1}^2 P(...


4

Your solution is the correct one because sample mean has the deviation $\sigma/\sqrt{n}$. Also, when we substitute $1620=\frac{900^2}{500}$, your professor doesn't seem to be using $n$ in its solution.


4

Great question! (Almost a professional statistician here!) Let's see if I can try to answer your question. Whuber gave a comment on what the typical strategy that someone might follow. I'll try to complement that by providing a couple papers that talk specifically about properties of tests when assumptions are violated. I'm doing this since, if I had a ...


1

Since the function $g$ is symmetric about the y-axis, and since this implies that $g(0)=0$, it is sufficient to find the function for values $z>0$. Thus, without loss of explanatory power, we will look at the transformation only over these values. Let $F_*$ and $Q_*$ denote the CDF and quantile function of the generalised error distribution. Using the ...


2

The answer to the question in the title is obviously "yes", because you can discretize every continuous distribution by splitting the real axis into cells $I_k=[x_{min}+(k-1)\Delta x,x+k\cdot\Delta x)$ and then define \begin{equation}P(k) := P(X\in I_k) = \int_{I_k} f(x)\,dx\end{equation} Alternatively, you could compute the CDF at only discrete points: $$F(...


1

Your assumption is flawed. While you might be able to set up a situation in which a normal distribution is fit to binomial data, the choice is not yours to make, but rather dictated by the form of the data and the phenomenon generating it. Binomial data is discrete, and if generated by the binomial distribution, it should not be fit by a continuous ...


1

A "probability distribution" can be uniquely described either by its CDF or the corresponding probability measure. Contrarily, a density function is often not a unique description of a probability distribution, and so it would not usually be used for statements of the equality of distributions.$^\dagger$ Unfortunately, there is no standard notation for ...


1

A heavy-tailed distribution on the other hand has a heavier tail, in the sense that the probability of observing a large value x is proportional x-a, where a is a positive integer. Exponential distributions are reasonably well described by the first and third quartile as their deviation around the mean is limited, and these quartiles will likely capture most ...


1

No, the sample mean of an independent sample from a Poisson distribution is NOT Poisson distributed. It is clear from the fact that a Poisson random variable can only have integer values, but the mean of such a sample does not need to be an integer. But, the sum of the sample ($n$ times the mean) do have a Poisson distribution, and all wanted probabilities ...


2

So you assume a continuous mixture of Poisson distributions, say $$ X \mid \Lambda=\lambda \sim \mathcal{Pois}(\lambda),$$ where $\Lambda$ then has some distribution on the positive line. If we assume that $\Lambda$ has a gamma distribution, results that $X$ has a marginal negative binomial distribution. You asks what happens if $\Lambda$ has a normal ...


1

I am assuming (perhaps wrongly) that intensity is recorded in some standard way (e.g. like weight being recorded in kilograms). In this case, the measures are on the same scale now. If you standardize, they will be on different scales, but each sample will have the same mean and sd. Which do you want? That depends on what you are trying to do. Why do the ...


1

One possible way is to compute the CDF and hen differentiate with respect to $y$. Here's a start, try to complete it. Let $y \in (0,1)$, \begin{align}Pr(Y \le y)&= Pr(-4X+4\sqrt{X}-y \le 0)\\ &=Pr\left(\sqrt{X} \le \frac{1-\sqrt{1-y}}{2}\right) + Pr\left(\sqrt{X} \ge \frac{1+\sqrt{1-y}}{2}\right)\\ &=\left( \frac{1-\sqrt{1-y}}{2}\right)^2 +1- \...


1

If $Y(t) = \sum_{n=1}^\infty \mathsf 1_{(0,t]}(S_n)$ is the counting process associated with the renewal times $\{S_n\}$, then by definition $$\{Y(t) = n\} = \{S_n\leqslant Y(t) < S_{n+1}\}. $$ This is simply because $n$ renewals have occured by time $t$, and the $(n+1)^{\mathrm{th}}$ renewal has yet to occur. As for $\{S_{n+1}<t\}\subset\{S_n<t\}$, ...


3

Assuming you want to visualize how the binomial approaches the normal as n grows I'd do it as follows: Choose some value for the second parameter, $p$ (like say 0.5 or 0.2) draw a standard normal cdf and compute and plot the cdf of a standardized binomial at some $n=n_1$ draw a standard normal cdf and compute and plot the cdf of a standardized binomial at ...


2

It is unclear from your question exactly what your objective is for this analysis, but if I understand correctly, you would like to be able to quantify the similarity (or "distance") between two MtG decks, so that you can alter a deck to make it more or less similar to other decks in the sample. As I will show below, this exercise is essentially equivalent ...


1

As a supplement to whuber's answer, I've written a script in Python which goes through each step of the sampling scheme. Note that this is meant for illustrative purposes and is not necessarily performant. Example output: n=10, s=20, k=4 Starting grid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . X X X X X X X X X X Filled in grid X X . . X ...


2

Most of the answers here seem to cover two approaches - Bayesian and the order statistic. I'd like to add a viewpoint from the binomial, which I think the easiest to grasp. The intuition for a beta distribution comes into play when we look at it from the lens of the binomial distribution. The difference between the binomial and the beta is that the former ...


1

When using logarithmic sampling, it means that you are sampling uniformly on the log-transformed interval. Here are 20 random values logarithmically sampled between 0 and 10: And here are those same values plotted on the log-axis: So, quick answer, small values are sampled more frequently!


1

One notable distance that can be considered in Fourier space is the maximum mean discrepancy (MMD). One first selects a positive semi-definite kernel $k : \mathcal X \times \mathcal X \to \mathbb R$ corresponding to a reproducing kernel Hilbert space (RKHS) $\mathcal H_k$. The MMD is then \begin{align} \operatorname{MMD}_k(\mathbb P, \mathbb Q) &= \...


2

The exact answer is going to depend greatly on the type of network, the inputs, how it's trained.... For a simple way to see this: If we're at a (local) optimum, the full gradient (across the entire training dataset) will be zero. In the interpolating regime common to modern neural networks, the individual gradient for each training point may even be ...


7

I want to ... uh ... "attenuate" @whuber's amazing answer, which @TomZinger says is too difficult to follow. By that I mean I want to re-describe it in terms that I think Tom Zinger will understand, because it's clearly the best answer here. And as Tom gradually uses the method and finds that he needs, say, to know the distribution of the samples rather than ...


40

I agree with X'ian that the problem is under-specified. However, there is an elegant, scalable, efficient, effective, and versatile solution worth considering. Because the product of the sample mean and sample size equals the sample sum, the problem concerns generating a random sample of $n$ values in the set $\{1,2,\ldots, k\}$ that sum to $s$ (assuming $...


7

You have the assumption that the errors are multivariate normal $\epsilon \sim \mathcal MVN(0,\Sigma)$, where the errors is the complete vector $\epsilon = (\epsilon_1,...,\epsilon_N)$ assuming that there are $N$ observations. In some cases observations will be independent hence the covariance matrix does not need to allow for serial correlation hence all ...


5

Assumptions: We are talking about a linear regression model $X$ is not random Then $$y = X\beta + \epsilon$$ $$\hat{y} = Hy = X(X'X)^{-1}X'X\beta + H\epsilon$$ $$\implies \hat{y} \sim N(X\beta ,H\sigma^2)$$


14

The question is under-specified in that the constraints on the frequencies \begin{align}n_1+2n_2+3n_3+4n_4&=100M\\n_1+n_2+n_3+n_4&=100\end{align} do not determine a distribution: "random" is not associated with a particular distribution, unless the OP means "uniform". For instance, if there exists one solution $(n_1^0,n_2^0,n_3^0,n_4^0)$ to the above ...


4

You can use sample() and select specific probabilities for each integer. If you sum the product of the probabilities and the integers, you get the expected value of the distribution. So, if you have a mean value in mind, say $k$, you can solve the following equation: $$k = 1\times P(1) + 2\times P(2) + 3\times P(3) + 4\times P(4)$$ You can arbitrarily choose ...


1

Here is a simple algorithm: Create $n-1$ random integers in the range $[1,4]$ and calculate the $n^{th}$ integer for the mean to be equal to the specified value. If that number is smaller than $1$ or larger than $4$, then one by one distribute the surplus/lacking onto other integers, e.g. if the integer is $5$, we have $1$ surplus; and we may add this to the ...


4

$Z$ has two possibilities: $a, a+b$ with probabilities $1-p,p$ respectively. We can use your formulation to write it compactly, but note that we can only substitute $a,a+b$ into this equation: $$P_Z(z)=p^{{Z-a}\over b}(1-p)^{1-{{Z-a}\over b}}$$ Just as we can only substitute $y\in\{0,1\}$ in the original Bernoulli PMF. For other values, the PMF is $0$ ...


5

First question: The "standard deviation" is defined mathematically, there is no other way to calculate it. It isn't meant to capture all the information about the shape of the distribution; no one numeric value could hope to do so except in the case of one-parameter distributions. Note that your expression for the variance: $$V(x) = \sigma^{2} = \int_{-...


0

I would suggest calculating the cophenetic distance(also called cophenetic coefficient). For items belonging to multiple sets, we can evaluate the average cophenetic distance of the object from the other set. Here is a link to the brief description of the same: https://en.wikipedia.org/wiki/Cophenetic It can be evaluated from this function in python: https:...


1

Forget about it, checking the scipy code I realized that the formula in my textbook is just false. Correct formula starts with $\frac{12}{N(N+1)}$ not with $\frac{12}{N(N-1)}$


1

The probability of $X^t$ is given by a truncated normal distribution (truncation of lower tail at $t$) and you can compute it using Bayes theorem. Let's say $X=X^t + t$, where $X$ is another random variable describing the total lifetime of the observed process, then you are interested in finding the posterior probability $$ p\left(X=x \mid t^* \right) = \...


2

I'm late to the party but it seems this question still needs an answer, and, surprising to me, I don't see any other questions on this topic. So I'll go ahead and offer this. To summarize, the QMPE approach is to formulate a likelihood function in terms of the quantiles of the observed data and not the observed data themselves. Then parameters are sought ...


Top 50 recent answers are included