New answers tagged sampling
0
Upon reading the question, it is not clear to me that you have completely followed, what at least one researcher has described, the steps consistent with the associated high-standard ascribed to the stratified random sampling scheme.
To assist, I refer you to this dissertation work which enumerates seven critical steps. Further, the steps themselves make one ...
1
The second screenshot (if possible, try to avoid these and typeset the text in TeX to make the site more searchable) still is "incomplete" in that the last S.E. still depends on unknown quantities, viz. $\sigma_1$ and $\sigma_2$.
Hence, $S.E.(s_1-s_2)$ will have to be replaced with some estimator thereof, call it $\widehat{S.E.}(s_1-s_2)$, in order ...
0
As a back-of-the-envelope calculation, you can estimate the means for the two salary bands as 0.83 and 0.69. The standard error of each of these estimates is less than $\frac{1}{2\sqrt{n}} < \frac{1}{2\sqrt{1024}} < \frac{1}{64}$ < 0.0156. So the difference between your mean estimates for the two salary groups is more than 9 times the standard error ...
1
It is typically not ideal for a model to be trained on 'an equal number of instances from each class', unless that is representative of the real population. See Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?. The point of stratified random sampling isn't to remove class imbalance, because, as you point out, it does not do ...
0
To answer your question directly, I'll start by considering a specific
example. Suppose $X \sim \mathsf{Norm}(\mu=100,\sigma=15)$ and the sample size is $n=100.$ Then it
is reasonable that your histogram will have a bin $(80,90).$
The number $Y$ of observations in this bin is binomial: $Y\sim\mathsf{Binom}(n=100, p),$ where $p = P(80 < X < 90) = 0.1613,...
2
Compared to the difficulty of getting confidence intervals for quantiles in the frequentist setting, Bayes handles this most elegantly. It is easiest to do by taking a few thousand draws of the bivariate posterior distribution of $\mu$ and $\sigma$, computing the quantile, e.g., $\mu + \sigma \Phi^{-1}(q)$, and analyzing the distribution of these derived ...
1
An exact simulation is possible if you can find another sequence from which you can draw simulations $\{p_i\}_{i>0}$ such that $\max\left\{\frac{q_i}{p_i}\right\}=M<\infty$ by means of rejection sampling.
The algorithm goes as follows:
Obtain a sample $j$ from distribution $Q$ and a sample $u$ from $\mathrm{Unif}(0,M)$.
Check if $u<\frac{q_j}{p_j}$....
3
The answer is yes.
Tabulating such a sample produces a sequence of $m$ integers with values between $0$ and $r$ inclusive. "Uniform" suggests making the counts exchangeable in the sense that any sample in which the integer $i$ appears $k_i$ times, which can be summarized as the vector $\mathbf{k} = (k_0,k_1,k_2, \ldots, k_{m-1}),$ is just as ...
1
The two quantities
$$
F(\tilde y|D) = \int_\Theta \int_{\mathcal X} F(\tilde{y}|\tilde{x},\theta) p(\theta|D) p(\tilde{x})\, d\tilde{x}\, d \theta \tag{1}
$$
and
$$
f(\tilde{y}|D) = \int_\Theta \int_{\mathcal X} f(\tilde{y}|\theta, \tilde{x}) p(\theta| D) p(\tilde{x}) \, d\tilde{x}\, d \theta \tag{2}
$$
are identical in that one is the cdf and the other one ...
0
There are algorithms that sample random variates using both the PDF and CDF of a distribution. One example is the inversion-rejection method described in chapter 7 of Non-Uniform Random Variate Generation (Devroye 1986), which works for any unimodal distribution for which the mode is known.
5
Isn't it just like sampling in general? You ideally want to sample IID from the target distribution, so that sample mean looks like population mean. If you don't sample IID, you have a sampling bias and you have to correct for that, which costs you samples in the sense that only a subset of your samples are independent (because autocorrelation decays).
1
You can approach the function but not get equal
Suppose we are given a coin with arbitrary (unknown) head probability $p$, I am wondering if there is an easy-to-implement algorithm for generating a $\min\{p, 0.5\}$ coin for any $p\in [0,1]$. Function $f(p) = \min\{p, 0.5\} \geq \min\{p, 1-p\}$ so the famous paper by Keane and O’Brien guarantees such an ...
1
Perhaps someone can post a more enlightening answer as to why, but judging from the last two US elections, I'm forced to conclude exactly what I had suspected prior to the 2016 elections:
It appears our polling methodologies in fact do not work for elections.
There isn't yet a consensus on why this appears to be the case, but some hypotheses I've found have ...
2
One paper I am aware of is called "A Practical Implementation of the Bernoulli Factory" by A. C. Thomas and J. H. Blanchet, 2012. Essentially, if the function is piecewise linear (which includes the function at issue here), the method involves finding a sequence of polynomials that converge from above to that function. But unfortunately, the paper ...
2
I'm assuming you have free access to the population of strings and so your only concerns are computational. The only real speed up you can get is by trading memory for speed, which might be relevant when your population is large.
When the strings are long it takes you $O(n) $ to convert from the string to the number you are interested in.
You can, before ...
2
For your case it seems simplest to use inverse transform sampling.
Then you need to express the quantile function. For your case this requires I integrating your pdf which will be a piecewice exponential function and linear function. Then inverting this, which will be some function with logarithms.
$$F(x) = \begin{cases}
0 & x<0\\
c_1-e^{-\lambda_1x}, ...
0
Just treat it as a conditional distribution. So sample $x$ according to its distribution, then nest a for loop for the appropriate bounds and append to a df.
df <- vector()
x = rdist(coefficients = [coef], size = 1)
n = int(size of trials)
count = 0
while (count < n){
if ( x > 0 & x<= a){
df = append(df, rpois(1,lamda1*x)
}
...
4
In the real world approach, we collect data first and then see how it
is distributed. In other words, the data came before the distribution.
Therefore, I would think that the artificial distribution must have
likewise came from artificial data, even though I didn't start out
with the data. I started out with the distribution. given that in the
real world it ...
4
You have a contingency table. Under the null hypothesis where there is no relationship between column and row variable, each cell count can be estimated from its row * column probability as you have in the code.
When you simulated the data by using a random uniform distribution, you basically cut the counts without consideration for the row or column ...
1
Since $f$ is bijective, you could also implement its inverse $f^{-1}(h) = x$. For example, to invert the first layer of the network $h_{I_1}^1 = x_{I_1}, h_{I_2}^1 = x_{I_2} + M(x_{I_1})$, you can compute $x_{I_1} = h_{I_1}^1, x_{I_2} = h_{I_2}^1 - m(h_{I_1}^1)$. Section 3.2 describes how to do this for any general coupling layer.
To sample (unconditionally)...
4
Your problem is related to the problem of simulating Boolean circuits that take inputs with a separate probability of being 0 or 1. This is called the stochastic logic problem.
In this sense, Qian and Riedel (2008) proved that a function can arise this way if and only if it's a Bernstein polynomial whose control points all lie in [0, 1]. In fact these are ...
1
I don't know how to generate that function, but there is a simple way to generate increasingly good approximations of it by consuming an increasingly large of inputs. Namely, you observe $N$ inputs, where $N$ is an odd integer. Then:
If the majority of those inputs is $0$ you "decide" that $p<1/2$. So you take an addtional input an output that.
...
1
Make the assumption that $X_1,\dots,X_n\overset{iid}{\sim} \text{Distribution}$ have a mean $\mu$ and a variance $\sigma^2$. The $\text{Distribution}$ need not be normal (though it could be).
I've given the full derivations. My suggestion is to scroll down one line at a time and try to anticipate my next line so that you can prove it yourself.
MEAN
$$\mathbb{...
0
As @astel notes, there are much better choices available. Stef van Buuren describes options for imputation of categorical data in Section 3.6 of his freely accessible Flexible Imputation of Missing Data. Briefly, the best way to proceed is with probabilistic approaches based on logistic (or multinomial, for multi-class problems) regression, using as much ...
0
Random imputation is certainly a valid imputation method, though it is not often used as there are better alternatives. It’s advantages are; it preserves the distribution of the data, it is easy to implement, is computationally efficient and has the benefit of only imputing values that are observed in the dataset.It is also unbiased under the missing ...
0
If this is a multivariate dataset, randomly imputing missing values based on the records available from that same variable is generally not a good way to proceed.
The records with missing values may represent a non-homogeneous subset of the overall population, and have different statistical properties from the overall data.
There may be strong correlations ...
3
As I mentioned in my comment, matching a histogram to a density in general requires some scaling considerations, as explained in this answer.
However, there are several issues with your code.
$Z$ is $N(0,1)$, but for some reason you generate $\nu+1$ standard normal RVs and take the mean, which has distribution $N(0, 1/\sqrt{\nu+1})$.
$V$ has $\nu$ degrees ...
Top 50 recent answers are included
