The Stack Overflow podcast is back! Listen to an interview with our new CEO.

All Questions

Tagged with
Filter by
Sorted by
Tagged with
16
votes
1answer
31k views

Generating random samples from a custom distribution

I am trying to generate random samples from a custom pdf using R. My pdf is: $$f_{X}(x) = \frac{3}{2} (1-x^2), 0 \le x \le 1$$ I generated uniform samples and then tried to transform it to my custom ...
14
votes
1answer
7k views

How to draw random samples from a non-parametric estimated distribution?

I have a sample of 100 points which are continuous and one-dimensional. I estimated its non-parametric density using kernel methods. How can I draw random samples from this estimated distribution?
13
votes
3answers
589 views

Estimate the size of a population being sampled by the number of repeat observations

Say I have a population of 50 million unique things, and I take 10 million samples (with replacement)... The first graph is I've attached shows how many times I sample the same "thing", which is ...
13
votes
3answers
11k views

Cluster Big Data in R and Is Sampling Relevant?

I'm new to data science and have a problem finding clusters in a data set with 200,000 rows and 50 columns in R. Since the data have both numeric and nominal variables, methods like K-means which ...
11
votes
3answers
12k views

How to resample in R without repeating permutations?

In R, if I set.seed(), and then use the sample function to randomize a list, can I guarantee I won't generate the same permutation? ie... ...
11
votes
2answers
7k views

Sampling with replacement in R randomForest

The randomForest implementation does not allow sampling beyond the number of observations, even when sampling with replacement. Why is this? Works fine: ...
9
votes
2answers
1k views

Inverse CDF sampling for a mixed distribution

The out-of-context short version Let $y$ be a random variable with CDF $$ F(\cdot) \equiv \cases{\theta & y = 0 \\ \theta + (1-\theta) \times \text{CDF}_{\text{log-normal}}(\cdot; \mu, \sigma) &...
9
votes
3answers
6k views

How to re-sample an XTS time series in R?

I have an irregularly spaced XTS time series (with POSIXct values as index type). How can I build a new time series sampled at ...
8
votes
3answers
3k views

Using Uniform Distribution to Generate Correlated Random Samples in R

[On recent questions I was looking into generating random vectors in R, and I wanted to share that "research" as an independent Q&A on a specific point.] Generating random data with correlation ...
7
votes
2answers
690 views

How to simulate effectiveness of treatment in R?

Let's say I want to write a simulation for the table below to decide if Xylitol treatment and ear infections are independent. How would I go about doing this?
7
votes
1answer
587 views

Probability of failure in a finite population

I regularly inspect finite populations for failures (we make custom products in batches of ~500-800). Currently, we inspect every product for failure, which is quite a bit of work. I want to reduce ...
7
votes
2answers
7k views

How do I sample without replacement using a sampling-with-replacement function?

I vaguely recall from grad school that the following is a valid approach to do a weighted sampling without replacement: Start with an initially empty "sampled set". Draw a (single) weighted sample ...
7
votes
2answers
489 views

How to test if social structure is non-random and resulting from genetic relatedness – and how to deal with demography effect

I am trying to construct (undirected) social network based on co-occurence of individuals. Clustering algorithm will be later applied on this network to find some distinct subgroups. Issue is that ...
7
votes
2answers
2k views

How to calculate sample size for comparing the area under the curve of two models?

Because I would like to calculate the sample size for comparing the area under the curve (AUC) of 2 models (cross-sectional study, predictor = continuous variable). Can you point me which function in ...
7
votes
0answers
1k views

How to generate 2 correlated Beta random variables

I was wondering if it might be possible to generate 2 correlated $Beta$ random variables? In other words, I want to generate two Beta random variables which can be said to have come from two Beta ...
6
votes
1answer
15k views

mtry and unbalanced use of predictor variables in Random Forest

I am working on the Random Forest prediction, with the focus on the importance of predictor variables, and have a question regarding understanding of mtry and the actual usage of variables in the ...
6
votes
1answer
2k views

Problems generating a sample from a custom distribution with log

I'm trying to generate a sample from a family of distributions. In particular I would like to be able to obtain a sample from the survival function: $$1-F(x) = c x^{-a} \log^b(x)$$ with a proper ...
6
votes
1answer
150 views

Generate tail of distribution by a given sample in R

I have a sample of measurements from a real life device which misses all the measurements that are less than some threshold (given device is not precise enough). From theory and also measurements ...
6
votes
3answers
2k views

Multistage sampling in R

I've got a dataset similar to this: ...
6
votes
1answer
1k views

Difference between calculated inclusion probability and what is returned by sampling function?

I have a (small) population from which I wish to sample. I assign probabilities proportional to $y$. I enumerate the possible samples and then determine the probability of each sample occurring based ...
6
votes
1answer
1k views

Regarding the sampling procedure in Adaboost algorithm

The AdaBoost algorithm states that it is to train a classifier based on the training data according to a weight vector. Assume the size of training data is N, the weight vector is of dimension N as ...
5
votes
1answer
739 views

How to calculate the chance of getting completely unbalanced groups? (with R)

I want to simulate or calculate probabilities of combinations of group membership for different sample sizes (e.g., n= 3, 4, 5, 10, or 100) for two groups (of the same sample size). Each outcome could ...
5
votes
1answer
3k views

What are the rules / guidelines for downsampling?

I have a data set with ~ 7 million rows, of which ~ 100k are positives. I'm looking to shrink the data by keeping all the ...
5
votes
1answer
79 views

Testing whether sampling (convex polytope) is uniform

Currently, I am sampling points from: i) a convex polytope (i.e. Ax <= b) ii) a high dimensional simplex The algorithms I am using are hit-and-run and a simple version of Bayesian bootstrap. I ...
4
votes
4answers
1k views

How exactly does null hypothesis testing work

I'm trying to gain a better understanding of how null hypothesis testing works. I have 3 questions related to the code below: Am I right in saying the probability of each t value in ...
4
votes
2answers
4k views

Sample from distribution given by histogram

Given a histogram obtained using given data points, how do I randomly sample from the distribution predicted by the histogram? Any conceptual comment / R code would be welcome.
4
votes
2answers
2k views

A robust R package to do MCMC and Gibbs sampling

I need to make linear model for which I need to do Gibbs sampling in MCMC simulations. The model needed to be fitted is a linear mixed model. Please suggest me for a robust R package for this task.
4
votes
2answers
828 views

Incidence density sampling in R

I was wondering if there is an R package to perform incidence density sampling for a case-control design. That is, for matching in a case-control design; accounting for multiple matching covariates.
4
votes
1answer
2k views

techniques for sampling graphs? (possibly implemented in r packages)

Let's say I have a very large graph that proves impractical for visualization ends and I wanted to sample a random subgraph. (I know that I can filter out a subgraph via measures like degree, ...
4
votes
2answers
446 views

How to reconstruct database from cross-tabulated data?

I am given a set of almost 100 crosstabulations (most 2D, but some three dimensional) based on a single, yet unavailable dataset. My task is to perform basic statistical tests ($\chi^2$, Mann-Whitney, ...
4
votes
1answer
36 views

Permutation using probability versus direct frequencies in R?

I'm trying to understand if there is any difference to the following approaches in permutating values based on their frequencies. Here is an example of the raw data, which gets binned by ...
4
votes
1answer
339 views

understanding proposal distribution sequential importance sampling in R

From the article of wikipedia http://en.wikipedia.org/wiki/Particle_filter I see that one generate samples from the proposal $\pi(x_k^{(L)}\vert x_{o:k-1}^{(L)},y_{1:k})$, however, the role of $y_{1:...
4
votes
1answer
686 views

Absent categorical data levels in Bootstrap samples

I have a huge dataset ($n$ around five million, $p$ around three thousand) for a classification problem, where my interest is predictive class probabilities for test data, not the target. I shall be ...
4
votes
1answer
3k views

SMOTE sampling in caret package in R

when using caret packge in the trainControl you can use "smote" sampling. what is the default parameters the train in caret are using for smote?? parameters such as: perc.over = 300, k = 8, perc....
4
votes
1answer
4k views

Importance Sampling to evaluate integral in R

I have asked the question here also. However, there might be something wrong with my theoretical understanding hence I'm asking here as it is more relevant. Kindly do not diss without looking first. ...
4
votes
1answer
487 views

Correct variance estimation for random sample post-stratification weights

I'm dealing with a random sample and I have a question about its poststratification weights. Let's assume that: this is a random sample of addresses, but post-stratification weights have been ...
4
votes
0answers
175 views

Samples size with sample proportion close to 0 or 1

For a future monitoring program on small water bodies we want to calculate the sample size. The bodies of water are so small that their number easily exceeds 100.000 in the monitoring area and ...
4
votes
1answer
297 views

Sampling from under/over-dispersed count data in R

I am currently working a some datasets with count data in R, in which the response is the number of activities of a given type that were performed in one day by a population. For each type, I build ...
3
votes
3answers
2k views

In R, how to sample from the output of combn(a,b) if the “a choose b” is too large?

I would like to sample 1 million vectors from the set of vectors $$\mathbf{v} = \{v_1,v_2,...,v_n \} $$ subject to $\sum_{i=1}^n v_i = 1$ $\forall i: v_i \ge 0$ each $v_i$ has at most 2 decimal ...
3
votes
2answers
2k views

What is behind JAGS (Just Another Gibbs Sampler)?

I have been using JAGS but I am not quite sure how it actually simulates it values. I need to know in a general sense what's going on in the background.
3
votes
3answers
2k views

Downsampling excessively sampled curves

I'm looking for a quick (event if not so accurate) way to deal with over-sampled curves using R. Consider the following example in which x contains 1000 values in ...
3
votes
1answer
72 views

Is there a more efficient method for creating a probability distribution?

I have written a short function in R to estimate the expected number of mutations that will be observed in a set of DNA sequences. The parameters are the mutation rate (x), the length of the DNA ...
3
votes
3answers
260 views

How to sample from a distribution so that mean of samples equals expected value?

Given a random variable $X$, how do I have obtain $N$ random variates of $X$ so that the mean value of my samples equals the expected value of $X$? E.g. let $X$ have uniform distribution on the ...
3
votes
1answer
1k views

Sampling from empirical distribution

I have a vector of y (min is > 0, max could be 1), for which, i have no idea what distribution is. But based on the data we have, vector y, we can get the empirical cumulative probability distribution,...
3
votes
1answer
750 views

Has anyone publicly shared an implementation of RUSBoost in R?

There's no package available on CRAN, so I was hoping someone in the community had written their own function/package. I see it's been done in MATLAB, so I may just have to start with that and write ...
3
votes
1answer
73 views

Repeated catch–mark–release (urn problem)

Imagine I have a small pond with some fish in it and I want to estimate its population size. I don't have a lot of resources at hand, in fact all I do have is a fishing pole, pen and paper, and a lot ...
3
votes
1answer
279 views

Sample selection and variation in the variable of interest when using panel data

I wonder how to reason when selecting samples. I am doing a panel data regression analysis about how the euro membership correlates with the budget deficit in the member countriues. Using R that is: ...
3
votes
0answers
83 views

How to testing the probability of two factors being found together?

I know how many times a particular factors has been found: values<-cbind(f=c("S1","S10","S3","S4","S5","S6","S7","S8","S9"),count=c(9,8,4,24,20,4,8,21,5)) I ...
2
votes
4answers
3k views

smart sampling techniques in r

I have a large data set of about 1.8 million rows with 80 variables. I would like to find a good technique (code or package) in R that can reduce the amount of training data without damaging the ...
2
votes
4answers
5k views

Sampling data to have specific mean and standard deviation [duplicate]

I have a data that I want to sample such the resultant distribution of values should have specified mean and standard deviation. I can think of rejection sampling to achieve this however that seems to ...