We’re rewarding the question askers & reputations are being recalculated! Read more.

# All Questions

Tagged with
117 questions
Filter by
Sorted by
Tagged with
40 views

### SMOTE in unbalanced dataset with binary features

after reading different posts about unbalanced datasets I didn't make my mind clear about my specific problem so that's why I'm posting a new question. In my case, I have a dataset with around 20K ...
28 views

### How can I generate a completely balanced combination

Attaching a code that generates unique combinations of 8 letters but, it still lacks a condition wherein all letters should have equal counts per column. For my code, basically, letter a to h should ...
36 views

### Validation Set Accuracy Significantly Higher than Hold out Test Set

I'm building a binary classifier, where each record is a task, and the response variable is whether it was completed on time. I'm using random forest My data set spans from 2000-2015 My hold out ...
30 views

343 views

### How can I create a multivariate random normal distribution from separate normal distributions? [duplicate]

In R I can sample from a multivariate normal distribution as follows: ...
36 views

### Permutation using probability versus direct frequencies in R?

I'm trying to understand if there is any difference to the following approaches in permutating values based on their frequencies. Here is an example of the raw data, which gets binned by ...
60 views

### Multistage Survey Sample for Sock Color

Here is the setup: We want to estimate the proportion of people in a country who wear black socks versus not-black socks. We have 100 cities (primary sampling units) in the country. In each city ...
92 views

### Unexpected pattern in posterior predictive check with set.seed()

I have stumbled upon R behavior involving set.seed(), sample() and rbinom() which I don't ...
106 views

### Checking the legitimacy of a simple one-way Anova simulation

Background: My question is based on This Paper (see its Appendix for R code). The paper is trying to basically obtain the empirical distribution of a statistic called $\eta^2$ (eta-squared) via large ...
19 views

### Distribution of a subset and its parent

We have a worldwide distribution (A) of number of friends (e.g. ~Poisson or in my case an empirical one); we also have a subset which gives the distribution (B) of number of friends for country X. In ...
30 views

### Subjectivity of models and their performance tests. Sampling issue

I am trying to classify cases according to binary response variable. For this purpose I chose random forest and classification tree methods. So, as it is usually done, I randomly split the data into ...
475 views

### Data balancing as using ROSE(Random Over Sampling Examples) function

I want to balance the distribution of data using the ROSE function. The balance of the data was adjusted, but the value of the independent variable was changed ...
56 views

### Imputing missing counts in a time series (with variance?)

We're using a sonar to enumerate migrating fish. The date & time are recorded with each event that a fish passes the sonar station. We've noted two very strong time-varying components to the ...
1k views

### How to generate 2 correlated Beta random variables

I was wondering if it might be possible to generate 2 correlated $Beta$ random variables? In other words, I want to generate two Beta random variables which can be said to have come from two Beta ...
183 views

### Samples size with sample proportion close to 0 or 1

For a future monitoring program on small water bodies we want to calculate the sample size. The bodies of water are so small that their number easily exceeds 100.000 in the monitoring area and ...
3k views

### SMOTE sampling in caret package in R

when using caret packge in the trainControl you can use "smote" sampling. what is the default parameters the train in caret are using for smote?? parameters such as: perc.over = 300, k = 8, perc....
950 views

### Manually generate random sample in Gaussian mixture model

I want to generate (manually) a random sample in the Gaussian mixture model: $$f_{\theta}(x) = \sum_{k = 1}^{K}\pi_k f_{\mathcal N(\mu_k, \sigma^2_k)}(x)$$ Here is my work: ...
3k views

### NbClust with large data sets. Sampling?

I have a data set which is 3 columns and about 50,000 rows. I am trying to do some cluster analysis and would like to speed it up. Most examples online use data sets like ...
72 views

### Is there a more efficient method for creating a probability distribution?

I have written a short function in R to estimate the expected number of mutations that will be observed in a set of DNA sequences. The parameters are the mutation rate (x), the length of the DNA ...
2k views

### Random Forest - Unbalanced Dataset for Training & Test

I am doing a classification modelling(using R and random forests) for a website where only 2% of the visitors convert. Now given the behavior and attributes of a visitor I want to predict probability ...
306 views

### Subsample to follow a normal distribution

I have a problem, that sounds very simple in theory but I fail to implement a good solution. Let my data be a sample of a continuous variable that that follows a normal distribution (m1,v1), ...
30 views

### Age-correcting survey sub-samples

I have a survey of N = 338 responses. If I analize the Age column, I can see that it is more or less normally distributed (mean =...
302 views

### Sampling from under/over-dispersed count data in R

I am currently working a some datasets with count data in R, in which the response is the number of activities of a given type that were performed in one day by a population. For each type, I build ...
341 views

### How are the null distributions created for hypothesis testing?

I'm struggling to connect the idea of a sampling distribution of a test statistic to the null distribution. If I run many simple linear regressions on samples extracted from a population of data I ...
503 views

### Stratified sampling with replacement of participants in R

I am wondering whether in R, there is a possibility to draw a stratified sample with replacement of participants. Often, in surveys, we need each group to have certain size. Hence if one respondent in ...
73 views

### Repeated catch–mark–release (urn problem)

Imagine I have a small pond with some fish in it and I want to estimate its population size. I don't have a lot of resources at hand, in fact all I do have is a fishing pole, pen and paper, and a lot ...
79 views

### dependent t-test giving opposite result

I have a dataframe as below. ...
39 views

### How to do the right sampling for a classification model

I'm looking for the best sampling method for classification modeling (logistic regression OR Xgboost ) that will decrease the number of observations in the training data set while keeping the bias and ...
267 views

### How to sample from a distribution so that mean of samples equals expected value?

Given a random variable $X$, how do I have obtain $N$ random variates of $X$ so that the mean value of my samples equals the expected value of $X$? E.g. let $X$ have uniform distribution on the ...
19 views

### increase class proportion in a class proportion vector without changing the sum of the vector

Say I have a vector x which defines the number of samples in a given class: x <- c(250,362,10,246,30) In this case, there are five classes. The resulting ...
61 views

### How can I estimate the area by sample points and Traverses

I have a study region with an area of 1 that has several lakes, one with an island. The total lake area happens to be 0.27. In total 20 random points were placed into this study region. Furthermore, ...
612 views

### Sampling from copula given a particular value of marginal

I am working in the bivariate case with copulas as follows: I have two marginal Gamma distributions $f_1=Ga(\alpha_1, 1)$ and $f_2=Ga(\alpha_2, 1)$ that are bound by a copula Frank copula $C$, with ...
210 views

### want to calculate significance of pairwise sharing between lists. Standard hypergeometric test seems inappropriate

I want to calculate if the percentage of sharing of strings (genes in this case) between lists is significantly more than expected by chance for multiple pairwise comparisons between lists of strings. ...
882 views

### Statistical test for random sample of data

I'm trying to determine if some particular measurements - in this case taken from a subset of genes of interest (50 genes) - show a significant difference to the rest of the population (15000 genes), ...
125 views

### Any good working method for sampling from estimated distribution in R

Suppose I have a number of points $\{x_i\}_{i=1}^N$ and I want to estimate the smooth distribution and sample from it. I have tried at the moment a lot of things, and all of them show poor ...
I'm trying to implement an algorithm that estimates time-homogeneous Markov chain with continuous state space. The probability of transition from state $i$ to state $i+1$ has continuous conditional ...