We’re rewarding the question askers & reputations are being recalculated! Read more.

All Questions

Tagged with
Filter by
Sorted by
Tagged with
0
votes
1answer
40 views

SMOTE in unbalanced dataset with binary features

after reading different posts about unbalanced datasets I didn't make my mind clear about my specific problem so that's why I'm posting a new question. In my case, I have a dataset with around 20K ...
0
votes
1answer
28 views

How can I generate a completely balanced combination

Attaching a code that generates unique combinations of 8 letters but, it still lacks a condition wherein all letters should have equal counts per column. For my code, basically, letter a to h should ...
0
votes
0answers
36 views

Validation Set Accuracy Significantly Higher than Hold out Test Set

I'm building a binary classifier, where each record is a task, and the response variable is whether it was completed on time. I'm using random forest My data set spans from 2000-2015 My hold out ...
0
votes
0answers
30 views

Generating variables using R

Question: Assume 2 treatments, 1 and 0. Given 120 participants, generate underlying truth with Age as a discrete uniform distribution over integers between [20-60] Gender (G) as $I\{G = F\} \sim \...
0
votes
0answers
23 views

Random Samples from Spliced Distribution

I am studying Clauset, Shalizi, and Newman, Power Law Distributions in Empirical Data (preprint available here) in R. Packages used: ...
0
votes
2answers
267 views

Sampling highly imbalance multi-class response variable

I have a dataset (11000 x 117) with response variable having multiple classes. Here is a plot of class distribution: Some of the classes have only 1 sample in the entire dataset and some have 2, 3 ...
1
vote
0answers
45 views

How to randomly downsample from matrix [closed]

I'm trying to downsample from a matrix of numbers. Each number is the number of times we saw a particular event. By downsample I mean I want to pseudo-randomly select values from each row so they ...
1
vote
0answers
50 views

When to use ordinary, balanced, antithetic, or permutation resampling for bootstrap?

I am using boot and would appreciate any explanation as to when using each of these resampling methods would be recommended in practice. I have come across Do who says that: Simulation results ...
1
vote
0answers
34 views

Does specifying normalizing constant significantly improves Hamiltonian Monte Carlo?

From my understanding the energy function needs only be specified such that it is proportional to the log density, and not specifying the normalizing constant should not greatly impact the sampling ...
0
votes
0answers
28 views

Permutation of datasets to check accuracy of class labels

Currently I am using a proteomics dataset to compare 4 different disorders to healthy controls. Ultimately, I want to find unique and overlapping factors (that are deviant from healthy controls) in ...
1
vote
1answer
20 views

ML Sampling question - Can I increase sample size of current dataset with older copies that have changed over time [closed]

I am working on a multinomial machine learning algorithm that labels stocks with buy/sell signals. My code updates with the most recent quantitative data about the stocks daily, so obviously the data ...
0
votes
0answers
20 views

Best way to measure treatment effect across two treatments

I have two sets of patient-level data for two distinct treatments, unfortunately, the data is only over a 14 week period and I'm hoping to build a predictive model to estimate/simulate what the data ...
1
vote
1answer
35 views

I want to simulate a random sample of length n from DAG of correlated Bernoulli's

Suppose I have a DAG of 4 vertices. Each vertex consists of a Bernoulli of parameter $p$. It is the following: (Z) ---> (Y) (Z) ---> (W) (X) ---> (Y) ---> (W) I hope it is clear. Anyway, I ...
3
votes
1answer
1k views

Sampling from empirical distribution

I have a vector of y (min is > 0, max could be 1), for which, i have no idea what distribution is. But based on the data we have, vector y, we can get the empirical cumulative probability distribution,...
0
votes
1answer
247 views

Number of bootstrap replicates versus number of simulations

I have some confusion over bootstrap simulation in R. Here is the question: I am asked to use the following parameters to produce simulation: 500 bootstrap replicates 1000 simulations Sample size of ...
1
vote
0answers
210 views

Simulating Random Strings using Binomial/Multinomial Distributions

I am simulating some DNA sequences (containing characters A, C, G, and T) in R through specifying i) the number of sequences to generate (num.seqs) ii) the length of the generated sequences (length....
2
votes
0answers
23 views

Reliability engineering design values from tests AS/NZS1170

To get design values (the strength of something, for example) designing to Australian/ New Zealand loading code you do $n$ tests and estimate the coefficient of variation of your parent population, $...
0
votes
0answers
343 views

How can I create a multivariate random normal distribution from separate normal distributions? [duplicate]

In R I can sample from a multivariate normal distribution as follows: ...
4
votes
1answer
36 views

Permutation using probability versus direct frequencies in R?

I'm trying to understand if there is any difference to the following approaches in permutating values based on their frequencies. Here is an example of the raw data, which gets binned by ...
2
votes
1answer
60 views

Multistage Survey Sample for Sock Color

Here is the setup: We want to estimate the proportion of people in a country who wear black socks versus not-black socks. We have 100 cities (primary sampling units) in the country. In each city ...
1
vote
1answer
92 views

Unexpected pattern in posterior predictive check with set.seed()

I have stumbled upon R behavior involving set.seed(), sample() and rbinom() which I don't ...
1
vote
1answer
106 views

Checking the legitimacy of a simple one-way Anova simulation

Background: My question is based on This Paper (see its Appendix for R code). The paper is trying to basically obtain the empirical distribution of a statistic called $\eta^2$ (eta-squared) via large ...
0
votes
0answers
19 views

Distribution of a subset and its parent

We have a worldwide distribution (A) of number of friends (e.g. ~Poisson or in my case an empirical one); we also have a subset which gives the distribution (B) of number of friends for country X. In ...
0
votes
1answer
30 views

Subjectivity of models and their performance tests. Sampling issue

I am trying to classify cases according to binary response variable. For this purpose I chose random forest and classification tree methods. So, as it is usually done, I randomly split the data into ...
0
votes
0answers
475 views

Data balancing as using ROSE(Random Over Sampling Examples) function

I want to balance the distribution of data using the ROSE function. The balance of the data was adjusted, but the value of the independent variable was changed ...
2
votes
0answers
56 views

Imputing missing counts in a time series (with variance?)

We're using a sonar to enumerate migrating fish. The date & time are recorded with each event that a fish passes the sonar station. We've noted two very strong time-varying components to the ...
7
votes
0answers
1k views

How to generate 2 correlated Beta random variables

I was wondering if it might be possible to generate 2 correlated $Beta$ random variables? In other words, I want to generate two Beta random variables which can be said to have come from two Beta ...
4
votes
0answers
183 views

Samples size with sample proportion close to 0 or 1

For a future monitoring program on small water bodies we want to calculate the sample size. The bodies of water are so small that their number easily exceeds 100.000 in the monitoring area and ...
4
votes
1answer
3k views

SMOTE sampling in caret package in R

when using caret packge in the trainControl you can use "smote" sampling. what is the default parameters the train in caret are using for smote?? parameters such as: perc.over = 300, k = 8, perc....
1
vote
2answers
950 views

Manually generate random sample in Gaussian mixture model

I want to generate (manually) a random sample in the Gaussian mixture model: $$f_{\theta}(x) = \sum_{k = 1}^{K}\pi_k f_{\mathcal N(\mu_k, \sigma^2_k)}(x)$$ Here is my work: ...
1
vote
1answer
3k views

NbClust with large data sets. Sampling?

I have a data set which is 3 columns and about 50,000 rows. I am trying to do some cluster analysis and would like to speed it up. Most examples online use data sets like ...
3
votes
1answer
72 views

Is there a more efficient method for creating a probability distribution?

I have written a short function in R to estimate the expected number of mutations that will be observed in a set of DNA sequences. The parameters are the mutation rate (x), the length of the DNA ...
1
vote
2answers
2k views

Random Forest - Unbalanced Dataset for Training & Test

I am doing a classification modelling(using R and random forests) for a website where only 2% of the visitors convert. Now given the behavior and attributes of a visitor I want to predict probability ...
2
votes
2answers
306 views

Subsample to follow a normal distribution

I have a problem, that sounds very simple in theory but I fail to implement a good solution. Let my data be a sample of a continuous variable that that follows a normal distribution (m1,v1), ...
1
vote
0answers
30 views

Age-correcting survey sub-samples

I have a survey of N = 338 responses. If I analize the Age column, I can see that it is more or less normally distributed (mean =...
4
votes
1answer
302 views

Sampling from under/over-dispersed count data in R

I am currently working a some datasets with count data in R, in which the response is the number of activities of a given type that were performed in one day by a population. For each type, I build ...
1
vote
0answers
341 views

How are the null distributions created for hypothesis testing?

I'm struggling to connect the idea of a sampling distribution of a test statistic to the null distribution. If I run many simple linear regressions on samples extracted from a population of data I ...
0
votes
1answer
503 views

Stratified sampling with replacement of participants in R

I am wondering whether in R, there is a possibility to draw a stratified sample with replacement of participants. Often, in surveys, we need each group to have certain size. Hence if one respondent in ...
3
votes
1answer
73 views

Repeated catch–mark–release (urn problem)

Imagine I have a small pond with some fish in it and I want to estimate its population size. I don't have a lot of resources at hand, in fact all I do have is a fishing pole, pen and paper, and a lot ...
1
vote
0answers
79 views

dependent t-test giving opposite result

I have a dataframe as below. ...
0
votes
1answer
39 views

How to do the right sampling for a classification model

I'm looking for the best sampling method for classification modeling (logistic regression OR Xgboost ) that will decrease the number of observations in the training data set while keeping the bias and ...
3
votes
3answers
267 views

How to sample from a distribution so that mean of samples equals expected value?

Given a random variable $X$, how do I have obtain $N$ random variates of $X$ so that the mean value of my samples equals the expected value of $X$? E.g. let $X$ have uniform distribution on the ...
0
votes
1answer
19 views

increase class proportion in a class proportion vector without changing the sum of the vector

Say I have a vector x which defines the number of samples in a given class: x <- c(250,362,10,246,30) In this case, there are five classes. The resulting ...
0
votes
0answers
61 views

How can I estimate the area by sample points and Traverses

I have a study region with an area of 1 that has several lakes, one with an island. The total lake area happens to be 0.27. In total 20 random points were placed into this study region. Furthermore, ...
1
vote
1answer
612 views

Sampling from copula given a particular value of marginal

I am working in the bivariate case with copulas as follows: I have two marginal Gamma distributions $f_1=Ga(\alpha_1, 1)$ and $f_2=Ga(\alpha_2, 1)$ that are bound by a copula Frank copula $C$, with ...
2
votes
0answers
210 views

want to calculate significance of pairwise sharing between lists. Standard hypergeometric test seems inappropriate

I want to calculate if the percentage of sharing of strings (genes in this case) between lists is significantly more than expected by chance for multiple pairwise comparisons between lists of strings. ...
2
votes
1answer
882 views

Statistical test for random sample of data

I'm trying to determine if some particular measurements - in this case taken from a subset of genes of interest (50 genes) - show a significant difference to the rest of the population (15000 genes), ...
0
votes
1answer
125 views

Any good working method for sampling from estimated distribution in R

Suppose I have a number of points $\{x_i\}_{i=1}^N$ and I want to estimate the smooth distribution and sample from it. I have tried at the moment a lot of things, and all of them show poor ...
1
vote
0answers
682 views

R: estimate joint density and sample from conditional densities

I'm trying to implement an algorithm that estimates time-homogeneous Markov chain with continuous state space. The probability of transition from state $i$ to state $i+1$ has continuous conditional ...
0
votes
1answer
1k views

How to exact prediction from over sampled data(Undoing oversampling)?

We are oversampling the data to use in logistic regression. Aim is to predict CTR(click probability) which is rare event scenario. I have predicted the probabilities of click but CTR results are ...