All Questions

Tagged with
Filter by
Sorted by
Tagged with
0
votes
1answer
25 views

How can I generate a completely balanced combination

Attaching a code that generates unique combinations of 8 letters but, it still lacks a condition wherein all letters should have equal counts per column. For my code, basically, letter a to h should ...
0
votes
0answers
35 views

Validation Set Accuracy Significantly Higher than Hold out Test Set

I'm building a binary classifier, where each record is a task, and the response variable is whether it was completed on time. I'm using random forest My data set spans from 2000-2015 My hold out ...
0
votes
0answers
29 views

Generating variables using R

Question: Assume 2 treatments, 1 and 0. Given 120 participants, generate underlying truth with Age as a discrete uniform distribution over integers between [20-60] Gender (G) as $I\{G = F\} \sim \...
0
votes
0answers
21 views

Random Samples from Spliced Distribution

I am studying Clauset, Shalizi, and Newman, Power Law Distributions in Empirical Data (preprint available here) in R. Packages used: ...
0
votes
2answers
255 views

Sampling highly imbalance multi-class response variable

I have a dataset (11000 x 117) with response variable having multiple classes. Here is a plot of class distribution: Some of the classes have only 1 sample in the entire dataset and some have 2, 3 ...
9
votes
3answers
6k views

How to re-sample an XTS time series in R?

I have an irregularly spaced XTS time series (with POSIXct values as index type). How can I build a new time series sampled at ...
4
votes
1answer
297 views

Sampling from under/over-dispersed count data in R

I am currently working a some datasets with count data in R, in which the response is the number of activities of a given type that were performed in one day by a population. For each type, I build ...
14
votes
1answer
7k views

How to draw random samples from a non-parametric estimated distribution?

I have a sample of 100 points which are continuous and one-dimensional. I estimated its non-parametric density using kernel methods. How can I draw random samples from this estimated distribution?
1
vote
0answers
43 views

How to randomly downsample from matrix [closed]

I'm trying to downsample from a matrix of numbers. Each number is the number of times we saw a particular event. By downsample I mean I want to pseudo-randomly select values from each row so they ...
1
vote
0answers
25 views

Does specifying normalizing constant significantly improves Hamiltonian Monte Carlo?

From my understanding the energy function needs only be specified such that it is proportional to the log density, and not specifying the normalizing constant should not greatly impact the sampling ...
1
vote
0answers
49 views

When to use ordinary, balanced, antithetic, or permutation resampling for bootstrap?

I am using boot and would appreciate any explanation as to when using each of these resampling methods would be recommended in practice. I have come across Do who says that: Simulation results ...
9
votes
2answers
1k views

Inverse CDF sampling for a mixed distribution

The out-of-context short version Let $y$ be a random variable with CDF $$ F(\cdot) \equiv \cases{\theta & y = 0 \\ \theta + (1-\theta) \times \text{CDF}_{\text{log-normal}}(\cdot; \mu, \sigma) &...
0
votes
0answers
26 views

Permutation of datasets to check accuracy of class labels

Currently I am using a proteomics dataset to compare 4 different disorders to healthy controls. Ultimately, I want to find unique and overlapping factors (that are deviant from healthy controls) in ...
1
vote
1answer
20 views

ML Sampling question - Can I increase sample size of current dataset with older copies that have changed over time [closed]

I am working on a multinomial machine learning algorithm that labels stocks with buy/sell signals. My code updates with the most recent quantitative data about the stocks daily, so obviously the data ...
0
votes
0answers
19 views

Best way to measure treatment effect across two treatments

I have two sets of patient-level data for two distinct treatments, unfortunately, the data is only over a 14 week period and I'm hoping to build a predictive model to estimate/simulate what the data ...
1
vote
1answer
35 views

I want to simulate a random sample of length n from DAG of correlated Bernoulli's

Suppose I have a DAG of 4 vertices. Each vertex consists of a Bernoulli of parameter $p$. It is the following: (Z) ---> (Y) (Z) ---> (W) (X) ---> (Y) ---> (W) I hope it is clear. Anyway, I ...
3
votes
1answer
1k views

Sampling from empirical distribution

I have a vector of y (min is > 0, max could be 1), for which, i have no idea what distribution is. But based on the data we have, vector y, we can get the empirical cumulative probability distribution,...
0
votes
1answer
232 views

Number of bootstrap replicates versus number of simulations

I have some confusion over bootstrap simulation in R. Here is the question: I am asked to use the following parameters to produce simulation: 500 bootstrap replicates 1000 simulations Sample size of ...
1
vote
0answers
203 views

Simulating Random Strings using Binomial/Multinomial Distributions

I am simulating some DNA sequences (containing characters A, C, G, and T) in R through specifying i) the number of sequences to generate (num.seqs) ii) the length of the generated sequences (length....
2
votes
0answers
23 views

Reliability engineering design values from tests AS/NZS1170

To get design values (the strength of something, for example) designing to Australian/ New Zealand loading code you do $n$ tests and estimate the coefficient of variation of your parent population, $...
2
votes
1answer
865 views

Statistical test for random sample of data

I'm trying to determine if some particular measurements - in this case taken from a subset of genes of interest (50 genes) - show a significant difference to the rest of the population (15000 genes), ...
0
votes
0answers
343 views

How can I create a multivariate random normal distribution from separate normal distributions? [duplicate]

In R I can sample from a multivariate normal distribution as follows: ...
4
votes
1answer
36 views

Permutation using probability versus direct frequencies in R?

I'm trying to understand if there is any difference to the following approaches in permutating values based on their frequencies. Here is an example of the raw data, which gets binned by ...
2
votes
1answer
60 views

Multistage Survey Sample for Sock Color

Here is the setup: We want to estimate the proportion of people in a country who wear black socks versus not-black socks. We have 100 cities (primary sampling units) in the country. In each city ...
5
votes
1answer
739 views

How to calculate the chance of getting completely unbalanced groups? (with R)

I want to simulate or calculate probabilities of combinations of group membership for different sample sizes (e.g., n= 3, 4, 5, 10, or 100) for two groups (of the same sample size). Each outcome could ...
4
votes
1answer
3k views

SMOTE sampling in caret package in R

when using caret packge in the trainControl you can use "smote" sampling. what is the default parameters the train in caret are using for smote?? parameters such as: perc.over = 300, k = 8, perc....
1
vote
1answer
92 views

Unexpected pattern in posterior predictive check with set.seed()

I have stumbled upon R behavior involving set.seed(), sample() and rbinom() which I don't ...
1
vote
1answer
105 views

Checking the legitimacy of a simple one-way Anova simulation

Background: My question is based on This Paper (see its Appendix for R code). The paper is trying to basically obtain the empirical distribution of a statistic called $\eta^2$ (eta-squared) via large ...
0
votes
0answers
19 views

Distribution of a subset and its parent

We have a worldwide distribution (A) of number of friends (e.g. ~Poisson or in my case an empirical one); we also have a subset which gives the distribution (B) of number of friends for country X. In ...
3
votes
2answers
2k views

What is behind JAGS (Just Another Gibbs Sampler)?

I have been using JAGS but I am not quite sure how it actually simulates it values. I need to know in a general sense what's going on in the background.
0
votes
1answer
30 views

Subjectivity of models and their performance tests. Sampling issue

I am trying to classify cases according to binary response variable. For this purpose I chose random forest and classification tree methods. So, as it is usually done, I randomly split the data into ...
1
vote
2answers
2k views

Random Forest - Unbalanced Dataset for Training & Test

I am doing a classification modelling(using R and random forests) for a website where only 2% of the visitors convert. Now given the behavior and attributes of a visitor I want to predict probability ...
0
votes
1answer
39 views

How to do the right sampling for a classification model

I'm looking for the best sampling method for classification modeling (logistic regression OR Xgboost ) that will decrease the number of observations in the training data set while keeping the bias and ...
0
votes
0answers
466 views

Data balancing as using ROSE(Random Over Sampling Examples) function

I want to balance the distribution of data using the ROSE function. The balance of the data was adjusted, but the value of the independent variable was changed ...
7
votes
2answers
2k views

How to calculate sample size for comparing the area under the curve of two models?

Because I would like to calculate the sample size for comparing the area under the curve (AUC) of 2 models (cross-sectional study, predictor = continuous variable). Can you point me which function in ...
2
votes
0answers
55 views

Imputing missing counts in a time series (with variance?)

We're using a sonar to enumerate migrating fish. The date & time are recorded with each event that a fish passes the sonar station. We've noted two very strong time-varying components to the ...
7
votes
0answers
1k views

How to generate 2 correlated Beta random variables

I was wondering if it might be possible to generate 2 correlated $Beta$ random variables? In other words, I want to generate two Beta random variables which can be said to have come from two Beta ...
4
votes
1answer
686 views

Absent categorical data levels in Bootstrap samples

I have a huge dataset ($n$ around five million, $p$ around three thousand) for a classification problem, where my interest is predictive class probabilities for test data, not the target. I shall be ...
2
votes
4answers
3k views

smart sampling techniques in r

I have a large data set of about 1.8 million rows with 80 variables. I would like to find a good technique (code or package) in R that can reduce the amount of training data without damaging the ...
4
votes
0answers
172 views

Samples size with sample proportion close to 0 or 1

For a future monitoring program on small water bodies we want to calculate the sample size. The bodies of water are so small that their number easily exceeds 100.000 in the monitoring area and ...
4
votes
2answers
446 views

How to reconstruct database from cross-tabulated data?

I am given a set of almost 100 crosstabulations (most 2D, but some three dimensional) based on a single, yet unavailable dataset. My task is to perform basic statistical tests ($\chi^2$, Mann-Whitney, ...
1
vote
2answers
898 views

Manually generate random sample in Gaussian mixture model

I want to generate (manually) a random sample in the Gaussian mixture model: $$f_{\theta}(x) = \sum_{k = 1}^{K}\pi_k f_{\mathcal N(\mu_k, \sigma^2_k)}(x)$$ Here is my work: ...
1
vote
1answer
3k views

NbClust with large data sets. Sampling?

I have a data set which is 3 columns and about 50,000 rows. I am trying to do some cluster analysis and would like to speed it up. Most examples online use data sets like ...
3
votes
1answer
72 views

Is there a more efficient method for creating a probability distribution?

I have written a short function in R to estimate the expected number of mutations that will be observed in a set of DNA sequences. The parameters are the mutation rate (x), the length of the DNA ...
2
votes
1answer
506 views

Tests for quality of permutation (Y-randomization)

Can you please suggest the metrics/tests for accessing the quality of Y-randomization (permutation)? I have some doubts in quality of Y-randomization that I am using with sample.int from R. For 100 Y-...
2
votes
2answers
295 views

Subsample to follow a normal distribution

I have a problem, that sounds very simple in theory but I fail to implement a good solution. Let my data be a sample of a continuous variable that that follows a normal distribution (m1,v1), ...
4
votes
1answer
4k views

Importance Sampling to evaluate integral in R

I have asked the question here also. However, there might be something wrong with my theoretical understanding hence I'm asking here as it is more relevant. Kindly do not diss without looking first. ...
1
vote
0answers
30 views

Age-correcting survey sub-samples

I have a survey of N = 338 responses. If I analize the Age column, I can see that it is more or less normally distributed (mean =...
-1
votes
1answer
671 views

How do I randomize a larger population, from an existing population in R?

I have a small 20k (176 variables) test dataset. I want to create a random bigger dataset for testing, using the rows (data) and columns (names) from the small dataset. Currently I have: ...
0
votes
1answer
1k views

How to exact prediction from over sampled data(Undoing oversampling)?

We are oversampling the data to use in logistic regression. Aim is to predict CTR(click probability) which is rare event scenario. I have predicted the probabilities of click but CTR results are ...