Questions tagged [computational-statistics]
Refers to the interface of statistics and computing; the use of algorithms and software for statistical purposes.
702
questions
1
vote
0
answers
33
views
mixed effect model question
Hi i have a certain task i want to solve:
For two months, participants played an app, in which they played 5 different therapeutic games (TGs). At the beginning of each session, they also completed a ...
- 11
0
votes
0
answers
14
views
UBCF rating predict for binary data in R package recommenderlab
In the R package recommenderlab's predict function for binary data (specifically unary for market basket analysis), why does the code do the following for the weighted UBCF model?
...
0
votes
0
answers
38
views
Resampling only $N$ particles out of $N(T+1)$ weighted particles
I have a bunch of weighted particles $(Z^{(i, k)}, W^{(i, k)})$ from a distribution $\mu(dz)$ where $i=1, \ldots, N$ and $k=0,\ldots, T$. These defines the following empirical approximation
$$
\hat{\...
- 657
0
votes
0
answers
20
views
SMC sampler weights when $K_n$ leaves $\pi_{n-1}$ invariant
I cannot seem to find this proof anywhere. Suppose I choose $K_n$ to leave $\pi_{n-1}$ invariant, and $L_{n-1}$ to be the reversal kernel. I want to show that the incremental weights
$$
w_n(x_{n-1}, ...
- 657
0
votes
1
answer
90
views
Guidance for statistical analysis on academic collaborations [closed]
I am currently engaged in a research project involving data analysis in the field of academic publications and author collaborations.
The dataset I'm working with includes information such as ...
2
votes
1
answer
239
views
Confusion on Chi-Squared test results
Basically, I have the data where I am trying to assess whether there is any correlation between the gender of the manager and the gender of people within their team. I decided to do the chi-squared ...
- 21
2
votes
0
answers
42
views
Bias and Variance of a Honest Random Forest
I am trying to read the paper Estimation and Inference of Heterogeneous Treatment
Effects using Random Forests. In the section 3.1(Theoretical Background), page 13 paragraph 2, The authors have ...
- 95
1
vote
0
answers
38
views
Most efficient way of converting a Quarto document to a presentation [closed]
I currently have a large body of statistics lecture notes that I wrote in Rmarkdown/Quarto document format, and I am looking to convert these notes to Quarto presentations in the simplest way possible....
0
votes
0
answers
25
views
Inference of Beta-Bernoulli Distribution
Assume $x_1, x_2, \cdots, x_n$ follows a $Bern(\pi_0)$, Let $y_{ik}$ follows $Beta(\alpha,\beta)$, $i\in \{1,\cdots, n\}$, and $k\in \{1,\cdots, K\}$. Let $z_k$ follows a Bernoulli Distribution with a ...
- 1
0
votes
0
answers
10
views
Logic behind the direction of coefficient in generalized estimating equation
I am not familiar with statistics and thus nor am I familiar with the methodological reasoning behind it so can someone please explain the reasoning behind the following results:
I have an outcome (...
- 1
0
votes
0
answers
81
views
Why do I get NaN p values in some variables when using mgcv to fit generalized additive mixed models?
I am currently trying to fit milk production data collected for three years by using generalized additive mixed model through mgcv package. The problem is, I am getting NaN p values in some variables. ...
0
votes
0
answers
10
views
Statistical models or problems equipped with methods that guarantee optimal or near solutions in finite time
What are some statistical models or problems that have computationally tractable solutions and are guaranteed to be optimal or near optimal? By computationally tractable, I mean finite time and ...
- 2,416
0
votes
0
answers
38
views
Is this algorithm for robust estimation of the covariance matrix sensible?
I have a high dimensional dataset $\bf{X} \subset \mathbb{R}^d$, which is multimodal and has outliers. I want to estimate a robust measure of association, something like the correlation between two ...
- 2,876
0
votes
0
answers
35
views
Is the Wikipedia Sample Entropy code wrong?
I'm following a tutorial paper on sample entropy, and I implemented the algorithm myself in Rust.
To check my work, I used Wikipedia's python code in this article. I have computed an example by hand ...
- 139
1
vote
0
answers
19
views
How do I numerically compute $I(X;CX+Y)$?
Given that $X\sim\text{Bernoulli}(\nu)$ for some $\nu\in(0,1)$, and $Y\sim N(0,1)$ are independent random variables. I want to compute the mutual information $I(X;CX+Y)$, where $C$ is some known non-...
- 197
2
votes
1
answer
114
views
Generating MLE in python - Problem witth the function [closed]
After my previous question (here) I tried to improve my work with this distribution. I'm using the parametrization $$f_X(x) = \frac{\theta^2 x^{\theta-1}(\gamma-\log(x))}{1+\theta\gamma} \mathbb{I}(0&...
- 177
4
votes
2
answers
139
views
How to iteratively calculate weighted standard error to report alongside a weighted mean
I have a group of individuals for which I would like to report a mean and a weighted error. The data that I observe on a daily basis are two independent $iid$ random variables with unknown ...
- 183
0
votes
0
answers
62
views
What is a good way to automate distribution fitting in python?
I have to do some distribution fitting of 120 data subsets.
They take the form of financial transactions amounts and timestamps.
Timestamp, BTC, EUR, USD
I know from some other analysis that each ...
- 1
4
votes
1
answer
140
views
bootstrap confidence interval and p-value calculations for finite population sizes
I am comparing the difference of medians between two groups of sample sizes $n1$ and $n2$. I would like to confirm that my boostrap approach for finite population size without pooling sample data ...
- 103
12
votes
6
answers
2k
views
How to generate from this distribution without inverse in R/Python?
I am working with a distribution with the following density: $$f(x) = - \frac{(\alpha+1)^2 x^\alpha \log(\beta x)}{1-(\alpha + 1)\log(\beta)}$$ and CDF $$\mathbb{P} (X \leq x) = \int_0^x - \frac{(\...
- 177
0
votes
0
answers
56
views
Exact computation of Bayes factor for multivariate normal
Question: Is there a known, exact expression for the Bayes factor between two multivariate normal hypotheses?
Let $H_1$ and $H_2$ be two subsets of $R^d$ with normal priors $\pi(\mu|H_j)$. The sets $...
- 1
0
votes
1
answer
61
views
XGBoost: Why is the "approximate algorithm" faster?
I am reading T. Chen, C. Guestrin, "XGBoost: A Scalable Tree Boosting System", 2016 (arXiv), which is seemingly full of typos. They propose the so-called "approximate algorithm" (...
- 325
0
votes
0
answers
37
views
Can I use Kendall's correlation to determine the correlation between continuous and binary variables?
I have a dataset where the first 6 columns correspond to binary entries referring to sick/not sick and 2 additional columns with age and a specific score (dim of the dataset 60x8). I need to generate ...
- 1
0
votes
0
answers
179
views
Testing whether a set of points on the unit sphere is uniformly distributed
The canonical way to do the test is to perform the spherical harmonic transform of the empirical distribution and then check that the power spectrum decays, but this is presumably fairly expensive. Is ...
- 317
1
vote
0
answers
201
views
Algorithm for Irwin Hall Distribution [closed]
I've been trying to create a function for the Irwin Hall distribution that doesn't face the same issue as the unifed package implementation. Because the function suffers from numerical issues, I ...
- 113
0
votes
0
answers
77
views
Overflow when computing binomial distribution for large n [duplicate]
How do you compute a binomial probability distribution for large $n$? If I try the following, I get an integer overflow in any programming language:
...
- 111
0
votes
0
answers
107
views
How does one create comparable metrics when the original metrics are not comparable?
The problem I have is that I have several groups (say 3 to make discussion concrete) with observations from a true but known distribution $p^*_1, p^*_2, p^*_3$ (or 3 populations). I can compute some ...
- 6,606
2
votes
0
answers
163
views
Rust or C++ for computational statistics? [closed]
I'll work on developing computer-intensive Bayesian sampling algorithms for spatio-temporal applications (e.g. MCMC, KF). So far, I'm thinking of coding my algorithms in C++. However, I've heard that ...
- 75
0
votes
0
answers
60
views
Sampling From Four-Parameter Beta Distribution
Most statistical computing packages have functions to sample out of a two-parameter Beta distribution, but not many offer functions for sampling out of a four-parameter Beta distribution. If I want to ...
- 113
0
votes
0
answers
19
views
Mean of the difference of two independent CMP distributions
For self studies I am looking for the mean of of the difference of two independent Conway-Maxwell-Poisson (CMP) distributions.
The complete PMF of the difference is not needed, only the mean matters.
...
1
vote
0
answers
20
views
How to filter out outliers from dataset? [duplicate]
I am trying to filter out outliers given a data set (maximum of 50 samples). For example:
dataSet = (10.0, 10.0, 10.0, 10.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, ...
- 11
0
votes
0
answers
16
views
Computational modelling without complex datasets
I'm currently learning about some higher-level machine learning and computational modelling techniques like decoding and representational similarity analysis, and I have a theoretical question that I'...
- 13
0
votes
0
answers
30
views
Could we code a program to compute VC-dimension of any given hypothesis class?
I've been studying machine learning theory and the fundamental theorem of the statistical learning for a while, but I still didn't found a general algorithm that could compute the VC dimension of any ...
- 1
1
vote
0
answers
88
views
Introducing the third spatial dimension (Z-coordinate) in the generalized dissimilarity model in R
Generalized dissimilarity modeling (R package gdm) is a tool to study the relative effects of the user-defined environmental gradients and the spatial distance decay on the pair-wise dissimilarity ...
- 133
0
votes
0
answers
25
views
How to simulate multivariate posterior distribution with a flat prior in general?
If I know that the posterior $p(\theta_1,\dots,\theta_m|y)$ can be written $p(\theta_1|\theta_2,\dots,\theta_m,y)p(\theta_2|\theta_3,\dots,\theta_m|y)\dots p(\theta_m|y)$ where $p(\cdot|y)$ in each ...
- 1,365
0
votes
0
answers
80
views
Numerical Stability when Inverse CDF Sampling from Truncated Density
Let $f(x)$ be the pdf of a random variable that we want to truncate to the interval $[a,b]$ and then sample from it. Let $F(x)$ denote the corresponding cdf. We can use inverse cdf sampling and ...
- 690
0
votes
0
answers
31
views
How Many Interaction Terms Am I Supposed To Include?
For my final school project, my group and I are doing multiple regression models using the Fragile Families dataset. I've determined that there are multiple interaction terms between covariates, but ...
0
votes
0
answers
66
views
Conditional expectation of Uniform given sum of Bernoulli trials
Given:
[]
Find the conditional probability distribution of theta given Sn and compute the conditional expectation.
I believe the distribution of Sn will be a binomial with mean ntheta and variance (...
- 1
1
vote
0
answers
39
views
Compute joint probability of two variables without using copulas
I have two timeseries one for a three-day accumulated rainfall and another one for a daily storm surge height. I would like to calculate the probability of a certain rainfall depth with a certain ...
1
vote
0
answers
31
views
IS PCA redundant when the first PC equals the mean of the data? [closed]
I have a situation where the principle component of my data is almost equal to the mean of the data. Does this make PCA redundant? Does PCA not work in this setting?
- 11
0
votes
0
answers
492
views
Shouldn't I do standardization when data is not normally distributed?
I am trying to scale the data prior to clustering analysis, and got a question.
The goal of scaling at this point is to unify(eliminate) the unit of all input variables so that make influence of each ...
- 23
0
votes
0
answers
17
views
Non-parametric test for separating samples from two populations
I have samples $\mathcal{Z} = \{Z_1,\ldots,Z_n\}$. These samples either come from the unknown distribution $\mathcal{D}_1$ or $\mathcal{D}_2$. (Note: I can't make any assumptions about these ...
0
votes
0
answers
45
views
Calculating the computer memory required to estimate a fixed effects logit model
What is the calculation to determine the computer memory necessary to estimate a fixed effects logit model? Will this calculation vary across statistical software? For example, let's say I want to ...
- 103
3
votes
1
answer
380
views
What are some good books for Computational Bayesian Statistics?
There is already a community wiki on introductory textbooks for Bayesian statistics.
However, I am looking for a book(s) on Computational Bayesian Statistics for intermediate learner who already has a ...
0
votes
0
answers
32
views
Calculating μ(A\ B) in probability space
I am new in learning sigma-algebra. My question is related to computing μ(A\ B). I just need the hints and ideas on how to do it, I will do it.
Let (X,Ao,μ) be a probability space. It is given that A, ...
- 1
1
vote
1
answer
157
views
Solving estimating equation using R
I am working with different types of data and comparing a variety of estimating equation approaches which share a multi-dimensional parameter $\beta$. Given a set of data $\boldsymbol{X}$, is there an ...
- 36
2
votes
1
answer
422
views
Coding the likelihood function for logistic regression
I would appreciate help in understanding if I made a correct interpretation and coding of the likelihood function for logistic regression.
Background: For a task I am going to write a function in <...
- 23
0
votes
0
answers
36
views
How to statistically determine if two files have identical content?
I'm writing a duplicate file finder application, and I do not want to have to read every byte of every file to determine which files are duplicates.
Hard drive storage is segmented into clusters of X ...
- 101
9
votes
1
answer
231
views
Why does somebody argue that the number of bootstrap replications should not be a multiple of 10?
At a recent conference somebody claimed that the size of the bootstrap replications should always be 999 rather than 1000.
Which argument supports this claim?
- 329
3
votes
1
answer
56
views
Statistics for large k-mers with low counts
Suppose I have a big genome and a small one (ex. mitochondrion) And I would like to know which words or k-mers of length k (ex. "AGCCGTA" - k = 7) are under/over-represented in both genomes, ...
