Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [computational-statistics]

Refers to the interface of statistics and computing; the use of algorithms and software for statistical purposes.

-1
votes
2answers
25 views

K-Means clustering and correlation

I ran K-Means on my dataset, it's a small dataset of 200 countries x 6 export sectors. My results formed three clusters. Now I want to check whether these three clusters are correlated with another ...
3
votes
1answer
55 views

What's a mean field variational family?

I'm working through variational Bayesian methods at the moment, and I think I have a grasp of the bigger picture. Where I sometimes have trouble is with the exact details of how it can be implemented. ...
2
votes
0answers
14 views

Using Markov random field spatial weights to account for spatial autocorrelation

I am looking at the relationship between life expectancy and smoking rate within the London boroughs. I thus created a bayesx spatial regression model including a term which assigns spatial ...
0
votes
1answer
36 views

Proof for “The sum of the observed values $Y_i$ equals the sum of the estimated / fitted values $\hat Y_i$”

I needed some help trying to understand why the sum of the observed values $Y_i$ equals the sum of the estimated values $\hat{Y}_i$.
2
votes
1answer
318 views

How does scipy calculate the binomial CDF so fast?

I ran scipy.stats.binom.cdf(500006, 1000000, 0.5) and it took less than a milisecond. This is crazy as binomial CDF involves summing up a bunch of binomial coefficients. What approximation algorithm ...
0
votes
1answer
24 views

Which analysis method question

I am currently conducting a short research paper for my university course and am struggling to determine which statistical analysis would be appropriate for the data I have collected. I surveyed two ...
2
votes
1answer
40 views

Covariance of Constrained Maximum Likelihood Estimators

I plan to numerically estimate the parameters of a GLM but with constraints imposed on some of the parameters. In this case, does the general approach of estimating the covariance matrix of my MLE ...
3
votes
1answer
34 views

Obtain global linear regression estimate from subsamples

I want to estimate $\widehat\beta$ in a simple linear regression with scikit. $$y = X \beta + \varepsilon$$ The problem is that the dimension of the complete $X$ is too large to fit into memory. Is ...
2
votes
0answers
28 views

Simulation - problem of maximization inside a circle

I am doing some projects related to statistics simulation using R based on "Introduction to Scientific Programming and Simulation Using R". In the Students projects session (chapter 24), I am doing ...
4
votes
2answers
701 views

Subtracting very small probabilities - How to compute? [duplicate]

This question is an extension of a related question about adding small probabilities. Suppose you have log-probabilities $\ell_1 \geqslant \ell_2$, where the corresponding probabilities $\exp(\ell_1)$...
0
votes
0answers
47 views

Must MSA be re-performed after annual instrument calibration?

I'm working in an automotive company that must follow IATF (International Automotive Task Force) requirements. I administrate a test-bench that performs radio frequency tests using an appropriate ...
2
votes
1answer
28 views

Vectorised computation of logsumexp

In this related post there is an explanation of how you can add together two very small probabilities using the logsumexp function, and how this can be programmed into base ...
1
vote
0answers
34 views

Data perturbation with normal variables

I am doing some projects related to statistics simulation using R based on "Introduction to Scientific Programming and Simulation Using R". In the Students projects session (chapter 24), I am doing ...
0
votes
1answer
64 views

Correcting data using poisson-regression

I'm new to stats and I was wondering if anyone had any good resources that could explain to me: How one can correct their data (false-positives) using Poisson-regression. I've been looking for some ...
0
votes
0answers
12 views

Effect of dimensionality for time taken to cluster data with k-means

In a dataset if I have $N$ features and for k-means clustering it might take $T$ seconds. If the dimensionality increased to $2N$, how would the time taken to run k-means clustering increase?
2
votes
0answers
28 views

Computing Wassertein Distance

For two probability measures $\mu$ and $\nu$, the Wassertein Distance is defined as $$W_p (\mu , \nu) = \left[ \inf\limits_{\gamma \in \Gamma} |x-y|^p \, d\gamma (x,y) \right] ^{\frac{1}{p}} \, , $$ ...
0
votes
1answer
34 views

Propensity to pay modelling

I am trying to build a propensity to pay model given an intervention to a customer. Context: The population I am dealing with are customers who were supposed to pay some amount on a certain date ...
1
vote
0answers
20 views

Total variation for draws from probability densities

I was wondering if there is a metric that is similar to that of total variation, which gives a measure of how different two probability distributions, $f$ and $g$ are, but works for samples? So ...
0
votes
1answer
48 views

Tracking Moving Objects with Kalman Filters— Over-fitting over time?

I've been learning about Kalman Filters, and the classic example given is tracking an object via radar/gps. My issue here is that each time you get a new data point, you update the error in the ...
2
votes
1answer
73 views

Adding very small probabilities - How to compute?

In some problems, probabilities are so small that they are best represented in computational facilities as log-probabilities. Computational problems can arise when you try to add these small ...
1
vote
1answer
31 views

Pearson correlation in data set with “jump”

I'm trying to calculate the correlation between the condition number of a finite elements matrix and the coarseness of the mesh that it represents. However, when trying to calculate the Pearson ...
1
vote
0answers
26 views

How do I measure information loss when converting categorical data to numerical?

Assume that a dataset has a mix of categorical and numerical attributes. The dataset has to undergo numeric processing which necessitates the conversion of the categorical attributes to numeric/...
1
vote
0answers
17 views

Suitable multiple testing procedure for three very correlated phenotypes

I have ran an analysis using polygenic risk scores including genetic variants at different p-value thresholds and have the following outcomes: IQ ages 8 and 9, strengths and difficulties questionnaire ...
0
votes
1answer
13 views

SPSS Statistics

The variance measures how far each number in the set is from the mean I have 151190 records of a table. and i have to analyze the data on SPSS. after the analysis i have some results. Mean=74617.92 ...
1
vote
1answer
53 views

Rules based Model (Function) - Derive Probability & Ensembling

Basically, let's assume I have a simple rules-based function/model (if weight >= 150) -> return true. Simple binary answer (true or false) from a single feature input. If I have a range of samples/...
0
votes
0answers
13 views

how to simulate with randdirichlet in SAS

My data has 100 obs, in 4 categories, each category has 22 obs, 25 obs, 35 obs and 18 obs . I want to create a random sample for these 4 proportions with randdirichlet in sas. Is the following the ...
4
votes
1answer
50 views

Techniques for improving mixing when sampling from a multidimensional posterior

I'm currently trying to use the Metropolis-Hastings algorithm to sample from a posterior distribution of the form $$p(\theta | y ) \propto \prod_{ij} \phi (\theta_{ij}) \times \prod_{i=1}^n \pi_{y_i}...
0
votes
0answers
7 views

How to compare a test case with standards with multiple (100+) variables?

I have a question about how to statistically get an answer for a question my group has. We have a test dataset where we have 100 independent variables with values ranging from 0-1. We then have a ...
0
votes
0answers
11 views

Compute p-value for very high z scores [duplicate]

For a statistical test I need to compute the p-value given a z score. I am using the Python method: scipy.stats.norm.cdf the problem is that for z score > 8, ...
1
vote
2answers
50 views

When is bootstrapping helpful and used?

When is bootstrapping helpful and when should it be used? I have watched several videos so I understand what bootstrapping does (samples with replacement from a single sample many times and create a ...
0
votes
1answer
131 views

Perplexity of a Non-Statistical Language Model

I have a piece of software that, given a input phrase, returns an ordered list of the next most likely words (entire vocab is ordered 1 to n). This is essentially an Language Model with the exception ...
3
votes
1answer
515 views

When to switch off the continuity correction in chisq.test function?

From this Research paper Table1 Association of RAD51-AS1 expression with clinicopathological features of EOC patients I see that p-value is calculated based on Chi-...
1
vote
0answers
21 views

What role(s) does Statistical Computing / Computational Statistics play in Clinical Trials?

I am going to pursue a PhD in Statistics, and the two fields that I am really interested are Statistical Computing / Computational Statistics and Clinical Trials (mainly because I like doing ...
0
votes
0answers
203 views

Computing Steady Probabilities for Markov Process

I hope I am asking this question on the correct SE, if not, please redirect it. I am trying to compute steady state probabilities for a Markov process using python. Knowing the transition rate values ...
0
votes
0answers
14 views

Avoiding underflow errors when marginalizing over a nuisance parameter in Bayesian inference

I was reading this question about how to marginalize over nuisance parameters in Bayesian inference, and the concern I have is how to deal with underflow errors. If we are interested only in the ...
1
vote
0answers
15 views

What tool do you use to Analyze Qualtrics CSV data

I’ve conducted a survey and have the csv export file as the results. What tool do you recommend to analyze the relationships between different factors? I think I can manually do ETL and put the data ...
3
votes
0answers
65 views

Computing the CDF of the minimum of particular dependent random variables

For each $i=1,\dots,n$ let $Z_i\sim\text{Poisson}(\lambda_i)$, and suppose $\{Z_i\}$ are independent. Also for each $i=1,\dots,n$, let $\{Y_{ij}\}_{j\in\mathbb{N}}$ be an infinite sequence of iid ...
0
votes
1answer
94 views

Filtering out bursts with a consistent range from a time series

I have time series of bursts that look like this: … and zoomed in: Now, there are also spurious bursts (which I call noise) in the data, which look like this: … and zoomed in: As you can see, the ...
4
votes
0answers
32 views

Computational statistics review

I'm looking for a mathematically rigorous review of key topics in computational statistics, such as numerical integration, EM algorithms, MCMC, and sampling algorithms. Are there any good lecture ...
0
votes
0answers
17 views

Computational Optimization of Cross-Validation in Training Models

TL;DR: If I run a 5-fold cross-validation which assigns 1 fold per core when my CPU only has 4 cores, is the 5th as costly (with respect to computation time) as the first 4 folds? Context: I am ...
0
votes
0answers
23 views

variance of projected data onto a vector

I am reading this great article on PCA and there is something that I do not understand. In the paragraph Eigendecomposition of a covariance matrix, the author states that if we were to project our ...
1
vote
0answers
68 views

Traffic density estimation using deep learning

I am working on a project implementing deep learning and computer vision to estimate the traffic density of any random given road segment/roundabout or intersection. I am given a camera mounted on the ...
3
votes
0answers
17 views

Looking for a recursive formula for asymptotic variance of importance sampling estimator (self-normalized)

Looking for a recursive formula to approximate variance of importance sampling estimator $Var_q\big[\delta_{IS}\big]\approx\sum_{i=1}^n\tilde w(X_i)^2\big[h(X_i)-\delta_{IS}\big]^2$. This is an ...
1
vote
1answer
36 views

Summarizing a multi-dimensional distribution

I have an empirical distribution of observations along a number of dimensions. The dimensions are either categorical or ordinal (low frequency, equally spaced time steps from 0 to cut-off). At the ...
2
votes
1answer
56 views

When we state hypothesis does it mean null hypothesis or alternative hypothesis

In one of assignments in a study programme I got this statement: Hypothesis: University towns have their mean housing prices less effected by recessions. Run a t-test to compare the ratio of the ...
0
votes
0answers
72 views

Create an A/B Sample Size Calculator using Evan Miller's Post

To learn more about A/B Testing sample sizes selection I am attempting to use Evan Miller's popular blog-post to recreate a sample size calculator (https://www.evanmiller.org/sequential-ab-testing....
0
votes
0answers
22 views

Is this model really saturated? Is there a good alternative to an ANOVA for saturated models?

I have chemical data from groundwater wells that were exposed to two separate amendments of vegetable oil and monitored over several time points. I am trying to set up a 2-way ANOVA to test if the ...
1
vote
1answer
14 views

Training a classifier on 2 catalogs combined

I'm currently trying to train a machine to identify flashes of light in the sky as supernova (SN) or not. To do this, I'm combining 2 different catalogs: A big catalog (99% SN) with brighter flashes ...
1
vote
0answers
41 views

Machine Learning - Predicting Multiple Response Variables (Clothing Sizes)

For context, when a retailer buys in a clothing item they have to stock it in various sizes. The ratio of stock for each size is referred to as the "size ratio" of the product. E.g. S=0.2, M=0.5, L=...
1
vote
1answer
58 views

Compute $P(X > Y)$ for two random variables with unknown distributions from Markov chains

I would like to compute the probability $P(X > Y)$ with R. I used JAGS to sample from the posterior distribution of each variable, so I have a Markov chain for each variable (of length $3\times 10^{...