Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [computational-statistics]

Refers to the interface of statistics and computing; the use of algorithms and software for statistical purposes.

1
vote
0answers
11 views

create joint prob distribution or empirical relation for two variables

There are two variables, X1 and X2. The experimental study shows that they are highly correlated. Are there any reliable ways to create an empirical mapping(or equation) between X1 and X2. Assuming ...
1
vote
0answers
15 views

What does it mean when Risk function turns out to be a number?

I have a statistical decision making theory problem.I have to calculate the Risk Function for each of 4 decision rules.However,it turns out that the fourth Risk function is not a function of θ and it ...
2
votes
1answer
41 views

Why median is NA for some of the group outcomes in survival analysis?

I'm trying to do survival analysis using the Followup information, patient_vital_status and the expression of gene. I'm using like below: ...
0
votes
0answers
25 views

Use Shapley Values for explaining whole Data Frame instead of a Single prediction

I am working on a Machine Learning model. One of the requests is to explain the models 'decisions' to the business. Therefore I am using Shapley Values (Game Theory). I found an interesting example ...
6
votes
1answer
171 views

Does Approximate Bayesian Computation (ABC) follow the Likelihood Principle?

I know that ABC is commonly used when the likelihood is intractable, so likelihood principle is not an interest in that case. But, I am curious whether the ABC satisfies the likelihood principle when ...
1
vote
0answers
27 views

What is the process for detecting a 0.2 gallon/hour leak in an underground petrol tank with an accuracy of 95% or more?

Current EPA regulation stipulates that SIR (Statistical Inventory Reconciliation) vendors must be able to detect a leak of 0.2 gallons/hour with a probability of 0.95 or higher, and a false alarm rate ...
1
vote
0answers
24 views

Using R to maximize a two parameter Weibull model via multivariate extension of Newton-Raphson method

I am just getting back into using R for the first time in a while, and wrote some code to perform the aforementioned task in the title. I was wondering if anyone could take a look at it and see if ...
0
votes
0answers
42 views

Confidence Region of a multivariate KDE in Python?

I have an estimated bivariate kernel density based on a set of observations (𝑥11,𝑥12,...,𝑥1𝑛) and (𝑥21,𝑥22,...,𝑥2𝑛) and would like to draw confidence regions in the (𝑥1,𝑥2) space. This is in ...
3
votes
1answer
57 views

Which is more numerically stable for OLS: pinv vs QR

If I am doing standard OLS and want to calculate beta values (OLS estimators), which of the following is the more numerically stable method? And why? Assuming that the columns of $X$ are already mean-...
0
votes
1answer
30 views

A simple concept question regarding Bootstrapping

I am having a difficulty understanding whether I can use the bootstrapping method for prediction. First off, my data is as follows. where personal Income is the dependent variable (y), and GPA, Age, ...
1
vote
0answers
10 views

Downsample a stochastic process without losing correlation statistics?

I have a stochastic variable $X(t)$ which changes at a discrete set of random times $t_1, t_2, \dots$. I can simulate this stochastic process to obtain a series $X(t_1), X(t_2),\dots$ However, the ...
1
vote
1answer
24 views

QR decomposition computational efficiency

I am struggling to find a reference for this: In terms of big Oh notation does anyone know of any expressions for the computational time taken by commonly used algorithms for QR decompositions?
-1
votes
2answers
50 views

K-Means clustering and correlation

I ran K-Means on my dataset, it's a small dataset of 200 countries x 6 export sectors. My results formed three clusters. Now I want to check whether these three clusters are correlated with another ...
3
votes
1answer
60 views

What's a mean field variational family?

I'm working through variational Bayesian methods at the moment, and I think I have a grasp of the bigger picture. Where I sometimes have trouble is with the exact details of how it can be implemented. ...
2
votes
0answers
21 views

Using Markov random field spatial weights to account for spatial autocorrelation

I am looking at the relationship between life expectancy and smoking rate within the London boroughs. I thus created a bayesx spatial regression model including a term which assigns spatial ...
0
votes
2answers
47 views

Proof for “The sum of the observed values $Y_i$ equals the sum of the estimated / fitted values $\hat Y_i$”

I needed some help trying to understand why the sum of the observed values $Y_i$ equals the sum of the estimated values $\hat{Y}_i$.
2
votes
1answer
331 views

How does scipy calculate the binomial CDF so fast?

I ran scipy.stats.binom.cdf(500006, 1000000, 0.5) and it took less than a milisecond. This is crazy as binomial CDF involves summing up a bunch of binomial coefficients. What approximation algorithm ...
0
votes
1answer
25 views

Which analysis method question

I am currently conducting a short research paper for my university course and am struggling to determine which statistical analysis would be appropriate for the data I have collected. I surveyed two ...
2
votes
1answer
43 views

Covariance of Constrained Maximum Likelihood Estimators

I plan to numerically estimate the parameters of a GLM but with constraints imposed on some of the parameters. In this case, does the general approach of estimating the covariance matrix of my MLE ...
3
votes
1answer
39 views

Obtain global linear regression estimate from subsamples

I want to estimate $\widehat\beta$ in a simple linear regression with scikit. $$y = X \beta + \varepsilon$$ The problem is that the dimension of the complete $X$ is too large to fit into memory. Is ...
2
votes
0answers
29 views

Simulation - problem of maximization inside a circle

I am doing some projects related to statistics simulation using R based on "Introduction to Scientific Programming and Simulation Using R". In the Students projects session (chapter 24), I am doing ...
5
votes
2answers
715 views

Subtracting very small probabilities - How to compute? [duplicate]

This question is an extension of a related question about adding small probabilities. Suppose you have log-probabilities $\ell_1 \geqslant \ell_2$, where the corresponding probabilities $\exp(\ell_1)$...
0
votes
0answers
56 views

Must MSA be re-performed after annual instrument calibration?

I'm working in an automotive company that must follow IATF (International Automotive Task Force) requirements. I administrate a test-bench that performs radio frequency tests using an appropriate ...
2
votes
1answer
44 views

Vectorised computation of logsumexp

In this related post there is an explanation of how you can add together two very small probabilities using the logsumexp function, and how this can be programmed into base ...
1
vote
0answers
40 views

Data perturbation with normal variables

I am doing some projects related to statistics simulation using R based on "Introduction to Scientific Programming and Simulation Using R". In the Students projects session (chapter 24), I am doing ...
0
votes
1answer
65 views

Correcting data using poisson-regression

I'm new to stats and I was wondering if anyone had any good resources that could explain to me: How one can correct their data (false-positives) using Poisson-regression. I've been looking for some ...
0
votes
0answers
13 views

Effect of dimensionality for time taken to cluster data with k-means

In a dataset if I have $N$ features and for k-means clustering it might take $T$ seconds. If the dimensionality increased to $2N$, how would the time taken to run k-means clustering increase?
2
votes
0answers
30 views

Computing Wassertein Distance

For two probability measures $\mu$ and $\nu$, the Wassertein Distance is defined as $$W_p (\mu , \nu) = \left[ \inf\limits_{\gamma \in \Gamma} |x-y|^p \, d\gamma (x,y) \right] ^{\frac{1}{p}} \, , $$ ...
0
votes
1answer
61 views

Propensity to pay modelling

I am trying to build a propensity to pay model given an intervention to a customer. Context: The population I am dealing with are customers who were supposed to pay some amount on a certain date ...
0
votes
1answer
52 views

Tracking Moving Objects with Kalman Filters— Over-fitting over time?

I've been learning about Kalman Filters, and the classic example given is tracking an object via radar/gps. My issue here is that each time you get a new data point, you update the error in the ...
3
votes
1answer
95 views

Adding very small probabilities - How to compute?

In some problems, probabilities are so small that they are best represented in computational facilities as log-probabilities. Computational problems can arise when you try to add these small ...
1
vote
1answer
37 views

Pearson correlation in data set with “jump”

I'm trying to calculate the correlation between the condition number of a finite elements matrix and the coarseness of the mesh that it represents. However, when trying to calculate the Pearson ...
1
vote
0answers
39 views

How do I measure information loss when converting categorical data to numerical?

Assume that a dataset has a mix of categorical and numerical attributes. The dataset has to undergo numeric processing which necessitates the conversion of the categorical attributes to numeric/...
1
vote
0answers
18 views

Suitable multiple testing procedure for three very correlated phenotypes

I have ran an analysis using polygenic risk scores including genetic variants at different p-value thresholds and have the following outcomes: IQ ages 8 and 9, strengths and difficulties questionnaire ...
0
votes
1answer
17 views

SPSS Statistics

The variance measures how far each number in the set is from the mean I have 151190 records of a table. and i have to analyze the data on SPSS. after the analysis i have some results. Mean=74617.92 ...
1
vote
1answer
55 views

Rules based Model (Function) - Derive Probability & Ensembling

Basically, let's assume I have a simple rules-based function/model (if weight >= 150) -> return true. Simple binary answer (true or false) from a single feature input. If I have a range of samples/...
0
votes
0answers
14 views

how to simulate with randdirichlet in SAS

My data has 100 obs, in 4 categories, each category has 22 obs, 25 obs, 35 obs and 18 obs . I want to create a random sample for these 4 proportions with randdirichlet in sas. Is the following the ...
4
votes
1answer
66 views

Techniques for improving mixing when sampling from a multidimensional posterior

I'm currently trying to use the Metropolis-Hastings algorithm to sample from a posterior distribution of the form $$p(\theta | y ) \propto \prod_{ij} \phi (\theta_{ij}) \times \prod_{i=1}^n \pi_{y_i}...
0
votes
0answers
7 views

How to compare a test case with standards with multiple (100+) variables?

I have a question about how to statistically get an answer for a question my group has. We have a test dataset where we have 100 independent variables with values ranging from 0-1. We then have a ...
0
votes
0answers
11 views

Compute p-value for very high z scores [duplicate]

For a statistical test I need to compute the p-value given a z score. I am using the Python method: scipy.stats.norm.cdf the problem is that for z score > 8, ...
1
vote
2answers
58 views

When is bootstrapping helpful and used?

When is bootstrapping helpful and when should it be used? I have watched several videos so I understand what bootstrapping does (samples with replacement from a single sample many times and create a ...
0
votes
1answer
159 views

Perplexity of a Non-Statistical Language Model

I have a piece of software that, given a input phrase, returns an ordered list of the next most likely words (entire vocab is ordered 1 to n). This is essentially an Language Model with the exception ...
3
votes
1answer
795 views

When to switch off the continuity correction in chisq.test function?

From this Research paper Table1 Association of RAD51-AS1 expression with clinicopathological features of EOC patients I see that p-value is calculated based on Chi-...
1
vote
0answers
21 views

What role(s) does Statistical Computing / Computational Statistics play in Clinical Trials?

I am going to pursue a PhD in Statistics, and the two fields that I am really interested are Statistical Computing / Computational Statistics and Clinical Trials (mainly because I like doing ...
0
votes
0answers
265 views

Computing Steady Probabilities for Markov Process

I hope I am asking this question on the correct SE, if not, please redirect it. I am trying to compute steady state probabilities for a Markov process using python. Knowing the transition rate values ...
1
vote
0answers
16 views

What tool do you use to Analyze Qualtrics CSV data

I’ve conducted a survey and have the csv export file as the results. What tool do you recommend to analyze the relationships between different factors? I think I can manually do ETL and put the data ...
3
votes
0answers
67 views

Computing the CDF of the minimum of particular dependent random variables

For each $i=1,\dots,n$ let $Z_i\sim\text{Poisson}(\lambda_i)$, and suppose $\{Z_i\}$ are independent. Also for each $i=1,\dots,n$, let $\{Y_{ij}\}_{j\in\mathbb{N}}$ be an infinite sequence of iid ...
0
votes
1answer
94 views

Filtering out bursts with a consistent range from a time series

I have time series of bursts that look like this: … and zoomed in: Now, there are also spurious bursts (which I call noise) in the data, which look like this: … and zoomed in: As you can see, the ...
4
votes
0answers
33 views

Computational statistics review

I'm looking for a mathematically rigorous review of key topics in computational statistics, such as numerical integration, EM algorithms, MCMC, and sampling algorithms. Are there any good lecture ...
0
votes
0answers
18 views

Computational Optimization of Cross-Validation in Training Models

TL;DR: If I run a 5-fold cross-validation which assigns 1 fold per core when my CPU only has 4 cores, is the 5th as costly (with respect to computation time) as the first 4 folds? Context: I am ...