Questions tagged [computational-statistics]

Refers to the interface of statistics and computing; the use of algorithms and software for statistical purposes.

Filter by
Sorted by
Tagged with
0
votes
1answer
16 views

Changepoint detection and confidence score

I am familiar with Bayes Theorem and hypothesis testing, but not much above that. However, I have a problem that I cannot seem to formulate in frequentist terms - to my knowledge. From what I know ...
0
votes
0answers
11 views

Is that possible to merge both a rate base and logical base recommendation system?

I am not sure if there exists an answer in previous questions, but I couldn't find. Consider I show pictures to the people and they rate that picture 1-5 if they want to make a detailed rating or if ...
2
votes
1answer
67 views

Detecting extrema under uncertainty?

An old version of this question was poorly articulated. Here is another go: I have fifty objects. With a different, independent, unbiased scale for each object, I measure their weights 100 times each ...
1
vote
0answers
29 views

Correlation between two 3d arrays [closed]

I know that a is related to b is related to c. I have data for two years: In 2018, d=10. In 2019, d=0. I would like to know the correlation between a, b and c, for both d=0 and d=10 in order to ...
2
votes
1answer
57 views

LDA/NMF Topic Modeling vs Topic Modeling using “skip gram” approach

I am having a little friendly debate with my coworker on how to properly/optimally do topic modeling. I am just using the regular traditional nmf/lda approach and he decided to do it using "skip grams"...
0
votes
0answers
18 views

Ordinal multinomial logistic regression on one-hot encoded data

I have a task I am unable to tackle by principle. I'm working on survey data for one of our clients such that my design matrix is made of one-hot vectors with 15 features (originally 3 variables with ...
0
votes
0answers
2 views

Fair comparisons when there are varying number of respondents

I'm working on an evolutionary algorithm and trying to compare the fitness in individuals who meet the minimum requirements for feasibility. As such I get a varying amount of respondents each ...
0
votes
0answers
23 views

Random-effects-meta-analysis-simulation: zero-estimates for tau^2

I am working on simulating a random-effects-model for comparison of the DerSimonian-Laird-method vs. Hartung-Knapp-Sidik-Jonkman-method in R. To do so, I chose different combinations of mu (true ...
5
votes
1answer
37 views

Stable and efficient computation of binomial expectations

Suppose we want to compute the expected value of some function $f(X)$ where $X \sim \text{Bin}(n,\theta)$. Taking $\mathbf{f} = (f_0,...,f_n)$ to be the function values over all possible outcomes of ...
0
votes
0answers
18 views

Bayesian Estimation of a Mean and Standard Deviation (2D)

(Originally Posted at: https://stackoverflow.com/questions/56399700/bayesian-estimation-of-a-mean-and-standard-deviation-2d) I'm currently following Think Bayes, an introductory text to Bayesian ...
0
votes
1answer
17 views

Binary classification with binary predictors

I have a dataset with a binary response variable and 45 binary predictors. Which possible classification algorithms can I use? And how can i perform feature selection when most methods like best ...
1
vote
0answers
21 views

fitting high dim copula to residuals of a garch model very slow in R [closed]

I'm looking for some help on understanding on the fitting procedure of a normal (or any other for that matter) copula in R. My main goal is to either improve computational speed, or revise my strategy....
0
votes
1answer
81 views

Statistics: what kind of test to use to test this hypothesis?

I am unsure what kind of a hypothesis test to use in my case: So we are filling 500 ml bottles with water; It's a known fact that before a max of 3% of all the bottles could be under-filled; Now I ...
0
votes
1answer
53 views

Computationally verifying the equivalence of ridge regression estimates and Bayesian regression estimates

I'm trying to show that the numerical estimates of ridge regression's parameter estimates are the same as the MAP parameter estimates of a Bayesian regression model with normal prior distributions. So ...
0
votes
0answers
22 views

What is the best model to forecast ACT scores using practice test scores and past student data?

I understand that I may not be asking this question correctly, and would appreciate any feedback possible in order to help set me on the path to figuring this out... I work at a high school where ...
0
votes
0answers
38 views

Best statistical method should I use to calculate p value?

I'm comparing two different populations with unequal variances and non normal distributions using python. For sample #1 I'm drawing a random sample of n=30 from a population of 200. For sample #2 I'...
0
votes
0answers
61 views

What's the difference between loadings from partial least squares (PLS) regression and beta coefficients from multiple linear regression?

I have a set of independent variables (X1, X2, ..., X10) and I have run a PLS to find a combination of the X1, X2, ..., X10 that best predicts an outcome Y (a single-variable outcome). As a result, I ...
1
vote
0answers
19 views

Error bars of Monte Carlo expectation with correlated samples

I will try to phrase the question in a general way, then give my specific case as an example. Suppose I want to evaluate $Q = \mathbb E \left[ f\left(X, Y \right) \right]$ where $X$ and $Y$ are ...
0
votes
1answer
28 views

How to test the difference of two linear regression slopes with 2 independent and one dependent variable

I am trying to determine if the CO2 emissions growth rate of developing countries is higher than the growth rate of developed countries. So essentially I need to compare two linear regression slopes ...
0
votes
0answers
16 views

Pointwise Mutual Information using spacy or just detailed explaination

So, I have been trying to play around with NLP recently and decided to work on a project involving Emotional Analysis. I have been following this particular research, http://www.cse.yorku.ca/~aan/...
1
vote
0answers
12 views

create joint prob distribution or empirical relation for two variables

There are two variables, X1 and X2. The experimental study shows that they are highly correlated. Are there any reliable ways to create an empirical mapping(or equation) between X1 and X2. Assuming ...
1
vote
0answers
19 views

What does it mean when Risk function turns out to be a number?

I have a statistical decision making theory problem.I have to calculate the Risk Function for each of 4 decision rules.However,it turns out that the fourth Risk function is not a function of θ and it ...
2
votes
1answer
233 views

Why median is NA for some of the group outcomes in survival analysis?

I'm trying to do survival analysis using the Followup information, patient_vital_status and the expression of gene. I'm using like below: ...
0
votes
1answer
464 views

Use Shapley Values for explaining whole Data Frame instead of a Single prediction [closed]

I am working on a Machine Learning model. One of the requests is to explain the models 'decisions' to the business. Therefore I am using Shapley Values (Game Theory). I found an interesting example ...
6
votes
1answer
181 views

Does Approximate Bayesian Computation (ABC) follow the Likelihood Principle?

I know that ABC is commonly used when the likelihood is intractable, so likelihood principle is not an interest in that case. But, I am curious whether the ABC satisfies the likelihood principle when ...
1
vote
0answers
36 views

What is the process for detecting a 0.2 gallon/hour leak in an underground petrol tank with an accuracy of 95% or more?

Current EPA regulation stipulates that SIR (Statistical Inventory Reconciliation) vendors must be able to detect a leak of 0.2 gallons/hour with a probability of 0.95 or higher, and a false alarm rate ...
1
vote
0answers
29 views

Using R to maximize a two parameter Weibull model via multivariate extension of Newton-Raphson method

I am just getting back into using R for the first time in a while, and wrote some code to perform the aforementioned task in the title. I was wondering if anyone could take a look at it and see if ...
0
votes
0answers
83 views

Confidence Region of a multivariate KDE in Python?

I have an estimated bivariate kernel density based on a set of observations (𝑥11,𝑥12,...,𝑥1𝑛) and (𝑥21,𝑥22,...,𝑥2𝑛) and would like to draw confidence regions in the (𝑥1,𝑥2) space. This is in ...
3
votes
1answer
171 views

Which is more numerically stable for OLS: pinv vs QR

If I am doing standard OLS and want to calculate beta values (OLS estimators), which of the following is the more numerically stable method? And why? Assuming that the columns of $X$ are already mean-...
0
votes
1answer
30 views

A simple concept question regarding Bootstrapping

I am having a difficulty understanding whether I can use the bootstrapping method for prediction. First off, my data is as follows. where personal Income is the dependent variable (y), and GPA, Age, ...
2
votes
0answers
23 views

Downsample a stochastic process without losing correlation statistics?

I have a stochastic variable $X(t)$ which changes at a discrete set of random times $t_1, t_2, \dots$. I can simulate this stochastic process to obtain a series $X(t_1), X(t_2),\dots$ However, the ...
1
vote
1answer
38 views

QR decomposition computational efficiency

I am struggling to find a reference for this: In terms of big Oh notation does anyone know of any expressions for the computational time taken by commonly used algorithms for QR decompositions?
-1
votes
2answers
168 views

K-Means clustering and correlation

I ran K-Means on my dataset, it's a small dataset of 200 countries x 6 export sectors. My results formed three clusters. Now I want to check whether these three clusters are correlated with another ...
3
votes
1answer
76 views

What's a mean field variational family?

I'm working through variational Bayesian methods at the moment, and I think I have a grasp of the bigger picture. Where I sometimes have trouble is with the exact details of how it can be implemented. ...
2
votes
0answers
23 views

Using Markov random field spatial weights to account for spatial autocorrelation

I am looking at the relationship between life expectancy and smoking rate within the London boroughs. I thus created a bayesx spatial regression model including a term which assigns spatial ...
0
votes
2answers
66 views

Proof for “The sum of the observed values $Y_i$ equals the sum of the estimated / fitted values $\hat Y_i$”

I needed some help trying to understand why the sum of the observed values $Y_i$ equals the sum of the estimated values $\hat{Y}_i$.
2
votes
1answer
366 views

How does scipy calculate the binomial CDF so fast?

I ran scipy.stats.binom.cdf(500006, 1000000, 0.5) and it took less than a milisecond. This is crazy as binomial CDF involves summing up a bunch of binomial coefficients. What approximation algorithm ...
0
votes
1answer
27 views

Which analysis method question

I am currently conducting a short research paper for my university course and am struggling to determine which statistical analysis would be appropriate for the data I have collected. I surveyed two ...
2
votes
1answer
43 views

Covariance of Constrained Maximum Likelihood Estimators

I plan to numerically estimate the parameters of a GLM but with constraints imposed on some of the parameters. In this case, does the general approach of estimating the covariance matrix of my MLE ...
3
votes
1answer
49 views

Obtain global linear regression estimate from subsamples

I want to estimate $\widehat\beta$ in a simple linear regression with scikit. $$y = X \beta + \varepsilon$$ The problem is that the dimension of the complete $X$ is too large to fit into memory. Is ...
2
votes
0answers
29 views

Simulation - problem of maximization inside a circle

I am doing some projects related to statistics simulation using R based on "Introduction to Scientific Programming and Simulation Using R". In the Students projects session (chapter 24), I am doing ...
5
votes
2answers
740 views

Subtracting very small probabilities - How to compute? [duplicate]

This question is an extension of a related question about adding small probabilities. Suppose you have log-probabilities $\ell_1 \geqslant \ell_2$, where the corresponding probabilities $\exp(\ell_1)$...
0
votes
0answers
67 views

Must MSA be re-performed after annual instrument calibration?

I'm working in an automotive company that must follow IATF (International Automotive Task Force) requirements. I administrate a test-bench that performs radio frequency tests using an appropriate ...
2
votes
1answer
80 views

Vectorised computation of logsumexp

In this related post there is an explanation of how you can add together two very small probabilities using the logsumexp function, and how this can be programmed into base ...
1
vote
1answer
66 views

Data perturbation with normal variables

I am doing some projects related to statistics simulation using R based on "Introduction to Scientific Programming and Simulation Using R". In the Students projects session (chapter 24), I am doing ...
0
votes
1answer
69 views

Correcting data using poisson-regression

I'm new to stats and I was wondering if anyone had any good resources that could explain to me: How one can correct their data (false-positives) using Poisson-regression. I've been looking for some ...
0
votes
0answers
14 views

Effect of dimensionality for time taken to cluster data with k-means

In a dataset if I have $N$ features and for k-means clustering it might take $T$ seconds. If the dimensionality increased to $2N$, how would the time taken to run k-means clustering increase?
2
votes
0answers
34 views

Computing Wassertein Distance

For two probability measures $\mu$ and $\nu$, the Wassertein Distance is defined as $$W_p (\mu , \nu) = \left[ \inf\limits_{\gamma \in \Gamma} |x-y|^p \, d\gamma (x,y) \right] ^{\frac{1}{p}} \, , $$ ...
0
votes
1answer
154 views

Propensity to pay modelling

I am trying to build a propensity to pay model given an intervention to a customer. Context: The population I am dealing with are customers who were supposed to pay some amount on a certain date ...
0
votes
1answer
54 views

Tracking Moving Objects with Kalman Filters— Over-fitting over time?

I've been learning about Kalman Filters, and the classic example given is tracking an object via radar/gps. My issue here is that each time you get a new data point, you update the error in the ...