Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [computational-statistics]

Refers to the interface of statistics and computing; the use of algorithms and software for statistical purposes.

-1
votes
0answers
5 views

Hands on prediction models using paradigms from a matrix that contains leds

I was interested in learning prediction models using paradigms in order to observe if there is a difference using different methods. The idea is to use multiple 6x6 matrices that contain 0/1 data in ...
1
vote
0answers
17 views

Total variation for draws from probability densities

I was wondering if there is a metric that is similar to that of total variation, which gives a measure of how different two probability distributions, $f$ and $g$ are, but works for samples? So ...
0
votes
1answer
39 views

Tracking Moving Objects with Kalman Filters— Over-fitting over time?

I've been learning about Kalman Filters, and the classic example given is tracking an object via radar/gps. My issue here is that each time you get a new data point, you update the error in the ...
1
vote
1answer
38 views

Adding very small probabilities - How to compute?

In some problems, probabilities are so small that they are best represented in computational facilities as log-probabilities. Computational problems can arise when you try to add these small ...
1
vote
1answer
24 views

Pearson correlation in data set with “jump”

I'm trying to calculate the correlation between the condition number of a finite elements matrix and the coarseness of the mesh that it represents. However, when trying to calculate the Pearson ...
0
votes
0answers
38 views

Are algorithms that solve the k-center problem part of machine learning or computational geometry?

I am currently working especially with the k-center problem, which is e.g. used to determine optimal locations for k warehouses. This is done by defining k circles that cover a given set of points (...
1
vote
0answers
17 views

How do I measure information loss when converting categorical data to numerical?

Assume that a dataset has a mix of categorical and numerical attributes. The dataset has to undergo numeric processing which necessitates the conversion of the categorical attributes to numeric/...
1
vote
0answers
17 views

Suitable multiple testing procedure for three very correlated phenotypes

I have ran an analysis using polygenic risk scores including genetic variants at different p-value thresholds and have the following outcomes: IQ ages 8 and 9, strengths and difficulties questionnaire ...
0
votes
1answer
12 views

SPSS Statistics

The variance measures how far each number in the set is from the mean I have 151190 records of a table. and i have to analyze the data on SPSS. after the analysis i have some results. Mean=74617.92 ...
1
vote
1answer
51 views

Rules based Model (Function) - Derive Probability & Ensembling

Basically, let's assume I have a simple rules-based function/model (if weight >= 150) -> return true. Simple binary answer (true or false) from a single feature input. If I have a range of samples/...
0
votes
0answers
10 views

how to simulate with randdirichlet in SAS

My data has 100 obs, in 4 categories, each category has 22 obs, 25 obs, 35 obs and 18 obs . I want to create a random sample for these 4 proportions with randdirichlet in sas. Is the following the ...
0
votes
0answers
17 views

Derive machine learning features from dependant variables in each data sample such that maximum information is retained

I have a collection of values for two different variables X1 and X2 for each timestamp, as briefly represented below: Timestamp 1: X1=[20,50,175], X2=[100,200,300] Timestamp 2: X1=[15,55,55,150,500],...
4
votes
1answer
39 views

Techniques for improving mixing when sampling from a multidimensional posterior

I'm currently trying to use the Metropolis-Hastings algorithm to sample from a posterior distribution of the form $$p(\theta | y ) \propto \prod_{ij} \phi (\theta_{ij}) \times \prod_{i=1}^n \pi_{y_i}...
0
votes
0answers
15 views

Application of Factor Analysis to a table of count data

I am being asked to apply a statistical technique that I do not think is optimal. I have asked before, but was told my question was vague. I have re-worded it appropriately here. I am being asked to ...
0
votes
0answers
7 views

How to compare a test case with standards with multiple (100+) variables?

I have a question about how to statistically get an answer for a question my group has. We have a test dataset where we have 100 independent variables with values ranging from 0-1. We then have a ...
0
votes
0answers
11 views

Compute p-value for very high z scores [duplicate]

For a statistical test I need to compute the p-value given a z score. I am using the Python method: scipy.stats.norm.cdf the problem is that for z score > 8, ...
1
vote
2answers
44 views

When is bootstrapping helpful and used?

When is bootstrapping helpful and when should it be used? I have watched several videos so I understand what bootstrapping does (samples with replacement from a single sample many times and create a ...
0
votes
1answer
88 views

Perplexity of a Non-Statistical Language Model

I have a piece of software that, given a input phrase, returns an ordered list of the next most likely words (entire vocab is ordered 1 to n). This is essentially an Language Model with the exception ...
1
vote
1answer
238 views

When to switch off the continuity correction in chisq.test function?

From this Research paper Table1 Association of RAD51-AS1 expression with clinicopathological features of EOC patients I see that p-value is calculated based on Chi-...
0
votes
0answers
28 views

On average mean estimation is higher if the sample size is increased. Long tailed distribution [duplicate]

As an experiment I am drawing random sample from long tailed distribution. Is it possible that I am likely to get higher sample mean, as I increase the sample size? It is confusing in reporting the ...
1
vote
0answers
20 views

What role(s) does Statistical Computing / Computational Statistics play in Clinical Trials?

I am going to pursue a PhD in Statistics, and the two fields that I am really interested are Statistical Computing / Computational Statistics and Clinical Trials (mainly because I like doing ...
0
votes
0answers
125 views

Computing Steady Probabilities for Markov Process

I hope I am asking this question on the correct SE, if not, please redirect it. I am trying to compute steady state probabilities for a Markov process using python. Knowing the transition rate values ...
0
votes
0answers
11 views

Avoiding underflow errors when marginalizing over a nuisance parameter in Bayesian inference

I was reading this question about how to marginalize over nuisance parameters in Bayesian inference, and the concern I have is how to deal with underflow errors. If we are interested only in the ...
1
vote
0answers
13 views

What tool do you use to Analyze Qualtrics CSV data

I’ve conducted a survey and have the csv export file as the results. What tool do you recommend to analyze the relationships between different factors? I think I can manually do ETL and put the data ...
3
votes
0answers
60 views

Computing the CDF of the minimum of particular dependent random variables

For each $i=1,\dots,n$ let $Z_i\sim\text{Poisson}(\lambda_i)$, and suppose $\{Z_i\}$ are independent. Also for each $i=1,\dots,n$, let $\{Y_{ij}\}_{j\in\mathbb{N}}$ be an infinite sequence of iid ...
0
votes
1answer
94 views

Filtering out bursts with a consistent range from a time series

I have time series of bursts that look like this: … and zoomed in: Now, there are also spurious bursts (which I call noise) in the data, which look like this: … and zoomed in: As you can see, the ...
4
votes
0answers
29 views

Computational statistics review

I'm looking for a mathematically rigorous review of key topics in computational statistics, such as numerical integration, EM algorithms, MCMC, and sampling algorithms. Are there any good lecture ...
0
votes
0answers
13 views

Computational Optimization of Cross-Validation in Training Models

TL;DR: If I run a 5-fold cross-validation which assigns 1 fold per core when my CPU only has 4 cores, is the 5th as costly (with respect to computation time) as the first 4 folds? Context: I am ...
0
votes
0answers
19 views

variance of projected data onto a vector

I am reading this great article on PCA and there is something that I do not understand. In the paragraph Eigendecomposition of a covariance matrix, the author states that if we were to project our ...
1
vote
0answers
48 views

Traffic density estimation using deep learning

I am working on a project implementing deep learning and computer vision to estimate the traffic density of any random given road segment/roundabout or intersection. I am given a camera mounted on the ...
3
votes
0answers
16 views

Looking for a recursive formula for asymptotic variance of importance sampling estimator (self-normalized)

Looking for a recursive formula to approximate variance of importance sampling estimator $Var_q\big[\delta_{IS}\big]\approx\sum_{i=1}^n\tilde w(X_i)^2\big[h(X_i)-\delta_{IS}\big]^2$. This is an ...
1
vote
1answer
35 views

Summarizing a multi-dimensional distribution

I have an empirical distribution of observations along a number of dimensions. The dimensions are either categorical or ordinal (low frequency, equally spaced time steps from 0 to cut-off). At the ...
2
votes
1answer
47 views

When we state hypothesis does it mean null hypothesis or alternative hypothesis

In one of assignments in a study programme I got this statement: Hypothesis: University towns have their mean housing prices less effected by recessions. Run a t-test to compare the ratio of the ...
0
votes
0answers
53 views

Create an A/B Sample Size Calculator using Evan Miller's Post

To learn more about A/B Testing sample sizes selection I am attempting to use Evan Miller's popular blog-post to recreate a sample size calculator (https://www.evanmiller.org/sequential-ab-testing....
0
votes
0answers
15 views

EMA and WMA computation

Hello (I am new to statistic, I am 20 years old just starting out, please don't be so rude) Given these numbers: 2, 3, 4, 8, 10, 15, 22 and here the number of days (each number for the ...
0
votes
0answers
19 views

Is this model really saturated? Is there a good alternative to an ANOVA for saturated models?

I have chemical data from groundwater wells that were exposed to two separate amendments of vegetable oil and monitored over several time points. I am trying to set up a 2-way ANOVA to test if the ...
1
vote
1answer
14 views

Training a classifier on 2 catalogs combined

I'm currently trying to train a machine to identify flashes of light in the sky as supernova (SN) or not. To do this, I'm combining 2 different catalogs: A big catalog (99% SN) with brighter flashes ...
1
vote
0answers
33 views

Machine Learning - Predicting Multiple Response Variables (Clothing Sizes)

For context, when a retailer buys in a clothing item they have to stock it in various sizes. The ratio of stock for each size is referred to as the "size ratio" of the product. E.g. S=0.2, M=0.5, L=...
1
vote
1answer
54 views

Compute $P(X > Y)$ for two random variables with unknown distributions from Markov chains

I would like to compute the probability $P(X > Y)$ with R. I used JAGS to sample from the posterior distribution of each variable, so I have a Markov chain for each variable (of length $3\times 10^{...
2
votes
0answers
38 views

Edgeworth Expansion of Discrete Random Variable

Background Consider a random variable $Z$ given by $$ Z=\sum_{i=1}^nX_i,$$ where the $X_i$ are iid random variables which are linear combinations (with both positive and negative coefficients) of ...
1
vote
0answers
41 views

How to compare two data sets with similar standard deviation and different type

I know my title is a little bit weird but here's an example. I was working on a project about RPG game. To be more simple, there are four values: physical - attack & defence magical - attack &...
0
votes
0answers
48 views

Comparing trends in 2 time series with different units,scales,directions of change

I have two time series data as shown below in the figures. I want to compare if the rate of change is same in the both the line plots regardless of the direction of change(i.e increasing trend or ...
1
vote
0answers
104 views

Simulated Annealing Parameter Tuning

My question concerns parameter tuning for simulated annealing (SA). I've the following toy equation $$ y = (x^2+x) \times cos(2x) + 20 \text{ if } x \in (-10, 10) $$ My problem is that the solution ...
2
votes
0answers
23 views

Selection of differentially expressed genes

I don't have any statistical background. Have some questions. I see in some research papers they select differentially expressed genes based on fold change and p.value. And in some other papers I ...
-1
votes
1answer
52 views

A proper function for nonlinear regression with 3 predictors

There are three independent variables in my experimental work, namely flow rate (0.5 ≤ Q ≤ 9 where ΔQ = 0.5), particle size (a = {6, 10, 15}), and a geometric parameter (AR = {AR1, AR2, AR3, AR4}). ...
0
votes
2answers
57 views

Can you confirm the complexity of an algorithm using simulations?

Let's say we're solving a regression problem where $X$ is an $n \times p$ matrix, $Y$ is $n \times 1$, and $\beta$ is $p \times 1.$ Then if we use the naive approach to solving the least squares ...
0
votes
1answer
27 views

What is the most efficient way to compute the weighted least squares estimator?

If we have the estimator $(X^{T}WX)^{-1}X^{T}Wy$ where the diagonal of W contains the inverted weights, what is the fastest/most efficient way to solve it? I know with OLS, the estimator is typically ...
1
vote
1answer
26 views

numberical implementation of linear regression with “loose variable”

I understand how to solve a linear system $X \beta = y$ the solution is $\beta = (X^{T}X)^{-1} X^{T} y$ The problem is I could have an entry $\beta_i$ where it has no exposure in $X$. i.e. $X$ has ...
7
votes
1answer
71 views

sampling cost of $O(d)$ versus $O(2^d)$

I came across the following simulation problem: given a set $\{\omega_1,\ldots,\omega_d\}$ of known real numbers, a distribution on $\{-1,1\}^d$ is defined by $$\mathbb{P}(X=(x_1,\ldots,x_d))\propto (...
0
votes
0answers
16 views

Data Correlation for computing software metrics

I have the below sets of data per application, you can call them as software metrics. These metrics vary depending on the size of an application. ...