Questions tagged [computational-statistics]

Refers to the interface of statistics and computing; the use of algorithms and software for statistical purposes.

Filter by
Sorted by
Tagged with
3
votes
1answer
441 views

Adding very small probabilities - How to compute?

In some problems, probabilities are so small that they are best represented in computational facilities as log-probabilities. Computational problems can arise when you try to add these small ...
2
votes
1answer
99 views

Pearson correlation in data set with “jump”

I'm trying to calculate the correlation between the condition number of a finite elements matrix and the coarseness of the mesh that it represents. However, when trying to calculate the Pearson ...
1
vote
0answers
166 views

How do I measure information loss when converting categorical data to numerical?

Assume that a dataset has a mix of categorical and numerical attributes. The dataset has to undergo numeric processing which necessitates the conversion of the categorical attributes to numeric/...
1
vote
0answers
21 views

Suitable multiple testing procedure for three very correlated phenotypes

I have ran an analysis using polygenic risk scores including genetic variants at different p-value thresholds and have the following outcomes: IQ ages 8 and 9, strengths and difficulties questionnaire ...
0
votes
2answers
97 views

SPSS Statistics

The variance measures how far each number in the set is from the mean I have 151190 records of a table. and i have to analyze the data on SPSS. after the analysis i have some results. Mean=74617.92 ...
1
vote
1answer
77 views

Rules based Model (Function) - Derive Probability & Ensembling

Basically, let's assume I have a simple rules-based function/model (if weight >= 150) -> return true. Simple binary answer (true or false) from a single feature input. If I have a range of samples/...
6
votes
2answers
418 views

Evaluating the hazard function when the CDF is close to 1?

I need to evaluate a hazard function $h(t;\theta) = \dfrac{f(t;\theta)}{1-F(t;\theta)}$, where $f$ and $F$ are a pdf and a cdf, respectively, at many values of $t$ (and for several values of the ...
4
votes
1answer
234 views

Techniques for improving mixing when sampling from a multidimensional posterior

I'm currently trying to use the Metropolis-Hastings algorithm to sample from a posterior distribution of the form $$p(\theta | y ) \propto \prod_{ij} \phi (\theta_{ij}) \times \prod_{i=1}^n \pi_{y_i}...
1
vote
2answers
1k views

When is bootstrapping helpful and used?

When is bootstrapping helpful and when should it be used? I have watched several videos so I understand what bootstrapping does (samples with replacement from a single sample many times and create a ...
0
votes
1answer
308 views

Perplexity of a Non-Statistical Language Model

I have a piece of software that, given a input phrase, returns an ordered list of the next most likely words (entire vocab is ordered 1 to n). This is essentially an Language Model with the exception ...
4
votes
1answer
3k views

When to switch off the continuity correction in chisq.test function?

From this Research paper Table1 Association of RAD51-AS1 expression with clinicopathological features of EOC patients I see that p-value is calculated based on Chi-...
1
vote
0answers
40 views

What role(s) does Statistical Computing / Computational Statistics play in Clinical Trials?

I am going to pursue a PhD in Statistics, and the two fields that I am really interested are Statistical Computing / Computational Statistics and Clinical Trials (mainly because I like doing ...
0
votes
0answers
474 views

Computing Steady Probabilities for Markov Process

I hope I am asking this question on the correct SE, if not, please redirect it. I am trying to compute steady state probabilities for a Markov process using python. Knowing the transition rate values ...
1
vote
0answers
41 views

What tool do you use to Analyze Qualtrics CSV data

I’ve conducted a survey and have the csv export file as the results. What tool do you recommend to analyze the relationships between different factors? I think I can manually do ETL and put the data ...
3
votes
0answers
93 views

Computing the CDF of the minimum of particular dependent random variables

For each $i=1,\dots,n$ let $Z_i\sim\text{Poisson}(\lambda_i)$, and suppose $\{Z_i\}$ are independent. Also for each $i=1,\dots,n$, let $\{Y_{ij}\}_{j\in\mathbb{N}}$ be an infinite sequence of iid ...
0
votes
1answer
117 views

Filtering out bursts with a consistent range from a time series

I have time series of bursts that look like this: … and zoomed in: Now, there are also spurious bursts (which I call noise) in the data, which look like this: … and zoomed in: As you can see, the ...
5
votes
2answers
77 views

Computational statistics review

I'm looking for a mathematically rigorous review of key topics in computational statistics, such as numerical integration, EM algorithms, MCMC, and sampling algorithms. Are there any good lecture ...
1
vote
0answers
143 views

Traffic density estimation using deep learning

I am working on a project implementing deep learning and computer vision to estimate the traffic density of any random given road segment/roundabout or intersection. I am given a camera mounted on the ...
4
votes
1answer
76 views

Looking for a recursive formula for asymptotic variance of importance sampling estimator (self-normalized)

Looking for a recursive formula to approximate variance of importance sampling estimator $Var_q\big[\delta_{IS}\big]\approx\sum_{i=1}^n\tilde w(X_i)^2\big[h(X_i)-\delta_{IS}\big]^2$. This is an ...
1
vote
1answer
44 views

Summarizing a multi-dimensional distribution

I have an empirical distribution of observations along a number of dimensions. The dimensions are either categorical or ordinal (low frequency, equally spaced time steps from 0 to cut-off). At the ...
2
votes
1answer
93 views

When we state hypothesis does it mean null hypothesis or alternative hypothesis

In one of assignments in a study programme I got this statement: Hypothesis: University towns have their mean housing prices less effected by recessions. Run a t-test to compare the ratio of the ...
1
vote
1answer
1k views

Create an A/B Sample Size Calculator using Evan Miller's Post

To learn more about A/B Testing sample sizes selection I am attempting to use Evan Miller's popular blog-post to recreate a sample size calculator (https://www.evanmiller.org/sequential-ab-testing....
0
votes
0answers
126 views

Is this model really saturated? Is there a good alternative to an ANOVA for saturated models?

I have chemical data from groundwater wells that were exposed to two separate amendments of vegetable oil and monitored over several time points. I am trying to set up a 2-way ANOVA to test if the ...
1
vote
1answer
17 views

Training a classifier on 2 catalogs combined

I'm currently trying to train a machine to identify flashes of light in the sky as supernova (SN) or not. To do this, I'm combining 2 different catalogs: A big catalog (99% SN) with brighter flashes ...
2
votes
0answers
126 views

Machine Learning - Predicting Multiple Response Variables (Clothing Sizes)

For context, when a retailer buys in a clothing item they have to stock it in various sizes. The ratio of stock for each size is referred to as the "size ratio" of the product. E.g. S=0.2, M=0.5, L=...
1
vote
1answer
68 views

Compute $P(X > Y)$ for two random variables with unknown distributions from Markov chains

I would like to compute the probability $P(X > Y)$ with R. I used JAGS to sample from the posterior distribution of each variable, so I have a Markov chain for each variable (of length $3\times 10^{...
2
votes
0answers
80 views

Edgeworth Expansion of Discrete Random Variable

Background Consider a random variable $Z$ given by $$ Z=\sum_{i=1}^nX_i,$$ where the $X_i$ are iid random variables which are linear combinations (with both positive and negative coefficients) of ...
1
vote
1answer
116 views

How to compare two data sets with similar standard deviation and different type

I know my title is a little bit weird but here's an example. I was working on a project about RPG game. To be more simple, there are four values: physical - attack & defence magical - attack &...
0
votes
0answers
412 views

Comparing trends in 2 time series with different units, scales, directions of change?

I have two time series data as shown below in the figures. I want to compare if the rate of change is same in the both the line plots regardless of the direction of change (i.e increasing trend or ...
4
votes
0answers
935 views

Simulated Annealing Parameter Tuning

My question concerns parameter tuning for simulated annealing (SA). I've the following toy equation $$ y = (x^2+x) \times cos(2x) + 20 \text{ if } x \in (-10, 10) $$ My problem is that the solution ...
2
votes
0answers
39 views

Selection of differentially expressed genes

I don't have any statistical background. Have some questions. I see in some research papers they select differentially expressed genes based on fold change and p.value. And in some other papers I ...
-1
votes
1answer
65 views

A proper function for nonlinear regression with 3 predictors

There are three independent variables in my experimental work, namely flow rate (0.5 ≤ Q ≤ 9 where ΔQ = 0.5), particle size (a = {6, 10, 15}), and a geometric parameter (AR = {AR1, AR2, AR3, AR4}). ...
1
vote
0answers
197 views

In a neural network, how are weights adjusted between the third to last and second to last layer of neurons?

I haven't been able to find much information that explains back propagation in simple terms. I would like to know what is happening (what is the algorithm) to be able to code one in java. I know that ...
1
vote
2answers
146 views

Can you confirm the complexity of an algorithm using simulations?

Let's say we're solving a regression problem where $X$ is an $n \times p$ matrix, $Y$ is $n \times 1$, and $\beta$ is $p \times 1.$ Then if we use the naive approach to solving the least squares ...
0
votes
1answer
134 views

What is the most efficient way to compute the weighted least squares estimator?

If we have the estimator $(X^{T}WX)^{-1}X^{T}Wy$ where the diagonal of W contains the inverted weights, what is the fastest/most efficient way to solve it? I know with OLS, the estimator is typically ...
1
vote
1answer
34 views

numberical implementation of linear regression with “loose variable”

I understand how to solve a linear system $X \beta = y$ the solution is $\beta = (X^{T}X)^{-1} X^{T} y$ The problem is I could have an entry $\beta_i$ where it has no exposure in $X$. i.e. $X$ has ...
9
votes
1answer
131 views

sampling cost of $O(d)$ versus $O(2^d)$

I came across the following simulation problem: given a set $\{\omega_1,\ldots,\omega_d\}$ of known real numbers, a distribution on $\{-1,1\}^d$ is defined by $$\mathbb{P}(X=(x_1,\ldots,x_d))\propto (...
0
votes
0answers
19 views

Data Correlation for computing software metrics

I have the below sets of data per application, you can call them as software metrics. These metrics vary depending on the size of an application. ...
-1
votes
1answer
91 views

interpretation of this interaction

I am looking for help with the interpretation of a complicated interaction term. I am using a fixed effects model. The interaction term is: Code: ...
0
votes
1answer
82 views

Computing new variable of nominal and scale z scores variables

I'm analyzing a set of data where participants' reaction time and number of errors made during the task were measured. As there were two trials (where same data was measured) I was thinking I could ...
2
votes
1answer
1k views

Functional Principal Component Analysis (FPCA): Mean Function

How does one calculate a mean function in Functional Principal Component Analysis (FPCA) given a data set with an unknown distribution? (A theoretical approach if possible) I am wanting to implement ...
4
votes
2answers
873 views

Extracting power of a power law from data

My question is more about the methodology. Assuming in some experiment we have measured quantity $y$ per each unit of time $x.$ So $y$ and $x$ form our data set here. Moreover, we know that they are ...
4
votes
1answer
791 views

Why is L-BFGS optimization faster when binary features have been standardized?

I have a question related to standardization of binary features within a regularized logistic regression. Suppose that you have a model where all the features are binary (the result of applying the ...
6
votes
1answer
416 views

sampling from an unnormalised distribution

If one has to sample (with replacement) from a population $(x_1,x_2,\ldots)$ with weights $(\omega_1,\omega_2,\ldots)$, possibly infinite (although this is asking too much without further details), a ...
15
votes
2answers
1k views

What are some important uses of random number generation in computational statistics?

How and why are random number generators (RNGs) important in computational statistics? I understand that randomness is important when choosing samples for many statistical tests to avoid bias towards ...
2
votes
0answers
64 views

Why matrix inverse is not numerical stable?

I was told by many people, many times, Never explicitly compute matrix inverse, because it is not numerical stable. Instead, use LU, QR, SVD decomposition to solve the linear system. But why? Is ...
19
votes
4answers
1k views

What kinds of statistical problems are likely to benefit from quantum computing?

We are at the advent of quantum computing, with quantum languages anticipating hardware quantum computers now available at high and low levels for simulated quantum computers. Quantum computing brings ...
0
votes
0answers
92 views

Project Euler Problem 213 (continued…) [duplicate]

This question is a continuation of How should one approch Project Euler problem 213 ("Flea Circus")? I followed the approach of Glen_b outlined in his answer in the page referenced above. I ...
3
votes
0answers
418 views

Importance sampling: what is this bias?

I am experiencing what seems to be a bias in importance sampling, which, given that it's an unbiased procedure, should not be there. Consider linear regression $$ y = X\beta+\epsilon $$ where there ...
3
votes
2answers
7k views

Gradient descent explodes if learning rate is too large

I've implemented my own gradient descent algorithm for an OLS, code below. It work's, however, when the learning rate is too large (i.e. learn_rate >= .3), my approach is unstable. The coefficient's ...

1 2 3
4
5
13