Questions tagged [computing]

For on-topic questions involving statistical computing. Please include also some statistical methods tag.

Filter by
Sorted by
Tagged with
0
votes
0answers
103 views

Transfer learning when Dropout in source model

Anyone can you pls help me to understand that. If we have dropout layers in between the convolutional layers of source model. When we do transfer learning from source to target model, the convotional ...
5
votes
1answer
50 views

Stable and efficient computation of binomial expectations

Suppose we want to compute the expected value of some function $f(X)$ where $X \sim \text{Bin}(n,\theta)$. Taking $\mathbf{f} = (f_0,...,f_n)$ to be the function values over all possible outcomes of ...
162
votes
21answers
67k views

Does Julia have any hope of sticking in the statistical community?

I recently read a post from R-Bloggers, that linked to this blog post from John Myles White about a new language called Julia. Julia takes advantage of a just-in-time compiler that gives it wicked ...
3
votes
1answer
265 views

Find true negatives in a confusion matrix

I'm trying to find the True negative in a confusion matrix, I have computed successfully from scratch the precision and recall/sensibility, now i need to compute the accuracy and specificity. This is ...
19
votes
3answers
847 views

Julia: Taking stock of how it has been doing

I came across a 2012 question that had a very good discussion about Julia as an alternative to R / Python for various types of Statistical Work. Here lies the original Question from 2012 about Julia'...
36
votes
4answers
59k views

How to sample from a normal distribution with known mean and variance using a conventional programming language?

I've never had a course in statistics, so I hope I'm asking in the right place here. Suppose I have only two data describing a normal distribution: the mean $\mu$ and variance $\sigma^2$. I want to ...
1
vote
1answer
41 views

Is it methodologically correct to, in some specific situations, merge related binary variables into one continuous?

I have 10 yes/no binary variables regarding different types of economic risks associated with running a startup. I have a database of around 100 different startup companies evaluated on each of those ...
8
votes
1answer
414 views

Symbolic computer algebra for statistics

What functionality should exist in a CAS that was specifically geared toward Statistics? Symbolic algebra systems like Mathematica and Maple are often used for calculus, logic, and physics problems ...
4
votes
2answers
195 views

Avoiding matrix inversions with the multivariate Gaussian distribution?

Let $(X, Y)$ be a Gaussian random vector with mean $\mu$, variance $\Sigma$ and both $X$ and $Y$ are vectors. No structure is assumed on $\Sigma$ (aside from being symmetric positive definite). Is ...
6
votes
2answers
381 views

Evaluating the hazard function when the CDF is close to 1?

I need to evaluate a hazard function $h(t;\theta) = \dfrac{f(t;\theta)}{1-F(t;\theta)}$, where $f$ and $F$ are a pdf and a cdf, respectively, at many values of $t$ (and for several values of the ...
15
votes
9answers
5k views

What books provide an overview of computational statistics as it applies to computer science?

As a software engineer, I'm interested in topics such as statistical algorithms, data mining, machine learning, Bayesian networks, classification algorithms, neural networks, Markov chains, Monte ...
1
vote
0answers
25 views

What role(s) does Statistical Computing / Computational Statistics play in Clinical Trials?

I am going to pursue a PhD in Statistics, and the two fields that I am really interested are Statistical Computing / Computational Statistics and Clinical Trials (mainly because I like doing ...
27
votes
12answers
17k views

Command-line tool to calculate basic statistics for stream of values [closed]

Is there any command-line tool that accepts the flow of numbers (in ascii format) from standard input and gives the basic descriptive statistics for this flow, such as min, max, average, median, RMS, ...
1
vote
0answers
175 views

In a neural network, how are weights adjusted between the third to last and second to last layer of neurons?

I haven't been able to find much information that explains back propagation in simple terms. I would like to know what is happening (what is the algorithm) to be able to code one in java. I know that ...
3
votes
1answer
1k views

Difference between pROC and ROCR in compute time and accuracy

I've been calculating receiver operating characteristic (ROC) curves on very large datasets for my thesis. I tried to run these in the pROC R package but the ...
0
votes
1answer
49 views

Computing new variable of nominal and scale z scores variables

I'm analyzing a set of data where participants' reaction time and number of errors made during the task were measured. As there were two trials (where same data was measured) I was thinking I could ...
23
votes
4answers
11k views

C++ libraries for statistical computing

I've got a particular MCMC algorithm which I would like to port to C/C++. Much of the expensive computation is in C already via Cython, but I want to have the whole sampler written in a compiled ...
1
vote
0answers
215 views

Why is a fully marginalized likelihood so much more difficult to computer than a partially marginalized posterior?

I have a question in terms of Bayesian computation. Let's contrast parameter estimation and model selection. When performing a Bayesian hypothesis test, this involves computing fully marginalized ...
1
vote
1answer
122 views

How to derive this version of variance formula

A probably very easy computational question, but I don't really understand how it's done: I try to compute the point estimates of a normal distribution as $N \sim (\beta, \sigma)$. Using the method ...
1
vote
0answers
23 views

Expected value of 3-repeated keys for a cuckoo hash table

I'm trying to solve a problem for my data structures class and am having problem computing the expectation for this scenario. I have wrote the question to be as independed from CS as possible I have ...
1
vote
0answers
21 views

Summarizing results obtained in different scenarios by different methods

I have generated different scenarios which depends on certain parameters $x_1, \dots, x_k$. I have evaluated each scenario with different methods. Summarizing, I have a value $S(m,x_1, \dots, x_k)$ ...
2
votes
1answer
145 views

Back-calculating the log-likelihood ratio statistic from a hazard ratio

I am wondering whether and how can I back-calculate the log-likelihood ratio statistic from a Cox proportional hazard analysis with 1 degree of freedom. For instance, if I have an hazard ratio (HR) ...
3
votes
1answer
490 views

Computing by hand vs. R's magic wand

I have fitted a polynomial to a data set that I have. Thus I have obtained coefficients $\beta_i$ for $i=0,1,2$ and have a relationship of the form is $$Y=\beta_0+\beta_{1}X+\beta_{2}X^{2}+\varepsilon$...
4
votes
1answer
1k views

Does it make sense to use PCA when the determinant of the correlation matrix is (almost) zero?

I'm running a PCA over a data set of $N \times p$ size ($N\approx 1000$ being the number of measurements and $p\approx 200$ being the number of dimensions/predictors). I expect many of the predictors ...
5
votes
2answers
1k views

Speeding up hat matrices like $X(X'X)^{-1}X'$ (projection matrices) and other aspects of custom-built estimator when software runs out of memory

Is there a way to speed up $Z(Z'Z)^{-1}Z'$ type matrices? I am implementing the expression below directly using a matrix language and my program frequently crashes while if I run OLS on them using a ...
3
votes
3answers
6k views

C code programming for statistics

Where can we find C code for statistical computing? R is kind of slow when doing a lot of iterations. C is a lot faster but coding is always heavy-duty. I am wondering does anybody have any good ...
2
votes
2answers
2k views

Issues with using glmnet package for MATLAB [closed]

I am trying to use the glmnet MATLAB package to train my elastic net model on some huge data. My features are of size 13200, and I have around 6000 samples of these....
6
votes
1answer
3k views

What is the current 'standard' for modern statistical computing hardware?

I am in the market for a new system (probably a laptop) that would be be used primarily for Bayesian/MCMC analyses. If I had unlimited funds I would obviously buy very high end hardware and be done ...
2
votes
1answer
142 views

Anomaly Detection in a set of points

I have a set of points in a matrix of size 100 x 100(total 10000 points). I know that there are roughly 500 anomaly points in it. There is a corresponding truth file which contains the true anomalous ...
10
votes
4answers
277 views

Testing statistical software

What techniques/approaches are useful in testing statistical software? I'm particularly interested in programs that do parametric estimation using maximum likelihood. Comparing results to those from ...
16
votes
4answers
4k views

Who uses R with multicore, SNOW or CUDA package for resource intense computing?

Who of you in this forum uses ">R with the multicore, snow packages, or CUDA, so for advanced calculations that need more power than a workstation CPU? On which hardware do you compute these scripts? ...
7
votes
1answer
2k views

How important is using the exact method for ties in a Cox model, and how long should this take?

I'm analyzing the results of some simulation work using a Cox proportional hazard model, and I have what I perceive are a great many ties in the data, representing when a particular individual in the ...
1
vote
1answer
388 views

Inverse of the central difference operator

The central difference is a method to approximate numerically the derivative of a function sampled at discrete intervals. In R, one would do: ...
1
vote
1answer
126 views

Is there a qualitatively useful statistic or approach to an ill-behaving sample average?

I was running some timing simulations on a computer. As is often done with low latency things like this, I was running the relevant block in a loop $k$ times and then recording the total time over $k$ ...
1
vote
0answers
290 views

Computing the expected value using the sample mean gives poor convergence?

This is for Project Euler problem #371: http://projecteuler.net/problem=371 My solution was to generate a huge random pool of numbers between 0 and 999 inclusive, repeatedly ask the question 'How ...
16
votes
6answers
9k views

Computation speed in R?

I have been tasked with moving one of our current large stochastic models out of SAS and into a new language. Personally, I prefer a traditional compiled language, but the PI wants me to check out R, ...
4
votes
1answer
223 views

Multiple simulations of a system under different conditions - paired data?

I am currently generating data by simulating a model of chemical system under different conditions (temperature) over time. In each simulation, the starting structure being modeled is exactly the same ...
7
votes
2answers
3k views

How to make R's gamm work faster?

Last night I started a complex calculation with gamm() and it took me... user system elapsed 9259.76 326.05 9622.64 (s) ......
12
votes
1answer
308 views

Online, scalable statistical methods

This was inspired by Efficient online linear regression, which I found very interesting. Are there any texts or resources devoted to large-scale statistical computing, by which computing with ...
4
votes
0answers
531 views

How to compute the weight matrix for WLS estimation of a multi-group ordinal CFA model

I am attempting to perform a two-group confirmatory factor analysis (CFA) of one continuous factor on six ordinal predictors in OpenMx (...
3
votes
1answer
946 views

Which method to use for anomaly detection?

I think people here could guide me in solving a problem related to anomaly detection in Computer Science. The term anomaly here refers to some undesired event occuring in the system like a virus ...
2
votes
2answers
3k views

How to get around Numerical Overflow in Stata?

I am working with a large data set (approximately 50K observations) and trying to running a Maximum likelihood estimation on 5 unknowns in Stata. I encountered an error message of "Numerical ...
6
votes
3answers
608 views

Visualization of a multivariate function

This is somewhat vague, but suppose you have a black box function $f(x_1,x_2,\ldots,x_k)$, for which you have code, and you are interested in the behaviour of $f$ when the $x_i$ are i.i.d. standard ...