Refers to the interface of statistics and computing; the use of algorithms and software for statistical purposes.

learn more… | top users | synonyms

0
votes
0answers
16 views

Why the shape of mahalanobis values of a set of data is elliptical? [on hold]

When I searched about mahalanobis distance, i came to know that the shape of mahalanobis values of a set of data is elliptical. Then i searched why the shape is elliptical? But i didn't get any ...
0
votes
0answers
19 views

Appropriate testing for outliers

I want to perform an outlier-test for my data. The distribution of the data looks like a hyperbola (there are many values near 0, all values are >0). Sample size of data is 5000. I want to find an ...
10
votes
2answers
375 views

How do ABC and MCMC differ in their applications?

To my understanding Approximate Bayesian Computation (ABC) and Markov Chain Monte Carlo (MCMC) have very similar aims. Below I describe my understanding of these methods and how I perceive the ...
0
votes
3answers
24 views

fitting distribution with zero values [closed]

Can I estimate the claim severity distribution with R if the data contains zero claims? or i must remove them from the data?
0
votes
0answers
19 views

Distributed PCA

I have a large data set. Large means many instances (~50000000) and many features (~25000). I will call my data matrix X where the rows are instances and columns are features. Lets say the number of ...
3
votes
4answers
191 views

Why is gradient descent required?

When we can differentiate the cost function and find parameters by solving equations obtained through partial differentiation with respect to every parameter and find out where the cost function is ...
1
vote
0answers
35 views

How does one infer the noise/error model given measurements?

What resources are available for applying inference/computational statistics to infer the underlying error/noise model, given X measurements from some apparatus? (Below, I am mostly referring to ...
0
votes
1answer
38 views

What's up with this variance computation?

I'm trying to re-implement the Ckmeans.1D.DP algorithm in Python. I have the actual dynamic optimization part down, but I'm a little confused by the BIC computation they use for selecting the number ...
0
votes
0answers
6 views

finding and comparing temporal and location-specific pass patterns in a soccer game

I have data on several soccer matches, where ball passes have been recorded in terms of XY-location where the pass starts, XY location where the pass ends, the team making the pass, the player making ...
0
votes
0answers
13 views

Topics needed to learn to for predicting future of a data set?

I am planning to work this summer by writing a code which will eventually become an app, that has the ability to predict what the prices will be atleast 6 month ahead of time. I have a data set of the ...
1
vote
0answers
47 views

Server Requirements for machine learning on 300 million row database [closed]

I am trying to build a logistic regression model using Revolution R 64 bit. My data is 300 million rows by 12 columns. The data is stored on AWS Redshift, but takes about 6 hrs to import and save as ...
0
votes
0answers
8 views

How to find the generalized inverse of a matrix in SageMath?

Given a matrix $A_{m\times n}$, a matrix $G_{n\times m}$ is said to be a generalized inverse of $A$, if it satisfies, $$AGA=A$$ Now, since for a matrix $A$, generalized inverse of $A$ is plenty if ...
0
votes
0answers
16 views

Sampling from Irwin-Hall distribution using triangular distribution

So I need to sample from the Irwin-Hall distribution using rejection sampling with the triangular distribution. I built 2 functions: The first is d_irwin which receives an $x\in supp(g)$ and the n we ...
1
vote
0answers
28 views

Is it ok to ignore data in this instance?

I'm running a program in python I wrote to look at the first spiking time of a neuron in a stochastic network. I'm running about 2500 realizations for each set of parameter values to calculate the ...
1
vote
0answers
35 views

Sampling from Irwin-Hall (Uniform Sum) distribution using rejection sampling

Irwin-Hall Distribution is a probability distribution for a random variable defined as the sum of a number of independent random variables,each having a uniform distribution True or False: The ...
9
votes
2answers
223 views

Inverse covariance matrix vs covariance matrix in PCA

In PCA, does it make a difference if we pick principal components of the inverse covariance matrix OR if we drop eigenvectors of the covariance matrix corresponding to large eigenvalues? This is ...
0
votes
0answers
15 views

Robustness criterion (1978)

I am referring to bradley's journal relating to the robustness criterion. According to his liberal criterion, the simulated type 1 error rate have to between 0.5 alpha < p < 1.5 alpha. since i ...
1
vote
0answers
22 views

Binomial vs inverse binomial transform in obtaining moments

I am trying to obtain raw co-moment matrices from central co-moment matrices and I have found formulas for the transition from central to raw moments here . I have also found the opposite, that is ...
0
votes
0answers
39 views

How to compute the conditional probability distribution with given joint distributions and marginal distribution?

I am trying to compute the conditional distribution p(a|c) with given equation consists of a set of marginal distribution and joint distribution. Variables A,B,C have Markov property A->B->C. $p(a|c) ...
0
votes
1answer
16 views

What tests to analyze the following hypotheses with nominal data?

I have a set of questionnaire data to analyze the working culture for respondents. (There are four existing cultures; a,b,c and d) Variables including Gender, Organization, Public/Private Sector, ...
0
votes
0answers
17 views

Spark big matrix computation

I am trying to see if Spark is fit for my problem. I would like to use kernel methods, as presented in Kernel method. For example, kernel k-means. One important feature of the kernel methods is that ...
1
vote
0answers
39 views

Is this simple paired-sample permutation test valid?

I have about 10 pairs of scores, $(x_1, y_1),~ ...,~(x_{10}, y_{10})$, with all $x_i$ and $y_i$ being between 0.0 and 1.0. I'm interested in testing whether the mean difference over pairs is ...
3
votes
2answers
43 views

Cluster real numbers

I have a set of precise measurements, and what I want to do is count the frequency (how many time it appears) for each value. The problem is that these are very precise measurements and with a naive ...
0
votes
1answer
32 views

value of discrete items when only have their value when sold in groups

I'm doing an analysis for my job and would greatly appreciate any help you can offer! My company sells pallets full of items. Each item has a category (1 category per item, many items per category) ...
0
votes
0answers
31 views

predict mechanical component failure using machine learning

I'm going to do a research project on the possibility to predict mechanical component failure within multidisciplinary system, using sensors data. different sensors are being sampled in different ...
4
votes
1answer
72 views

Posterior distribution of parameter

I am looking at pg. 257 of the textbook "Statistical computing with R" by Maria L. Rizzo, where an example is given of a simple model of stock returns: We have 5 stocks, and at the end of a year of ...
0
votes
0answers
14 views

Best possible statistical approach to compare n pair of variables along a set of genomic intervals

I am dealing a bioinformatics problem and need help in approaching it in the right direction. I have a set of variables that represent different kinds of genomic data and are real valued. I am trying ...
0
votes
0answers
10 views

Find high occurring low values

In numerical data, where outlier is defined not by infrequency but value (this is a real-world data-set where low values would be indicative of noise). Although this is real world data, there is no ...
0
votes
0answers
10 views

creating tidy data sets collaboratively

I am wondering if there is a body of knowledge out there that describes how to best produce tidy data sets in a setting where a lot of people contribute to creating different pieces of the data. So ...
0
votes
0answers
18 views

Testing Random Number Generators(RNG)

There are numerous tests of RNGs. Consider that we tested RNG with lots of different seeds and the output is as follows: with certain seeds the RNG passes the test, with other seeds RNG doesn't pass. ...
0
votes
0answers
14 views

Figuring the algorithm / common calculation on a set

I have a column that is size in MM and another that reflects the change in price for said size. I'm trying to figure out the algorithm/common calculation behind these generated numbers so that I would ...
6
votes
1answer
75 views

Algebraic classifiers, more information?

I have read Algebraic classifiers: a generic approach to fast cross-validation, online training, and parallel training and was amazed by the performance of the derived algorithms. However, it seems ...
0
votes
1answer
30 views

Interpretaion of Logistic Coefficients

All, I ran a logistic Regression on a set of variables both categorical and continuous with a binary event as dependent variable. Now post modelling, I observe a set of categorical variables showing ...
0
votes
0answers
32 views

Calculate the non centrality parameter (ncp) for non central chi squared distribution

I have the 0.975-quantile value (0.191) and the degree of freedom (k=1). How can i calculate the non centrality parameter ? in R if possible. Q0.975(k=1, ncp= ??) = 0.191
1
vote
1answer
57 views

What does make SVM a “soft computing” method?

Soft computing is defined in [1] by the capability of "operating with uncertain, imprecise and incomplete information in a manner that reflects human thinking". So, based on my limited understanding, ...
0
votes
0answers
63 views

Build HMM of text data in R

I'm trying to make my own HMM tagging in R but don't know how to estimate parameter values since the packages I have been working with haven't worked with my data. The latest package I have been ...
2
votes
1answer
57 views

Efficient kernel estimation of density over grid of points

I was wondering whether it is possible to calculate density estimates for $n \times n$ matrix in $o(n^2)$ computational time. Imagine we have a bivariate random vector $(X,Y)$ with a joint density ...
0
votes
0answers
23 views

Finding proper cut off for filtering?

I had a set of data (149876 rows and 1200 columns)and I have found outliers points for each row. Now I want to determine a proper threshold for filtering my outliers. The distribution of my data is ...
1
vote
1answer
36 views

Random unit vectors in $\mathbb{C}^n$ and $\mathbb{R}^{2n}$

Suppose $\mathbf{u}\in\mathbb{C}^n$ is a complex random vector with circular symmetry, uniformly distributed on the unit complex $n$-sphere, so we have $\|\mathbf{u}\|=1$. In other words, $\mathbf{u}$ ...
0
votes
0answers
40 views

help with computing standard score for scarce discretely quantized data

I'm not a stats major and I'd appreciate any help I can get. I've got data with each point defined by a score (between 0-1) and a frequency (0-100%). At 100%, the score is most reliable, at 1% the ...
1
vote
0answers
141 views

calculating PMI for co-occurrences of words

I am in the process of building a question answering system. I am interested in calculating the PMI for words $x$ and $y$ occurring within 5 words of each other in a document. I have the formula and ...
0
votes
1answer
73 views

confusion matrix and Jaccard index computation in O(n) for cluster comparison

My problem: I have two clustering (or partitions) of a set of numbers $\{1,...,n\}$, each with $k$ cluster (or subsets), that i have to compare and, to do that, i'd like to try some different indexes ...
1
vote
0answers
30 views

Multivariate Q-Q plot to compare datasets

I'm trying to compare two datasets, each of two dimensions with unequal size. I tried using Q-Q plot to compare the datasets however I find it's implementation quite difficult. Can anyone suggest some ...
0
votes
2answers
37 views

How to interpret SPSS p-value for one tailed tests

Suppose H0 = Mean 1 - Mean 2 > or = 0 and HA = Mean 1 - Mean 2 < 0. We do an unpaired t-test in SPSS to get our p-value. The SPSS p-value is 0,534. This P-value is two tailed, so we have to devide ...
3
votes
1answer
36 views

Calculate the mean of the minimum of any k samples

Given a fixed list of N numbers from an unknown distribution, and a k (k <= N), say we ...
0
votes
0answers
11 views

Appropriate model to score a test dynamically?

I have a test bank of questions with particular difficulties (say, a number from 1 to 10), and people take "tests" consisting of some subset of those questions. Each question is graded on a 1 to 10 ...
9
votes
2answers
322 views

How to sample from discrete distribution on the non-negative integers?

I have the following discrete distribution, where $\alpha,\beta$ are known constants: $$ p(x;\alpha,\beta) = \frac{\text{Beta}(\alpha+1, \beta+x)}{\text{Beta}(\alpha,\beta)} \;\;\;\;\text{for } x = ...
0
votes
1answer
29 views

Is it possible to apply PCA to the reconstructed data of ICA?

I have read that it is not possible to verify the variance and order of independent component extracted from ICA. So, I reconstructed the spectra from matrix A and ST(independent component) then ...
0
votes
1answer
268 views

How can I calculate the mean and standard deviation given three percentiles using Matlab?

I have the percentiles of 3 values. The percentiles are 5th, 50th and 95th. The values respectively are 340, 365 and 385. How can I find the median and the standard deviation given that data follow a ...
4
votes
1answer
54 views

Duplicating data in ML training

Lets say I have a data set $X_1...X_N$ with the corresponding class of each event $Y(0,1)$. I'm training this set and getting a result for classification. What happens if I simply duplicate the ...