Refers to the interface of statistics and computing; the use of algorithms and software for statistical purposes.

learn more… | top users | synonyms

1
vote
0answers
20 views

Negative estimated values [on hold]

I'm using Weka API to make a prediction with SVR. My dataset contains 499 records. In the outpout of the program, I get some negative predicted values. Is that due to problem of Overfitting? If yes, ...
0
votes
0answers
37 views

Recommend a optimization book with more coding examples?

I am interested in continuous optimization problems. However, I feel it is very difficult for me to understand the classic books such as Convex Optimization or Numerical Optmization. My problem with ...
5
votes
3answers
98 views

What is the best way to detect repetition in xyz data for purposes of splitting data?

I'll use this picture to explain What I want to do is define some patterns as trained patterns. Then given data I want to be able to determine if the pattern exists in the dataset, and if it does ...
2
votes
2answers
291 views

The value of an Effect Size

I calculated a Cohen's d value of d= -2.1. I understand that there are small, medium, and large effect sizes. But in my case the d value is negative? Would it still be considered large since abs(-2.1)...
2
votes
1answer
27 views

What is your configuration to do statistical analysis on large data sets?

The stat packages which I've been using are not very well suited for working with large data sets, i.e. more than 1 billion records. They all tend to try loading all into memory, and install one one ...
2
votes
1answer
26 views

Is it possible to add up the accuracy rates of 2 predictors?

Weather channel 1 has a 65% accuracy rate of predicting tomorrows weather Weather channel 2 has a 59% accuracy rate of predicting tomorrows weather Is it somehow possible to take into account ...
0
votes
0answers
11 views

How to measure the impact of a product on the global revenue?

We have products daily revenue and when each product started being sold by the company. We also have different break-down of product revenue by location, store, sales person, etc. We need to measure ...
0
votes
1answer
49 views

Does it make sense to cluster based on Euclidean distances between rows of a cosine matrix?

I calculated the Cosine distances for binary data and got the relations between different variables. I need to cluster them. I tried passing the cosine matrix directly to the (clustering) function ...
0
votes
0answers
31 views

How to compute correlation for two data sets?

I am a CS student and naive in statistics. I have 2 related data sets, to which I want to find correlation. Let me explain the case first: This is a question related to computer architecture. One ...
4
votes
1answer
57 views

Gibbs sampling and conditional distribution

I need to simulate the posterior distribution of intraclass correlation coefficient $\pi(\rho|y)$ where $y$ is the data set and $\rho=\frac{\sigma_a^2}{\sigma_a^2+\sigma_e^2}$ with $\sigma^2_a\sim IG(\...
0
votes
0answers
21 views

Appropriate testing for outliers

I want to perform an outlier-test for my data. The distribution of the data looks like a hyperbola (there are many values near 0, all values are >0). Sample size of data is 5000. I want to find an ...
12
votes
2answers
485 views

How do ABC and MCMC differ in their applications?

To my understanding Approximate Bayesian Computation (ABC) and Markov Chain Monte Carlo (MCMC) have very similar aims. Below I describe my understanding of these methods and how I perceive the ...
0
votes
0answers
19 views

Distributed PCA

I have a large data set. Large means many instances (~50000000) and many features (~25000). I will call my data matrix X where the rows are instances and columns are features. Lets say the number of ...
3
votes
4answers
241 views

Why is gradient descent required?

When we can differentiate the cost function and find parameters by solving equations obtained through partial differentiation with respect to every parameter and find out where the cost function is ...
1
vote
0answers
36 views

How does one infer the noise/error model given measurements?

What resources are available for applying inference/computational statistics to infer the underlying error/noise model, given X measurements from some apparatus? (Below, I am mostly referring to ...
0
votes
1answer
38 views

What's up with this variance computation?

I'm trying to re-implement the Ckmeans.1D.DP algorithm in Python. I have the actual dynamic optimization part down, but I'm a little confused by the BIC computation they use for selecting the number ...
0
votes
0answers
8 views

finding and comparing temporal and location-specific pass patterns in a soccer game

I have data on several soccer matches, where ball passes have been recorded in terms of XY-location where the pass starts, XY location where the pass ends, the team making the pass, the player making ...
0
votes
0answers
13 views

Topics needed to learn to for predicting future of a data set?

I am planning to work this summer by writing a code which will eventually become an app, that has the ability to predict what the prices will be atleast 6 month ahead of time. I have a data set of the ...
1
vote
0answers
54 views

Server Requirements for machine learning on 300 million row database [closed]

I am trying to build a logistic regression model using Revolution R 64 bit. My data is 300 million rows by 12 columns. The data is stored on AWS Redshift, but takes about 6 hrs to import and save as ...
0
votes
0answers
13 views

How to find the generalized inverse of a matrix in SageMath?

Given a matrix $A_{m\times n}$, a matrix $G_{n\times m}$ is said to be a generalized inverse of $A$, if it satisfies, $$AGA=A$$ Now, since for a matrix $A$, generalized inverse of $A$ is plenty if ...
0
votes
0answers
17 views

Sampling from Irwin-Hall distribution using triangular distribution

So I need to sample from the Irwin-Hall distribution using rejection sampling with the triangular distribution. I built 2 functions: The first is d_irwin which receives an $x\in supp(g)$ and the n we ...
1
vote
0answers
28 views

Is it ok to ignore data in this instance?

I'm running a program in python I wrote to look at the first spiking time of a neuron in a stochastic network. I'm running about 2500 realizations for each set of parameter values to calculate the ...
1
vote
0answers
36 views

Sampling from Irwin-Hall (Uniform Sum) distribution using rejection sampling

Irwin-Hall Distribution is a probability distribution for a random variable defined as the sum of a number of independent random variables,each having a uniform distribution True or False: The ...
9
votes
2answers
244 views

Inverse covariance matrix vs covariance matrix in PCA

In PCA, does it make a difference if we pick principal components of the inverse covariance matrix OR if we drop eigenvectors of the covariance matrix corresponding to large eigenvalues? This is ...
0
votes
0answers
17 views

Robustness criterion (1978)

I am referring to bradley's journal relating to the robustness criterion. According to his liberal criterion, the simulated type 1 error rate have to between 0.5 alpha < p < 1.5 alpha. since i ...
1
vote
0answers
23 views

Binomial vs inverse binomial transform in obtaining moments

I am trying to obtain raw co-moment matrices from central co-moment matrices and I have found formulas for the transition from central to raw moments here . I have also found the opposite, that is ...
0
votes
0answers
52 views

How to compute the conditional probability distribution with given joint distributions and marginal distribution?

I am trying to compute the conditional distribution p(a|c) with given equation consists of a set of marginal distribution and joint distribution. Variables A,B,C have Markov property A->B->C. $p(a|c) ...
0
votes
1answer
16 views

What tests to analyze the following hypotheses with nominal data?

I have a set of questionnaire data to analyze the working culture for respondents. (There are four existing cultures; a,b,c and d) Variables including Gender, Organization, Public/Private Sector, ...
0
votes
0answers
18 views

Spark big matrix computation

I am trying to see if Spark is fit for my problem. I would like to use kernel methods, as presented in Kernel method. For example, kernel k-means. One important feature of the kernel methods is that ...
1
vote
0answers
40 views

Is this simple paired-sample permutation test valid?

I have about 10 pairs of scores, $(x_1, y_1),~ ...,~(x_{10}, y_{10})$, with all $x_i$ and $y_i$ being between 0.0 and 1.0. I'm interested in testing whether the mean difference over pairs is ...
3
votes
2answers
45 views

Cluster real numbers

I have a set of precise measurements, and what I want to do is count the frequency (how many time it appears) for each value. The problem is that these are very precise measurements and with a naive ...
0
votes
1answer
33 views

value of discrete items when only have their value when sold in groups

I'm doing an analysis for my job and would greatly appreciate any help you can offer! My company sells pallets full of items. Each item has a category (1 category per item, many items per category) ...
0
votes
0answers
40 views

predict mechanical component failure using machine learning

I'm going to do a research project on the possibility to predict mechanical component failure within multidisciplinary system, using sensors data. different sensors are being sampled in different ...
4
votes
1answer
72 views

Posterior distribution of parameter

I am looking at pg. 257 of the textbook "Statistical computing with R" by Maria L. Rizzo, where an example is given of a simple model of stock returns: We have 5 stocks, and at the end of a year of ...
0
votes
0answers
14 views

Best possible statistical approach to compare n pair of variables along a set of genomic intervals

I am dealing a bioinformatics problem and need help in approaching it in the right direction. I have a set of variables that represent different kinds of genomic data and are real valued. I am trying ...
0
votes
0answers
10 views

Find high occurring low values

In numerical data, where outlier is defined not by infrequency but value (this is a real-world data-set where low values would be indicative of noise). Although this is real world data, there is no ...
0
votes
0answers
10 views

creating tidy data sets collaboratively

I am wondering if there is a body of knowledge out there that describes how to best produce tidy data sets in a setting where a lot of people contribute to creating different pieces of the data. So ...
0
votes
0answers
18 views

Testing Random Number Generators(RNG)

There are numerous tests of RNGs. Consider that we tested RNG with lots of different seeds and the output is as follows: with certain seeds the RNG passes the test, with other seeds RNG doesn't pass. ...
0
votes
0answers
14 views

Figuring the algorithm / common calculation on a set

I have a column that is size in MM and another that reflects the change in price for said size. I'm trying to figure out the algorithm/common calculation behind these generated numbers so that I would ...
6
votes
1answer
76 views

Algebraic classifiers, more information?

I have read Algebraic classifiers: a generic approach to fast cross-validation, online training, and parallel training and was amazed by the performance of the derived algorithms. However, it seems ...
0
votes
1answer
31 views

Interpretaion of Logistic Coefficients

All, I ran a logistic Regression on a set of variables both categorical and continuous with a binary event as dependent variable. Now post modelling, I observe a set of categorical variables showing ...
0
votes
0answers
33 views

Calculate the non centrality parameter (ncp) for non central chi squared distribution

I have the 0.975-quantile value (0.191) and the degree of freedom (k=1). How can i calculate the non centrality parameter ? in R if possible. Q0.975(k=1, ncp= ??) = 0.191
1
vote
1answer
58 views

What does make SVM a “soft computing” method?

Soft computing is defined in [1] by the capability of "operating with uncertain, imprecise and incomplete information in a manner that reflects human thinking". So, based on my limited understanding, ...
0
votes
0answers
71 views

Build HMM of text data in R

I'm trying to make my own HMM tagging in R but don't know how to estimate parameter values since the packages I have been working with haven't worked with my data. The latest package I have been ...
2
votes
1answer
58 views

Efficient kernel estimation of density over grid of points

I was wondering whether it is possible to calculate density estimates for $n \times n$ matrix in $o(n^2)$ computational time. Imagine we have a bivariate random vector $(X,Y)$ with a joint density $f(...
0
votes
0answers
24 views

Finding proper cut off for filtering?

I had a set of data (149876 rows and 1200 columns)and I have found outliers points for each row. Now I want to determine a proper threshold for filtering my outliers. The distribution of my data is ...
1
vote
1answer
36 views

Random unit vectors in $\mathbb{C}^n$ and $\mathbb{R}^{2n}$

Suppose $\mathbf{u}\in\mathbb{C}^n$ is a complex random vector with circular symmetry, uniformly distributed on the unit complex $n$-sphere, so we have $\|\mathbf{u}\|=1$. In other words, $\mathbf{u}$ ...
0
votes
0answers
41 views

help with computing standard score for scarce discretely quantized data

I'm not a stats major and I'd appreciate any help I can get. I've got data with each point defined by a score (between 0-1) and a frequency (0-100%). At 100%, the score is most reliable, at 1% the ...
1
vote
0answers
176 views

calculating PMI for co-occurrences of words

I am in the process of building a question answering system. I am interested in calculating the PMI for words $x$ and $y$ occurring within 5 words of each other in a document. I have the formula and ...
0
votes
1answer
85 views

confusion matrix and Jaccard index computation in O(n) for cluster comparison

My problem: I have two clustering (or partitions) of a set of numbers $\{1,...,n\}$, each with $k$ cluster (or subsets), that i have to compare and, to do that, i'd like to try some different indexes ...