Refers to the interface of statistics and computing; the use of algorithms and software for statistical purposes.

learn more… | top users | synonyms

-2
votes
0answers
51 views

What are common ill-conditioned problems/mistakes in linear regression? [on hold]

Many problems are ill-conditioned and cannot be reliably solved using double precision computer systems. Here is what I know that could happen in linear model, that the problem is ill-conditioned and ...
6
votes
1answer
109 views

How do iterative methods for solving maximizing likelihood problems work?

Does anyone know about the computational iteration processes for maximum likelihood estimation? If these set of equations cannot be solved practically then how does the computer solve them?
0
votes
0answers
19 views

Numerical optimization of Monte-Carlo simulated moments

I hope not to post an off-topic related question here as it is related to programming, however, I am faced to a specific problem which I 1) tackle in a statistically oriented language and 2) which I ...
1
vote
1answer
34 views

computation of square root of matrix

I am studying the Canonical correlation analysis(CCA) the formula https://en.wikipedia.org/wiki/Canonical_correlation is involving the -1/2 degree of a matrix. My question is: how can i do the ...
2
votes
1answer
29 views

Since we use Bootsrap to approximate the SE, can we use Bootstrap to find prediction errors?

Instead of using Cross-validation, or K-fold Cross-validation, can we use Bootstrap to generate random samples and use one of them as test set, and others as training set?
1
vote
1answer
39 views

how to find similarity based on only one column value

I am not sure if it is possible and that is why I am asking the question here. I have a data looks like below ...
0
votes
1answer
36 views

Random Forest explanation

I am having trouble understanding Random Forest, especially some terms. What is a node what is node size? What are ...
0
votes
0answers
14 views

Unbalanced classification problem: Can more data be more harmful?

In a two-class problem, I have 1000 raw data to train my model, which 900 belongs to class 1 and 100 belongs to class 2. If I add another 500 data into the pool but they all belong to class 1, will ...
1
vote
0answers
23 views

Negative estimated values [closed]

I'm using Weka API to make a prediction with SVR. My dataset contains 499 records. In the outpout of the program, I get some negative predicted values. Is that due to problem of Overfitting? If yes, ...
5
votes
5answers
147 views

What is the best way to detect repetition in xyz data for purposes of splitting data?

I'll use this picture to explain What I want to do is define some patterns as trained patterns. Then given data I want to be able to determine if the pattern exists in the dataset, and if it does ...
2
votes
2answers
302 views

The value of an Effect Size

I calculated a Cohen's d value of d= -2.1. I understand that there are small, medium, and large effect sizes. But in my case the d value is negative? Would it still be considered large since abs(-2.1)...
2
votes
1answer
29 views

What is your configuration to do statistical analysis on large data sets?

The stat packages which I've been using are not very well suited for working with large data sets, i.e. more than 1 billion records. They all tend to try loading all into memory, and install one one ...
2
votes
1answer
28 views

Is it possible to add up the accuracy rates of 2 predictors?

Weather channel 1 has a 65% accuracy rate of predicting tomorrows weather Weather channel 2 has a 59% accuracy rate of predicting tomorrows weather Is it somehow possible to take into account ...
0
votes
0answers
12 views

How to measure the impact of a product on the global revenue?

We have products daily revenue and when each product started being sold by the company. We also have different break-down of product revenue by location, store, sales person, etc. We need to measure ...
0
votes
1answer
55 views

Does it make sense to cluster based on Euclidean distances between rows of a cosine matrix?

I calculated the Cosine distances for binary data and got the relations between different variables. I need to cluster them. I tried passing the cosine matrix directly to the (clustering) function ...
0
votes
0answers
33 views

How to compute correlation for two data sets?

I am a CS student and naive in statistics. I have 2 related data sets, to which I want to find correlation. Let me explain the case first: This is a question related to computer architecture. One ...
4
votes
1answer
60 views

Gibbs sampling and conditional distribution

I need to simulate the posterior distribution of intraclass correlation coefficient $\pi(\rho|y)$ where $y$ is the data set and $\rho=\frac{\sigma_a^2}{\sigma_a^2+\sigma_e^2}$ with $\sigma^2_a\sim IG(\...
0
votes
0answers
23 views

Appropriate testing for outliers [on hold]

I want to perform an outlier-test for my data. The distribution of the data looks like a hyperbola (there are many values near 0, all values are >0). Sample size of data is 5000. I want to find an ...
12
votes
2answers
503 views

How do ABC and MCMC differ in their applications?

To my understanding Approximate Bayesian Computation (ABC) and Markov Chain Monte Carlo (MCMC) have very similar aims. Below I describe my understanding of these methods and how I perceive the ...
0
votes
0answers
21 views

Distributed PCA

I have a large data set. Large means many instances (~50000000) and many features (~25000). I will call my data matrix X where the rows are instances and columns are features. Lets say the number of ...
3
votes
4answers
287 views

Why is gradient descent required?

When we can differentiate the cost function and find parameters by solving equations obtained through partial differentiation with respect to every parameter and find out where the cost function is ...
1
vote
0answers
36 views

How does one infer the noise/error model given measurements?

What resources are available for applying inference/computational statistics to infer the underlying error/noise model, given X measurements from some apparatus? (Below, I am mostly referring to ...
0
votes
1answer
38 views

What's up with this variance computation?

I'm trying to re-implement the Ckmeans.1D.DP algorithm in Python. I have the actual dynamic optimization part down, but I'm a little confused by the BIC computation they use for selecting the number ...
0
votes
0answers
11 views

finding and comparing temporal and location-specific pass patterns in a soccer game

I have data on several soccer matches, where ball passes have been recorded in terms of XY-location where the pass starts, XY location where the pass ends, the team making the pass, the player making ...
0
votes
0answers
14 views

Topics needed to learn to for predicting future of a data set?

I am planning to work this summer by writing a code which will eventually become an app, that has the ability to predict what the prices will be atleast 6 month ahead of time. I have a data set of the ...
1
vote
0answers
65 views

Server Requirements for machine learning on 300 million row database [closed]

I am trying to build a logistic regression model using Revolution R 64 bit. My data is 300 million rows by 12 columns. The data is stored on AWS Redshift, but takes about 6 hrs to import and save as ...
0
votes
0answers
14 views

How to find the generalized inverse of a matrix in SageMath?

Given a matrix $A_{m\times n}$, a matrix $G_{n\times m}$ is said to be a generalized inverse of $A$, if it satisfies, $$AGA=A$$ Now, since for a matrix $A$, generalized inverse of $A$ is plenty if ...
0
votes
0answers
18 views

Sampling from Irwin-Hall distribution using triangular distribution [on hold]

So I need to sample from the Irwin-Hall distribution using rejection sampling with the triangular distribution. I built 2 functions: The first is d_irwin which receives an $x\in supp(g)$ and the n we ...
1
vote
0answers
28 views

Is it ok to ignore data in this instance?

I'm running a program in python I wrote to look at the first spiking time of a neuron in a stochastic network. I'm running about 2500 realizations for each set of parameter values to calculate the ...
1
vote
0answers
38 views

Sampling from Irwin-Hall (Uniform Sum) distribution using rejection sampling

Irwin-Hall Distribution is a probability distribution for a random variable defined as the sum of a number of independent random variables,each having a uniform distribution True or False: The ...
9
votes
2answers
257 views

Inverse covariance matrix vs covariance matrix in PCA

In PCA, does it make a difference if we pick principal components of the inverse covariance matrix OR if we drop eigenvectors of the covariance matrix corresponding to large eigenvalues? This is ...
0
votes
0answers
19 views

Robustness criterion (1978)

I am referring to bradley's journal relating to the robustness criterion. According to his liberal criterion, the simulated type 1 error rate have to between 0.5 alpha < p < 1.5 alpha. since i ...
1
vote
0answers
25 views

Binomial vs inverse binomial transform in obtaining moments

I am trying to obtain raw co-moment matrices from central co-moment matrices and I have found formulas for the transition from central to raw moments here . I have also found the opposite, that is ...
0
votes
0answers
57 views

How to compute the conditional probability distribution with given joint distributions and marginal distribution?

I am trying to compute the conditional distribution p(a|c) with given equation consists of a set of marginal distribution and joint distribution. Variables A,B,C have Markov property A->B->C. $p(a|c) ...
0
votes
1answer
16 views

What tests to analyze the following hypotheses with nominal data?

I have a set of questionnaire data to analyze the working culture for respondents. (There are four existing cultures; a,b,c and d) Variables including Gender, Organization, Public/Private Sector, ...
0
votes
0answers
19 views

Spark big matrix computation

I am trying to see if Spark is fit for my problem. I would like to use kernel methods, as presented in Kernel method. For example, kernel k-means. One important feature of the kernel methods is that ...
1
vote
0answers
43 views

Is this simple paired-sample permutation test valid?

I have about 10 pairs of scores, $(x_1, y_1),~ ...,~(x_{10}, y_{10})$, with all $x_i$ and $y_i$ being between 0.0 and 1.0. I'm interested in testing whether the mean difference over pairs is ...
3
votes
2answers
48 views

Cluster real numbers

I have a set of precise measurements, and what I want to do is count the frequency (how many time it appears) for each value. The problem is that these are very precise measurements and with a naive ...
0
votes
1answer
33 views

value of discrete items when only have their value when sold in groups

I'm doing an analysis for my job and would greatly appreciate any help you can offer! My company sells pallets full of items. Each item has a category (1 category per item, many items per category) ...
0
votes
0answers
55 views

predict mechanical component failure using machine learning

I'm going to do a research project on the possibility to predict mechanical component failure within multidisciplinary system, using sensors data. different sensors are being sampled in different ...
4
votes
1answer
72 views

Posterior distribution of parameter

I am looking at pg. 257 of the textbook "Statistical computing with R" by Maria L. Rizzo, where an example is given of a simple model of stock returns: We have 5 stocks, and at the end of a year of ...
0
votes
0answers
16 views

Best possible statistical approach to compare n pair of variables along a set of genomic intervals

I am dealing a bioinformatics problem and need help in approaching it in the right direction. I have a set of variables that represent different kinds of genomic data and are real valued. I am trying ...
0
votes
0answers
11 views

Find high occurring low values

In numerical data, where outlier is defined not by infrequency but value (this is a real-world data-set where low values would be indicative of noise). Although this is real world data, there is no ...
0
votes
0answers
10 views

creating tidy data sets collaboratively

I am wondering if there is a body of knowledge out there that describes how to best produce tidy data sets in a setting where a lot of people contribute to creating different pieces of the data. So ...
0
votes
0answers
18 views

Testing Random Number Generators(RNG)

There are numerous tests of RNGs. Consider that we tested RNG with lots of different seeds and the output is as follows: with certain seeds the RNG passes the test, with other seeds RNG doesn't pass. ...
0
votes
0answers
14 views

Figuring the algorithm / common calculation on a set

I have a column that is size in MM and another that reflects the change in price for said size. I'm trying to figure out the algorithm/common calculation behind these generated numbers so that I would ...
6
votes
1answer
77 views

Algebraic classifiers, more information?

I have read Algebraic classifiers: a generic approach to fast cross-validation, online training, and parallel training and was amazed by the performance of the derived algorithms. However, it seems ...
0
votes
1answer
31 views

Interpretaion of Logistic Coefficients

All, I ran a logistic Regression on a set of variables both categorical and continuous with a binary event as dependent variable. Now post modelling, I observe a set of categorical variables showing ...
0
votes
0answers
35 views

Calculate the non centrality parameter (ncp) for non central chi squared distribution

I have the 0.975-quantile value (0.191) and the degree of freedom (k=1). How can i calculate the non centrality parameter ? in R if possible. Q0.975(k=1, ncp= ??) = 0.191
1
vote
1answer
60 views

What does make SVM a “soft computing” method?

Soft computing is defined in [1] by the capability of "operating with uncertain, imprecise and incomplete information in a manner that reflects human thinking". So, based on my limited understanding, ...