Problems focusing on (but not necessarily limited to) using Python for statistical computation.

learn more… | top users | synonyms

0
votes
0answers
6 views

How to add regression functions in python, or create a new regression function from given coefficients? [migrated]

I have one regression function, g1(x) = 5x - 1 for one data point. I have another regression function, g2(x) = 3x + 4. I want ...
0
votes
0answers
5 views

Trend estimation for participation?

I have logs for users and posts in a blog platform for 3 years. I can easily find out how many posts each user have made per day/month/year/etc. What I want to find out is if the frequency of posts ...
0
votes
1answer
20 views

Similarity measure between numbers

I have a huge matrix (10*10k). I'd like to know if there is a way to find similarities between lines. Let's give an example of matrix: 4*5 ...
0
votes
1answer
14 views

Converting SAS EWMA code to Python

I have the following SAS code that uses PROC FORECAST that I would like to replicated with the Python Pandas pandas.stats.moments.ewma ...
0
votes
0answers
32 views

Binomial proportion confidence interval

I have a sample and I've classified each member into 3 groups A,B,C. To get uncertainties on the percentages of these groups, I've been told I should use the wilson score interval. As someone who is ...
1
vote
0answers
20 views

Given 3 discriminant functions I was able to classify 2D patterns, but how do I plot decision boundaries via matplotlib?

I have implemented the discriminant function and was able to classify the 2D patterns (via Python), but I have troubles thinking about an approach to plot the decision boundaries. Hope anyone has an ...
1
vote
0answers
29 views

Pybrain trainer runtime error [closed]

I am trying to use pybrain for the first time on 20x20 handwritten digit MNIST data. I keep getting a runtime error when I setup the trainer and I can;t figure out what the problem is? code: ...
1
vote
0answers
30 views

Detecting time-shifted time series

Assume there are two time series of real values. How to test hypothesis, that these series are quite the same, but there is a time shift between them? Sorry, if the question is too basic. Thanks in ...
0
votes
1answer
25 views

Result from Step by step PCA implementation differs from `matplotlib.mlab.PCA()`, would be nice if someone can help me finding the source

I was reading this nice article and tried to implement this step by step guide in Python, and then I compared the results using the Python function from the matplotlib library: ...
1
vote
2answers
44 views

Should statsmodels's GLM produce the same results as R's lm?

Should Python's statsmodels.api.GLM(train_y, train_X, family=sm.families.Binomial()).fit().predict(test_X) always produce the same results as R's ...
1
vote
3answers
57 views

What classification method to use

I have two sets of data from two experiments. All data can be divided in three classes, e.g.: $$\begin{array}{cccc}\rm{Parameter}1&\rm{Parameter}2 ...&\rm{Parameter}N&\rm{Class}\\\hline ...
1
vote
1answer
30 views

Correlation of many 1d vectors

Let's say I have many 1 dimensional vectors of the same dimensions which I want to compare to each other. What I'm really looking for is to check whether they are all very similar, or not, and to ...
1
vote
1answer
30 views

Normalized Variation of Information (NVI) visualization of distances to several references

I am using NVI (Normalized Variation of Information) to measure the quality of clustering obtained ( a family of them) compared to the Ground Truth and also to the result obtained with another method. ...
1
vote
0answers
72 views

Using BIC to estimate the number of k in KMEANS

I am currently trying to compute the BIC for my toy data set (ofc iris (: ). I want to reproduce the results as shown here (Fig. 5). That paper is also my source for the BIC formulas. I have 2 ...
0
votes
0answers
19 views

Statistics probability with python [migrated]

Let X1,X2,X3 be independent random variables, each of which has the exponential distribution with parameter λ=2. Use random simulation to estimate the probability that X1+X2+X3≥1. I was told to use ...
2
votes
0answers
34 views

Variational inference engines

After doing some research on the topic, I have noticed a surprising deficit of inference packages and libraries that rely on message-passing or optimization methods for Python and R. To the best of ...
0
votes
0answers
37 views

Significant difference between two correlation matrices

We have two huge correlation matrices at different experimental conditions. If we want to identify the significant differences between these matrices , what would be the ideal method. I have ...
1
vote
0answers
55 views

Logistic regression with binary inputs

Is logistic regression an appropriate classifier when the input data are binary? Say we are conducting an experiment where the subject is presented with blue and green circles of varying shades, and ...
0
votes
1answer
20 views

Neural Networks output range in simulation

I am learning some model based on examples ${((x_{i1},x_{i2},....,x_{ip}),y_i)}_{i=1...N}$ using a neural network of Feed Forward Multilayer Perceptron (newff) (using python library neurolab). I ...
0
votes
0answers
21 views

building a feature set for scikit learn

Im using RandomForestClassifier for a probability prediction task. I have a featureset of around 50 features and two possible labels - ...
1
vote
1answer
36 views

How to initiate bias node in a restricted Boltzman machine

I am new to Neural Networks and trying to implement RBM. I am stuck on initializing the visible layer's bias value. Is it supposed to initialize to some random number or there is some probabilistic ...
2
votes
2answers
107 views

How to compute the standard errors of a logistic regression's coefficients

I am using Python's scikit-learn to train and test a logistic regression. scikit-learn returns the regression's coefficients of the independent variables, but it does not provide the coefficients' ...
2
votes
0answers
102 views

How would you visualize a segmented funnel? (and could you do it with Python?)

I saw this post on Moz which presented a segmented marketing funnel: This kind of thing would have quite a bit of value in my job. What I have no idea is how to visualize raw data to show a ...
0
votes
0answers
14 views

Implementation of prototype selection in application to nearest neighbor classifier

I am looking for an implementation of prototype selection algorithm and it is highly preferable that implementation would be in Python. I am using sikit-learn, but there is none.
-2
votes
1answer
80 views

Document Similarity Gensim

I am trying to get related documents for a list of 10,000 documents from the same set of 10,000 docs. I am using two algorithms for testing: gensim lsi and gensim similarity. Both give terrible ...
4
votes
2answers
145 views

I have my data set, now what?

I have a basic understanding of basic statistics, but I believe I've gotten myself out of my depth. I have a data set with a dependent variable (time span) and three quantitative independent ...
0
votes
1answer
66 views

Computation of a binomial probability: nan result

I am computing binomial probabilities for many parameters. I am using the library scipy.stats: b = scipy.stats.binom(n,p) probability = b.pmf(k) I have noticed ...
0
votes
1answer
114 views

Multidimensional scaling using Python

I have 6,000 points for which I have all pairwise distances in a distance matrix. I want to get an idea whether these data were generated by a mixture of Gaussian distributions so I'm trying to get a ...
0
votes
1answer
155 views

Beginner - How can I use ranked values in my Logistic Regression?

I am running a Logistic Regression on some data to predict if a webpage is "good" or "bad". I got the dataset from a finished Kaggle competiton here (train.tsv). I extract the second column of this ...
3
votes
0answers
46 views

Asymmetric confidence intervals on bootstrap estimates

I've performed bootstrapping on my leastsq parameters and now I have a load of data from which I can get the mean and standard deviation for each parameter. Lovely. But when I look at a histogram of ...
0
votes
1answer
45 views

lhs Unit Eigenvectors in Python for DeGroot model Convergence for Social Networking problems

The influence array at time 0 is: 1/3, 1/3 1/3 1/2, 1/2, 0 0, 0.25, 0.75 It is suppose to converge to the array: ...
0
votes
0answers
60 views

Improving Python Exponential smoothing

I am going to improve my code to the Exponential smoothing I submitted to Statsmodel which can be found here. The code handles 15 different variation Standard Exponential Smoothing models including ...
0
votes
0answers
30 views

How to interpret probability plot?

I was reading about how to find the distribution of data from this question I generated the following probability plot from the scipy package.scipy prob plot Can someone please help me figure out ...
0
votes
1answer
46 views

Trouble understanding two scale graph

I am trying to plot a graph which has two y-axes and one x-axis. Consider three variables A,B,C. My first data set G1, has values of A and B. While getting this dataset in the script I kept C ...
2
votes
1answer
99 views

What's so Poisson about a Poisson Point Process? (or, can I generate one using random ordered pairs?)

I know there is an R spatstat function to generate a ppp (Poisson Point Process), but I'm working in python, and I am not clear what spatstat.ppp is doing behind the scenes. If I generate a an ...
1
vote
1answer
60 views

Regression test coefficient on variable is greater than coefficient on interaction term

With a model such as: $y \approx B_0 + B_1\cdot \log(x) + B_i\cdot \log(x):\text{group}_i +B_j\cdot group_i$, where group can take on several values ($i = 2$ to $15$, let's say): In an OLS ...
0
votes
0answers
27 views

Why does this simulation of prediction accuracy and volatility returns weird results?

This is a simulation of the effects of prediction accuracy and volatility but it's results show that accuracy jumps drastically around a change in decimal points for the volatility. The output shows ...
1
vote
0answers
41 views

Heuristic Feature Selection for Gradient Boosting

I originally posted on stackoverflow and was told to move it here: If I am trying to select from two different sets of features for a Gradient Boosting Machine but I do not want to run through ...
1
vote
1answer
43 views

Different HC3 standard error estimates when normalising weights for weighted least squares fit using Python statsmodels

When I normalize the weights I use for fitting a line with weighted least squares, the parameters of the fitted line and the 'normal' standard errors stay exactly the same, as I would expect. The HC3 ...
3
votes
1answer
87 views

Generate two variables with precise pre-specified correlation

I want to generate two variables with (pseudo-) random numbers with an exact pearson's r. How do I do that? Python and/or R solutions would be nice! I am able to generate random data that ...
0
votes
3answers
206 views

Regression layer in convolutional neural network

I am trying to do regression with a deep convolutional network. I am using the code from http://deeplearning.net/tutorial/lenet.html Currently, the network is using logistic regression as its last ...
0
votes
1answer
36 views

error increasing with no of estimators in adaboost

My error gets increased when i increase the n_estimators value in ...
1
vote
0answers
70 views

MLE for bivariate distribution

I am trying to estimate parameters defining the rate of a Poisson bivariate distributions. (...following Maher's 1982 paper "Modelling association football scores"). Basically I have random variables ...
0
votes
0answers
338 views

Principal Component Analysis and Regression in Python

I'm trying to figure out how to reproduce in Python some work that I've done in SAS. Using this dataset, where multicollinearity is a problem, I would like to perform principal component analysis in ...
0
votes
1answer
76 views

SSB - Sum of squares between clusters

I got a little confused with the squares and the sums. As far as I know, the variance or total sum of squares (TSS) is smth like $\sum_{i}^{n} (x_i - \bar x)^2$ and the sum of squares within (SSW) ...
4
votes
0answers
44 views

How do I model the probability of two empirical distributions?

I have two distributions: A, and B. Each distribution is filled with the numbers 1.0-10.0. These distributions are NOT simple functions, like the gaussian, but are merely empirical counts. ...
1
vote
1answer
66 views

Trying to determine the similarity of the distributions of labels

I asked this question on stackoverflow first since it is more of an implementation question, but I got no answers yet, therefore I'm asking it here again. I have a dataset which is annotated by 2 ...
3
votes
2answers
134 views

Is it possible to train a one-class SVM to have zero training error?

I'm trying to work on an anomaly detection problem, so I am currently exploring my options on which algorithm is best to use for me. I've been looking at the one-class SVM in the scikit-learn library ...
2
votes
2answers
149 views

Exponential smoothing models backcasting and determining initial values python

I have made python code for exponential smoothing (ES) that takes in about 15 different cases including: Simple Exponential Smoothing (SES) Simple Seasonal models (both multiplicative and additive) ...
5
votes
2answers
114 views

Is it possible to do a test of significance for a string occurrence in two datasets

I have two sets of data compiled from different sources. Both sets of data contain words with frequencies of occurrence. I would like to check if a certain word exists in both datasets and if it does, ...