Problems focusing on (but not necessarily limited to) using Python for statistical computation.

learn more… | top users | synonyms

0
votes
0answers
14 views

How to add cluster centers to the already transformed arrays with T-SNE Scikit Learn?

let's get this scikit original code, which is basically the one I'm using. My X is 2000x100 and in order to plot the clusters (plot on the right) I want to transform it with with the TSNE algorithm ...
1
vote
1answer
35 views

Generate independent random values from a bivariate normal distribution

I am trying to independently select two sets of numbers (set 1 and set 2) from a bivariate normal distribution. I want the ...
-2
votes
0answers
11 views

Does theano support multi-machine version [on hold]

We use theano in the GPU in a single machine, I am just wondering if it supports multi-machine programming so that the speed will raise up?
2
votes
0answers
31 views

Finding idf only for text mining

We find tf-idf for training phase in text mining, however, in test phase, we need the tf for each element in test set, but should use idf in train set, so is there any api in python that can calculate ...
0
votes
0answers
20 views

Pandas generating features vector with apply()

I have a data set with two features: class (int) and content (text). Each row of content needs to be vectorized to a set of boolean features matching regexes. My vectorized function return a ...
1
vote
1answer
31 views

Binomial Distribution Where N is Generated by a Poisson Process (pymc)

I'm not sure if this is the best way to go about this, because I'm fairly new to Bayesian methods. I'm trying to model a process where the number of trials $n$ used in a binomial process is generated ...
0
votes
0answers
10 views

I inserted n variables into statsmodels.formula.api.ols and returned > n predictions?

This makes no sense to me? I have split the data into training and testing data with sklearn. ...
0
votes
0answers
12 views

What is the main difference between Numba Pro and Theano/pyautodiff for GPU calculations? [closed]

Both Numba Pro and pyautodiff based on Theano supports conversion of Python code into GPU machine code. Theano will also allow symbolic derivation of the resulted syntax tree, but this is outside the ...
0
votes
0answers
16 views

Gaze estimation, choosing algorithm and parameters

I am trying to build a program for estimating point of gaze on the computer screen from the x and y coordinates of the pupil centres from webcam video .(x and y coordinates correspond to pixel ...
0
votes
0answers
47 views

Did I use gradient descent correctly?

In an attempt to learn Gradient Descent. I've created my own dataset which is a the total bill of a meal with tips. As I created the perfect data so I make every meal with 10% tip. I purposely ...
2
votes
1answer
14 views

Comparing performance of a single classifier on two datasets

I'm using a single classifier on two different datasets, consisting of non-overlapping set of observations. The performance on one dataset is higher than the other, and I want to show that the ...
0
votes
1answer
21 views

Weighted re sampling why cdf not pdf or pmf?

From thinkstats2 the O'Reilly text. ...
0
votes
0answers
25 views

Information retrieval from strings using neural network

I am dealing with input strings that have max length of 200 characters, they are the USSD popup msgs and the SMS text strings on android smart phones extracted as and when they appear via an android ...
0
votes
2answers
68 views

Median of ratio of independent variates with Beta distributions

Let $X, Y$ be independent random variables where $X \sim Beta(\alpha_1,\beta_1)$, $Y \sim Beta(\alpha_2,\beta_2)$, and $Z = X/Y$. Recall $X, Y$ are supported on $(0,1)$, so $z > 0$. I've ...
0
votes
0answers
21 views

“without” not in NLTK Stopwords corpus, while “with” is

I will preface this by saying that I do not have experience in linguistics. Unnecessary Background: I was doing some NLP in python using NLTK, and began by removing stopwords from my text of ...
0
votes
0answers
7 views

Principant Component Analysis being too slow (MLPY Python) [migrated]

I am using the PCAFast method from the MLPY API in python (http://mlpy.sourceforge.net/docs/3.2/dim_red.html) The method is executed pretty fast when it learns a feature matrix generated as follows: ...
0
votes
1answer
76 views

Gradient descend algorithm ascending for learning rate

This is not an Homework assignment. I am trying to implement the gradient descend explained in the link (Page 5) [http://cs229.stanford.edu/notes/cs229-notes1.pdf][1].I am able to successfully run the ...
0
votes
0answers
15 views

Reimplement Binomial Test R-> Python

I am trying to re-implement a Binomial test in R that uses the alternative= "greater" optionin Python using scipy. Right now I am looking at ...
0
votes
1answer
48 views

Can a Precision-Recall curve or a ROC curve be horizontal?

I am working on a binary classification task on imbalanced data. Since the accuracy is not so meaningful in this case. I use Scikit-Learn to compute the Precision-Recall curve and ROC curve in order ...
0
votes
1answer
11 views

What's meaning of BOS and EOS in CRFSuite feature list and what is the role of them?

In NER(Named Entity Recognition) example in python-crf package website we see this function as feature generator: ...
0
votes
0answers
20 views

Dissecting the code of statistical algorithms for education

I was curious if anyone has made or seen any videos that dissect the code of statistical algorithms as a means of learning statistics and coding. Ideally the videos would explain step by step both the ...
1
vote
1answer
10 views

sklearn estimate_bandwidth in MeanShift algorithm with small dataset

How should I approach the estimate_bandwidth function with only very small datasets? My problem is that I am analyzing datasets that range from 500 entries to 2. ...
1
vote
1answer
17 views

Is it possible for test error to be lower than training error

Is it possible to have test error lower than training error? I have a classification problem with 2000 samples, 500 of which are positives, 1500 are negatives. I split my data into 70% training data, ...
0
votes
0answers
18 views

Multivariate discretization method / library for huge data

Does anyone know any multivariate discretization method that can be used for large amounts of data. A library / Python library would be awesome but algorithms would also do. Also I'm not sure if ...
1
vote
1answer
30 views

Granger Causality Test - Non-Stationary

I have two time series that are both non-stationary at level. The ADF test says they have a unit root. When applying the first difference of each time series, they are now stationary. I guess this is ...
0
votes
0answers
42 views

Testing for Granger Causality

I have two time series (Stocks and GDP) that I want to check for Granger causality. After reading the literature and documentations of various statistics software documentations (py statsmodels), I'm ...
1
vote
0answers
29 views

Temporal Difference Learning Not Converging

I'm trying to implement a $Q(\lambda)$ algorithm from this paper (warning: link is a download of a PDF) and can't seem to get it to find anything that close to the optimal policy. From what I can ...
1
vote
1answer
36 views

spurious regression/co-integration

I have two I(1) time series and I regressed one against the other and found that it had low to moderate R-squared but my DW statistic is about 0.015. I know the literature says this is the case of ...
1
vote
2answers
61 views

best tool for plotting 3 3D lines in one plot

I'm trying to recreate an image using either R, Matlab or Python (the languages I know) but I'm having troubles finding a good library for the task. The image I'd like to recreate is the one below. ...
0
votes
1answer
23 views

Confirming calculations with simulations?

This question may be a little abstract, but I would like to understand how to develop a mentality towards performing statistical simulations. For example: If I have a normal distribution, and I ...
2
votes
0answers
32 views

Spatial clustering based on response

Statistics version: I have a few measurements of a function that takes three inputs and produces a few 2D fields of outputs: f(a,b,c;x,y), with f being a vector of several quantities. I would like to ...
1
vote
1answer
59 views

Implementing bayesian networks in python for gaze estimation using visual saliency

I am developing an appearance based gaze estimation system based on opencv and python. I have currently developed a prototype which can estimate the gaze based on active calibration, which is ...
1
vote
2answers
529 views

Using RNN (LSTM) for predicting the timeseries vectors (Theano)

I have very simple problem but I cannot find a right tool to solve it. I have some sequence of vectors of the same length. Now I would like to train LSTM RNN on train sample of these sequences and ...
1
vote
0answers
40 views

Does feature size affect polynomial regression?

(I'm still trying to learn all this, sorry for any wrong terms or mistakes I might have made in this question) By feature size, I mean the value of the numbers. For example, let's say I have input ...
1
vote
0answers
13 views

Can I use an unknown number of variables to model my time-series?

I have a bunch of data-sets showing the relationship between two observables, "force" and "time". See example plot You see the regularity of the features: There is a region of linearly increasing ...
0
votes
0answers
42 views

Granger Causality / VAR / Negative Correlations

I have two questions on Granger causality. I feel puzzled after reading a dozen of papers on the topic and it appears to me that I need to clarify my understanding of Granger causality. Question 1 : ...
1
vote
1answer
27 views

Identifying filtered features after feature selection with scikit learn

Here is my Code for feature selection method in Python: ...
1
vote
2answers
62 views

How do I use Lasso and elastic net as feature selectors?

I have a data set with 900,000 rows and 8 features. I want to look at the significance of each feature so that I can evaluate whether the features I add are viable or not. One method I am using after ...
0
votes
1answer
28 views

How do I improve the accuracy of my supervised document classification model? [closed]

Given 1000 legal judgement documents, 900 of which are labeled, my task is to predict the label for the remaining 100 documents. The labeled documents belong to 41 different categories of Law, with ...
0
votes
0answers
46 views

Why can't I reconstruct parameters of a synthetic data set?

The following Python function creates synthetic binary labeled data that is supposed to perfectly follow the logistic regression model: ...
0
votes
0answers
27 views

Uplift model with a continuous outcome?

Does anyone know any good packages (preferably in R/python) or references that are specifically about building the uplift model with a "continuous" outcome? I've used the upliftRF from R and made it ...
0
votes
0answers
29 views

CSV file with several values, get median in groups of ten [closed]

I have a csv file with 4000 lines and 270 columns. So I want to read each column and take the median of groups of 10 elements. How do I do this in Python on a fast way, taking into account the number ...
0
votes
0answers
25 views

Sensitivity to scaling of multivariate data with HMM

I have some multivariate data, say 40 features. Some features are scaled between 0 and 1, and some are scaled between 0 and 1e8. For reference, I am using sci-kit learn's HMM implementation (yes, I ...
2
votes
1answer
58 views

How “interesting” a data series is

I have a large dataset containing several objects. Each object has many attributes which is arranged in a time series. Is there a suggested method to find the top n "most interesting" attributes? The ...
2
votes
0answers
62 views

Julia: Taking stock of how it has been doing

I came across a 2012 question that had a very good discussion about Julia as an alternative to R / Python for various types of Statistical Work. Here lies the original Question from 2012 about ...
3
votes
2answers
106 views

Sampling from von Mises-Fisher distribution in Python?

I am looking for a simple way to sample from a multivariate von Mises-Fisher distribution in Python. I have looked in the stats module in scipy and the numpy module but only found the univariate von ...
0
votes
0answers
26 views

Statsmodels: What can cause LinAlg error?

I am trying to do a mixed linear model for a dataset I am helping a colleague analyze, as in an earlier question I had posted, GLMs were unable to handle random effects. With the appropriate imports ...
0
votes
0answers
38 views

Statsmodels: Incorporate random effects vs. fixed effects

In JMP Pro, it is possible to include certain columns of predictor variables as "random effects" and others as "fixed effects". How do we do the same in ...
1
vote
0answers
44 views

Calculating Mutual Information for feature selection

In order to determine the importance of some individual features coming from labelled time series, I am trying to calculate the Mutual Information (as showed in "Who do you sync you are?: smartphone ...
1
vote
0answers
22 views

Scipy.stats.anderson_ksamp negative return values for test statistic

Ok, so I've been trying to run this test on the the iris dataset to see if it flags the clusters within the data as samples that aren't from the same population. from sklearn import datasets iris = ...