Problems focusing on (but not necessarily limited to) using Python for statistical computation.

learn more… | top users | synonyms

0
votes
1answer
11 views

Sparse Collaborative Filtering

Does anyone know of any Python code examples for sparse collaborative filtering. Everything I can find revolves around using prebuilt packages (e.g. Mahout, GraphLab), but I'm learning to learn the ...
0
votes
0answers
8 views

Define feature for text classification using NLTK [on hold]

I'm working on Aspect Based sentiment analysis , I have a training set (text ,and aspectTerms) for each review. Using NLTK , I wan to build a NaiveBays Classifier that predict aspects of test unseen ...
4
votes
1answer
35 views

Understanding multiple KS tests

I have read these two questions Why are p-values uniformly distributed under the null hypothesis? and Understanding scipy Kolmogorov-Smirnov test And this inspired me the following experiment. I ...
1
vote
0answers
21 views

Johansen Test in python

How to get the trace statistics from the Johansen test for cointegration in python. Also, i tried to search it myself on google, found the following website ...
0
votes
0answers
11 views

Multiple chains in PyMC - problem with repeatability

I have seen related questions on the problems with re-seeding in PyMC (e.g. pymc random seed doesn't guarantee the same posterior samples?), but the answers I have seen do not fully address the ...
0
votes
0answers
12 views

pybrain NN for classification

I have created my own NN to identify properly handwritten letters from a bunch of records. I used ClassificationDataSet with data downaloded from ...
1
vote
1answer
10 views

Recommendation Engine With Physical Distance Cutoff

I am looking to develop a recommendation engine for local stores to users. There are approximately 1 million stores in the database and around 1 million users. The 1Mx1M matrix for a user-based ...
0
votes
0answers
19 views

One-class SVM vs NN with backprop… Or is there something better?

I'm pretty new to unary classification, so I've been playing around with different approaches to one-class document classification in Python. NN seemed promising at first, but has some undesirable ...
0
votes
1answer
50 views

Gradient of Log-Likelihood

Considering the following functions I'm having a tough time finding the appropriate gradient function for the log-likelihood as defined below: $a_k(x)=\sum_{i=1}^D w_{ki}\cdot x_i$ $P(y_k|x) = ...
1
vote
0answers
52 views

Ordinary least squares regression giving wrong prediction [migrated]

I am using statsmodels OLS to fit a series of points to a line: ...
0
votes
1answer
42 views

Probabilistic classifier useful for both numeric and nominal features

First, I should say I'm beginner in data mining and use Matlab. I have a huge dataset: approximately 40 millions observation. It consists of 24 features including numeric and nominal values like ...
0
votes
0answers
5 views

Passing a symbolic theano.tensor into a compiled theano.function [migrated]

I'm trying to refactor my code to be easier to change architectures easily. Currently, I'm constructing a recurrent neural network as follows. ...
0
votes
0answers
23 views

How can i do time series forecasting with missing data

I am relatively new to time series forecasting, I have worked previously with continuous data at regular intervals successfully, Now I have a data set with missing values, for example look at the ...
0
votes
0answers
29 views

The mean and standard deviation of an Kernel Density Estimation

[I have crossed posted this from StackOverflow. Not sure if this is allowed, but maybe there is a Python answer, or maybe there is a good stats answer] I have fitted a ...
2
votes
0answers
28 views

Random Forest Probabilistic Prediction vs majority vote

Scikit learn seems to use probabilistic prediction instead of majority vote for the model aggregation technique without an explanation as to why (1.9.2.1. Random Forests). Is there a clear ...
1
vote
2answers
40 views

How to find if a difference is statistically significant?

I have a web crawler that scrapes items on an hourly basis and expect to get 375 items every time. In the past 10 runs, I have scraped the following number of items: [360, 361, 363, 364, 368, 372, ...
1
vote
1answer
37 views

Visualize difference between 2 classifiers

I trained 2 binary classifiers with the same data (a Decision Tree and a Random Forest). They both made a prediction on the same test data. Now, I want to visualize the difference in classification ...
2
votes
1answer
49 views

How do I force the L-BFGS-B to not stop early? Projected gradient is zero

I'm trying to use the SciPy implementation of the fmin_l_bfgs_b algorithm using the following code: ...
0
votes
1answer
47 views

How to OLS Regress Y on 1/x denominator in R or Python

I have a problem where my general equation is $Y = C + 1/(\beta x)$, where $C$ is a constant. I want to find a b in OLS fashion to minimize RSS I have already transformed my equation thus far: $Y - ...
0
votes
0answers
21 views

Implementing autoregressive model on financial returns using sklearn

I have a model where I'm trying to regress S&P quarterly returns on lagged economic indicators using sklearn. I've read a lot about sklearn but this will be my first time implementing it. I ...
1
vote
1answer
110 views

I would like to analyze the data using R and Python. How to handle 10 millions records of data? [closed]

I would like to analyze the data using R and Python, and I am unable to open text file due to huge number of records. Is there any free database software to install in PC and load the data into it and ...
1
vote
1answer
59 views

Classification score for Random Forest

I'm learning about the Decision Tree and Random Forests. But there is something I don't really understand. I have a training set and a cross-validation set. I need to train different Random Forests, ...
0
votes
0answers
16 views

Trend analysis on time intervales [on hold]

I am trying to do trend analysis on tweets. I am still confused. The classical model is based on computing the frequency of items, to say that the item with big number of frequency is the trending ...
0
votes
1answer
29 views

How to find optimal penaltyparameter C for SVM (regression)

I am training an svm regressor using python sklearn.svm.SVR From the example given on the sklearn website, the above line of code defines my svm. ...
0
votes
0answers
15 views

Python “feature_importances” for most important factors

I'm a little unsure as whether this belongs in stackoverflow or cross validated. I have found a few posts on this topic , but I have not found the following question. Is it accurate to run the feature ...
0
votes
0answers
41 views

lowess: R (& python-statsmodels) vs MATLAB (& biopython)

There seems to be two different interpretations of what LOWESS really means: one from R (also used by python-statsmodels), and one from MATLAB (also used by biopython), see comparison below. Could ...
2
votes
2answers
126 views

Reproduce Cross Correlation results in Python

I'm trying and failing to reproduce the results found in this paper, where the cross correlation between two distributions is calculated: ...
0
votes
0answers
7 views

Is there a way to get the p-values/standard errors via a function in scikit learn?

Can't seem to figure this out very easily. I am looking for a similar output as that of statsmodels; however, statsmodels doesn't support some of the regressions that I want to do. I basically want ...
0
votes
0answers
22 views

Regression Analysis with String values in column of DataFrame

I have a dataset which is having multiple columns where some column has string values with comma seperated ...
0
votes
1answer
16 views

Scikit-learn model representation

Suppose, i have some models created and fitted in scikit-learn. ...
0
votes
0answers
23 views

NLP- Sentiment analysis [on hold]

Most of solutions are based on training classifier to detect the sentiment of text. I am trying to analyze political reviews, I don't wanna divide my data on train/Test. What is the approach that ...
0
votes
0answers
9 views

Obtain the graph of the autocorrelation function in ARIMA models [migrated]

I am implementing an ARIMA model in Python for forecasting U.S. GDP. I am interested in obtaining the graph for the autocorrelation function. I obtained the values for ACF but I can not see the ...
0
votes
0answers
16 views

Convert time scalar field to velocity vector field

I am trying to figure out if it's possible to create vector velocity field from some raster containing spatial distribution of scalar variable (timestamp of some mass-media event in my case). Here is ...
0
votes
0answers
31 views

Using PyBrain after training a network

I'm using PyBrain to create a neural network. I'm still pretty new to neural networks and their concepts. I've so far only run train() over the network, as ...
0
votes
2answers
91 views

two samples z test in Python [closed]

Does Scipy have a z test to compare the mean of two samples? I searched this page but couldn't find one.
0
votes
0answers
43 views

Random Forest Regressor - Incorporating Sample Weights in scikit-learn

I am running randomforestregressor in python. The target variable I am modeling is the frequency of an event occurring per unit of time. Each record in my data includes whether or not the event ...
0
votes
2answers
88 views

Predict interesting articles: increase accuracy

I'm trying to write a gui to display articles, and predict which articles I could like, based on the articles I previously liked. This post is the continuation of this one: ...
0
votes
0answers
28 views

How to fit an equation to a regression (and get coefficients) using scikit or statsmodels?

This may be simple but I can't find out how to do this. Suppose I have the following equation: f = a*(1+e^(c*x))^(-1) + by + d f, x, y, and d are vectors. I want to get estimates of the ...
0
votes
0answers
18 views

How do I implement custom sparse connections in a neural network?

We'd like to implement neural network which is not fully connected - we want to explicitly set which output neurons connect to which input neurons (there's no hidden layer). We use Theano but we can ...
1
vote
1answer
84 views

Simple model selection example in PYMC

I am currently experimenting with PYMC and I am trying out a simple example so that I start learning how things work (I am also a Python beginner, but an experienced machine learner). I have set ...
1
vote
1answer
77 views

What are some useful robust and scalable approaches towards anomaly detection of a time series data?

What are some useful robust and scalable approaches/methods towards anomaly detection of a time series data? I am mainly looking for some practical approaches carried out using Python, R, Java, etc. ...
0
votes
0answers
53 views

How to plot a precision-recall curve when doing cross-validation?

I'm using cross-validation to evaluate the performance of a classifier with scikit-learn and I want to plot the Precision-Recall curve. I found an example on ...
0
votes
0answers
14 views

Borgatti Key Player Problem (KPP) in Python

Has anyone come across any Python implementation of Borgatti's proposed Key Player Problem (KPP) algorithms? I'm interested in solutions using NetworkX and particularly interested in implementations ...
0
votes
0answers
36 views

Given a covariance matrix from a Linear regression, how do I calculate the standard error of the coefficients?

I have an OLS with autocorrelation in the residuals. I'm using python statsmodels, and found that there is the sandwich_covariance matrix, which can cal Reference to Newey-West covariance matrix: ...
0
votes
1answer
108 views

Convert Daily Data to Monthly Data in Python : Time Series Analysis

I am new to data analysis with python. I have daily data of flu cases for a five year period which I want to do Time Series Analysis on. Am using the Pandas library. It is easy to plot this data and ...
1
vote
0answers
27 views

Approximating cox model with time varying covariates using poisson

How do you reformat a dataset in order to perform a cox regression with time-varying covariates as a poisson regression. I'm trying to run a survival analysis regression in python with time varying ...
0
votes
1answer
124 views

What is the best way to do a seasonal ARMA (or ARIMA) in python?

Scikit learn and statsmodels don't seem to support this type of ARMA. I tried to use rpy2 python library, but that proved to be far too difficult to integrate, as my IDE was not able to recognize my ...
1
vote
2answers
183 views

Time Series Anomaly Detection with Python

I need to implement anomaly detection on several time-series datasets. I've never done this before and was hoping for some advice. I'm very comfortable with python, so I would prefer the solution be ...
0
votes
0answers
18 views

Slow Lasso Performance Using sklearn

I am trying to use scikit-learn's LassoCV and/or ElasticNetCV functions to model a dataset with a large (>800) number of ...
0
votes
0answers
38 views

Which naive Bayes?

I am attempting to use a naïve Bayes classifier in python (using scikit-learn), with two examples. The first example has 6 classes and 2 hypotheses, the 2nd example has 2 classes and 6 hypotheses. ...