Problems focusing on (but not necessarily limited to) using Python for statistical computation.

learn more… | top users | synonyms

0
votes
0answers
12 views

Weekly data normalization - Python

I have a weekly dataset and I have to normalize this data. Data is something like this : ...
0
votes
1answer
25 views

Questions about weather prediction in scikit learn

Hello I am a high school student doing research on weather. I have a dataset that has four columns each labeled with time, pressure, and lat/long. I am confused on the cross validation process. What ...
0
votes
0answers
16 views

How to do ranking with scikit-learn random forest model

I have a training dataset that I've developed, that has the following format: ------------------------------ | User ID | Item | Label | ------------------------------ | 001 | umbrella | 0 ...
0
votes
1answer
25 views

Log posterior function in PYMC

my question concerns the logp function in the PYMC package in Python. Ultimately I want to calculate a quantity that goes by many names, namely the Bayes-factor/ evidence/ marginal-likelihood of the ...
0
votes
0answers
4 views

python list permutations [migrated]

I need to print all the permutations of numbers in pyhton. so far I wrote this: def permutation(listNum, i): ...
1
vote
0answers
41 views

Prediction based on multiple time series - Python

I have 3 predictors and 1 variable that represents ground truth. They all are linked time series. My purpose is with the 3 predictors to try to forecast the ground truth data. For example : ...
-1
votes
1answer
32 views

Prediction on multiple regression - Python

I have 3 list of value and 1 ground truth data. They all belongs to the same time series. My purpose is with 3 list try to forecast the ground truth data. For example : ...
1
vote
1answer
51 views

Simple Multivariate Bayesian Method in Python

I am trying to follow the Bayesian method described in this text. The python notebook goes through the example of creating two Poisson functions describing a change in SMS frequency at some point tau. ...
0
votes
0answers
32 views

MonteCarlo simulations to test light curve variability

I have an average orbital light curve for a source, that is, binned count rate vs orbital phase, where the count rate are averaged over a number of orbit. I want to run MonteCarlo simulations to find ...
0
votes
1answer
30 views

Python - SkLearn Logistic Regression: One-by-one train instance

Here is my question, I have a huge train set so I can't load it in memory and apply this code. ...
1
vote
2answers
92 views

Python vs R for Text Mining Preprocessing

I've been reading some articles on cleaning text data before doing text mining analysis on it. I have experience in both Python and R and am wondering if one of these languages is an obviously better ...
0
votes
1answer
28 views

Regression with a kernel

I have a fixed kernel and a set of points. I do SVC with the flavor of SVM classification i'm working on (assume it's just a regular SVM) and i obtain a classifier represented by an explicit vector of ...
0
votes
0answers
9 views

Trouble applying Coupled Hidden Markov model [migrated]

I want to train a Coupled Hidden Markov model with two chains like this with baum_welch in python [or other programming language, if there are open code available :)] but when I use baum_welch to ...
1
vote
0answers
14 views

Sampling subset to span entire range of full set (*not* to be representative), in order to construct some sort of lookup table

I have a large number of $N$ (20770) measurements. I need to perfom a calculation on all of them, but this is computationally too expensive. Therefore, I am looking for a way to select a subset of $p$ ...
0
votes
0answers
7 views

working of ndim in numpy [migrated]

import numpy as np >>> a=np.array([1,2,3,4]) >>> a array([1, 2, 3, 4]) >>> a.ndim 1 How the dimension is 1 .I have given a equation of ...
0
votes
0answers
26 views

Enterprise Use of SAS & SPSS and Impact of Open Source Platforms (R & Python) [duplicate]

I'm curious to hear from people that use SAS & SPSS why a lot of the Fortune 100/500 enterprises still use SAS & SPSS despite the high cost of the license. Also would love to understand what ...
1
vote
0answers
19 views

Pre-processing before digit recognition for NN & CNN trained with MNIST dataset

I'm trying to classify handwriting digits, written by myself and a few friends, by usign NN and CNN. In order to train the NN, MNIST dataset is used. The problem is the NN trained with MNIST dataset ...
0
votes
0answers
16 views

Custom Impulse Response Functions for VAR

I am working on a VAR model for the Safe Assets in the U.S. economy. The demand for Safe Assets has increased as needs to pledge them as collateral, or the response after the '08 crisis has been risk ...
1
vote
1answer
42 views

Marginal Likelihood in PYMC

I am using the PYMC toolbox in python in order to carry out a model selection problem using MCMC. What I would like to have for each model is the marginal log-likelihood (i.e. model evidence). The ...
0
votes
0answers
13 views

Repeated utility values in Value Iteration (Markov Decision Process)

I am trying to implement the value iteration algorithm of the Markov Decision Process using python. I have one implementation. But, this is giving me many repeated values for the utilities. My ...
0
votes
0answers
9 views

How to do panel data analysis in Bayesian model with pymc [migrated]

everyone. I have a question on how to do panel data analysis in Bayesian model with pymc. The data is like: ...
2
votes
0answers
106 views

A smarter way for evaluating combinations of samples to optimize an overall score

I have the following problem (which I simplified for clarity) that I want to find k out of n samples (here: rows) that yield the maximum score (sum of the values in ...
4
votes
1answer
86 views

Review of Box-Jenkins methodology

i just finished developing an ARMAX model with python (mostly statsmodels) in order to forecast some data. My next step is to test the data (24 time series) with the given ARMAX model. As i need to ...
1
vote
1answer
30 views

Automatic selection of lowest information criterion comes with warning

I am building a forecasting model (ARMA) and found the very useful code-object arma_order_select_ic(see code below). It all works, however, each calculation comes ...
2
votes
1answer
53 views

Differences in Spearman coefficient between R and pandas

I've noticed a small difference between pandas and R with regards to how they calculate Spearman coefficients. It seems as if some rounding occurs. I see no such difference when calculating Kendall or ...
1
vote
0answers
18 views

RandomForest - why isn't it predicting well with manually-selected test sets?

I am using python sklearn.ensemble to do a RandomForestClassifier on about 800K rows of data, coupled with sklearn.cross_validation to generate the train/test sets. When it completes, it says on the ...
2
votes
0answers
13 views

False Positives in Real Time Classification

I am doing a sliding window binary classification. I have time series data and I am running a time window over this data and let a classifier produce a decision probability. Based on this probability, ...
3
votes
1answer
52 views

Is it possible to seed RANSAC with a given line?

I am analyzing a stream of data and I want to seed every new instance with the best guess output (line) of the previous, so as to eventually converge. Given that Scikit Learn - RANSAC is an iterative ...
3
votes
2answers
102 views

Visualization of random distribution with 3 variables

Suppose I have a joint distribution of three random variables $x,y,z$, $P(x,y,z)$. For simplicity, let's suppose those three rvs. are discrete. The distribution will be represented in Python as a ...
0
votes
1answer
20 views

sklearn.tree.export_graphviz values do not add up to samples

When I run tree.export_graphviz() after training a sklearn.ensemble.RandomForestClassifier() on my data, I get some leaf nodes where the samples count doesn't match the value array, like this: ...
0
votes
1answer
16 views

Sparse Collaborative Filtering

Does anyone know of any Python code examples for sparse collaborative filtering. Everything I can find revolves around using prebuilt packages (e.g. Mahout, GraphLab), but I'm learning to learn the ...
4
votes
1answer
49 views

Understanding multiple KS tests

I have read these two questions Why are p-values uniformly distributed under the null hypothesis? and Understanding scipy Kolmogorov-Smirnov test And this inspired me the following experiment. I ...
2
votes
0answers
72 views

Johansen Test in python

How to get the trace statistics from the Johansen test for cointegration in python. Also, i tried to search it myself on google, found the following website ...
1
vote
1answer
15 views

Recommendation Engine With Physical Distance Cutoff

I am looking to develop a recommendation engine for local stores to users. There are approximately 1 million stores in the database and around 1 million users. The 1Mx1M matrix for a user-based ...
0
votes
0answers
40 views

One-class SVM vs NN with backprop… Or is there something better?

I'm pretty new to unary classification, so I've been playing around with different approaches to one-class document classification in Python. NN seemed promising at first, but has some undesirable ...
0
votes
1answer
56 views

Gradient of Log-Likelihood

Considering the following functions I'm having a tough time finding the appropriate gradient function for the log-likelihood as defined below: $a_k(x)=\sum_{i=1}^D w_{ki}\cdot x_i$ $P(y_k|x) = ...
0
votes
1answer
54 views

Probabilistic classifier useful for both numeric and nominal features

First, I should say I'm beginner in data mining and use Matlab. I have a huge dataset: approximately 40 millions observation. It consists of 24 features including numeric and nominal values like ...
0
votes
0answers
37 views

How can i do time series forecasting with missing data

I am relatively new to time series forecasting, I have worked previously with continuous data at regular intervals successfully, Now I have a data set with missing values, for example look at the ...
0
votes
0answers
40 views

The mean and standard deviation of an Kernel Density Estimation

[I have crossed posted this from StackOverflow. Not sure if this is allowed, but maybe there is a Python answer, or maybe there is a good stats answer] I have fitted a ...
2
votes
0answers
39 views

Random Forest Probabilistic Prediction vs majority vote

Scikit learn seems to use probabilistic prediction instead of majority vote for the model aggregation technique without an explanation as to why (1.9.2.1. Random Forests). Is there a clear ...
1
vote
2answers
42 views

How to find if a difference is statistically significant?

I have a web crawler that scrapes items on an hourly basis and expect to get 375 items every time. In the past 10 runs, I have scraped the following number of items: [360, 361, 363, 364, 368, 372, ...
1
vote
1answer
39 views

Visualize difference between 2 classifiers

I trained 2 binary classifiers with the same data (a Decision Tree and a Random Forest). They both made a prediction on the same test data. Now, I want to visualize the difference in classification ...
2
votes
1answer
71 views

How do I force the L-BFGS-B to not stop early? Projected gradient is zero

I'm trying to use the SciPy implementation of the fmin_l_bfgs_b algorithm using the following code: ...
0
votes
1answer
50 views

How to OLS Regress Y on 1/x denominator in R or Python

I have a problem where my general equation is $Y = C + 1/(\beta x)$, where $C$ is a constant. I want to find a b in OLS fashion to minimize RSS I have already transformed my equation thus far: $Y - ...
0
votes
0answers
30 views

Implementing autoregressive model on financial returns using sklearn

I have a model where I'm trying to regress S&P quarterly returns on lagged economic indicators using sklearn. I've read a lot about sklearn but this will be my first time implementing it. I ...
1
vote
1answer
114 views

I would like to analyze the data using R and Python. How to handle 10 millions records of data? [closed]

I would like to analyze the data using R and Python, and I am unable to open text file due to huge number of records. Is there any free database software to install in PC and load the data into it and ...
1
vote
1answer
82 views

Classification score for Random Forest

I'm learning about the Decision Tree and Random Forests. But there is something I don't really understand. I have a training set and a cross-validation set. I need to train different Random Forests, ...
0
votes
1answer
39 views

How to find optimal penaltyparameter C for SVM (regression)

I am training an svm regressor using python sklearn.svm.SVR From the example given on the sklearn website, the above line of code defines my svm. ...
0
votes
0answers
22 views

Python “feature_importances” for most important factors

I'm a little unsure as whether this belongs in stackoverflow or cross validated. I have found a few posts on this topic , but I have not found the following question. Is it accurate to run the feature ...
0
votes
0answers
69 views

lowess: R (& python-statsmodels) vs MATLAB (& biopython)

There seems to be two different interpretations of what LOWESS really means: one from R (also used by python-statsmodels), and one from MATLAB (also used by biopython), see comparison below. Could ...