Problems focusing on (but not necessarily limited to) using Python for statistical computation.

learn more… | top users | synonyms

-1
votes
0answers
14 views

How to find a Threshold Value [on hold]

I have large file with data over 100000 items. I have to find the optimal Threshold.Any suggestion how to find an optimal Threshold value for future data analysis. The input file has following data ...
0
votes
1answer
26 views

Best optimization package for employee scheduling problem? [on hold]

I am looking to solve the optimization problem described below. Which optimization software package would be best suited for this, considering the requirements specified below? Requirements: 1) Can ...
1
vote
2answers
46 views

Time series - correlation and lag time

I am studying the correlation between a set of input variables and a response variable, price. These are all in time series. 1) Is it necessary that I smooth out the curve where the input variable is ...
3
votes
1answer
36 views

Standard Deviation of an Exponentially-weighted Mean

I wrote a simple function in Python to calculate the exponentially weighted mean: ...
1
vote
1answer
48 views

How do I perform this complicated ANOVA type analysis in R?

I have the following type of dataset: | | | variable_r | subject | gender | age_group | Cond_1 | Cond_2 | ...
0
votes
0answers
15 views

What library in Python implements a Logistic Model Tree?

I am looking for something that implements a an algorithm along the lines of Landwehr et al.'s 2003 "Logistic Model Trees". I've found LMT() in the RWeka package implements the algorithm in R, but am ...
1
vote
0answers
32 views

Test means of populations with lots of zeros, is sampling the way to go?

I am trying to test whether means of two populations with lots of zeros are different. Here is the following python code example: ...
0
votes
0answers
23 views

Multivariate Linear Regression in Python

How to compute the overall standard error of a linear regression model using Python? Which library should I use? I am looking for something like this, however, I can't see how to get the overall ...
0
votes
0answers
25 views

Find linear SVM feature weights using libsvm

I'm trying to use linear SVM to do some feature selection. I'm using libsvm, but I cannot figure out how to find feature weights. The model file created looks something like this: ...
0
votes
0answers
12 views

Liblinear vs sklearn implementation

I'm using both sklearn (linearSVC) and liblinear (python wrapper) to see if they match. From liblinear documentation these are the available options: ...
0
votes
0answers
35 views

Autoregressive distributed lag models ADL(p,q): seeking “how to” preferably in Matlab (Stata/R/Python/C# etc)

Could anyone provide me the details of how to determine the lag order of the distributed lags for an $\text{ADL}(p,q)$ model in Matlab or another statistical package (and very much preferably in ...
1
vote
1answer
124 views

Plot the probability mass function

I am trying to plot the probability mass function of a sample of a discrete metric. If it was continuous, I know that using pandas it would be as simple as calling: ...
2
votes
1answer
41 views

finding global minima of a random forest estimator

I have a random forest regression model with 1000 trees, having 16 parameters (using python scikit-learn). The estimator can predict a target value with cross validated r2 score of 0.87 +/- 0.03. I ...
0
votes
1answer
12 views

Feature selection with a binary dependent variable

Given we have a binary dependent variable and 100s of features and ~50k observations, is there a generally accepted way to trim the features via some type of machine learning concept? I was trying a ...
0
votes
0answers
5 views

how to set key for deterministic variable in pymc [migrated]

I'm trying to plot the difference between two variables. I'm following the example set here (search for true_p_A and it will be in the right section) Here is my code ...
0
votes
1answer
26 views

Calculating the information gain on the features with python

I'm looking for a python library that computes the information gain for the features given a training matrix. Are you aware of any?
0
votes
3answers
45 views

Biased coin toss simulation — which random generator is most appropriate?

I have a doubt whether the uniformly distributed random generator is the most appropriate to use for simulating biased coin toss. ...
0
votes
0answers
13 views

How to calculate the correlation between a variable and a matrix of variables

I'm trying to improve a factory quality control. I have some variables from the melting process (something like ten control variables) that changes trough time (a matrix of the values of those ...
0
votes
0answers
24 views

Chi-squared test with scipy: what's the difference between chi2_contingency and chisquare?

I'd like to run a chi-squared test in Python with scipy. I've created code to do this, but I don't know if what I'm doing is right, because the scipy docs are quite sparse. Background first: I have ...
0
votes
1answer
21 views

How to get both MSE and R2 from a sklearn GridSearchCV?

I can use a GridSearchCV on a pipeline and specify scoring to either be 'MSE' or 'R2'. I can then access ...
0
votes
0answers
18 views

Divide a huge data set ( integers) into some small sets.(I have a very ugly method, seek for a nicer way) [migrated]

I have a lot of integers in a file, about 45 million. I want to visualize these data, to find the distribution of these data. It's too large for a software like Mathematica to load and plot. So I use ...
0
votes
2answers
43 views

Plotting a Gaussian in Python

I am trying to plot a histogram of my data, and I seem to be a little confused here. I am using matplotlib in Python. Here is the code from their website: ...
0
votes
1answer
29 views

Alternative to forecast() and ets() in Python?

I'm looking for a Python alternative to R's ETS() from forecast(). It's my understanding that ETS() is one of the best performing forecasting program and I would like to use it. However I am ...
0
votes
0answers
22 views

What is R squared for a neural network and what does it signify?

I calculated R square for my neural network based on a formula I found somewhere, which goes something like: http://i.stack.imgur.com/DojZC.png It should be something around 0.98-0.99. But, when I ...
7
votes
3answers
257 views

Why Python's scikit-learn LDA results are different from LDA in R or a step-by-step approach

I was using the Linear Discriminant Analysis (LDA) from the scikit-learn machine learning library (Python) for dimensionality reduction and was a little bit curious about the results. I am wondering ...
0
votes
1answer
28 views

PDF not matching histogram of synthetic ratios of independent beta

The PDF of the ratios of independent beta variables is described in http://www.tandfonline.com/doi/abs/10.1080/03610920008832632#.U9J02vldUcC To explore the implications, i created an implementation ...
3
votes
1answer
89 views

Training a Tic Tac Toe brain - am I on the right track?

My only experience with Machine Learning is Andrew Ng's Coursera course, but I did work through that just fine and passed with 100%. I decided to practice by making up some problems and solving them. ...
0
votes
1answer
45 views

Fisher's z-transform in Python?

I want to test a sample correlation $r$ for significance ($n=16$), using p-values, in Python. As I have understood from this question, I can achieve that by using Fisher's z-transform. Is there a ...
0
votes
1answer
60 views

What is the equation for random forest?

I need an equation for random forest so that I can score fresh data I receive every week, based on beta estimates I got after building model using this ensemble methodology. Every week I do not want ...
1
vote
2answers
70 views

Methods in R or Python to perform feature selection in unsupervised learning

What are the available methods/implementation in R/Python to discard/select unimportant/important features in data? My data does not have labels (unsupervised). The data has ~100 features with mixed ...
2
votes
0answers
36 views

Defining constraint on prior with potential class

I have written an MCMC code in order to estimate parameters Xpos, Ypos, MASS and concentration with a set of input data gal_pos, ...
0
votes
1answer
14 views

missing value patterns

I am doing some data preparation with Python using Pandas and I am working with a dataset that has about 80 variables with missing values and I want to capture any patterns of missingness to cut down ...
0
votes
0answers
34 views

CDF/ ECDF plot for data with two attributes

I have a data in the following format: $$ \begin{array}{rr} \textbf{colm_1} & \textbf{colm_2}\\ 3 & 1\\ 10 & 0\\ 3 & 0\\ 100 & 1\\ . & .\\ . & . \end{array} $$ colm_1 are ...
3
votes
1answer
393 views

Island Hopping with Metropolis Algorithm

From John Kruschke's book, Chapter 7, Pg. 120 (summarised for succinctness): A politician is constantly travelling from island to island on a chain of islands... His goal is to visit all the ...
0
votes
0answers
25 views

Kalman Filter Correction efficiency

I was wondering if Kalman Filter used in a way to correct and reduce forecast errors is useful in real life forecast.Since we are using output forecast data and measurement data from t-1 to correct ...
1
vote
2answers
51 views

Singular covariance matrix in exploratory Factor Analysis

I'm kind of a noob to EFA and am trying to use the FANode object in Python. This is from the MDP library. I am using it on survey data to see which variables are tied together. Whenever I run it on my ...
0
votes
0answers
30 views

How to model non-linear time courses with nlme

I have some time course data which plotted looks like this: I am trying to fir a model to it using nlme.lme(). I am interfacing with R via RPy, and using the ...
3
votes
2answers
62 views

What are some methods for generating simulated time series data for use in modeling?

I have a data set which consists of 50 observed years for which I have date and inflow values between a river and a reservoir. The data is formatted as follows: ...
0
votes
0answers
19 views

kalman filter initialization parameter

Hello I don't have any idea of how to start implementing Kalman Filter in python! I have a DataFrame ( table) with in one column my forecast values and in another column my actual datas (real). The ...
0
votes
0answers
21 views

linearity of a time series

I am currently trying to correct forecast data using Kalman filter (python). I do not know where to start. I wanted to know how can I do a test to Know if my time series is linear or non linear? Is ...
0
votes
0answers
9 views

Turning MiniBatchKMeans into Fuzzy MiniBatchKMeans

I'm using Scikit-Learn, which has an implementation of MiniBatchKMeans. I'm very inexperienced with ML, so I'm wondering how (if ...
1
vote
0answers
38 views

Running R caret in Python script [closed]

I am currently trying to transition from R into Python to streamline the process of working in web-based applications. I know that SciKit Learn has a ton of functionalities parallel to R's, including ...
2
votes
1answer
48 views

How to prepare interactions of categorical variables in scikit-learn?

What is the best way to prepare interactions of categorical features before fitting with scikit-learn? With statsmodels I could conveniently say in R-style ...
2
votes
0answers
42 views

forecast improvement using Kalman FIlter clearing

I have been facing a wall after doing a forecast of wind speed time series data using ARIMA with python. I have result with a nrmse growth going from 2% to 15% and now what I want is to use kalman ...
1
vote
1answer
115 views

What is the Probability Distribution of NLTK Naive Bayes?

As I know Naïve Bayes has various distributions, as said in Sci-kit learn manual: The different naive Bayes classifiers differ mainly by the assumptions they make regarding the distribution of ...
1
vote
0answers
45 views

Latent Dirichlet Allocation in PyMC

As an exercise to improve my skills in PyMC (Python's Markov chain Monte Carlo library), I am trying to implement Latent Dirichlet Allocation as described here: ...
0
votes
0answers
20 views

Calculating statistically rare events from a set of observations

I have a Pandas data frame as follows: Event Location| Number of events | %age success A | 10 | 0.5 B | 2 | 1 C | 1 | 0 D | 100 | 0.3 E | 1 | 0 F | 1 | 1 ... ... The data represent the ...
1
vote
1answer
53 views

Statsmodels OLS and MSE

So, I have data set and I calculate the model parameters and errors using statsmodels: result = sm.OLS(y, X).fit() result.summary() Now, ...
4
votes
0answers
51 views

Significance of Spatial Data

Background: It's common in many aspects of science to want to determine if the trends shown in spatial data are significant. In my field of familiarity, climate science, authors often show the ...
0
votes
1answer
37 views

Random forest ML algorithm suitable for use on cluster based HPC?

I have developed a script using pythons scipy package to analyse a rather large model that I wish to solve, the model contains over 12gb of data, including over 500 parameters. Now running small ...