Problems focusing on (but not necessarily limited to) using Python for statistical computation.

learn more… | top users | synonyms

4
votes
1answer
53 views

Applying a Gaussian radial basis function kernel PCA to nonlinear data

I have an assignment to implement a Gaussian radial basis function-kernel principal component analysis (RBF-kernel PCA) and have some challenges here. It would be great if someone could point me to ...
0
votes
0answers
13 views

Python module request: Spectral density estimation for multivariate time series

I have worked with scipy.signal.welch and spectrum.pptm to calculate power spectral density with Welch and Multitaper methods. However as far as I can see these functions are meant for one dimensional ...
0
votes
0answers
19 views

Time-series detection algorithm for multi-seasonal data using Python

My data: I have two seasonal patterns in my hourly data... daily and weekly. For example... each day in my dataset has roughly the same shape based on hour of the day. However, certain days like ...
0
votes
0answers
8 views

Script for downloading Medline abstracts with PMID [migrated]

I am looking for a script that will enable me to download a set of abstracts (or article meta-data of an article) from PubMed by supplying it with a list of PubMed ID (PMID) numbers (e.g., from a .csv ...
2
votes
1answer
35 views

PYMC Confusion: are observed nodes fixed or stochastic?

I've been trying to gain a better understanding of factor potentials in PYMC. In reading this article by Cam Davidson-Pilon on Yhat, I got confused about how observed nodes are understood by PYMC. ...
0
votes
0answers
14 views

Choosing between different methods when the first one raises error message for linear regression

I have a linear regression problem $$Ax=b$$ My initial approach that helped to solve some of my questions was using SVD and obtaining the chi-square and some other values that I am interested but it ...
1
vote
0answers
30 views

Optimization in R vs Python, constrained, unconstrained and automatic differentiation?

I am an economics/stat guy who uses quite a bit of optimization (maximum likelihood, simulated maximum likelihood), constrained optimization (mathematical programming w/ equilibrium conditions), ...
1
vote
1answer
34 views

How to sample from “biased” binomial distribution (ideally in python/numpy)

I need to sample a variable from a distribution that's like a binomial distribution except with a "bias", I'm not sure what it may be called: $p(X=k)$ is proportional to $k.B(n,p)(k)$ where $B(n,p)$ ...
0
votes
1answer
27 views

Autoclass in R/Python? [closed]

Are there any packages that implement the Autoclass/ Naive Bayes Clustering algorithm in R or Python? Alternatively, what are some other clustering algorithms that can handle both categorical and ...
0
votes
0answers
12 views

PyMC: Which parameter is passed to the model by sampler? How to find the value passed by custom sampler?

I am relatively new for PyMC and as a whole I have some less knowledge in probability. As far as I understand MCMC algorithm, its main task is to propose samples of parameters to the model (in ...
0
votes
1answer
32 views

Defining the Stochastic and Deterministic variables with pymc3

I am trying to use write my own stochastic and deterministic variables with pymc3, but old ...
0
votes
0answers
29 views

Numerical estimation of MLE in Python — normal distribution and gradient is close to zero away from the mean

I am exploring how to model a data set using normal distributions with both mean and variance defined as linear functions of independent variables. Something like $\mathcal{N} \sim \left (f(x), ...
0
votes
1answer
40 views

Best optimization package for employee scheduling problem? [closed]

I am looking to solve the optimization problem described below. Which optimization software package would be best suited for this, considering the requirements specified below? Requirements: 1) Can ...
1
vote
2answers
75 views

Time series - correlation and lag time

I am studying the correlation between a set of input variables and a response variable, price. These are all in time series. 1) Is it necessary that I smooth out the curve where the input variable is ...
3
votes
1answer
42 views

Standard Deviation of an Exponentially-weighted Mean

I wrote a simple function in Python to calculate the exponentially weighted mean: ...
1
vote
1answer
57 views

How do I perform this complicated ANOVA type analysis in R?

I have the following type of dataset: | | | variable_r | subject | gender | age_group | Cond_1 | Cond_2 | ...
0
votes
0answers
23 views

What library in Python implements a Logistic Model Tree?

I am looking for something that implements a an algorithm along the lines of Landwehr et al.'s 2003 "Logistic Model Trees". I've found LMT() in the RWeka package implements the algorithm in R, but am ...
1
vote
0answers
36 views

Test means of populations with lots of zeros, is sampling the way to go?

I am trying to test whether means of two populations with lots of zeros are different. Here is the following python code example: ...
0
votes
0answers
35 views

Multivariate Linear Regression in Python

How to compute the overall standard error of a linear regression model using Python? Which library should I use? I am looking for something like this, however, I can't see how to get the overall ...
0
votes
0answers
32 views

Find linear SVM feature weights using libsvm

I'm trying to use linear SVM to do some feature selection. I'm using libsvm, but I cannot figure out how to find feature weights. The model file created looks something like this: ...
0
votes
0answers
49 views

Autoregressive distributed lag models ADL(p,q): seeking “how to” preferably in Matlab (Stata/R/Python/C# etc)

Could anyone provide me the details of how to determine the lag order of the distributed lags for an $\text{ADL}(p,q)$ model in Matlab or another statistical package (and very much preferably in ...
1
vote
1answer
139 views

Plot the probability mass function

I am trying to plot the probability mass function of a sample of a discrete metric. If it was continuous, I know that using pandas it would be as simple as calling: ...
2
votes
1answer
47 views

finding global minima of a random forest estimator

I have a random forest regression model with 1000 trees, having 16 parameters (using python scikit-learn). The estimator can predict a target value with cross validated r2 score of 0.87 +/- 0.03. I ...
0
votes
1answer
14 views

Feature selection with a binary dependent variable

Given we have a binary dependent variable and 100s of features and ~50k observations, is there a generally accepted way to trim the features via some type of machine learning concept? I was trying a ...
0
votes
1answer
59 views

Calculating the information gain on the features with python

I'm looking for a python library that computes the information gain for the features given a training matrix. Are you aware of any?
0
votes
3answers
76 views

Biased coin toss simulation — which random generator is most appropriate?

I have a doubt whether the uniformly distributed random generator is the most appropriate to use for simulating biased coin toss. ...
0
votes
0answers
23 views

How to calculate the correlation between a variable and a matrix of variables

I'm trying to improve a factory quality control. I have some variables from the melting process (something like ten control variables) that changes trough time (a matrix of the values of those ...
0
votes
0answers
34 views

Chi-squared test with scipy: what's the difference between chi2_contingency and chisquare?

I'd like to run a chi-squared test in Python with scipy. I've created code to do this, but I don't know if what I'm doing is right, because the scipy docs are quite sparse. Background first: I have ...
0
votes
1answer
32 views

How to get both MSE and R2 from a sklearn GridSearchCV?

I can use a GridSearchCV on a pipeline and specify scoring to either be 'MSE' or 'R2'. I can then access ...
0
votes
2answers
56 views

Plotting a Gaussian in Python

I am trying to plot a histogram of my data, and I seem to be a little confused here. I am using matplotlib in Python. Here is the code from their website: ...
0
votes
1answer
37 views

Alternative to forecast() and ets() in Python?

I'm looking for a Python alternative to R's ETS() from forecast(). It's my understanding that ETS() is one of the best performing forecasting program and I would like to use it. However I am ...
0
votes
1answer
31 views

What is R squared for a neural network and what does it signify?

I calculated R square for my neural network based on a formula I found somewhere, which goes something like: http://i.stack.imgur.com/DojZC.png It should be something around 0.98-0.99. But, when I ...
7
votes
3answers
302 views

Why Python's scikit-learn LDA results are different from LDA in R or a step-by-step approach

I was using the Linear Discriminant Analysis (LDA) from the scikit-learn machine learning library (Python) for dimensionality reduction and was a little bit curious about the results. I am wondering ...
0
votes
1answer
30 views

PDF not matching histogram of synthetic ratios of independent beta

The PDF of the ratios of independent beta variables is described in http://www.tandfonline.com/doi/abs/10.1080/03610920008832632#.U9J02vldUcC To explore the implications, i created an implementation ...
3
votes
1answer
99 views

Training a Tic Tac Toe brain - am I on the right track?

My only experience with Machine Learning is Andrew Ng's Coursera course, but I did work through that just fine and passed with 100%. I decided to practice by making up some problems and solving them. ...
0
votes
1answer
68 views

Fisher's z-transform in Python?

I want to test a sample correlation $r$ for significance ($n=16$), using p-values, in Python. As I have understood from this question, I can achieve that by using Fisher's z-transform. Is there a ...
0
votes
1answer
77 views

What is the equation for random forest?

I need an equation for random forest so that I can score fresh data I receive every week, based on beta estimates I got after building model using this ensemble methodology. Every week I do not want ...
1
vote
2answers
93 views

Methods in R or Python to perform feature selection in unsupervised learning

What are the available methods/implementation in R/Python to discard/select unimportant/important features in data? My data does not have labels (unsupervised). The data has ~100 features with mixed ...
2
votes
0answers
41 views

Defining constraint on prior with potential class

I have written an MCMC code in order to estimate parameters Xpos, Ypos, MASS and concentration with a set of input data gal_pos, ...
0
votes
1answer
19 views

missing value patterns

I am doing some data preparation with Python using Pandas and I am working with a dataset that has about 80 variables with missing values and I want to capture any patterns of missingness to cut down ...
0
votes
0answers
37 views

CDF/ ECDF plot for data with two attributes

I have a data in the following format: $$ \begin{array}{rr} \textbf{colm_1} & \textbf{colm_2}\\ 3 & 1\\ 10 & 0\\ 3 & 0\\ 100 & 1\\ . & .\\ . & . \end{array} $$ colm_1 are ...
3
votes
1answer
405 views

Island Hopping with Metropolis Algorithm

From John Kruschke's book, Chapter 7, Pg. 120 (summarised for succinctness): A politician is constantly travelling from island to island on a chain of islands... His goal is to visit all the ...
0
votes
0answers
30 views

Kalman Filter Correction efficiency

I was wondering if Kalman Filter used in a way to correct and reduce forecast errors is useful in real life forecast.Since we are using output forecast data and measurement data from t-1 to correct ...
1
vote
2answers
56 views

Singular covariance matrix in exploratory Factor Analysis

I'm kind of a noob to EFA and am trying to use the FANode object in Python. This is from the MDP library. I am using it on survey data to see which variables are tied together. Whenever I run it on my ...
0
votes
0answers
31 views

How to model non-linear time courses with nlme

I have some time course data which plotted looks like this: I am trying to fir a model to it using nlme.lme(). I am interfacing with R via RPy, and using the ...
3
votes
2answers
68 views

What are some methods for generating simulated time series data for use in modeling?

I have a data set which consists of 50 observed years for which I have date and inflow values between a river and a reservoir. The data is formatted as follows: ...
0
votes
0answers
21 views

kalman filter initialization parameter

Hello I don't have any idea of how to start implementing Kalman Filter in python! I have a DataFrame ( table) with in one column my forecast values and in another column my actual datas (real). The ...
0
votes
0answers
22 views

linearity of a time series

I am currently trying to correct forecast data using Kalman filter (python). I do not know where to start. I wanted to know how can I do a test to Know if my time series is linear or non linear? Is ...
0
votes
0answers
15 views

Turning MiniBatchKMeans into Fuzzy MiniBatchKMeans

I'm using Scikit-Learn, which has an implementation of MiniBatchKMeans. I'm very inexperienced with ML, so I'm wondering how (if ...
1
vote
0answers
57 views

Running R caret in Python script [closed]

I am currently trying to transition from R into Python to streamline the process of working in web-based applications. I know that SciKit Learn has a ton of functionalities parallel to R's, including ...