Problems focusing on (but not necessarily limited to) using Python for statistical computation.

learn more… | top users | synonyms

0
votes
0answers
12 views

What is R squared for a neural network and what does it signify?

I calculated R square for my neural network based on a formula I found somewhere, which goes something like: http://i.stack.imgur.com/DojZC.png It should be something around 0.98-0.99. But, when I ...
5
votes
1answer
79 views

Why Python's scikit-learn LDA results are different from LDA in R or a step-by-step approach

I was using the Linear Discriminant Analysis (LDA) from the scikit-learn machine learning library (Python) for dimensionality reduction and was a little bit curious about the results. I am wondering ...
0
votes
1answer
26 views

PDF not matching histogram of synthetic ratios of independent beta

The PDF of the ratios of independent beta variables is described in http://www.tandfonline.com/doi/abs/10.1080/03610920008832632#.U9J02vldUcC To explore the implications, i created an implementation ...
2
votes
0answers
43 views

Training a Tic Tac Toe brain - am I on the right track?

My only experience with Machine Learning is Andrew Ng's Coursera course, but I did work through that just fine and passed with 100%. I decided to practice by making up some problems and solving them. ...
0
votes
1answer
25 views

Fisher's z-transform in Python?

I want to test a sample correlation $r$ for significance ($n=16$), using p-values, in Python. As I have understood from this question, I can achieve that by using Fisher's z-transform. Is there a ...
0
votes
1answer
40 views

What is the equation for random forest?

I need an equation for random forest so that I can score fresh data I receive every week, based on beta estimates I got after building model using this ensemble methodology. Every week I do not want ...
1
vote
0answers
36 views

methods to perform feature selection in unsupervissed learning

What are the available methods/implementation in R/Python to discard/select unimportant/important features in data? My data does not have labels (unsupervised). The data has ~100 features with mixed ...
2
votes
0answers
25 views

Defining constraint on prior with potential class

I have written an MCMC code in order to estimate parameters Xpos, Ypos, MASS and concentration with a set of input data gal_pos, ...
0
votes
1answer
13 views

missing value patterns

I am doing some data preparation with Python using Pandas and I am working with a dataset that has about 80 variables with missing values and I want to capture any patterns of missingness to cut down ...
0
votes
0answers
30 views

CDF/ ECDF plot for data with two attributes

I have a data in the following format: $$ \begin{array}{rr} \textbf{colm_1} & \textbf{colm_2}\\ 3 & 1\\ 10 & 0\\ 3 & 0\\ 100 & 1\\ . & .\\ . & . \end{array} $$ colm_1 are ...
3
votes
1answer
390 views

Island Hopping with Metropolis Algorithm

From John Kruschke's book, Chapter 7, Pg. 120 (summarised for succinctness): A politician is constantly travelling from island to island on a chain of islands... His goal is to visit all the ...
0
votes
0answers
21 views

Kalman Filter Correction efficiency

I was wondering if Kalman Filter used in a way to correct and reduce forecast errors is useful in real life forecast.Since we are using output forecast data and measurement data from t-1 to correct ...
1
vote
2answers
45 views

Singular covariance matrix in exploratory Factor Analysis

I'm kind of a noob to EFA and am trying to use the FANode object in Python. This is from the MDP library. I am using it on survey data to see which variables are tied together. Whenever I run it on my ...
0
votes
0answers
29 views

How to model non-linear time courses with nlme

I have some time course data which plotted looks like this: I am trying to fir a model to it using nlme.lme(). I am interfacing with R via RPy, and using the ...
3
votes
2answers
57 views

What are some methods for generating simulated time series data for use in modeling?

I have a data set which consists of 50 observed years for which I have date and inflow values between a river and a reservoir. The data is formatted as follows: ...
0
votes
0answers
15 views

kalman filter initialization parameter

Hello I don't have any idea of how to start implementing Kalman Filter in python! I have a DataFrame ( table) with in one column my forecast values and in another column my actual datas (real). The ...
0
votes
0answers
27 views

Python scatter plot of 4D data [migrated]

I have 4D array of data that I would like to scatter plot. The data can be seen as x- and y- coordinates for each pair of values of two additional parameters. I would like to "flatten" the plot to a ...
0
votes
0answers
20 views

linearity of a time series

I am currently trying to correct forecast data using Kalman filter (python). I do not know where to start. I wanted to know how can I do a test to Know if my time series is linear or non linear? Is ...
0
votes
0answers
1 views

Plotting date or time as x axis with ggplot [migrated]

My data is of the form: datetime, count 2011-01-01 00:00:00, 10 2011-01-01 01:00:00, 15 2011-01-01 02:00:00, 20 ... Using ggplot, I want to plot following 2 ...
0
votes
0answers
6 views

Turning MiniBatchKMeans into Fuzzy MiniBatchKMeans

I'm using Scikit-Learn, which has an implementation of MiniBatchKMeans. I'm very inexperienced with ML, so I'm wondering how (if ...
1
vote
0answers
26 views

Running R caret in Python script [closed]

I am currently trying to transition from R into Python to streamline the process of working in web-based applications. I know that SciKit Learn has a ton of functionalities parallel to R's, including ...
2
votes
1answer
37 views

How to prepare interactions of categorical variables in scikit-learn?

What is the best way to prepare interactions of categorical features before fitting with scikit-learn? With statsmodels I could conveniently say in R-style ...
2
votes
0answers
40 views

forecast improvement using Kalman FIlter clearing

I have been facing a wall after doing a forecast of wind speed time series data using ARIMA with python. I have result with a nrmse growth going from 2% to 15% and now what I want is to use kalman ...
0
votes
0answers
8 views

Negative Binomial Mixture in PyMC [migrated]

I am trying to fit a Negative binomial mixture with PyMC. It seems I do something wrong, because the predictive doesn't look at all similar to the input data. And the predictions for the parameters ...
0
votes
0answers
44 views

What is the Probability Distribution of NLTK Naive Bayes?

As I know Naïve Bayes has various distributions, as said in Sci-kit learn manual “The different naive Bayes classifiers differ mainly by the assumptions they make regarding the distribution of P(x_i ...
1
vote
0answers
31 views

Latent Dirichlet Allocation in PyMC

As an exercise to improve my skills in PyMC (Python's Markov chain Monte Carlo library), I am trying to implement Latent Dirichlet Allocation as described here: ...
0
votes
0answers
17 views

Calculating statistically rare events from a set of observations

I have a Pandas data frame as follows: Event Location| Number of events | %age success A | 10 | 0.5 B | 2 | 1 C | 1 | 0 D | 100 | 0.3 E | 1 | 0 F | 1 | 1 ... ... The data represent the ...
1
vote
1answer
41 views

Statsmodels OLS and MSE

So, I have data set and I calculate the model parameters and errors using statsmodels: result = sm.OLS(y, X).fit() result.summary() Now, ...
4
votes
0answers
47 views

Significance of Spatial Data

Background: It's common in many aspects of science to want to determine if the trends shown in spatial data are significant. In my field of familiarity, climate science, authors often show the ...
0
votes
1answer
36 views

Random forest ML algorithm suitable for use on cluster based HPC?

I have developed a script using pythons scipy package to analyse a rather large model that I wish to solve, the model contains over 12gb of data, including over 500 parameters. Now running small ...
0
votes
0answers
21 views

Perform analysis of deviance on R/RPy aov() models

I am using stats.aov() in R to perform an ANOVA on my 2x2 within subject exeriment design. I am unsure how best to model the data so I have formulated 3 models, ...
1
vote
0answers
114 views

Complete machine learning library for Java/Scala

Python is plenty of ML libraries (like the great scikit-learn). Is there any good for java/scala, containing many algos (regression, classification, clustering, cross-validation, feature processing), ...
0
votes
0answers
18 views

Edge Probabilty Using KL-Divergence Code in Python

Its a little complicated question, so please bear with me. I am doing Image Segmentation using Swendsen-Wang method for Image Analysis./ (stat.fsu.edu/~abarbu/papers/jcgs.pdf) I have to calculate ...
0
votes
2answers
38 views

how do you do regression analysis on advertising impact

i have data that includes clicks, spend, signups and date. for 1 week, i turn off advertising spend to see what clicks and signups are. the next week, i turn advertising back on to see what the new ...
1
vote
1answer
55 views

Creating data that follows a specific data distribution

I have a variable z which has around 3000 values between 0.0 and 0.5. I have plotted here z in the y-axis and 3000 evenly spaced numbers over the interval 0.0 and 0.5 in the x-axis so that one can ...
2
votes
0answers
71 views

Interpolating time series wind data using correlation

Is there a way to use correlation to interpolate missing data? I know the wind speed in 6 locations every hour. This shows the correlation between the separate locations. Hourly Data I know the wind ...
0
votes
1answer
69 views

Metropolis Random Walk Algorithm in Python

I am trying to implement random walk metropolis in Python for generate from a pdf $f$, which is defined in the domain: $$(0, \infty) \times [3, 4] \times (0, \infty)$$ I am using Metropolis Random ...
0
votes
0answers
30 views

Build corpus with phrases

I have my documents as: doc1 = beautifull, very good, very bad, you are great doc2 = very bad, good restaurent, nice place to visit I want to make my corpus ...
0
votes
1answer
40 views

A way to use GPU hardware in matlab [closed]

I know that theano is a python library for using gpu hardware and make effective implementations. Is there any such library or a way to do the same in matlab?
0
votes
0answers
48 views

Difference in values of tf-idf matrix using scikit-learn and hand calculation

I am playing with scikit-learn to find the tf-idf values. I have a set of documents like: ...
2
votes
1answer
127 views

Predictive Modeler: How can learning Python and/or Java benefit me?

On a daily basis, I build predictive models (namely, logistic regression and credit scorecard models) using fairly large datasets (typically ~500k records and ~1k candidate variables) to predict ...
0
votes
0answers
34 views

MCMC Standard error

I need to make a simulation to calculate the integral of a pdf $f(x|D)$ over a region $T$.That is, I need to evaluate $$\int_T f(x|D)\; dx$$ to do this, I just defined the function $g(x)=1$ if $x \in ...
2
votes
0answers
51 views

Emotion recognition lip zone processing

I want to detect emotions by comparing the cropped lip zone with different templates and output an emotion based on the score. So far I've applied: Normalization Discrete and fast Fourier transform ...
1
vote
1answer
64 views

Create a matrix of tf-idf values from documents

I have a set of documents like: D1 = "The sky is blue." D2 = "The sun is bright." D3 = "The sun in the sky is bright." and a ...
0
votes
0answers
97 views

Writing the objective function and constraints for scipy.optimize.minimize from matrices

I'm trying to find the optimal threshold point from a ROC curve. I have two main constraints : tpr >=80 and fpr <=60. I tried three main minimization functions : ...
1
vote
0answers
49 views

Possible to evaluate GLM in Python/scikit-learn using the Poisson, Gamma, or Tweedie distributions as the family for the error distribution?

Trying to learn some Python and SKLearn, but for my work I need to run regressions that use error distributions from the Poisson, Gamma, and especially Tweedie families. I don't see anything in the ...
0
votes
0answers
41 views

Can I compare results from the Kruskal-Wallis test with different numbers of inputs?

I have repeatedly sampled 9 populations and calculated the error from a "true" estimate. I have done this where I have taken progressively less samples (A-G). So for example in A I have taken 10,000 ...
1
vote
2answers
151 views

K-medoid clustering in python

How do I implement k-medoid clustering algorithms like PAM and CLARA in python 2.7? I am currently using Anaconda, and working with ipython 2.7. I have tried scipy.clusters but they don't seem to ...
2
votes
3answers
453 views

Algorithm for winning casino roulette

I would like to try the following algorithm in order to win in the roulette: Be an observer until there are 3 same parity numbers in a row ($0$ has no defined parity in this context) Once there were ...
1
vote
0answers
22 views

How to enforce periodic boundary conditions when performing regression with sci-kit learn?

I have signals that resemble sine waves (or, more accurately, sums of sines). The data are normalized in the time domain so that there is only one cycle and all of the points lie between 0 and 1. An ...