Problems focusing on (but not necessarily limited to) using Python for statistical computation.

learn more… | top users | synonyms

1
vote
0answers
32 views

methods to perform feature selection in unsupervissed learning

What are the available methods/implementation in R/Python to discard/select unimportant/important features in data? My data does not have labels (unsupervised). The data has ~100 features with mixed ...
2
votes
0answers
19 views

Defining constraint on prior with potential class

I have written an MCMC code in order to estimate parameters Xpos, Ypos, MASS and concentration with a set of input data gal_pos, ...
0
votes
1answer
11 views

missing value patterns

I am doing some data preparation with Python using Pandas and I am working with a dataset that has about 80 variables with missing values and I want to capture any patterns of missingness to cut down ...
0
votes
0answers
29 views

CDF/ ECDF plot for data with two attributes

I have a data in the following format: $$ \begin{array}{rr} \textbf{colm_1} & \textbf{colm_2}\\ 3 & 1\\ 10 & 0\\ 3 & 0\\ 100 & 1\\ . & .\\ . & . \end{array} $$ colm_1 are ...
3
votes
1answer
385 views

Island Hopping with Metropolis Algorithm

From John Kruschke's book, Chapter 7, Pg. 120 (summarised for succinctness): A politician is constantly travelling from island to island on a chain of islands... His goal is to visit all the ...
0
votes
0answers
20 views

Kalman Filter Correction efficiency

I was wondering if Kalman Filter used in a way to correct and reduce forecast errors is useful in real life forecast.Since we are using output forecast data and measurement data from t-1 to correct ...
1
vote
2answers
41 views

Singular covariance matrix in exploratory Factor Analysis

I'm kind of a noob to EFA and am trying to use the FANode object in Python. This is from the MDP library. I am using it on survey data to see which variables are tied together. Whenever I run it on my ...
0
votes
0answers
28 views

How to model non-linear time courses with nlme

I have some time course data which plotted looks like this: I am trying to fir a model to it using nlme.lme(). I am interfacing with R via RPy, and using the ...
3
votes
2answers
56 views

What are some methods for generating simulated time series data for use in modeling?

I have a data set which consists of 50 observed years for which I have date and inflow values between a river and a reservoir. The data is formatted as follows: ...
0
votes
0answers
14 views

kalman filter initialization parameter

Hello I don't have any idea of how to start implementing Kalman Filter in python! I have a DataFrame ( table) with in one column my forecast values and in another column my actual datas (real). The ...
0
votes
0answers
27 views

Python scatter plot of 4D data [migrated]

I have 4D array of data that I would like to scatter plot. The data can be seen as x- and y- coordinates for each pair of values of two additional parameters. I would like to "flatten" the plot to a ...
0
votes
0answers
20 views

linearity of a time series

I am currently trying to correct forecast data using Kalman filter (python). I do not know where to start. I wanted to know how can I do a test to Know if my time series is linear or non linear? Is ...
0
votes
0answers
1 views

Plotting date or time as x axis with ggplot [migrated]

My data is of the form: datetime, count 2011-01-01 00:00:00, 10 2011-01-01 01:00:00, 15 2011-01-01 02:00:00, 20 ... Using ggplot, I want to plot following 2 ...
0
votes
0answers
6 views

Turning MiniBatchKMeans into Fuzzy MiniBatchKMeans

I'm using Scikit-Learn, which has an implementation of MiniBatchKMeans. I'm very inexperienced with ML, so I'm wondering how (if ...
1
vote
0answers
24 views

Running R caret in Python script [closed]

I am currently trying to transition from R into Python to streamline the process of working in web-based applications. I know that SciKit Learn has a ton of functionalities parallel to R's, including ...
2
votes
1answer
33 views

How to prepare interactions of categorical variables in scikit-learn?

What is the best way to prepare interactions of categorical features before fitting with scikit-learn? With statsmodels I could conveniently say in R-style ...
2
votes
0answers
40 views

forecast improvement using Kalman FIlter clearing

I have been facing a wall after doing a forecast of wind speed time series data using ARIMA with python. I have result with a nrmse growth going from 2% to 15% and now what I want is to use kalman ...
0
votes
0answers
8 views

Negative Binomial Mixture in PyMC [migrated]

I am trying to fit a Negative binomial mixture with PyMC. It seems I do something wrong, because the predictive doesn't look at all similar to the input data. And the predictions for the parameters ...
0
votes
0answers
37 views

What is the Probability Distribution of NLTK Naive Bayes?

As I know Naïve Bayes has various distributions, as said in Sci-kit learn manual “The different naive Bayes classifiers differ mainly by the assumptions they make regarding the distribution of P(x_i ...
1
vote
0answers
29 views

Latent Dirichlet Allocation in PyMC

As an exercise to improve my skills in PyMC (Python's Markov chain Monte Carlo library), I am trying to implement Latent Dirichlet Allocation as described here: ...
0
votes
0answers
15 views

Calculating statistically rare events from a set of observations

I have a Pandas data frame as follows: Event Location| Number of events | %age success A | 10 | 0.5 B | 2 | 1 C | 1 | 0 D | 100 | 0.3 E | 1 | 0 F | 1 | 1 ... ... The data represent the ...
1
vote
1answer
40 views

Statsmodels OLS and MSE

So, I have data set and I calculate the model parameters and errors using statsmodels: result = sm.OLS(y, X).fit() result.summary() Now, ...
-3
votes
0answers
41 views

Machine learning and statistical modeling [duplicate]

I would like to know that what one needs to do to start a career in statistical modeling or machine learning. Currently I am working in a banking field where I am using SAS for data analysis but not ...
4
votes
0answers
45 views

Significance of Spatial Data

Background: It's common in many aspects of science to want to determine if the trends shown in spatial data are significant. In my field of familiarity, climate science, authors often show the ...
0
votes
1answer
36 views

Random forest ML algorithm suitable for use on cluster based HPC?

I have developed a script using pythons scipy package to analyse a rather large model that I wish to solve, the model contains over 12gb of data, including over 500 parameters. Now running small ...
0
votes
0answers
20 views

Perform analysis of deviance on R/RPy aov() models

I am using stats.aov() in R to perform an ANOVA on my 2x2 within subject exeriment design. I am unsure how best to model the data so I have formulated 3 models, ...
1
vote
0answers
105 views

Complete machine learning library for Java/Scala

Python is plenty of ML libraries (like the great scikit-learn). Is there any good for java/scala, containing many algos (regression, classification, clustering, cross-validation, feature processing), ...
0
votes
0answers
17 views

Edge Probabilty Using KL-Divergence Code in Python

Its a little complicated question, so please bear with me. I am doing Image Segmentation using Swendsen-Wang method for Image Analysis./ (stat.fsu.edu/~abarbu/papers/jcgs.pdf) I have to calculate ...
0
votes
2answers
38 views

how do you do regression analysis on advertising impact

i have data that includes clicks, spend, signups and date. for 1 week, i turn off advertising spend to see what clicks and signups are. the next week, i turn advertising back on to see what the new ...
1
vote
1answer
55 views

Creating data that follows a specific data distribution

I have a variable z which has around 3000 values between 0.0 and 0.5. I have plotted here z in the y-axis and 3000 evenly spaced numbers over the interval 0.0 and 0.5 in the x-axis so that one can ...
2
votes
0answers
67 views

Interpolating time series wind data using correlation

Is there a way to use correlation to interpolate missing data? I know the wind speed in 6 locations every hour. This shows the correlation between the separate locations. Hourly Data I know the wind ...
0
votes
1answer
65 views

Metropolis Random Walk Algorithm in Python

I am trying to implement random walk metropolis in Python for generate from a pdf $f$, which is defined in the domain: $$(0, \infty) \times [3, 4] \times (0, \infty)$$ I am using Metropolis Random ...
0
votes
0answers
30 views

Build corpus with phrases

I have my documents as: doc1 = beautifull, very good, very bad, you are great doc2 = very bad, good restaurent, nice place to visit I want to make my corpus ...
0
votes
1answer
39 views

A way to use GPU hardware in matlab [closed]

I know that theano is a python library for using gpu hardware and make effective implementations. Is there any such library or a way to do the same in matlab?
0
votes
0answers
39 views

Difference in values of tf-idf matrix using scikit-learn and hand calculation

I am playing with scikit-learn to find the tf-idf values. I have a set of documents like: ...
2
votes
1answer
125 views

Predictive Modeler: How can learning Python and/or Java benefit me?

On a daily basis, I build predictive models (namely, logistic regression and credit scorecard models) using fairly large datasets (typically ~500k records and ~1k candidate variables) to predict ...
0
votes
0answers
33 views

MCMC Standard error

I need to make a simulation to calculate the integral of a pdf $f(x|D)$ over a region $T$.That is, I need to evaluate $$\int_T f(x|D)\; dx$$ to do this, I just defined the function $g(x)=1$ if $x \in ...
2
votes
0answers
50 views

Emotion recognition lip zone processing

I want to detect emotions by comparing the cropped lip zone with different templates and output an emotion based on the score. So far I've applied: Normalization Discrete and fast Fourier transform ...
1
vote
1answer
60 views

Create a matrix of tf-idf values from documents

I have a set of documents like: D1 = "The sky is blue." D2 = "The sun is bright." D3 = "The sun in the sky is bright." and a ...
0
votes
0answers
86 views

Writing the objective function and constraints for scipy.optimize.minimize from matrices

I'm trying to find the optimal threshold point from a ROC curve. I have two main constraints : tpr >=80 and fpr <=60. I tried three main minimization functions : ...
1
vote
0answers
46 views

Possible to evaluate GLM in Python/scikit-learn using the Poisson, Gamma, or Tweedie distributions as the family for the error distribution?

Trying to learn some Python and SKLearn, but for my work I need to run regressions that use error distributions from the Poisson, Gamma, and especially Tweedie families. I don't see anything in the ...
0
votes
0answers
38 views

Can I compare results from the Kruskal-Wallis test with different numbers of inputs?

I have repeatedly sampled 9 populations and calculated the error from a "true" estimate. I have done this where I have taken progressively less samples (A-G). So for example in A I have taken 10,000 ...
1
vote
2answers
134 views

K-medoid clustering in python

How do I implement k-medoid clustering algorithms like PAM and CLARA in python 2.7? I am currently using Anaconda, and working with ipython 2.7. I have tried scipy.clusters but they don't seem to ...
2
votes
3answers
438 views

Algorithm for winning casino roulette

I would like to try the following algorithm in order to win in the roulette: Be an observer until there are 3 same parity numbers in a row ($0$ has no defined parity in this context) Once there were ...
1
vote
0answers
20 views

How to enforce periodic boundary conditions when performing regression with sci-kit learn?

I have signals that resemble sine waves (or, more accurately, sums of sines). The data are normalized in the time domain so that there is only one cycle and all of the points lie between 0 and 1. An ...
1
vote
1answer
50 views

PCA realization in python

Input matrix M x N, M rows (number of samples), N cols (dimesionality of data). But I don't understand why we must transpose matrix when put it in np.cov? ...
0
votes
0answers
27 views

Testing whether or not a result is significantly different from another?

If I have a table with results (columns being different feature subsets and rows being different machine learning algorithms) how can I test whether or not some of the results (0-1 loss) is not ...
3
votes
1answer
112 views

Why does my posterior distribution have probability values greater than 1? [duplicate]

I'm attempting to learn Bayesian modelling with PyMC, so I have been going through Cam Pilon Davidson's Probabilistic Programming for Hackers. I literally copied his code from chapter 1 and used my ...
2
votes
1answer
40 views

Summarization of text documents (legal domain) using deep learning techniques

I am referring to the site deeplearning.net on how to implement the deep learning architectures. I have read quite a few research papers on document summarization (both single document and ...
3
votes
1answer
101 views

How do I detect the number of distributions in a set of data?

I'm trying to detect subpopulations of data in a set of data. The problem is that the data may have more than one subpopulation and the subpopulations may not be normally distributed. For example, ...