Problems focusing on (but not necessarily limited to) using Python for statistical computation.
1
vote
0answers
5 views
Python module for change point analysis
I'm looking for a Python module that performs a change-point analysis on a time-series. There are a number of different algorithms and I'd like to explore the efficacy of some of them without having ...
0
votes
0answers
40 views
Is there any Python equivalent of R's biglm?
I have used biglm in R and found it very useful. Now I need the same type of functionality in python. Any ideas? I have seen that patsy/statsmodels has an ...
4
votes
2answers
63 views
Understanding bootstrap method for confidence interval of correlation coefficients
Please correct me where I'm wrong:
My understanding of bootstrapping is that it is a way to estimate the distribution of some statistic (mean, standard error, Pearson's correlation coeff, etc), ...
1
vote
1answer
54 views
Regression model for periodic data
Moved from Stack Overflow
Could someone help me to find adequate regression model for my data?
I tried to find one by changing the model and initial approximation (ln 15-16) in this simple Python ...
1
vote
1answer
22 views
How do I identify which parameters are correlated to binary output?
I'm trying to program an SVM in Python to categorize proteins as "Go" or "No-Go".
I have a list of about 30 proteins, each with ~ 100 columns of structure-related parameters and 1 column of "True" or ...
-1
votes
0answers
25 views
How to plot a scatter diagram using rpy2 in python [closed]
I have a dataset like below in dictionary format,
...
-1
votes
1answer
36 views
0
votes
1answer
83 views
How to handle Regression data thats not linear
I'm new to stats and am using Python 2.7 to fit a regression model (Random Forest). When I plot the percentile plot of the prices before and after a log ...
0
votes
0answers
47 views
When to Log/Exp your Variables when performing Linear Regression?
I'm doing regression using Random Forests for predicting prices based on several attributes. Code is written in Python using Scikit-learn.
How do you decide whether you should transform your ...
0
votes
0answers
21 views
What is the way to represent factor variables in scikit-learn while using Random Forests?
I am solving a classification problem using Random Forests. For that I have decided to use Python library scikit-learn. But I am new to both Random Forest algorithm and this tool. My data contains ...
1
vote
0answers
13 views
Mean absolute percentage error (MAPE) in Scikit-learn
How can we calculate the Mean absolute percentage error (MAPE) of our predictions using Python and scikit-learn?
From the docs, we have only these 4 metric functions for Regressions:
...
0
votes
0answers
17 views
Python statsmodels OLS: how to save learned model to file [migrated]
I am trying to learn an ordinary least squares model using Python's statsmodels library, as described here.
sm.OLS.fit() returns the learned model. Is there a way to save it to the file and reload ...
0
votes
0answers
28 views
OLS in Python with Dummy Variables - Best Solution? [migrated]
I have a problem I am trying to solve in Python, and I have found multiple solutions (I think) but I am trying to figure out which one is the best. I am hoping to choose libraries that will be ...
0
votes
1answer
67 views
Why is Hedonic Regression used instead of Linear Regression
Why is Hedonic Regression used (especially in housing prices) instead of Linear Regression?
There do not seem to be any libraries in Python (and R) for Hedonic regression, is it too niched a ...
0
votes
0answers
20 views
Smoothing of log-distributed periodogram
I use the lomb-scargle periodogram to output information about chemical species in distinct time periods. This produces a distribution that is skewed heavily, with the majority of points and variance ...
0
votes
2answers
126 views
How to interpret p-value of Kolmogorov-Smirnov test (python)?
I have Two samples that I want to test (using python) if they are drawn from the same distribution. To do that I use the statistical function ks_2samp from scipy.stats. It returns 2 values and I find ...
3
votes
0answers
41 views
Link Anomaly Detection in Temporal Network
I came across this paper that uses link anomaly detection to predict trending topics, and I found it incredibly intriguing: The paper is "Discovering Emerging Topics in Social Streams via Link Anomaly ...
-1
votes
0answers
21 views
Results of TF-IDF Transformer [closed]
When using scikit-learn's tf-idf transformer, what does the values printed out here means? I printed out the col 2256 and the results include ...
3
votes
2answers
95 views
Chart/visual for negative binomial regression
My wife is presenting a study at a conference poster session. She has a correlation of sorts [1] where kids' self-assessment of asthma ("I felt a lot better" or "I felt a little better or not better") ...
1
vote
1answer
144 views
1sigma error on the mean
My x and y data (n) have a non normal distribution. I'm measuring the mean of the distribution (I'm using Python):
x_m=np.mean(x)
y_m=np.mean(y)
I would like to ...
0
votes
0answers
53 views
10-Fold Cross Validation for Naive Bayes
I created a training set and train my Naives Bayes Classifier which had 3 classes (pos, neu, neg).
...
3
votes
7answers
258 views
Visualizing high dimensional data
I have samples of two classes which are vectors in high dimensional space and I want to plot them in 2D or 3D.
I know about dimensionality reduction techniques, but I need a really simple and easy to ...
1
vote
2answers
145 views
PyMC: how can I define a function of two stochastic variables, with no closed-form distribution?
I'm learning PyMC and basically I have a random variable $Z = X + Y$ where (say) $X \sim \mathrm{Normal}(\theta_X)$ and $Y \sim \mathrm{Lognormal}(\theta_Y)$ and $Z$ has no simple closed-form ...
0
votes
0answers
48 views
Pearson Correlation Coefficient and gene expression matrix
This is a gene expression matrix. The first column are gene ids, the latter values are normalized gene expression values for controls (first 3 cols) and disease state (last 3 cols).
...
0
votes
0answers
26 views
How do you generate synthetic sparse binary linguistic data for logistic regression?
I am trying to generate synthetic linguistic data (boolean features) to fit a binary logistic regression model. This is similar to 8260771 on StackOverflow and several synthetic data questions on this ...
0
votes
0answers
55 views
How to find trends/correlations in a simple 2D dataset?
I have a dataset of approximately 100 objects. For each object, I have made measurements and have a handful of values (typically less than 10).
I also know several physical properties of my objects. ...
2
votes
1answer
41 views
Filtering outliers from geo-spatial-temporal log
I have downloaded my Latitude location history from Google for the time of about three years and now I'd like to, for starters, visualize where I've been.
It turns out that the history contains some ...
0
votes
0answers
37 views
Web Scraping : How do I maximize information scraped? [closed]
Full Disclosure:
I've also posted this problem on StackOverflow.
Here's the problem:
Users register for a site and can pick one of 8 job categories, or choose to skip this step. I want to classify ...
2
votes
0answers
46 views
How should I distribute a classifier to customers?
When consulting, I often do my exploratory analysis and prototyping in R, and deliver results on the initial dataset to the client. The client wants to use the trained classifier in a production ...
0
votes
1answer
45 views
Setting PyMC model with two different time series data
I've been working with PyMC for a bit, and am stuck on this one. I see example on fitting time series, in the tutorial and others like:
...
0
votes
0answers
47 views
ProjectTemplate equivalent for Python?
Is there ProjectTemplate equivalent in python? I.e. a template that defines and helps carry out an efficient workflow for a statistical project?
If there is nothing specifically designed for ...
0
votes
0answers
43 views
Weighting and time series with machine learning
I'm trying to produce a model to predict the price of a product on the basis of several factors effecting previous time-stamped sales. I am certain that older sales are less relevant to the prediction ...
2
votes
0answers
72 views
detecting circadian rhythm in a time series
I have a sensor that can detect minute changes in distance. It produces a time series.
I would like to point it at people and detect things like their sleeping pattern. How would one build a system ...
1
vote
1answer
27 views
Assigning a sample set to one of several populations based on composition
I have a problem that seems tractable, which I will attempt to describe with classic balls and buckets probability nomenclature.
Imagine someone brings you a bag of colored balls. This sample was ...
0
votes
0answers
25 views
Comparing datasets and scoring
I have 7 datasets of similar experiments each containing values of ~2000 parameters. There are a lot of missing values in few experiments. I want to score the datasets in such way that some 30 known ...
0
votes
1answer
91 views
Python module to compute stationary distribution of Markov chain
Is there any module for Python that computes the stationary distribution of a Markov chain, given the generator matrix?
Thanks for any help!
2
votes
1answer
173 views
How to use Box-Muller transform to generate n-dimensional normal random variables
I'm trying to generate random variables. I read about Box-Muller transform which is a way to generate a pair of normal variables, 2-d normal distrubution. But how do I expand that transform to ...
0
votes
0answers
79 views
Solution for long-term electricity prices forecasting
I'm trying to adopt a solution for long-term electricity annual prices forecasting (depending on past electricity prices, past oil prices, past consumption data, etc.)
I'm considering some solutions:
...
2
votes
1answer
164 views
Simple text classifier: classification taking forever?
I work for a small tech startup, and I want to classify or users into demographics based on the domain of their email address. When users sign up to our site, they can enter a job category, or pick ...
1
vote
0answers
47 views
Plotting a histogram estimate together with an analytical solution of the mean and covariance parameters
How do I plot a histogram estimate of this marginal distribution p(x1):
...
0
votes
0answers
68 views
Drawing floating numbers with [0, 1] from uniform distribution by using numpy
I'm currently trying to draw floating numbers from a uniform distribution.
The Numpy provides numpy.random.uniform.
...
1
vote
1answer
50 views
Packages or libraries for multiple-output learning
I have a learning problem from $X$ to $Y$ where:
$X$ = $n$ input numeric vectors of $m$ dimensions
$Y$ = $n$ output numeric vectors of $k$ dimensions
In other words:
I am ...
2
votes
1answer
125 views
Simple machine learning: bot detection
I've been aching to get my feet wet with a machine learning project, and I've found one that should be relatively simple, and actually has non-negligible business value for my organization. The ...
2
votes
1answer
155 views
What does it mean to develop a proper implementation of the grammar of graphics?
I want to develop an implementation of the grammar of graphics in python, and I need to wrap my head around a few things. I am actually having some difficulties in understanding some of the book's ...
2
votes
1answer
393 views
PCA and SVD, R prcomp
I am trying to replicate Shalizi's NY Times PCA example found in his Advanced Data Analysis with an Elementary Point of View book. I found sample code and data here
...
1
vote
1answer
168 views
Maximum likelihood fitting of truncated, mixed, two population systems
this is a repost from math.stackexchange (I didn't know how to migrate the post, apologies). I was informed that this would be a better place to ask.
TLDR: I am trying to do maximum likelihood ...
5
votes
2answers
734 views
Pandas / Statsmodel / Scikits-learn
Are Pandas, Statsmodels and scikits-learn different implementations of machine learning/statistical operations, or are these complementary to one another?
Which of these has the most comprehensive ...
0
votes
0answers
80 views
Mixing categorial and continuous data in Naive Bayes classifier using scikit-learn
I'm using scikit-learn in Python to develop a classification algorithm to predict gender of a certain customers. Amongst others I want to use the Naive Bayes classifier but my problem is that I have a ...
1
vote
1answer
112 views
VAR vs STAR for space-time autoregression in Python
I want to use autoregressive model to build a predictor for some sets of spatio-temporal data. For example, I have historical traffic data (speeds at various segments of freeways). similarly, I have ...
6
votes
1answer
1k views
R vs Python for Data Analysis [duplicate]
Possible Duplicate:
Python as a statistics workbench
I am just starting out with data analysis and machine learning. From the books that I am reading/have read Python and R seem to be the ...
