Problems focusing on (but not necessarily limited to) using Python for statistical computation.

learn more… | top users | synonyms

1
vote
1answer
9 views

Time Series Anomaly Detection with Python

I need to implement anomaly detection on several time-series datasets. I've never done this before and was hoping for some advice. I'm very comfortable with python, so I would prefer the solution be ...
0
votes
0answers
6 views

Slow Lasso Performance Using sklearn

I am trying to use scikit-learn's LassoCV and/or ElasticNetCV functions to model a dataset with a large (>800) number of ...
0
votes
0answers
22 views

Which naive Bayes?

I am attempting to use a naïve Bayes classifier in python (using scikit-learn), with two examples. The first example has 6 classes and 2 hypotheses, the 2nd example has 2 classes and 6 hypotheses. ...
1
vote
0answers
18 views

A log-normal distribution in Python

I have seen several questions in stackoverflow regarding how to fit a log-normal distribution. Still there are two clarifications that I need known. I have a ...
0
votes
0answers
10 views

Weka SMOreg and LIBSVM with linear kernel problems

I want to test a dataset in weka using either LIBSVM with an e-SVR or SMOreg for regression. I also choose a linear kernel in both (in SMOreg i use an exponent=1 in a non normalized polykernel). ...
0
votes
0answers
15 views

Inverse Box-Cox transform in Python

I am using SciPy's boxcox function to perform a Box-Cox transformation on a continuous variable. ...
1
vote
0answers
65 views

Any advice on how to improve my accuracy rate in text classification?

I'm trying to do a text classification task. Here are some specs: Context file size = 1M+ documents already labeled Number of top-labels = 17 Number of sub-labels = around 130 Each document is ...
4
votes
1answer
56 views

How to interpret autocorrelation plot in MCMC

I am getting familiar with Bayesian statistics by reading the book Doing Bayesian Data Analysis, by John K. Kruschke also known as the "puppy book". In chapter 9, hierarchical models are introduced ...
3
votes
0answers
25 views

Fitting Multivariate Bernoulli distribution

I want to fit a model to a number of observations, each of them being a k-dimensional binary vector $(x_1, x_2, ..., x_k)$ where $x_i \in \{0,1\}$. Naturally I would like to fit a multivariate ...
0
votes
1answer
28 views

what does a negative logloss value indicate

I am using logloss python function provided here and I am getting results as -2.99 when I use a machine learning algorithm on my dataset. What does that mean? The algorithm's predictions are bad (or) ...
0
votes
0answers
43 views

Python: Compute the Generalized Impulse Responses

I am using the tsa.vector_ar in Python, StatsModel Package and when I do a Forecast Error Variance Decomposition using the command ...
1
vote
0answers
19 views

Align two datasets in Python

I want to develop some python code to align datasets obtained by different instruments recording the same event. As an example, say I have two sets of measurements: ...
0
votes
0answers
33 views

How can you print the decision tree of a RandomForestClassifier

Recently, I have noticed that there is a method sklearn.tree.export_graphviz documented here. However, I do not know how I can apply it to a ...
1
vote
2answers
131 views

very high frequency time series analysis (seconds) and Forecasting (Python/R)

I have high frequency data (observations separated by seconds), which I'd like to analyse and eventually forecast short-term periods (1/5/10/15/60 min ahead) using ARIMA models. My whole data set is ...
0
votes
0answers
16 views

How do I get a classification report for my cross validated scores using sklean

I am running a logistic regression model using sklearn with 2 classes (1 and 0). Here is my code: ...
0
votes
0answers
16 views

How can I improve my sklearn logistic regression model

My objective is to classify sentences into useful (denote in boolean as 1) and not useful (denote in boolean as 0) categories. I have about 525 features where 300 features are the most frequent and ...
0
votes
0answers
53 views

Anyone use R or Python as statistical tool in industry? How to get coefficients of gradient booting models?

I tried gradient boosting models using both gbm in R and sklearn in Python. However, neither of them can provide the coefficients of the model. For gbm in R, it seems one can get the tree structure, ...
0
votes
1answer
28 views

what does the numbers in the classification report of sklearn mean?

I have below an example i pulled from sklearn 's sklearn.metrics.classification_report documentation. What i dont understand is why there are f1-score, precision and recall values for each class ...
1
vote
1answer
32 views

The likelihood that a time series is generated by certain ARMA(p,q) ?

I have a group ( only 20 of them, each one has 170 time pointers) of time series that I can consider as "GOOD", meaning, they have consistent statistical characteristics. I am not sure how they are ...
1
vote
1answer
20 views

Logistic-Regression: Prior correction at test time

Using sklean.linear_model.LogisticRegression for a binary classification problem. My classes are unbalanced. The positive class comprises about 20% of the training set. When fitting the model I use: ...
0
votes
0answers
9 views

Is it possible to create the data object in Python for SPSS [migrated]

I have a python script that is reading in an XML file into an array (in a CSV format I created). I'd like to be able to use that data directly instead of saving to a file. Is this possible? So it ...
0
votes
0answers
4 views

Printing nested lists using nested functions [migrated]

n = [1, [2, [3, [4, 5]]], [6, [7, [8, [9]], 10]]] In python, I want to write a nested function within the function 'print_list' that calls itself recursively for each sub-list. The nested function ...
4
votes
2answers
100 views

Tournament Plotting: Who is good against whom?

I would like to get insight into who is good against whom. Imagine a tournament setting* of 40 contestants and you're interested in seeing who is good against who in the top 10: how could we make this ...
2
votes
2answers
115 views

Different output for R lm() and python statsmodel OLS for linear regression

I'm exploring linear regressions in R and Python, and usually get the same results but this is an instance I do not. I added the sum of Agriculture and ...
0
votes
0answers
20 views

unary classification in PyBrain

I've just started using PyBrain for some data classification work, and I've gotten it working pretty well where I have data from all possible classes and I can train the network using all the classes. ...
0
votes
0answers
43 views

More recognizable Python implementation of Linear Discriminant Analysis?

I have been using scikit-learn's LDA implementation to do some experiments, and recently wanted to test out some modifications to the LDA derivation. I was looking at the Python implementation that ...
1
vote
0answers
41 views

Proper normalized cross-correlation

I'm confused with the widely used approach to compute the normalized cross-correlation which is something like this: Standardize the argument vectors (of equal length $n$). Slide one over the other ...
1
vote
0answers
35 views

Bias-variance decomposition with sklearn BaggingRegressor

There is an example given on the Scikit-Learn site that compares the bias-variance decomposition of the rmse of a single SVR model against a bagging ensemble. Unfortunately, the data is being ...
1
vote
0answers
18 views

How do I filter insignificant modes from a kernel density estimation

How do I detect the number of significant modes in a set of data? I have a set of 1 dimensional data where the number of modes are unknown. I'm familiar with python and scikit learn so I use the ...
0
votes
0answers
14 views

converting textblob object into DataFrame

Can we convert blob.noun_phrases into Pandas's DataFrame? The data type of blob.noun_phrases is class 'textblob.blob.WordList' type(blob.noun_phrases) class ...
-1
votes
0answers
33 views

tutorial on sampling methods and MC

I'm looking for good tutorials that cover the various sampling methods: simple sampling, MCMC, Gibbs Sampling, and Metropolis Hastings Algorithm. I barely know what is an MCMC. I would like to learn ...
0
votes
0answers
34 views

Plot 2D view of 3D joint ecdf with isolines

I have this matrix, histDict, which is a dictionary of data of variable X for each value of variable ...
4
votes
1answer
121 views

How to apply a Gaussian radial basis function kernel PCA to nonlinear data?

I have an assignment to implement a Gaussian radial basis function-kernel principal component analysis (RBF-kernel PCA) and have some challenges here. It would be great if someone could point me to ...
0
votes
0answers
24 views

Python module request: Spectral density estimation for multivariate time series

I have worked with scipy.signal.welch and spectrum.pptm to calculate power spectral density with Welch and Multitaper methods. However as far as I can see these functions are meant for one dimensional ...
0
votes
0answers
33 views

Time-series detection algorithm for multi-seasonal data using Python

My data: I have two seasonal patterns in my hourly data... daily and weekly. For example... each day in my dataset has roughly the same shape based on hour of the day. However, certain days like ...
2
votes
1answer
44 views

PYMC Confusion: are observed nodes fixed or stochastic?

I've been trying to gain a better understanding of factor potentials in PYMC. In reading this article by Cam Davidson-Pilon on Yhat, I got confused about how observed nodes are understood by PYMC. ...
0
votes
0answers
17 views

Choosing between different methods when the first one raises error message for linear regression

I have a linear regression problem $$Ax=b$$ My initial approach that helped to solve some of my questions was using SVD and obtaining the chi-square and some other values that I am interested but it ...
1
vote
0answers
48 views

Optimization in R vs Python, constrained, unconstrained and automatic differentiation?

I am an economics/stat guy who uses quite a bit of optimization (maximum likelihood, simulated maximum likelihood), constrained optimization (mathematical programming w/ equilibrium conditions), ...
1
vote
1answer
38 views

How to sample from “biased” binomial distribution (ideally in python/numpy)

I need to sample a variable from a distribution that's like a binomial distribution except with a "bias", I'm not sure what it may be called: $p(X=k)$ is proportional to $k.B(n,p)(k)$ where $B(n,p)$ ...
0
votes
1answer
35 views

Autoclass in R/Python? [closed]

Are there any packages that implement the Autoclass/ Naive Bayes Clustering algorithm in R or Python? Alternatively, what are some other clustering algorithms that can handle both categorical and ...
0
votes
0answers
14 views

PyMC: Which parameter is passed to the model by sampler? How to find the value passed by custom sampler?

I am relatively new for PyMC and as a whole I have some less knowledge in probability. As far as I understand MCMC algorithm, its main task is to propose samples of parameters to the model (in ...
0
votes
1answer
70 views

Defining the Stochastic and Deterministic variables with pymc3

I am trying to use write my own stochastic and deterministic variables with pymc3, but old ...
0
votes
0answers
59 views

Numerical estimation of MLE in Python — normal distribution and gradient is close to zero away from the mean

I am exploring how to model a data set using normal distributions with both mean and variance defined as linear functions of independent variables. Something like $\mathcal{N} \sim \left (f(x), ...
0
votes
1answer
69 views

Best optimization package for employee scheduling problem? [closed]

I am looking to solve the optimization problem described below. Which optimization software package would be best suited for this, considering the requirements specified below? Requirements: 1) Can ...
1
vote
2answers
107 views

Time series - correlation and lag time

I am studying the correlation between a set of input variables and a response variable, price. These are all in time series. 1) Is it necessary that I smooth out the curve where the input variable is ...
3
votes
1answer
61 views

Standard Deviation of an Exponentially-weighted Mean

I wrote a simple function in Python to calculate the exponentially weighted mean: ...
1
vote
1answer
58 views

How do I perform this complicated ANOVA type analysis in R?

I have the following type of dataset: | | | variable_r | subject | gender | age_group | Cond_1 | Cond_2 | ...
0
votes
0answers
31 views

What library in Python implements a Logistic Model Tree?

I am looking for something that implements a an algorithm along the lines of Landwehr et al.'s 2003 "Logistic Model Trees". I've found LMT() in the RWeka package implements the algorithm in R, but am ...
1
vote
0answers
39 views

Test means of populations with lots of zeros, is sampling the way to go?

I am trying to test whether means of two populations with lots of zeros are different. Here is the following python code example: ...
0
votes
1answer
72 views

Multivariate Linear Regression in Python

How to compute the overall standard error of a linear regression model using Python? Which library should I use? I am looking for something like this, however, I can't see how to get the overall ...