Problems focusing on (but not necessarily limited to) using Python for statistical computation.

learn more… | top users | synonyms

0
votes
0answers
7 views

NMSE - division by zero

I am using the normalized mean square error function in the Python Oger Toolbox which is defined as: ...
1
vote
0answers
27 views

What are the best packages for Image Processing in R? [on hold]

I just have started working on an image processing and classification problem. I am familiar with both R and Python. But I am not much experienced in scikit-learn and scikit-image libraries in Python. ...
0
votes
0answers
6 views

Train model based on correlations

I have a dataset of all trains in my country for a period in time, in a MySQL database. The form of this data is the following: ...
0
votes
0answers
10 views

Co-ranking matrices for dimensionality reduction [on hold]

I'm not sure if this question is better suited to stack overflow or here but here goes. I've been trying to implement ranking and co-ranking matrices based on this paper (section III). ...
1
vote
1answer
33 views
+50

Compare performance of 2 models

I have a dataset which I have split into 3 parts: a training set, a cross-validation set and a test set. I have used the training set and cross-validation set to train 2 models. For this, I have taken ...
0
votes
0answers
11 views

Fitness sharing in DEAP

Is there any way to implement fitness sharing/niching using DEAP? Specifically I'm looking for an implementation of the method defined here (Goldberg's fitness sharing) on page 98 of the pdf. If you ...
0
votes
1answer
35 views

Difference between statsmodel OLS and scikit linear regression

I have a question about two different methods from different libraries which seems doing same job. I am trying to make linear regression model. Here is the code which I using statsmodel library with ...
0
votes
0answers
10 views

Studentized range statistic (q*) in Python Scipy [on hold]

I do not mean to spam, but i do not know if different stackexchange communities interface with each other, but i am wondering about the following question ...
0
votes
0answers
16 views

Adding a continuous to logistic regression based on TF-IDF

My train dataset contains blog posts. I have an excerpt from a post, its total length in words and an arbitrary "Good" binary variable: ...
3
votes
1answer
50 views

PyMC3 Implementation of Probabilistic Matrix Factorization (PMF): MAP produces all 0s

I've started working with pymc3 over the past few days, and after getting a feel for the basics, I've tried implementing the Probabilistic Matrix Factorization model. For validation, I use a subset ...
1
vote
0answers
30 views

Sample time series to equal interval

I have data with timestamp and associated values. time interval between two consecutive data is not constant. How to standardize the the time series and associated value ? eg- Input data is Timestamp ...
0
votes
0answers
50 views

How good a fit is my linear regression - really?

So I've made a linear regression of my two variables using pythons np.polyfit. This is my code: ...
1
vote
1answer
19 views

Different probability values when using DecisionTreeClassifier and RandomForestClassifier

I'm studying the Random Forests and I made a little example to validate my knowledge. I create two classifiers, one with the DecisionTreeClassifier and another with RandomForestClassifier. After I ...
1
vote
0answers
40 views

Interpreting Granger Causality F-test

This question is a bit basic (I reviewed the previous postings on similar subjects, but still need help with this). Thanks in advance for any answer. The question is if A & B are two time-series ...
1
vote
1answer
23 views

How to interpret + and - precisions and recalls?

I understand the general calculation and concept of precision and recall. But when I am trying to predict people's ethnicity using some feature, say for example, predicting a binary class Chinese vs ...
0
votes
0answers
15 views

What is the residuals return in python scipy? [closed]

I use a python library (scipy) to do a polynomial fit, and one of the returns is a number for residual (like, 3.17389767e+08 from an example I'm working with), where I do: ...
0
votes
1answer
347 views

Why probability distribution function gives “~.40” probability when it should have been 1.0? [duplicate]

I am following code given here- http://www.bigdataexaminer.com/how-to-implement-these-5-powerful-probability-distributions-in-python/ Under "Normal Distribution" section, the graph peaks at .40 when ...
0
votes
1answer
25 views

OLS regression user defined function in Python

Is there a way to handle complex functions for OLS regression in Python? For example, if my function is $y = a - bx^{c} + e^{dx}$, then how I can use a Python library to estimate $a,b,c$ and $d$? i ...
1
vote
0answers
36 views

pyMC: implementing a joint distribution model

I'm attempting to model a multi-modal distribution that's affected by two separate distributions in pyMC and am having trouble implementing a joint or conditional distribution. Suppose I have N data ...
0
votes
0answers
12 views

feature slection in random forest in python

I have a dataset consisting of 24 numeric features and about 7000 rows, i am applying random forest to get the binary classification, So please tell me how to find only the relevant features to get ...
0
votes
0answers
20 views

Bandwidth and Sparsity in Quantile Regression

I am struggling to understand what bandwidth and sparsity mean in the context of quantile regressions and how they relate to the pseudo r-squared. This is what I get: ...
2
votes
2answers
122 views

What are the options when statisticians who only know R need more computing speed

Speed isn't a concern for many statistics projects but sometimes it is, for example MCMC. Assume that hardware improvement is not an option (looking for relatively free solutions that produce orders ...
0
votes
0answers
28 views

statsmodels: quantreg convergence cycle warning

I am getting the same Convergence cycle detected warning running a quantile regression with ...
0
votes
0answers
12 views

Testing of the Grassberger-Procaccia algorithm

I've wrote the implementation of Grassberger-Procaccia algorithm in python. It is a method of reconstruction of properties of dynamical system by analysis of single realization. I would like to test ...
1
vote
0answers
27 views

PyMC structure learning

Is it possible to learn the structure of a Bayesian network using PyMC? I have ~100 examples and all features are discrete. Here is an example of causal modeling, but the structure is already known. ...
0
votes
0answers
22 views

libpgm set root in Bayesian network

I'm using libpgm to learn the structure of a discrete Bayesian network given data. How do I set the root before calling discrete_estimatebn that will return the ...
0
votes
0answers
36 views

ARMA Model fitting using statsmodels.tsa.ARMA()

Two questions. 1.) When I use the statsmodels.tsa.ARMA() module, I enter my parameters and fit a model as follows: ...
0
votes
0answers
3 views

sklearn.linear_model.RandomizedLogisticRegression : Handle Categorical Value [migrated]

I want to use RandomizedLogisticRegression for selecting variable for my data set. But the problem is that, One of the feature in my data set is Gender. So it's ...
0
votes
0answers
75 views

Is this polynomial correct?

Disclaimer: I have no knowledge of stats. I fitted a polynomial to data points. I expected it to look like an exponential decreasing curve, but this seems to dip below zero, as well as the histogram ...
1
vote
1answer
50 views

Choosing reasonable parameters for a negative binomial distribution

My data is a list of observations and a count for each observation. The data is overdispersed, the mean is ~1,200 and the variance is ~18,000,000. I want to use a negative binomial model to assign ...
6
votes
1answer
93 views

Jenks Natural Breaks in Python: How to find the optimum number of breaks?

I found this Python implementation of the Jenks Natural Breaks algorithm and I could make it run on my Windows 7 machine. It is pretty fast and it finds the breaks in few time, considering the size of ...
0
votes
0answers
5 views

Semi supervised classification on multivariate data for outlier detection

I have a large multi dimensional unlabelled dataset of cars (price, mileage, horsepower, ...) for which I want to find outliers. I decided to use the sklearn OneClassSVM to build a decision boundary ...
1
vote
0answers
16 views

what could be the best method to find best scenario based on revenue(statistical analysis) in python?

I have a data set on which I want to do some statistical analysis. The sample data set is in a csv file and of following nature: ...
2
votes
0answers
41 views

How to increase the performance of random forest classifier?

I have a text classification task. These are the metrics for different languages at present: class1: 0.6823 class2: 0.7450 class3: 0.66 class4: 0.6719 How can I ...
0
votes
0answers
7 views

Average using grouping value in another vector (numpy / Python) [migrated]

I'd like to take the average of one vector based on grouping information in another vector. I've created a minimal example below based on averaging predictions for each user. How do I do that? ...
0
votes
0answers
6 views

apply fitted model to data and obtain loglikelihood [migrated]

I would like to do the following in Python, preferably with the statsmodels package (but if you know a solution with another package, I would be glad to hear about it as well): I have data ...
0
votes
0answers
19 views

How to measure convergence of data?

I am running a word-game simulations that assign values to words. In short, a computer is self-playing games of Scrabble and each move is recorded as (word, point value of the word). One value usually ...
1
vote
1answer
80 views

Which library is the easiest to start with for Deep Learning

I am quite proficient with Machine Learning libraries and now want to get into Deep Learning. I am even quite comfortable with neural networks as far as understanding back propagation algorithm is ...
1
vote
0answers
27 views

Balancing Per-Class Accuracy of Multiclass Classifier

Suppose I have a multi-class classifier like Naive Bayes, k-Nearest Neighbors, Decision Trees, Random Forest, etc. The classifier maps a feature vector to (let's say) 3 classes: A, B, or C. My ...
0
votes
0answers
10 views

How do i get prediction accuracy when testing unknown data on a saved model in Scikit-Learn?

i have a model i have trained for binary classification, i now want to use it to predict unknown class elements. ...
0
votes
1answer
36 views

Adaptive Rejection Sampling in python?

Adaptive Rejection Sampling is a sampling technique for uni-dimensional variables that takes profit of the log-concavity of the probability density. It is used, for instance, in Gibbs sampling, when ...
0
votes
1answer
45 views

Using Adaptive Linear Neurons (Adalines) and Perceptrons for 0-1 class problems

I am wondering how to adjust the Adaline algorithm to classify the classes 0 and 1 instead of -1 and 1. I found a section in Neural Networks and Statistical Learning by Ke-Lin Du, M. N. S. Swamy ...
1
vote
0answers
41 views

rnorm vs numpy.random.randn

For a regression example, I constructed some artificial data and ran ols, ...
4
votes
0answers
45 views

KDE vs. histogram [closed]

I plotted KDE (Gaussian kernel) and histogram of a dataset and the result is a bit confusing How can I explain lack of a spike around 0 in KDE? I assume that I don't understand something related to ...
0
votes
1answer
26 views

What are the good study materials on Association Rules?

I am looking to learn Association Rules, from basic level. I was looking for some good web based materials to start with. My objectives in the materials is to: (a) learn the aspect nicely from ...
1
vote
0answers
30 views

Sampling from Wishart: Any package or code that works with $dof>p-1$ instead of $dof \geq p$?

This is related to: rWishart: should be $dof>p-1$ or $dof \ge p$? Let df be the degrees of freedom of a Wishart distribution and $p$ the dimensions of its scale ...
1
vote
1answer
46 views

Kalman Filter Expectation Maximization

I'm not very familiar with the EM algorithm for the Kalman Filter. I've been using pykalman to do my analysis in Python. The package comes with a simple EM algo: ...
0
votes
0answers
24 views

unable to perform holt winters forecasting on time series data

I am trying to perform a holt Winters forecasting for future dates in python. There is a working code but it only predicts one point ahead so at the end of the series, I only one more point for the ...
1
vote
0answers
33 views

Python kNN vs. radius nearest neighbor regression

Python offers two nearest neighbor regressions: radius nearest neighbor and k-nearest neighbor. I'm trying to figure out a few things: 1. Under which circumstances would each be preferable? 2. How do ...
0
votes
0answers
24 views

divide among the use of statistical software package [duplicate]

Most of the application done by professionals, may it be in finance, economics or other industry is done in either R, SAS or Python. From what it looks like, most of the application in industries in ...