Problems focusing on (but not necessarily limited to) using Python for statistical computation.

learn more… | top users | synonyms

0
votes
0answers
11 views

how to improve linear regression model [on hold]

i am working on a simple linear regression model for practicing in order to learn machine learning . my model runs correctly however it get a bad score which means it is a bad model so any advice for ...
0
votes
0answers
9 views

How to perform pattern identification using ML?

I have the following problem: An event, takes place at a determined day of the week, hour, and with a pre-defined format (movie, music concert, lecture (3 items). Based on exit polls we determine 3 ...
0
votes
0answers
17 views

Likelihood of hypothesis in live data

Bayes rule is $P(H|E)=\frac{P(H)P(E|H)}{P(E)}$ I have a prior distribution from categorical data prior={'a':0.2,'b':0.6,'c':0.1,'d':0.1} Which forms my ...
0
votes
0answers
5 views

Show mean in the box plot in python? [migrated]

I am new to Matplotlib, and as I am learning how to draw box plot in python, I was wondering if there is a way to show mean in the box plots? Below is my code.. ...
1
vote
1answer
35 views

Python Astronomy Censored Data in Lifelines

I am trying to find a correlation between a given data set containing redshifts and turnover frequencies (I have a list of 320 galaxies, and the redshift and turnover frequency (a turnover frequency ...
0
votes
0answers
10 views

NMSE - division by zero

I am using the normalized mean square error function in the Python Oger Toolbox which is defined as: ...
1
vote
0answers
31 views

What are the best packages for Image Processing in R? [closed]

I just have started working on an image processing and classification problem. I am familiar with both R and Python. But I am not much experienced in scikit-learn and scikit-image libraries in Python. ...
0
votes
0answers
7 views

Train model based on correlations

I have a dataset of all trains in my country for a period in time, in a MySQL database. The form of this data is the following: ...
0
votes
0answers
13 views

Co-ranking matrices for dimensionality reduction [closed]

I'm not sure if this question is better suited to stack overflow or here but here goes. I've been trying to implement ranking and co-ranking matrices based on this paper (section III). ...
1
vote
3answers
91 views
+50

Compare performance of 2 models

I have a dataset which I have split into 3 parts: a training set, a cross-validation set and a test set. I have used the training set and cross-validation set to train 2 models. For this, I have taken ...
0
votes
0answers
13 views

Fitness sharing in DEAP

Is there any way to implement fitness sharing/niching using DEAP? Specifically I'm looking for an implementation of the method defined here (Goldberg's fitness sharing) on page 98 of the pdf. If you ...
0
votes
1answer
45 views

Difference between statsmodel OLS and scikit linear regression

I have a question about two different methods from different libraries which seems doing same job. I am trying to make linear regression model. Here is the code which I using statsmodel library with ...
0
votes
0answers
11 views

Studentized range statistic (q*) in Python Scipy [closed]

I do not mean to spam, but i do not know if different stackexchange communities interface with each other, but i am wondering about the following question ...
0
votes
0answers
17 views

Adding a continuous to logistic regression based on TF-IDF

My train dataset contains blog posts. I have an excerpt from a post, its total length in words and an arbitrary "Good" binary variable: ...
3
votes
1answer
67 views

PyMC3 Implementation of Probabilistic Matrix Factorization (PMF): MAP produces all 0s

I've started working with pymc3 over the past few days, and after getting a feel for the basics, I've tried implementing the Probabilistic Matrix Factorization model. For validation, I use a subset ...
1
vote
0answers
34 views

Sample time series to equal interval

I have data with timestamp and associated values. time interval between two consecutive data is not constant. How to standardize the the time series and associated value ? eg- Input data is Timestamp ...
0
votes
0answers
52 views

How good a fit is my linear regression - really?

So I've made a linear regression of my two variables using pythons np.polyfit. This is my code: ...
1
vote
1answer
25 views

Different probability values when using DecisionTreeClassifier and RandomForestClassifier

I'm studying the Random Forests and I made a little example to validate my knowledge. I create two classifiers, one with the DecisionTreeClassifier and another with RandomForestClassifier. After I ...
1
vote
0answers
45 views

Interpreting Granger Causality F-test

This question is a bit basic (I reviewed the previous postings on similar subjects, but still need help with this). Thanks in advance for any answer. The question is if A & B are two time-series ...
1
vote
1answer
23 views

How to interpret + and - precisions and recalls?

I understand the general calculation and concept of precision and recall. But when I am trying to predict people's ethnicity using some feature, say for example, predicting a binary class Chinese vs ...
0
votes
1answer
349 views

Why probability distribution function gives “~.40” probability when it should have been 1.0? [duplicate]

I am following code given here- http://www.bigdataexaminer.com/how-to-implement-these-5-powerful-probability-distributions-in-python/ Under "Normal Distribution" section, the graph peaks at .40 when ...
0
votes
1answer
27 views

OLS regression user defined function in Python

Is there a way to handle complex functions for OLS regression in Python? For example, if my function is $y = a - bx^{c} + e^{dx}$, then how I can use a Python library to estimate $a,b,c$ and $d$? i ...
1
vote
0answers
40 views

pyMC: implementing a joint distribution model

I'm attempting to model a multi-modal distribution that's affected by two separate distributions in pyMC and am having trouble implementing a joint or conditional distribution. Suppose I have N data ...
0
votes
0answers
13 views

feature slection in random forest in python

I have a dataset consisting of 24 numeric features and about 7000 rows, i am applying random forest to get the binary classification, So please tell me how to find only the relevant features to get ...
0
votes
0answers
21 views

Bandwidth and Sparsity in Quantile Regression

I am struggling to understand what bandwidth and sparsity mean in the context of quantile regressions and how they relate to the pseudo r-squared. This is what I get: ...
2
votes
2answers
124 views

What are the options when statisticians who only know R need more computing speed

Speed isn't a concern for many statistics projects but sometimes it is, for example MCMC. Assume that hardware improvement is not an option (looking for relatively free solutions that produce orders ...
0
votes
0answers
29 views

statsmodels: quantreg convergence cycle warning

I am getting the same Convergence cycle detected warning running a quantile regression with ...
0
votes
0answers
12 views

Testing of the Grassberger-Procaccia algorithm

I've wrote the implementation of Grassberger-Procaccia algorithm in python. It is a method of reconstruction of properties of dynamical system by analysis of single realization. I would like to test ...
1
vote
0answers
27 views

PyMC structure learning

Is it possible to learn the structure of a Bayesian network using PyMC? I have ~100 examples and all features are discrete. Here is an example of causal modeling, but the structure is already known. ...
0
votes
0answers
23 views

libpgm set root in Bayesian network

I'm using libpgm to learn the structure of a discrete Bayesian network given data. How do I set the root before calling discrete_estimatebn that will return the ...
0
votes
0answers
39 views

ARMA Model fitting using statsmodels.tsa.ARMA()

Two questions. 1.) When I use the statsmodels.tsa.ARMA() module, I enter my parameters and fit a model as follows: ...
0
votes
0answers
3 views

sklearn.linear_model.RandomizedLogisticRegression : Handle Categorical Value [migrated]

I want to use RandomizedLogisticRegression for selecting variable for my data set. But the problem is that, One of the feature in my data set is Gender. So it's ...
0
votes
0answers
78 views

Is this polynomial correct?

Disclaimer: I have no knowledge of stats. I fitted a polynomial to data points. I expected it to look like an exponential decreasing curve, but this seems to dip below zero, as well as the histogram ...
1
vote
1answer
55 views

Choosing reasonable parameters for a negative binomial distribution

My data is a list of observations and a count for each observation. The data is overdispersed, the mean is ~1,200 and the variance is ~18,000,000. I want to use a negative binomial model to assign ...
6
votes
1answer
112 views

Jenks Natural Breaks in Python: How to find the optimum number of breaks?

I found this Python implementation of the Jenks Natural Breaks algorithm and I could make it run on my Windows 7 machine. It is pretty fast and it finds the breaks in few time, considering the size of ...
0
votes
0answers
5 views

Semi supervised classification on multivariate data for outlier detection

I have a large multi dimensional unlabelled dataset of cars (price, mileage, horsepower, ...) for which I want to find outliers. I decided to use the sklearn OneClassSVM to build a decision boundary ...
1
vote
0answers
16 views

what could be the best method to find best scenario based on revenue(statistical analysis) in python?

I have a data set on which I want to do some statistical analysis. The sample data set is in a csv file and of following nature: ...
2
votes
0answers
43 views

How to increase the performance of random forest classifier?

I have a text classification task. These are the metrics for different languages at present: class1: 0.6823 class2: 0.7450 class3: 0.66 class4: 0.6719 How can I ...
0
votes
0answers
20 views

How to measure convergence of data?

I am running a word-game simulations that assign values to words. In short, a computer is self-playing games of Scrabble and each move is recorded as (word, point value of the word). One value usually ...
2
votes
1answer
83 views

Which library is the easiest to start with for Deep Learning

I am quite proficient with Machine Learning libraries and now want to get into Deep Learning. I am even quite comfortable with neural networks as far as understanding back propagation algorithm is ...
1
vote
0answers
32 views

Balancing Per-Class Accuracy of Multiclass Classifier

Suppose I have a multi-class classifier like Naive Bayes, k-Nearest Neighbors, Decision Trees, Random Forest, etc. The classifier maps a feature vector to (let's say) 3 classes: A, B, or C. My ...
0
votes
0answers
11 views

How do i get prediction accuracy when testing unknown data on a saved model in Scikit-Learn?

i have a model i have trained for binary classification, i now want to use it to predict unknown class elements. ...
0
votes
1answer
39 views

Adaptive Rejection Sampling in python?

Adaptive Rejection Sampling is a sampling technique for uni-dimensional variables that takes profit of the log-concavity of the probability density. It is used, for instance, in Gibbs sampling, when ...
0
votes
1answer
48 views

Using Adaptive Linear Neurons (Adalines) and Perceptrons for 0-1 class problems

I am wondering how to adjust the Adaline algorithm to classify the classes 0 and 1 instead of -1 and 1. I found a section in Neural Networks and Statistical Learning by Ke-Lin Du, M. N. S. Swamy ...
1
vote
0answers
42 views

rnorm vs numpy.random.randn

For a regression example, I constructed some artificial data and ran ols, ...
4
votes
0answers
50 views

KDE vs. histogram [closed]

I plotted KDE (Gaussian kernel) and histogram of a dataset and the result is a bit confusing How can I explain lack of a spike around 0 in KDE? I assume that I don't understand something related to ...
0
votes
1answer
26 views

What are the good study materials on Association Rules?

I am looking to learn Association Rules, from basic level. I was looking for some good web based materials to start with. My objectives in the materials is to: (a) learn the aspect nicely from ...
1
vote
0answers
30 views

Sampling from Wishart: Any package or code that works with $dof>p-1$ instead of $dof \geq p$?

This is related to: rWishart: should be $dof>p-1$ or $dof \ge p$? Let df be the degrees of freedom of a Wishart distribution and $p$ the dimensions of its scale ...
1
vote
1answer
53 views

Kalman Filter Expectation Maximization

I'm not very familiar with the EM algorithm for the Kalman Filter. I've been using pykalman to do my analysis in Python. The package comes with a simple EM algo: ...
0
votes
0answers
25 views

unable to perform holt winters forecasting on time series data

I am trying to perform a holt Winters forecasting for future dates in python. There is a working code but it only predicts one point ahead so at the end of the series, I only one more point for the ...