Problems focusing on (but not necessarily limited to) using Python for statistical computation.

learn more… | top users | synonyms

0
votes
0answers
3 views

sklearn.linear_model.RandomizedLogisticRegression : Handle Categorical Value [migrated]

I want to use RandomizedLogisticRegression for selecting variable for my data set. But the problem is that, One of the feature in my data set is Gender. So it's ...
0
votes
0answers
68 views

Is this polynomial correct?

Disclaimer: I have no knowledge of stats. I fitted a polynomial to data points. I expected it to look like an exponential decreasing curve, but this seems to dip below zero, as well as the histogram ...
1
vote
1answer
35 views

Choosing reasonable parameters for a negative binomial distribution

My data is a list of observations and a count for each observation. The data is overdispersed, the mean is ~1,200 and the variance is ~18,000,000. I want to use a negative binomial model to assign ...
6
votes
1answer
54 views

Jenks Natural Breaks in Python: How to find the optimum number of breaks?

I found this Python implementation of the Jenks Natural Breaks algorithm and I could make it run on my Windows 7 machine. It is pretty fast and it finds the breaks in few time, considering the size of ...
0
votes
0answers
5 views

Semi supervised classification on multivariate data for outlier detection

I have a large multi dimensional unlabelled dataset of cars (price, mileage, horsepower, ...) for which I want to find outliers. I decided to use the sklearn OneClassSVM to build a decision boundary ...
0
votes
0answers
12 views

visualize timeseries dataset using python [closed]

How to visualize a specific time period in time series dataset? I'm beginner in python.I believe with matplotlib but I can't find syntax
0
votes
0answers
32 views

ParagraphVector od doc2vec for classification tasks [closed]

I am really interested to use doc2vec on classification task. I'dont understand how they create the featurevector for classification. I am trying to sum all words and paragraph id of the sentence to ...
1
vote
0answers
15 views

what could be the best method to find best scenario based on revenue(statistical analysis) in python?

I have a data set on which I want to do some statistical analysis. The sample data set is in a csv file and of following nature: ...
2
votes
0answers
29 views

How to increase the performance of random forest classifier?

I have a text classification task. These are the metrics for different languages at present: class1: 0.6823 class2: 0.7450 class3: 0.66 class4: 0.6719 How can I ...
0
votes
0answers
7 views

Average using grouping value in another vector (numpy / Python) [migrated]

I'd like to take the average of one vector based on grouping information in another vector. I've created a minimal example below based on averaging predictions for each user. How do I do that? ...
0
votes
0answers
6 views

apply fitted model to data and obtain loglikelihood [migrated]

I would like to do the following in Python, preferably with the statsmodels package (but if you know a solution with another package, I would be glad to hear about it as well): I have data ...
0
votes
0answers
19 views

How to measure convergence of data?

I am running a word-game simulations that assign values to words. In short, a computer is self-playing games of Scrabble and each move is recorded as (word, point value of the word). One value usually ...
0
votes
0answers
29 views

High dimension Categorical Decision trees, Python? [closed]

I'm just getting into the whole machine learning thing after much reading.. I'm trying out This contest on Women's health http://www.drivendata.org/competitions/6/ There are about 1,000 columns of ...
0
votes
1answer
61 views

Which library is the easiest to start with for Deep Learning

I am quite proficient with Machine Learning libraries and now want to get into Deep Learning. I am even quite comfortable with neural networks as far as understanding back propagation algorithm is ...
1
vote
0answers
18 views

Balancing Per-Class Accuracy of Multiclass Classifier

Suppose I have a multi-class classifier like Naive Bayes, k-Nearest Neighbors, Decision Trees, Random Forest, etc. The classifier maps a feature vector to (let's say) 3 classes: A, B, or C. My ...
0
votes
0answers
8 views

How do i get prediction accuracy when testing unknown data on a saved model in Scikit-Learn?

i have a model i have trained for binary classification, i now want to use it to predict unknown class elements. ...
0
votes
1answer
31 views

Adaptive Rejection Sampling in python?

Adaptive Rejection Sampling is a sampling technique for uni-dimensional variables that takes profit of the log-concavity of the probability density. It is used, for instance, in Gibbs sampling, when ...
0
votes
1answer
37 views

Using Adaptive Linear Neurons (Adalines) and Perceptrons for 0-1 class problems

I am wondering how to adjust the Adaline algorithm to classify the classes 0 and 1 instead of -1 and 1. I found a section in Neural Networks and Statistical Learning by Ke-Lin Du, M. N. S. Swamy ...
0
votes
0answers
4 views

Load text Files to python weka wrapper [migrated]

I've installed weka python wrapper on Windows 7. And I tried running the sample code: ...
1
vote
0answers
37 views

rnorm vs numpy.random.randn

For a regression example, I constructed some artificial data and ran ols, ...
4
votes
0answers
37 views

KDE vs. histogram [closed]

I plotted KDE (Gaussian kernel) and histogram of a dataset and the result is a bit confusing How can I explain lack of a spike around 0 in KDE? I assume that I don't understand something related to ...
0
votes
1answer
22 views

What are the good study materials on Association Rules?

I am looking to learn Association Rules, from basic level. I was looking for some good web based materials to start with. My objectives in the materials is to: (a) learn the aspect nicely from ...
1
vote
0answers
28 views

Sampling from Wishart: Any package or code that works with $dof>p-1$ instead of $dof \geq p$?

This is related to: rWishart: should be $dof>p-1$ or $dof \ge p$? Let df be the degrees of freedom of a Wishart distribution and $p$ the dimensions of its scale ...
1
vote
1answer
40 views

Kalman Filter Expectation Maximization

I'm not very familiar with the EM algorithm for the Kalman Filter. I've been using pykalman to do my analysis in Python. The package comes with a simple EM algo: ...
0
votes
0answers
6 views

Higher order spectral analysis in python [migrated]

Are there higher-order spectral analysis software which can be used from python? By higher-order spectral analysis I mean: (Cross) Bispectral analysis (Cross) ...
0
votes
0answers
23 views

unable to perform holt winters forecasting on time series data

I am trying to perform a holt Winters forecasting for future dates in python. There is a working code but it only predicts one point ahead so at the end of the series, I only one more point for the ...
1
vote
0answers
22 views

Python kNN vs. radius nearest neighbor regression

Python offers two nearest neighbor regressions: radius nearest neighbor and k-nearest neighbor. I'm trying to figure out a few things: 1. Under which circumstances would each be preferable? 2. How do ...
0
votes
0answers
24 views

divide among the use of statistical software package [duplicate]

Most of the application done by professionals, may it be in finance, economics or other industry is done in either R, SAS or Python. From what it looks like, most of the application in industries in ...
0
votes
0answers
10 views

What is an “example based” metric?

In the context of multilabel learning I came across several "example based" metrics, for instance example based Recall, example based Precision etc. (see here) I do know the concept of Recall etc. ...
0
votes
0answers
33 views

RandomForest Classifier With Very High Success Rate

I'm having a weird problem that may suprise you all. My classification rate is too high on my test set. I'm using scikit-learn packages, and I'm very suspicious of these classification rates, as they ...
0
votes
0answers
11 views

Gaussian Mixture Model with Custom Distance Metric

I have some 1D data that I want to cluster using Mixture of Gaussian. However, the data "wraps around" at two extremes. Specifically, I have a list of angles from $-\pi$ to $\pi$ and the data near two ...
0
votes
0answers
23 views

Penalized ML estimation of non-linear probit

I have a model of the form $P(y_i=1) = \Phi(\frac{w_1^{\beta'x_i}-w_2^{\beta'x_i}}{\sigma' x_i})$ where $y_i$ is a binary response, $\Phi$ is the normal CDF, $w_1$ and $w_2$ are non-negative ...
0
votes
1answer
37 views

PyMC consistently under estimating results found in paper. Possibly not sampling enough?

I have been trying to build confidence in (my ability to correctly use) PyMC by working examples. Namely, I have been working on Chickering and Pearl 1997, and more specifically on their 'artificial' ...
0
votes
0answers
27 views

what's the best empirical macro/micro F1 score?

Theoretically it should be 1. In the following presentation it's said that "0.5 to 0.55 (micro) F1 score is best for multilabel classification problems" I tried to investigate this statement but ...
2
votes
2answers
79 views

Clustering a correlation matrix

I have a correlation matrix which states how every item is correlated to the other item. Hence for a N items, I already have a N*N correlation matrix. Using this correlation matrix how do I cluster ...
0
votes
0answers
9 views

What is the fastest way to get the document term matrix in scikit learn?

I am using scikit learn CountVectorizer on top of ~11K documents, each of size ~5000 words. It takes ~ 1 hour to generate the tdm (document term matrix). Is there a much faster way to generate the ...
2
votes
1answer
86 views

Kalman filter with control inputs in python?

i am trying to fit a simple kalman filter with input controls (in this case step input) in python. i am using filterpy (http://filterpy.readthedocs.org/). my code is: ...
0
votes
0answers
35 views

Why does Support Vector Regression slow down after several iterations?

TLDR: Why does SVR slow down on my machine after multiple runs in IPython/sklearn? I'm trying to grid-search to optimize some parameters in Support Vector Regression with a problem that has 2 ...
0
votes
1answer
41 views

Least-square fit with uneven distribution of data

I'd like to perform least-squares fit to data which is unevenly distributed on the x-axis. For example, if I was to bin the data, it would be something like x = 0~5: 10 data points x = 5~10: 20 ...
1
vote
0answers
16 views

How to find the probalities of a,b,c (mutually exclusive events) when i turn the problem into three binary classification problems?

I am using scikits linear logisitc regression to classify three events a,b and c. it works better (score) when i convert them into a binary classification model. such as: 1. M1 classifies a or b 2. ...
0
votes
0answers
46 views

Latent Semantic Analysis with automatically discovered priors - gensim

I have around 20,000 English words in a set, mostly nouns. Something like {berry,bloom,buddy,comic,front,mind,charlie,consultants,destination,enterprise,experts,lady,stores,weight,arms,autism,balloon} ...
0
votes
1answer
106 views

Implementation of nested cross-validation

I'm trying to figure out if my understanding of nested cross-validation is correct, therefore I wrote this toy example to see if I'm right: ...
3
votes
1answer
36 views

Likeness of brands using tweets - is Chi-square appropriate?

I'm trying to determine if brand x is more similar to brand y or brand z using tweets. I have a data set for the words that occurred in each brand's tweets, and I'm considering using the chi-square ...
3
votes
0answers
52 views

State of the Art versions of Generalized Additive Models

Generalized Additive Models [Tribshirani 86] was well received with over 1335 Citations. I am also aware of the popular version of GAM - the Multivariate Adaptive Regression Splines [MARS by Friedman ...
4
votes
2answers
82 views

Is the chi-squared test appropriate with many small counts in a 5x2 table?

I have two sample populations, A, and B, which are independent. ...
0
votes
1answer
48 views

Weekly data normalization - Python

I have a weekly dataset and I have to normalize this data. Data is something like this : ...
0
votes
1answer
54 views

Questions about weather prediction in scikit learn

Hello I am a high school student doing research on weather. I have a dataset that has four columns each labeled with time, pressure, and lat/long. I am confused on the cross validation process. What ...
0
votes
0answers
42 views

How to do ranking with scikit-learn random forest model

I have a training dataset that I've developed, that has the following format: ------------------------------ | User ID | Item | Label | ------------------------------ | 001 | umbrella | 0 ...
0
votes
1answer
51 views

Log posterior function in PYMC

my question concerns the logp function in the PYMC package in Python. Ultimately I want to calculate a quantity that goes by many names, namely the Bayes-factor/ evidence/ marginal-likelihood of the ...
1
vote
0answers
90 views

Prediction based on multiple time series - Python

I have 3 predictors and 1 variable that represents ground truth. They all are linked time series. My purpose is with the 3 predictors to try to forecast the ground truth data. For example : ...