Problems focusing on (but not necessarily limited to) using Python for statistical computation.

learn more… | top users | synonyms

2
votes
0answers
20 views

Why does my train data not fall in confidence interval with scikit-learn Gaussian Process?

I have a dataset with 1-dimensional input and output and I am applying Gaussian Process regression to it using scikit-learn: ...
0
votes
0answers
10 views

How do i convert 30 categories into numbers for scikit [migrated]

I am new to statistics, Python, machine learning and Scikit-learn. However, I am trying this project where I have a CSV with 35 columns of student data. The first column is an ID which I think I can ...
0
votes
0answers
14 views

Numpy MemoryError calculating correlation matrix (88k rows / 150 cols) [on hold]

I am getting the following error trying to calculate the correlation matrix on a dataset with 88k rows and 150 columns using Pandas and Numpy. Ultimately I am trying to find associated features on a ...
0
votes
0answers
27 views
+50

Which Naive Bayes in the Wikipedia article “Naive Bayes spam filtering”?

In this 2006 paper, it discusses that there are many Naive Bayes algorithms: Spam Filtering with Naive Bayes – Which Naive Bayes? The paper states that binary Multinomial NB performs best. I coded ...
0
votes
0answers
21 views

How much memory do I need to filter through dictionaries in python of 5million? [on hold]

I am creating a program that has a lot of data. I am currently running it and its been 10 minutes and its still running probably about to stop it. Anyways, I am trying to make my computer faster at ...
1
vote
0answers
36 views

How would you modify your model according to that confusion matrix?

How would you proceed with your model, if the confusion matrix looks like on the picture below? Classes 2, 3 and 4 get misclassified a lot among each other. EDIT: The Confusion Matrix was created ...
3
votes
1answer
33 views

Method for a hypothesis testing non normal distribution number of retweets

I am a stats-beginner, Using pandas I am analysing a small dataset. There are 60 data-points, 22 of which are from Group A and 38 are from Group B. The dataset is made up of the number of retweets ...
1
vote
1answer
26 views

How to plot the OR function along with the decision boundary of a Perceptron?

I've written a small program that predicts correctly the OR function output. The problem is that when I try to plot the decision boundary, I don't know what to do. Should I plot the final weights?. ...
1
vote
1answer
53 views

How to generate the posterior predictive distribution for hierarchal model in PYMC3

See iPython notebook for full example The below stochastic node y_pred enables me to generate the posterior predictive distribution: ...
0
votes
0answers
2 views

Sparse Coding Library example in scikit-learn [migrated]

I just started learning about machine learning and using scikit-learn. But scikit-learn is a little bit difficult for me to use and understand completely what it is doing. So cloud you help me to ...
0
votes
0answers
6 views

I'm getting an error message using PyMc that I don't understand [migrated]

Here is my setup and error message: creating fake data ...
0
votes
0answers
20 views

Regression plot and function for: heavy-tailed probability distribution

I've got data points from a simulation as coordinates in a text-files like so: ...
-1
votes
0answers
24 views

Hierarchical clustering and sklearn [closed]

I would like to know if there is a reason why in sklearn library is not implemented a partitional solution for hierarchical clustering. I think that is possible to apply recursively a K-means ...
3
votes
2answers
152 views

Why is this time-series stationary?

I am using python for time-series analysis of count data and came across a problem where I have a time-series that to me looks non-stationary but the Augmented Dickey-Fuller test (implemented in ...
-1
votes
0answers
13 views

Clustering headlines without full text [closed]

I have a data-set with headlines of news. The number of topics is 100. Also I have a tf-idf file, which is not generated by me. I saw data-set, and I can determine something like 50 clusters. 50 - ...
0
votes
2answers
51 views

verified procedure for calculating gradient descent?

I'm wondering if there is a good procedure people are using for gradient descent that is pretty well validated--something like a package for R or Python, or generic code many people adapt. After ...
0
votes
2answers
28 views

Multiclass classification - Optimizing for recall in subset of classes

As part of a wider analysis, I'm seeking a method to reliably recall the outer margins (~10% each) of a data set. I don't mind if precision is lower as a result. I've separated the data into 3 ...
1
vote
0answers
31 views

Ordinal logistic regression in Python

I would like to run an ordinal logistic regression in Python - for a response variable with three levels and with a few explanatory factors. The statsmodels package ...
0
votes
0answers
35 views

How to do MANOVA in Python?

I can quickly do a MANOVA test in R using this code: ...
1
vote
1answer
26 views

Getting distance of points from decision boundary with linear SVM?

I posted this originally in Stack Overflow but realize it might be more of a statistics question. I am using SKLearn to run SVC on my data. ...
0
votes
0answers
17 views

help in coding decision tree in python [migrated]

I am not sure if this is the right place to post this, but I have been trying to code up a simple decision tree class for a while and am getting lost at various points. Specifically, I'm not sure ...
0
votes
2answers
91 views

From where I can learn to use scikit learn

I have read text about machine learning and I feel that I have gained sufficient knowledge that I can start applying them practically. I have programming experience in python so I want to learn how to ...
1
vote
0answers
356 views

CV (Curriculum Vitae) Recommender system - guidance

Please note that I am a total beginner with machine learning and artificial intelligence and also a novice with Python (I'm sure I have a very non-Pythonic way of writing code). I have a college ...
0
votes
0answers
37 views

What could cause a K-means clustering algorithm to converge into a single cluster?

I am currently writing a K-means clustering algorithm in Python, and I seem to have coded myself into a corner... I begin my algorithm with data sets assigned randomly to the appropriate number of K ...
1
vote
2answers
55 views

sklearn - overfitting problem

I'm looking for recommendations as to the best way forward for my current machine learning problem The outline of the problem and what I've done is as follows: I have 900+ trials of EEG data, where ...
1
vote
0answers
23 views

Optimal method for identifying correlation of a stock price and various moving average durations for Python

I am trying to find the optimal method in trying to find the relationship between daily stock price and rolling correlation of another stock. For example, let's say I have a hunch that there is a ...
4
votes
1answer
129 views

Understanding derp regression

After having a bit too much tea I decided to write a silly way of estimating the parameters of a single-feature linear regression. For starters, I avoided using techniques like gradient descent, ...
0
votes
0answers
14 views

Classification of array data?

I have a set of array data that falls into one of two bins. Right now, the classification is done manually, but I'd like to try to use a learning approach. The classification will depend on the size ...
0
votes
1answer
49 views

Regression of two data data sets of the same experiment

I have two data sets collected by different methods about the same phenomena, one 'bare' and other where I believe there is a constant value added (the same phenomena, with the data collected by a ...
0
votes
1answer
50 views

Increasing sample size of multivariate data with bootstrapping

I'm trying to perform variable selection methods on a small data set (approximately 100 observations with 14 predictors each). Is there any way to increase the sample size with bootstrapping? Are ...
0
votes
0answers
28 views

Sentence classification feature selection

My task is to classify sentences based on their meaning. More specifically, I would like to have one class for sentences indicating current possession ("The store has < product > in stock...") as ...
3
votes
4answers
158 views

Machine Learning Book (Python)

I'm in search of a good book about Machine Learning. Requirements: Good theoretical depth (while being a ML beginner, I hold a degree in mathematics), but accompained with good examples and plenty ...
1
vote
0answers
24 views

Vectorizing code to calculate (squared) Mahalanobis Distiance [migrated]

Say I want to to calculate the squared Mahalanobis distance between two vectors $\vec{x}$ and $\vec{y}$ with covariance matrix $\mathbf{S}$. This is a fairly simple function defined by $$M^2(\vec{x} ...
0
votes
1answer
23 views

Scikit-learn Scaled Data - Means Not Zero

I tried to scale the data by referring to the link as follow: http://scikit-learn.org/stable/modules/preprocessing.html However, when I checked the data distribution, the mean returned is NOT ZERO. ...
1
vote
1answer
37 views

Model selection for linear regression with categorical variables

I regressed the dependent variable Rating (numeric) on Judge which is categorical. The output of the first model is given at the end of the question. Only Judge John Foy and Linda Murphy came out to ...
0
votes
0answers
7 views

Model selection for linear regression with categorical variables [duplicate]

I regressed the dependent variable Rating (numeric) on Judge which is categorical. The output of the first model is given at the ...
1
vote
2answers
135 views

How to standardize the variables in R for regression analysis

I have been looking at some tutorials and articles and couldn't get a scenario where two variables are in different scales and used in modeling. So, firstly lets assume I have one metric of numeric ...
0
votes
0answers
9 views

How does one call external datasets into scikit-learn? [migrated]

For example consider this dataset : (1) https://archive.ics.uci.edu/ml/machine-learning-databases/annealing/anneal.data or (2) http://data.worldbank.org/topic How does one call such external ...
0
votes
1answer
64 views

Python/R Exploratory Data Analysis for Classification [closed]

Are there preexisting functions in Python/R that create exploratory data analysis plots like the following:
0
votes
0answers
3 views
0
votes
0answers
30 views

How to add cluster centers to the already transformed arrays with T-SNE Scikit Learn?

let's get this scikit original code, which is basically the one I'm using. My X is 2000x100 and in order to plot the clusters (plot on the right) I want to transform it with with the TSNE algorithm ...
1
vote
1answer
43 views

Generate independent random values from a bivariate normal distribution

I am trying to independently select two sets of numbers (set 1 and set 2) from a bivariate normal distribution. I want the ...
2
votes
0answers
115 views

Finding idf only for text mining

We find tf-idf for training phase in text mining, however, in test phase, we need the tf for each element in test set, but should use idf in train set, so is there any api in python that can calculate ...
0
votes
0answers
25 views

Pandas generating features vector with apply()

I have a data set with two features: class (int) and content (text). Each row of content needs to be vectorized to a set of boolean features matching regexes. My vectorized function return a ...
1
vote
1answer
51 views

Binomial Distribution Where N is Generated by a Poisson Process (pymc)

I'm not sure if this is the best way to go about this, because I'm fairly new to Bayesian methods. I'm trying to model a process where the number of trials $n$ used in a binomial process is generated ...
0
votes
0answers
18 views

I inserted n variables into statsmodels.formula.api.ols and returned > n predictions?

This makes no sense to me? I have split the data into training and testing data with sklearn. ...
0
votes
0answers
17 views

Gaze estimation, choosing algorithm and parameters

I am trying to build a program for estimating point of gaze on the computer screen from the x and y coordinates of the pupil centres from webcam video .(x and y coordinates correspond to pixel ...
0
votes
0answers
50 views

Did I use gradient descent correctly?

In an attempt to learn Gradient Descent. I've created my own dataset which is a the total bill of a meal with tips. As I created the perfect data so I make every meal with 10% tip. I purposely ...
2
votes
1answer
20 views

Comparing performance of a single classifier on two datasets

I'm using a single classifier on two different datasets, consisting of non-overlapping set of observations. The performance on one dataset is higher than the other, and I want to show that the ...
0
votes
1answer
24 views

Weighted re sampling why cdf not pdf or pmf?

From thinkstats2 the O'Reilly text. ...