Problems focusing on (but not necessarily limited to) using Python for statistical computation.

learn more… | top users | synonyms

0
votes
0answers
6 views

I'm getting an error message using PyMc that I don't understand [migrated]

Here is my setup and error message: creating fake data ...
0
votes
0answers
17 views

Regression plot and function for: heavy-tailed probability distribution

I've got data points from a simulation as coordinates in a text-files like so: ...
-1
votes
0answers
20 views

Hierarchical clustering and sklearn [on hold]

I would like to know if there is a reason why in sklearn library is not implemented a partitional solution for hierarchical clustering. I think that is possible to apply recursively a K-means ...
3
votes
2answers
139 views

Why is this time-series stationary?

I am using python for time-series analysis of count data and came across a problem where I have a time-series that to me looks non-stationary but the Augmented Dickey-Fuller test (implemented in ...
-1
votes
0answers
12 views

Clustering headlines without full text [on hold]

I have a data-set with headlines of news. The number of topics is 100. Also I have a tf-idf file, which is not generated by me. I saw data-set, and I can determine something like 50 clusters. 50 - ...
-1
votes
0answers
17 views

plotting SVM in python [on hold]

I tried following the example here but i am having trouble applying it when i have 16 features. lin_svc is trained with those 16 features (i deleted the line to ...
1
vote
1answer
34 views

verified procedure for calculating gradient descent?

I'm wondering if there is a good procedure people are using for gradient descent that is pretty well validated--something like a package for R or Python, or generic code many people adapt. After ...
0
votes
2answers
25 views

Multiclass classification - Optimizing for recall in subset of classes

As part of a wider analysis, I'm seeking a method to reliably recall the outer margins (~10% each) of a data set. I don't mind if precision is lower as a result. I've separated the data into 3 ...
1
vote
0answers
25 views

Ordinal logistic regression in Python

I would like to run an ordinal logistic regression in Python - for a response variable with three levels and with a few explanatory factors. The statsmodels package ...
0
votes
0answers
25 views

How to do MANOVA in Python?

I can quickly do a MANOVA test in R using this code: ...
1
vote
1answer
24 views

Getting distance of points from decision boundary with linear SVM?

I posted this originally in Stack Overflow but realize it might be more of a statistics question. I am using SKLearn to run SVC on my data. ...
0
votes
0answers
17 views

help in coding decision tree in python [migrated]

I am not sure if this is the right place to post this, but I have been trying to code up a simple decision tree class for a while and am getting lost at various points. Specifically, I'm not sure ...
0
votes
2answers
86 views

From where I can learn to use scikit learn

I have read text about machine learning and I feel that I have gained sufficient knowledge that I can start applying them practically. I have programming experience in python so I want to learn how to ...
1
vote
0answers
320 views

CV (Curriculum Vitae) Recommender system - guidance

Please note that I am a total beginner with machine learning and artificial intelligence and also a novice with Python (I'm sure I have a very non-Pythonic way of writing code). I have a college ...
0
votes
0answers
37 views

What could cause a K-means clustering algorithm to converge into a single cluster?

I am currently writing a K-means clustering algorithm in Python, and I seem to have coded myself into a corner... I begin my algorithm with data sets assigned randomly to the appropriate number of K ...
1
vote
2answers
54 views

sklearn - overfitting problem

I'm looking for recommendations as to the best way forward for my current machine learning problem The outline of the problem and what I've done is as follows: I have 900+ trials of EEG data, where ...
1
vote
0answers
18 views

Optimal method for identifying correlation of a stock price and various moving average durations for Python

I am trying to find the optimal method in trying to find the relationship between daily stock price and rolling correlation of another stock. For example, let's say I have a hunch that there is a ...
4
votes
1answer
121 views

Understanding derp regression

After having a bit too much tea I decided to write a silly way of estimating the parameters of a single-feature linear regression. For starters, I avoided using techniques like gradient descent, ...
0
votes
0answers
12 views

Classification of array data?

I have a set of array data that falls into one of two bins. Right now, the classification is done manually, but I'd like to try to use a learning approach. The classification will depend on the size ...
0
votes
1answer
47 views

Regression of two data data sets of the same experiment

I have two data sets collected by different methods about the same phenomena, one 'bare' and other where I believe there is a constant value added (the same phenomena, with the data collected by a ...
0
votes
1answer
47 views

Increasing sample size of multivariate data with bootstrapping

I'm trying to perform variable selection methods on a small data set (approximately 100 observations with 14 predictors each). Is there any way to increase the sample size with bootstrapping? Are ...
0
votes
0answers
22 views

Sentence classification feature selection

My task is to classify sentences based on their meaning. More specifically, I would like to have one class for sentences indicating current possession ("The store has < product > in stock...") as ...
3
votes
4answers
149 views

Machine Learning Book (Python)

I'm in search of a good book about Machine Learning. Requirements: Good theoretical depth (while being a ML beginner, I hold a degree in mathematics), but accompained with good examples and plenty ...
1
vote
0answers
24 views

Vectorizing code to calculate (squared) Mahalanobis Distiance [migrated]

Say I want to to calculate the squared Mahalanobis distance between two vectors $\vec{x}$ and $\vec{y}$ with covariance matrix $\mathbf{S}$. This is a fairly simple function defined by $$M^2(\vec{x} ...
0
votes
1answer
21 views

Scikit-learn Scaled Data - Means Not Zero

I tried to scale the data by referring to the link as follow: http://scikit-learn.org/stable/modules/preprocessing.html However, when I checked the data distribution, the mean returned is NOT ZERO. ...
1
vote
1answer
36 views

Model selection for linear regression with categorical variables

I regressed the dependent variable Rating (numeric) on Judge which is categorical. The output of the first model is given at the end of the question. Only Judge John Foy and Linda Murphy came out to ...
0
votes
0answers
7 views

Model selection for linear regression with categorical variables [duplicate]

I regressed the dependent variable Rating (numeric) on Judge which is categorical. The output of the first model is given at the ...
1
vote
2answers
132 views

How to standardize the variables in R for regression analysis

I have been looking at some tutorials and articles and couldn't get a scenario where two variables are in different scales and used in modeling. So, firstly lets assume I have one metric of numeric ...
0
votes
0answers
9 views

How does one call external datasets into scikit-learn? [migrated]

For example consider this dataset : (1) https://archive.ics.uci.edu/ml/machine-learning-databases/annealing/anneal.data or (2) http://data.worldbank.org/topic How does one call such external ...
0
votes
1answer
52 views

Python/R Exploratory Data Analysis for Classification [closed]

Are there preexisting functions in Python/R that create exploratory data analysis plots like the following:
0
votes
0answers
3 views
0
votes
0answers
28 views

How to add cluster centers to the already transformed arrays with T-SNE Scikit Learn?

let's get this scikit original code, which is basically the one I'm using. My X is 2000x100 and in order to plot the clusters (plot on the right) I want to transform it with with the TSNE algorithm ...
1
vote
1answer
42 views

Generate independent random values from a bivariate normal distribution

I am trying to independently select two sets of numbers (set 1 and set 2) from a bivariate normal distribution. I want the ...
2
votes
0answers
96 views

Finding idf only for text mining

We find tf-idf for training phase in text mining, however, in test phase, we need the tf for each element in test set, but should use idf in train set, so is there any api in python that can calculate ...
0
votes
0answers
25 views

Pandas generating features vector with apply()

I have a data set with two features: class (int) and content (text). Each row of content needs to be vectorized to a set of boolean features matching regexes. My vectorized function return a ...
1
vote
1answer
48 views

Binomial Distribution Where N is Generated by a Poisson Process (pymc)

I'm not sure if this is the best way to go about this, because I'm fairly new to Bayesian methods. I'm trying to model a process where the number of trials $n$ used in a binomial process is generated ...
0
votes
0answers
17 views

I inserted n variables into statsmodels.formula.api.ols and returned > n predictions?

This makes no sense to me? I have split the data into training and testing data with sklearn. ...
0
votes
0answers
17 views

Gaze estimation, choosing algorithm and parameters

I am trying to build a program for estimating point of gaze on the computer screen from the x and y coordinates of the pupil centres from webcam video .(x and y coordinates correspond to pixel ...
0
votes
0answers
49 views

Did I use gradient descent correctly?

In an attempt to learn Gradient Descent. I've created my own dataset which is a the total bill of a meal with tips. As I created the perfect data so I make every meal with 10% tip. I purposely ...
2
votes
1answer
19 views

Comparing performance of a single classifier on two datasets

I'm using a single classifier on two different datasets, consisting of non-overlapping set of observations. The performance on one dataset is higher than the other, and I want to show that the ...
0
votes
1answer
23 views

Weighted re sampling why cdf not pdf or pmf?

From thinkstats2 the O'Reilly text. ...
0
votes
0answers
29 views

Information retrieval from strings using neural network

I am dealing with input strings that have max length of 200 characters, they are the USSD popup msgs and the SMS text strings on android smart phones extracted as and when they appear via an android ...
0
votes
2answers
72 views

Median of ratio of independent variates with Beta distributions

Let $X, Y$ be independent random variables where $X \sim Beta(\alpha_1,\beta_1)$, $Y \sim Beta(\alpha_2,\beta_2)$, and $Z = X/Y$. Recall $X, Y$ are supported on $(0,1)$, so $z > 0$. I've ...
0
votes
0answers
27 views

“without” not in NLTK Stopwords corpus, while “with” is

I will preface this by saying that I do not have experience in linguistics. Unnecessary Background: I was doing some NLP in python using NLTK, and began by removing stopwords from my text of ...
0
votes
1answer
81 views

Gradient descend algorithm ascending for learning rate

This is not an Homework assignment. I am trying to implement the gradient descend explained in the link (Page 5) [http://cs229.stanford.edu/notes/cs229-notes1.pdf][1].I am able to successfully run the ...
0
votes
0answers
20 views

Reimplement Binomial Test R-> Python

I am trying to re-implement a Binomial test in R that uses the alternative= "greater" optionin Python using scipy. Right now I am looking at ...
0
votes
1answer
70 views

Can a Precision-Recall curve or a ROC curve be horizontal?

I am working on a binary classification task on imbalanced data. Since the accuracy is not so meaningful in this case. I use Scikit-Learn to compute the Precision-Recall curve and ROC curve in order ...
0
votes
1answer
21 views

What's meaning of BOS and EOS in CRFSuite feature list and what is the role of them?

In NER(Named Entity Recognition) example in python-crf package website we see this function as feature generator: ...
0
votes
0answers
20 views

Dissecting the code of statistical algorithms for education

I was curious if anyone has made or seen any videos that dissect the code of statistical algorithms as a means of learning statistics and coding. Ideally the videos would explain step by step both the ...
1
vote
1answer
13 views

sklearn estimate_bandwidth in MeanShift algorithm with small dataset

How should I approach the estimate_bandwidth function with only very small datasets? My problem is that I am analyzing datasets that range from 500 entries to 2. ...