Problems focusing on (but not necessarily limited to) using Python for statistical computation.

learn more… | top users | synonyms

0
votes
0answers
15 views
0
votes
1answer
17 views

Confirming calculations with simulations?

This question may be a little abstract, but I would like to understand how to develop a mentality towards performing statistical simulations. For example: If I have a normal distribution, and I ...
0
votes
0answers
13 views

Python implementation of Gini impurity [on hold]

I have a 2x2 contingency table and want to compute the gain in Gini impurity (definition extracted from An Empirical Comparison of Selection Measures for Decision-Tree Induction, JOHN MINGERS) I ...
2
votes
0answers
17 views

Spatial clustering based on response

Statistics version: I have a few measurements of a function that takes three inputs and produces a few 2D fields of outputs: f(a,b,c;x,y), with f being a vector of several quantities. I would like to ...
1
vote
1answer
46 views

Implementing bayesian networks in python for gaze estimation using visual saliency

I am developing an appearance based gaze estimation system based on opencv and python. I have currently developed a prototype which can estimate the gaze based on active calibration, which is ...
0
votes
0answers
30 views

Why is my moving average fit so bad [on hold]

I am following this MatLab guide on deseasoning/detrending data, however, I seem to be getting a really bad fit with a henderson moving average. The end points drop off very suddenly and don't seem ...
0
votes
2answers
110 views

Using RNN (LSTM) for predicting the timeseries vectors (Theano)

I have very simple problem but I cannot find a right tool to solve it. I have some sequence of vectors of the same length. Now I would like to train LSTM RNN on train sample of these sequences and ...
1
vote
0answers
35 views

Does feature size affect polynomial regression?

(I'm still trying to learn all this, sorry for any wrong terms or mistakes I might have made in this question) By feature size, I mean the value of the numbers. For example, let's say I have input ...
1
vote
0answers
10 views

Can I use an unknown number of variables to model my time-series?

I have a bunch of data-sets showing the relationship between two observables, "force" and "time". See example plot You see the regularity of the features: There is a region of linearly increasing ...
0
votes
0answers
27 views

Granger Causality / VAR / Negative Correlations

I have two questions on Granger causality. I feel puzzled after reading a dozen of papers on the topic and it appears to me that I need to clarify my understanding of Granger causality. Question 1 : ...
0
votes
0answers
7 views

Bigdata cluster compatible distributed predictive model [migrated]

I might be asking a dumb question but my question is can I write a python program (lets say a classifier) using some library that scales in hadoop (not only using a simple parallel processing).The ...
1
vote
1answer
13 views

Identifying filtered features after feature selection with scikit learn

Here is my Code for feature selection method in Python: ...
1
vote
2answers
54 views

How do I use Lasso and elastic net as feature selectors?

I have a data set with 900,000 rows and 8 features. I want to look at the significance of each feature so that I can evaluate whether the features I add are viable or not. One method I am using after ...
0
votes
0answers
8 views

Suitable classifier for 'objects on a string' data [closed]

I don't know if this can be considered a subjective question, but I have no option but to ask someone. My problem: I have a series of strings (or lines, think metal strings, not text strings) on ...
0
votes
1answer
24 views

How do I improve the accuracy of my supervised document classification model? [closed]

Given 1000 legal judgement documents, 900 of which are labeled, my task is to predict the label for the remaining 100 documents. The labeled documents belong to 41 different categories of Law, with ...
0
votes
0answers
44 views

Why can't I reconstruct parameters of a synthetic data set?

The following Python function creates synthetic binary labeled data that is supposed to perfectly follow the logistic regression model: ...
0
votes
0answers
22 views

Uplift model with a continuous outcome?

Does anyone know any good packages (preferably in R/python) or references that are specifically about building the uplift model with a "continuous" outcome? I've used the upliftRF from R and made it ...
0
votes
0answers
25 views

CSV file with several values, get median in groups of ten [closed]

I have a csv file with 4000 lines and 270 columns. So I want to read each column and take the median of groups of 10 elements. How do I do this in Python on a fast way, taking into account the number ...
0
votes
0answers
23 views

Sensitivity to scaling of multivariate data with HMM

I have some multivariate data, say 40 features. Some features are scaled between 0 and 1, and some are scaled between 0 and 1e8. For reference, I am using sci-kit learn's HMM implementation (yes, I ...
2
votes
1answer
56 views

How “interesting” a data series is

I have a large dataset containing several objects. Each object has many attributes which is arranged in a time series. Is there a suggested method to find the top n "most interesting" attributes? The ...
2
votes
0answers
53 views

Julia: Taking stock of how it has been doing

I came across a 2012 question that had a very good discussion about Julia as an alternative to R / Python for various types of Statistical Work. Here lies the original Question from 2012 about ...
3
votes
1answer
62 views

Sampling from von Mises-Fisher distribution in Python?

I am looking for a simple way to sample from a multivariate von Mises-Fisher distribution in Python. I have looked in the stats module in scipy and the numpy module but only found the univariate von ...
0
votes
0answers
25 views

Statsmodels: What can cause LinAlg error?

I am trying to do a mixed linear model for a dataset I am helping a colleague analyze, as in an earlier question I had posted, GLMs were unable to handle random effects. With the appropriate imports ...
0
votes
0answers
20 views

Statsmodels: Incorporate random effects vs. fixed effects

In JMP Pro, it is possible to include certain columns of predictor variables as "random effects" and others as "fixed effects". How do we do the same in ...
1
vote
0answers
37 views

Calculating Mutual Information for feature selection

In order to determine the importance of some individual features coming from labelled time series, I am trying to calculate the Mutual Information (as showed in "Who do you sync you are?: smartphone ...
1
vote
0answers
14 views

Scipy.stats.anderson_ksamp negative return values for test statistic

Ok, so I've been trying to run this test on the the iris dataset to see if it flags the clusters within the data as samples that aren't from the same population. from sklearn import datasets iris = ...
1
vote
0answers
12 views

How to correct standard errors of a two stage least squares

Can you demo a working example, in python (pandas or numpy) or R, how you can correct the standard errors on a 2sls. Most resources say that the software will do that automatically for you, but I ...
0
votes
0answers
12 views

Computing weighted AIC scores [duplicate]

I am trying to compute the weighted AIC using the example posted here as a basis: $$ w_i = \frac{e^{(-0.5\mathsf{\Delta}_i)}}{\sum_{r=1}^Re^{(-0.5\mathsf{\Delta}_i)}}. $$ where ${\Delta}_i$ is the ...
-1
votes
1answer
24 views

Why does the Generalised Gamma distribution in Scipy have 2 shape parameters

Why does the Scipy Generalised Gamma distribution have 2 shape parameters when other implementations only require one. See wikipedia and R(GAMLSS) for examples.
0
votes
0answers
30 views

MAP Parameter Estimates with Bayesian Logistic Regression

I am trying to reproduce the CLG algorithm for the Laplace prior given in Genkin et al to find the MAP estimates for a logistic regression model. I am using Python (Anaconda 2.2) with Numpy to ...
0
votes
0answers
8 views

Save and Restore current state in PYMC [migrated]

Recently, I launched a Bayesian model run that are written in PYMC. Due to power outage, the results generated during halfway of the run are gone. So, the logical step is to look for ways to save the ...
0
votes
0answers
22 views

How to learn from dataset where there are repeated instances but with different values?

I'm facing an issue with my dataset. The issue is that I have for example 1000 instances (customers or students) but I have 4000 lines/records because some of the instances are presented in several ...
0
votes
0answers
30 views

Balancing random forest via cross validation. Difference between sample weight and cutoffs?

My random forest model of a simple binary target (0, 1) and is producing unbalanced results. i.e many more false positives than there are false negatives. In addition, '1' is a low percentage class, ...
1
vote
0answers
25 views

Latent Dirichlet Allocation example in PyMC doesn't seem to give correct results

I've been working on the PyMC example for Latent Dirichlet Allocation (LDA) in this other post. The problem is that it doesn't seem to actually find the correct topics. For the code below my ...
1
vote
2answers
100 views

How to do Principal Components Analysis from start to finish in Python or R? [closed]

I'm a Software Engineer trying to learn how to do a Principal Components Analysis in Python or R. I've found a few links which do a good job of explaining the concept from a high-level. However, I ...
0
votes
0answers
30 views

How to implement Kernel density estimation in multivariate/3D

I have dataset like the following fromat and im trying to find out the Kernel density estimation with optimal bandwidth. ...
0
votes
1answer
49 views

What kind of graph should I choose

I have gathered criminal activity and GDP from 2012 to 2015 for over 30 countries and I'm trying to confirm my hypothesis about how higher GDP decreases criminal activity, but I'm baffled over which ...
1
vote
0answers
24 views

Testing data for Same Shape

I want to run a Mann-Whitney test on 2 arbitrary sets of data but I read that one of the assumptions of the Mann-Whitney test is that the data is the same shape. How can I test if the 2 sets of data ...
1
vote
1answer
68 views

find a general formula between two parameters using curve-fit

I am trying to study the relationship between two parameters say A and B for an image. For a single scene S1, I take 20 images ...
1
vote
1answer
33 views

Python: GLM multidimensional input

I'm trying to fit a GLM model to some data. My response variable, y, is a vector of length 24, my inputs x are a 24 * 24 data frame. My code looks like this: ...
0
votes
0answers
32 views

2D Kernel density estimation with uncertainties

I would like to perform bivariate KDE with Gaussian kernels (preferably using Python, or R) of a dataset with heteroscedastic uncertainties. What would be the correct way to do this: to rescale a ...
2
votes
0answers
56 views

How to systematically remove collinear variables in Python?

Thus far, I have removed collinear variables as part of the data preparation process by looking at correlation tables and eliminating variables that are above a certain threshold. Is there a more ...
0
votes
0answers
19 views

How to predict values for a given model using a custom data set in statsmodels?

If I've trained some linear model using the statsmodels library, how can I predict certain values using custom independent variables? That is, I've trained the model using some observations ...
0
votes
0answers
34 views

TfidfVectorizer: should it be used on train only or train+test

When training a model it is possible to train the Tfidf on the corpus of only the training set or also on the test set. It seems not to make sense to include the test corpus when training the model, ...
1
vote
1answer
45 views

Classification in real time without prior knowledge of the number of classes

Is there an implemented algorithm (with python/R or java in preference) that can classify incoming data from an unknown generator with absolutely no prior knowledge or assumption. For example: Let G ...
2
votes
1answer
31 views

Ensemble time series prediction from two separate models

I have two different forecasts that are produced by ARMA models using two different data samples. The difference between the two data sets is their size: one used data from 2013-2014 and another used ...
0
votes
1answer
86 views

Various methods for predicting multiple dependent variables (python)

I would like to model and predict multiple dependent variables depending on one or more independent variables. The most straightforward method appears to be multivariate regression. I was wondering ...
0
votes
0answers
23 views

Weather analysis | company sales

I'm writing a python code that reads in a csv file of rain in inches for a given zip code and creates a normal distribution from the data. Ultimately, I want to be able to create some score for the ...
0
votes
0answers
16 views

Prediction - important features from a new element

On my data I did LSI and got a large matrix (>200000 samples, >6000 features, very sparse). I do SVD on it, keeping only 150 dimensions. When I get a new element, do a folding-in, calculate cosine ...
1
vote
2answers
76 views

regression with scikit-learn with multiple outputs, svr or gbm possible?

I have been trying regression with scikit-learn with a problem with multiple outputs like this: ...