Problems focusing on (but not necessarily limited to) using Python for statistical computation.

learn more… | top users | synonyms

1
vote
1answer
29 views

Classification in real time without prior knowledge of the number of classes

Is there an implemented algorithm (with python/R or java in preference) that can classify incoming data from an unknown generator with absolutely no prior knowledge or assumption. For example: Let G ...
2
votes
1answer
18 views

Ensemble time series prediction from two separate models

I have two different forecasts that are produced by ARMA models using two different data samples. The difference between the two data sets is their size: one used data from 2013-2014 and another used ...
0
votes
1answer
34 views

Various methods for predicting multiple dependent variables (python)

I would like to model and predict multiple dependent variables depending on one or more independent variables. The most straightforward method appears to be multivariate regression. I was wondering ...
0
votes
0answers
15 views

Weather analysis | company sales

I'm writing a python code that reads in a csv file of rain in inches for a given zip code and creates a normal distribution from the data. Ultimately, I want to be able to create some score for the ...
0
votes
0answers
10 views

Prediction - important features from a new element

On my data I did LSI and got a large matrix (>200000 samples, >6000 features, very sparse). I do SVD on it, keeping only 150 dimensions. When I get a new element, do a folding-in, calculate cosine ...
1
vote
2answers
25 views

regression with scikit-learn with multiple outputs, svr or gbm possible?

I have been trying regression with scikit-learn with a problem with multiple outputs like this: ...
0
votes
2answers
18 views

What is “Verbose” in scikit-learn package of Python?

What is "Verbose" in scikit-learn package of Python? In some models like neural network and svm we can set it's value to true. This is the documentation: verbose ...
0
votes
0answers
9 views

Implementing the Bayesian Information Criterion (BIC) Using PyKalman

I'm trying to use pykalman to do a Kalman filter on financial data and it seems to be generally working very well. However, when I attempt to extend the code using BIC $\mathrm{BIC} = {-2 \cdot ...
0
votes
1answer
33 views

What is batch size in neural network?

I'm using Python Keras package for neural network. This is the link. Is batch_size equals to number of test samples? From ...
0
votes
0answers
33 views

Time series forecasting using SVM

I am trying to set up a Python code for forecasting a time series, using SVM libraries of scikit-learn. My data contains $X$ values at a day interval for the last one years, and I need to predict $y$ ...
0
votes
1answer
35 views

Best supervised neural network package for python

What is the best supervised neural network package for python? I found that sci-kit package only have unsupervised neural network.
0
votes
1answer
33 views

GridSearchCV Regression vs Linear Regression vs Stats.model OLS

I am trying to build multiple linear regression model with 3 different method and I am getting different results for each one. I think that I have to get the same results but ...
2
votes
1answer
44 views

Fisher's exact test vs chi-squared

I've been trying to figure out the correct way to calculate the p-value for my data. I originally created a simulation that randomly selected numbers that were greater than or less than a certain ...
0
votes
0answers
12 views

3D Zernike moments vs. Spherical Harmonics. Which one has higher discriminative power as shape descriptor?

I am looking for a comprehensive study that has performed comparison of different 3D shape descriptors for classification/clustering problems. Particularly, I am interested in 3D Zernike moments vs. ...
0
votes
1answer
38 views

What does it mean to normalize the data by the autocorrelation at the 0-th lag?

I'm just digging into python as a newbie and saw this expression in the plot docs: normalize the data by the autocorrelation at the 0-th lag. I didn't see further details, and Google wasn't very ...
8
votes
5answers
331 views

What algorithm should I use to detect anomalies on time-series?

Background I'm working in Network Operations Center, we monitor computer systems and their performance. One of the key metrics to monitor is a number of visitors\customers currently connected to our ...
0
votes
1answer
12 views

What is the equivalent of the complexity parameter ( rpart in R ) in python for regression trees (sklearn)?

The complexity parameter decides when to stop splitting. what is its equivalent in python. As decreasing the cp tends to increase the accuracy in the prediction, so is there a similar parameter in ...
2
votes
0answers
28 views

pymc implementation of ThinkBayes 1.3 cookie problem

This is obviously overkill for this problem, but I thought it would help cement the concepts for me. The problem: Suppose there are two bowls of cookies. Bowl 1 contains 30 vanilla cookies and 10 ...
0
votes
0answers
21 views

How to simulate a multivariate Logistic-Normal distribution in Python

I'm trying to generate a text document using reverse "Correlated Topic Models", which is an advanced version of LDA (Latent Dirichlet Allocation). In this version the topics are generated over a ...
0
votes
0answers
20 views

Visualization Question [duplicate]

Simple question, has anybody ever seen a similar visualization to the one below? If so what is the name of such a graph and what software could be used to create it?
0
votes
0answers
11 views

How does the stats.gaussian_kde method calcute the pdf?

I am using the scipy.stats.gaussian_kde method from scipy to generate random samples from the data. It works fine! What I have ...
1
vote
0answers
45 views

Are two linear regression models significantly different?

This question extends What test should be used to tell if two linear regression lines are significantly different? to the more general case of having two estimated models. I have got the following ...
1
vote
1answer
23 views

Class-specific feature importance

I have rather a simple question which I have not had any luck finding the answer to. I'm training a Random Forest classifier using sklearn in Python 2.7, on a large dataset ~(80k,250) where ...
0
votes
0answers
41 views

estimate confidence interval for poisson process

I would like to know how I can estimate the confidence intervals for poisson process distributed variables. I have a pandas dataframe with a column of trials and a column of successes. I want to ...
0
votes
0answers
30 views

Rolling window forecasts in Python

I asked this question some days ago but haven't got any response. So I've taken it to myself to do the rolling window manually. My limited grasp on regression forecasting has stumped my progress a ...
0
votes
0answers
13 views

Logistic loss approximation

In many implementations of logistic loss (example sklearn) I see the following code(adapted from sklearn), where p is the prediction and y the true value: ...
1
vote
2answers
36 views

Multiple regression/correlation analysis, large dataset:ways, tools [closed]

I've got a large "clean" dataset (800 MB), containing 210k rows and 320 columns. There is 2 discrete string-type columns, others are numeric. One of such numeric columns is selected as depended ...
1
vote
0answers
36 views

Rolling volatility estimation using GARCH family of models in python

Problem: Correct usage of GARCH(1,1) Aim of research: Forecasting volatility/variance. Tools used: Python 3.3 with arch library I am trying to obtain out-of-sample estimation of volatility using a ...
0
votes
0answers
34 views

Moving window forecasting in Python

I am looking to create some code that will out-of-sample forecast the HAR-RV model. The model itself is formulated as the following, and the betas are estimated through HAC-OLS or Newey-West. ...
1
vote
1answer
26 views

Compute test error and std of a model

I would like to do the following: Train a classifier on a certain dataset Test the classifier on a certain test set Compute the test error and standard deviation Compute a 95% confidence interval ...
0
votes
0answers
27 views

How to input sparse feature

In theano everything is symbolic, so how to input sparse feature in , for example, neural network? The setting is: the task is text application. the input is a mini-batch. Since theano sparse module ...
1
vote
2answers
68 views

Normalizing matrix values python/R

I am trying to fill missing values in 1000 x 1000 matrices. Dataset1 contains such 1000 x 1000 matrice with value ranging 1-100. ...
0
votes
1answer
44 views

Robust softmax solutions for Theano?

I am implementing multilayer perceptrons with the softmax activation function over Theano. In some extreme cases I am running into problems with too high/low values in the softmax function that ...
0
votes
0answers
22 views

Implementing equation for Conditional Intensity Function

I have point processes for which I would like to compute the conditional intensity functions. I know R has a package for doing point process analysis, but I have already done a lot of work on this ...
0
votes
2answers
29 views

Which model fit best for semi sinusoidal data? [duplicate]

I have a record containing the maximum and the minimum monthly temperatures at a particular station. The record shows information for each month from January 1908 to March 2012. However, some of the ...
5
votes
1answer
78 views

Use of circular predictors in linear regression

I am trying to fit a model using wind data (0, 359) and time of day (0, 23), but I am concerned that they will poorly fit into a linear regression because they are not themselves linear parameters. I ...
0
votes
0answers
22 views

Python: In which cases will random forest and SVM classifiers can produce high accuracy?

I am using Random Forest and SVM classifiers to do classification, and I have 18322 samples which are unbalanced in 9 classes (3667, 1060, 1267, 2103, 2174, 1495, 884, 1462, 4210). I use 10-fold CV ...
1
vote
3answers
67 views

Highly significant Pearson's chi-squared test (goodness of fit) when observed & expected are very close

I have two arrays that I would like to do a Pearson's chi-squared test (goodness of fit). I want to test whether or not there is a significant difference between the expected and observed results. ...
0
votes
0answers
10 views

How to perform pattern identification using ML?

I have the following problem: An event, takes place at a determined day of the week, hour, and with a pre-defined format (movie, music concert, lecture (3 items). Based on exit polls we determine 3 ...
0
votes
0answers
20 views

Likelihood of hypothesis in live data

Bayes rule is $P(H|E)=\frac{P(H)P(E|H)}{P(E)}$ I have a prior distribution from categorical data prior={'a':0.2,'b':0.6,'c':0.1,'d':0.1} Which forms my ...
2
votes
1answer
44 views

Python Astronomy Censored Data in Lifelines

I am trying to find a correlation between a given data set containing redshifts and turnover frequencies (I have a list of 320 galaxies, and the redshift and turnover frequency (a turnover frequency ...
0
votes
0answers
13 views

NMSE - division by zero

I am using the normalized mean square error function in the Python Oger Toolbox which is defined as: ...
1
vote
0answers
51 views

What are the best packages for Image Processing in R? [closed]

I just have started working on an image processing and classification problem. I am familiar with both R and Python. But I am not much experienced in scikit-learn and scikit-image libraries in Python. ...
2
votes
3answers
114 views

Compare performance of 2 models

I have a dataset which I have split into 3 parts: a training set, a cross-validation set and a test set. I have used the training set and cross-validation set to train 2 models. For this, I have taken ...
0
votes
0answers
25 views

Fitness sharing in DEAP

Is there any way to implement fitness sharing/niching using DEAP? Specifically I'm looking for an implementation of the method defined here (Goldberg's fitness sharing) on page 98 of the pdf. If you ...
0
votes
1answer
72 views

Difference between statsmodel OLS and scikit linear regression

I have a question about two different methods from different libraries which seems doing same job. I am trying to make linear regression model. Here is the code which I using statsmodel library with ...
0
votes
0answers
21 views

Adding a continuous to logistic regression based on TF-IDF

My train dataset contains blog posts. I have an excerpt from a post, its total length in words and an arbitrary "Good" binary variable: ...
3
votes
1answer
93 views

PyMC3 Implementation of Probabilistic Matrix Factorization (PMF): MAP produces all 0s

I've started working with pymc3 over the past few days, and after getting a feel for the basics, I've tried implementing the Probabilistic Matrix Factorization model. For validation, I use a subset ...
1
vote
0answers
45 views

Sample time series to equal interval

I have data with timestamp and associated values. time interval between two consecutive data is not constant. How to standardize the the time series and associated value ? eg- Input data is Timestamp ...
0
votes
0answers
57 views

How good a fit is my linear regression - really?

So I've made a linear regression of my two variables using pythons np.polyfit. This is my code: ...