Questions tagged [python]

Python is a programming language commonly used for machine learning. Use this tag for any *on-topic* question that (a) involves `Python` either as a critical part of the question or expected answer, & (b) is not *just* about how to use `Python`.

Filter by
Sorted by
Tagged with
371
votes
26answers
133k views

Python as a statistics workbench

Lots of people use a main tool like Excel or another spreadsheet, SPSS, Stata, or R for their statistics needs. They might turn to some specific package for very special needs, but a lot of things can ...
232
votes
5answers
397k views

What is batch size in neural network?

I'm using Python Keras package for neural network. This is the link. Is batch_size equals to number of test samples? From ...
107
votes
2answers
85k views

What is an embedding layer in a neural network?

In many neural network libraries, there are 'embedding layers', like in Keras or Lasagne. I am not sure I understand its function, despite reading the documentation. For example, in the Keras ...
89
votes
6answers
115k views

What loss function for multi-class, multi-label classification tasks in neural networks?

I'm training a neural network to classify a set of objects into n-classes. Each object can belong to multiple classes at the same time (multi-class, multi-label). I read that for multi-class problems ...
76
votes
9answers
86k views

What algorithm should I use to detect anomalies on time-series?

Background I'm working in Network Operations Center, we monitor computer systems and their performance. One of the key metrics to monitor is a number of visitors\customers currently connected to our ...
74
votes
1answer
69k views

How to split the dataset for cross validation, learning curve, and final evaluation?

What is an appropriate strategy for splitting the dataset? I ask for feedback on the following approach (not on the individual parameters like test_size or ...
56
votes
9answers
13k views

How do R and Python complement each other in data science?

In many tutorials or manuals the narrative seems to imply that R and python coexist as complementary components of the analysis process. To my untrained eye, however, it seems that both languages sort ...
53
votes
2answers
30k views

Pandas / Statsmodel / Scikit-learn

Are Pandas, Statsmodels and Scikit-learn different implementations of machine learning/statistical operations, or are these complementary to one another? Which of these has the most comprehensive ...
53
votes
5answers
98k views

How does one interpret SVM feature weights?

I am trying to interpret the variable weights given by fitting a linear SVM. (I'm using scikit-learn): ...
53
votes
10answers
15k views

Machine Learning using Python

I am considering using Python libraries for doing my Machine Learning experiments. Thus far, I had been relying on WEKA, but have been pretty dissatisfied on the whole. This is primarily because I ...
52
votes
7answers
116k views

Why is the validation accuracy fluctuating?

I have a four layer CNN to predict response to cancer using MRI data. I use ReLU activations to introduce nonlinearities. The train accuracy and loss monotonically increase and decrease respectively. ...
48
votes
7answers
23k views

Survival Analysis tools in Python [closed]

I am wondering if there are any packages for python that is capable of performing survival analysis. I have been using the survival package in R but would like to port my work to python.
45
votes
3answers
88k views

Logistic Regression: Scikit Learn vs Statsmodels

I am trying to understand why the output from logistic regression of these two libraries gives different results. I am using the dataset from UCLA idre tutorial, predicting ...
41
votes
6answers
27k views

Why do I get a 100% accuracy decision tree?

I'm getting a 100% accuracy for my decision tree. What am I doing wrong? This is my code: ...
40
votes
2answers
137k views

Mean absolute percentage error (MAPE) in Scikit-learn [closed]

How can we calculate the Mean absolute percentage error (MAPE) of our predictions using Python and scikit-learn? From the docs, we have only these 4 metric functions for Regressions: metrics....
38
votes
1answer
71k views

what does the numbers in the classification report of sklearn mean?

I have below an example I pulled from sklearn 's sklearn.metrics.classification_report documentation. What I don't understand is why there are f1-score, precision and recall values for each class ...
37
votes
2answers
102k views

How to interpret p-value of Kolmogorov-Smirnov test (python)?

I have Two samples that I want to test (using python) if they are drawn from the same distribution. To do that I use the statistical function ks_2samp from scipy.stats. It returns 2 values and I find ...
36
votes
4answers
8k views

Data has two trends; how to extract independent trendlines?

I have a set of data that is not ordered in any particular way but when plotted clearly has two distinct trends. A simple linear regression would not really be adequate here because of the clear ...
35
votes
3answers
14k views

Building an autoencoder in Tensorflow to surpass PCA

Hinton and Salakhutdinov in Reducing the Dimensionality of Data with Neural Networks, Science 2006 proposed a non-linear PCA through the use of a deep autoencoder. I have tried to build and train a ...
35
votes
1answer
2k views

Link Anomaly Detection in Temporal Network

I came across this paper that uses link anomaly detection to predict trending topics, and I found it incredibly intriguing: The paper is "Discovering Emerging Topics in Social Streams via Link Anomaly ...
32
votes
4answers
31k views

Implementation of CRF in python

Is there a popular implementation of Conditional Random Fields in Python? I can't seem to find any that is widely used and popular!
31
votes
2answers
79k views

libsvm data format [closed]

I'm using the libsvm (http://www.csie.ntu.edu.tw/~cjlin/libsvm/) tool for support vector classification. However, I'm confused about the format of the input data. From the README: The format of ...
30
votes
6answers
32k views

Python module for change point analysis

I'm looking for a Python module that performs a change-point analysis on a time-series. There are a number of different algorithms and I'd like to explore the efficacy of some of them without having ...
26
votes
2answers
11k views

Why is Python's scikit-learn LDA not working correctly and how does it compute LDA via SVD?

I was using the Linear Discriminant Analysis (LDA) from the scikit-learn machine learning library (Python) for dimensionality reduction and was a little bit curious ...
25
votes
2answers
38k views

PCA in numpy and sklearn produces different results

Am i misunderstanding something. This is my code using sklearn ...
25
votes
4answers
29k views

Clustering a correlation matrix

I have a correlation matrix which states how every item is correlated to the other item. Hence for a N items, I already have a N*N correlation matrix. Using this correlation matrix how do I cluster ...
24
votes
5answers
12k views

How do I make my neural network better at predicting sine waves?

Here, have a look: You can see exactly where the training data ends. Training data goes from $-1$ to $1$. I used Keras and a 1-100-100-2 dense network with tanh activation. I calculate the result ...
24
votes
1answer
7k views

Use of circular predictors in linear regression

I am trying to fit a model using wind data (0, 359) and time of day (0, 23), but I am concerned that they will poorly fit into a linear regression because they are not themselves linear parameters. I ...
24
votes
1answer
26k views

Keras, how does SGD learning rate decay work?

If you look at the documentation http://keras.io/optimizers/ there is a parameter in the SGD for decay. I know this reduces the learning rate over time. However, I can not figure out how it works ...
24
votes
3answers
58k views

How to systematically remove collinear variables (pandas columns) in Python? [closed]

Thus far, I have removed collinear variables as part of the data preparation process by looking at correlation tables and eliminating variables that are above a certain threshold. Is there a more ...
22
votes
3answers
29k views

How to compute the standard errors of a logistic regression's coefficients

I am using Python's scikit-learn to train and test a logistic regression. scikit-learn returns the regression's coefficients of the independent variables, but it does not provide the coefficients' ...
22
votes
2answers
29k views

How to use scikit-learn's cross validation functions on multi-label classifiers

I'm testing different classifiers on a data set where there are 5 classes and each instance can belong to one or more of these classes, so I'm using scikit-learn's multi-label classifiers, ...
21
votes
4answers
30k views

Difference between statsmodel OLS and scikit linear regression

I have a question about two different methods from different libraries which seems doing same job. I am trying to make linear regression model. Here is the code which I using statsmodel library with ...
20
votes
2answers
24k views

Difference between selecting features based on “F regression” and based on $R^2$ values?

Is comparing features using F-regression the same as correlating features with the label individually and observing the $R^2$ value? I have often seen my ...
20
votes
2answers
125k views

How to name the ticks in a python matplotlib boxplot

Python matplotlib has a boxplot command. Normally, all the parts of the graph are numerically ticked. How can I change the ticks to names instead of positions? For illustration, I mean the Mon Tue ...
20
votes
3answers
913 views

Julia: Taking stock of how it has been doing

I came across a 2012 question that had a very good discussion about Julia as an alternative to R / Python for various types of Statistical Work. Here lies the original Question from 2012 about Julia'...
20
votes
3answers
925 views

How to simulate data to be statistically significant?

I am in 10th grade and I am looking to simulate data for a machine learning science fair project. The final model will be used on patient data and will predict the correlation between certain times of ...
20
votes
1answer
22k views

Jenks Natural Breaks in Python: How to find the optimum number of breaks?

I found this Python implementation of the Jenks Natural Breaks algorithm and I could make it run on my Windows 7 machine. It is pretty fast and it finds the breaks in few time, considering the size of ...
19
votes
2answers
7k views

Is it possible to automate time series forecasting?

I would like to build an algorithm that would be able to analyze any time series and "automatically" choose the best traditional/statiscal forecasting method (and its parameters) for the analyzed time ...
19
votes
3answers
34k views

Using RNN (LSTM) for predicting the timeseries vectors (Theano)

I have very simple problem but I cannot find a right tool to solve it. I have some sequence of vectors of the same length. Now I would like to train LSTM RNN on train sample of these sequences and ...
19
votes
1answer
25k views

Generate two variables with precise pre-specified correlation [duplicate]

I want to generate two variables with (pseudo-) random numbers with an exact pearson's r. How do I do that? Python and/or R solutions would be nice! I am able to generate random data that ...
18
votes
3answers
12k views

Among Matlab and Python, which language is good for statistical analysis?

Among Matlab and Python, which language is good for general statistical data analysis? What are the pros and cons, other than accessibility, for each?
18
votes
3answers
33k views

Collinear variables in Multiclass LDA training

I'm training a Multi-class LDA classifier with 8 classes of data. While performing training, I get a warning of: "Variables are collinear" I'm getting a training accuracy of over 90%. I'm using ...
18
votes
1answer
4k views

In multiple linear regression, why does a plot of predicted points not lie in a straight line?

I'm using multiple linear regression to describe relationships between Y and X1,X2. From theory I understood that multiple regression assumes linear relationships between Y and each of X (Y and X1, Y ...
18
votes
2answers
16k views

Ordinal logistic regression in Python

I would like to run an ordinal logistic regression in Python - for a response variable with three levels and with a few explanatory factors. The statsmodels package ...
18
votes
1answer
5k views

Getting started with bayesian structural models using MCMC

I'm trying to learn bayesian structural time series analysis. For a variety of reasons I need to use Python (mostly pymc3) not R so please do not suggest the ...
17
votes
4answers
2k views

What is the best programmatic way for determining whether two variables are linearly or non-linearly or not even related

What is the best programmatic way for determining whether two predictor variables are linearly or non-linearly or not even related, maybe using any of the packages scipy/statsmodels or anything else ...
17
votes
3answers
1k views

How can I model flips until N successes?

You and I decide to play a game where we take turns flipping a coin. The first player to flip 10 heads in total wins the game. Naturally, there is an argument about who should go first. Simulations ...
17
votes
1answer
19k views

Beta distribution fitting in Scipy

According to Wikipedia the beta probability distribution has two shape parameters: $\alpha$ and $\beta$. When I call scipy.stats.beta.fit(x) in Python, where ...
17
votes
2answers
25k views

Feature selection with Random Forests

I have a dataset with mostly financial variables (120 features, 4k examples) which are mostly highly correlated and very noisy (technical indicators, for example) so I would like to select about max ...

1
2 3 4 5
75