# Questions tagged [scipy]

SciPy is a Python-based ecosystem of open-source software for mathematics, science, and engineering.

77 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
1k views

When I run ...
598 views

### Goodness-of-fit for Discrete Distributions

I've been doing some data analysis with Scipy. So far I accomplished this with continuous distributions: I can fit a probability distribution to a set of data points using a maximum likelihood fit. ...
5k views

### Assessing A/B test results using Python

Problem's setting Assume having results of an A/B test. You let you users experience two variants of your website and you counted how many converted: ...
91 views

### Why is it much quicker to compute ridge regression than regular linear regression?

By my understanding, for a matrix with n samples and p features: Ridge regression using cholesky takes O(p^3) time Ordinary linear regression takes O(p^3) time Singular value decomposition if u, v ...
5k views

### MANOVA with SciPy

Is it possible to conduct a multivariate ANOVA with Python and SciPy? I would prefer to stick to the SciPy stack (NumPy, Statsmodels, Scikit-Learn, Matplotlib, etc.)
235 views

### Best method to detect structure in the residuals?

I need to fit an expression to a large number of datasets. In most, the fit is good; but for some, it is not (residuals have a clear structure). Variance in the data differs between the datasets, so ...
207 views

### Estimating the non- parametric conditional probability

I have a set of observation from two parameters, let say $x$ and $y$ and then I want to make the conditional probability of $x$ for the given $y$, $p(x|y)$. So first I use ...
101 views

### Logistic Regression not quite working

I'm playing around with logistic regression (using scikit-learn, which uses liblinear) I created some example data sets, and it often works well. Since I created the data sets, I already know the ...
117 views

### Fitting two parallel lines with mutually excluding samples

I have a set of (slightly noisy) data taken from two (or more) curves, but taken either from one or the other curve. As an example, consider this code (in reality, data is noisier, especially in how ...
707 views

### How to consider “pratt” zero method in Wilcoxon Test of R, which is available in Python Scipy?

I have a dataset of two columns which represent rank data. These ranks are ranged from 1, 2, and 3. To test the difference in medians between these two columns, I am considering the Wilcoxon rank-sum ...
582 views

### Standard errors different from sqrt of covariance matrix diagonal elements in scipy.odr

I'm writing a code to fit some data (with x_errors and y_errors) to a model with three parameters a, b and c, using the ODR method with scipy.odr. The fitting ...
1k views

### how to generate sliding window features in sci kit learn?

I have a data matrix where each row is measurement in time and each column is a feature. ...
326 views

### Calculating standard error in linear regression

I have seen in a couple of places that the following expression gives the standard error in linear regression: $$se = \sqrt{\frac{1-r^2}{N-2}}$$ where $r$ is the correlation coefficient. I was ...
646 views

### Selecting correct settings for the order of Minkowski distance

I am looking to compute the distance between vectors of word frequencies (and I am new to this). I am trying out the Minkowski distance as implemented in Scipy. The documentation asks me to specify a "...
326 views

### finding the interpolated values of an empirical probability distribution

I have an empirical probability function $p(z)$. In the first column $z$ and the second column contains $p(z)$ values. The data is given as following : data.cat ...
120 views

### Visualise significant differences in a contingency table?

I'm working with pandas and scipy, and I would like to find a quick way to visualise unusual values in a contingency table. Say this is my contingency table: ...
218 views

### Sample data from combination of two probability distributions

I want to generate a mock catalogue. I have access to two independent sets of real data and I want to use their properties to make the mock catalogue: The first one contains the information from ...
23 views

### Multi-variable nonlinear scipy curve_fit

I have been trying to fit my data to a custom equation.which is the following y=(a1/x)+a2*x2+b with curve fit i used curve fit with 1 independant variable it works perfectly but i cannot figure out ...
61 views

### Fitting mixture of Gamma variate functions at once (with python)

I am trying to automate the fitting of a signal composed of several Gamma variate functions with some added noise. However, I face some troubles and I do not know how to deal with it. First I do not ...
24 views

### What is the name of the significance test scipy.stats.pearsonr is using?

The Notes section in the docs describes it in detail but does not name it. I would like to avoid guessing. They use a beta distribution in this way: ...
2k views

### Goodness of fit measurement in Python

I have a (pretty noisy) 1D array of data in Python that I fit a linear function to -- see plot below. The x-axis is a wavelength scale. I would like to measure the "goodness of fit" of this new line ...
166 views

### How to improve Pairwise Euclidean Distance for Similarity Measure

I am trying to identify the most similar stations between two DataFrames like below: ...
68 views

### Can't understand the Beta distribution as described in this paper

I'm reading through the following paper (1). On page 2, section 2.1, there's a description of the selection of a beta distribution used: Since both the choice of s and the strategy for handling the ...
224 views

### How to estimate the leafsize of the kd-tree?

The kd-tree implementation proposed by the scipy python libray asks for the value of the leafsize parameter that is to say the maximum number of points a node can hold. It is by default set to 10. ...
15 views

### Constructing an appropriate null hypothesis?

I have a collection of 20 2D surfaces embedded in 3D. The mean curvature of each surface has been sampled. The number of samples per surface varies from ~700,000 to ~20,000,000. The choice of ...
687 views

### Calculating P-value for multivariate normal distributions?

I'm given a set of 500 ($\sigma_i, \mu_i$) that define a 500-dimensional multivariate normal distribution, where each dimension is effectively independent from all the others. Given a 500-...
26 views

### 3 Layer neural network (feedforward and backpropagation) - Desired error not necessarily achieved due to precision loss

I am trying to implement a 3 layer neural network with feedforward and backpropagation. I have tested my cost function and it is working fine. My gradient function also seems ok. but when I try to ...
27 views

### Mode of function kstest in module scipy.stats.stats

In the function kstest in module scipy.stats.stats, there are two options for the mode, described in the help file: ...
33 views

### How to make chisquare goodness of fit not vulnerable to a single high values?

There is an observation of a process, that I suppose to be a Poisson process. And I'm trying to make a test, to check goodness of fit of my data with Poisson distribution. ...
1k views

### χ² (chi-squared) statistic of scipy.stats.chi2_contingency vs sklearn.feature_selection.chi2

It appears from reading Scikit-learn χ² (chi-squared) statistic and corresponding contingency table that sklearn does not perform a standard contingency table analysis when calculating the χ² ...
631 views

### Computing the correlation / covariance of time series of multiple length and multiple variables

I have N time series of DIFFERENT lengths with i number of data points in each observation. I would like to compute the similarity of each time series and generate M number of clusters. I have tried ...
224 views

### How can I shrink/compress a gaussian survival function based on two or more datapoints?

I have some data that I have fitted with a normal distribution using scipy. Here's how the fit looks like (whether it is a good fit or not is not a problem here) ...
226 views

### Recommendation for fitting mainly linear time-series data to a curve

Most of the time-series data I'll be looking at is linear and uniform - a straightish light parallel to the x-axis. The exceptions I need to find are those that deviate recently, the last one, or two ...
617 views

### Discrepancy Chi-Squared test of independence in scipy

I don't understand the value scipy.stats.chi2_contingency is giving me. The expected values seem fine, but the $\chi^2$ value is different from what manual ...
409 views

### Interpretation and use of the correlation matrix and Python and/or Matlab available codes for errors calculations

I am writing here because I would like you to help me understanding how to interpret and correctly use the correlation matrix. From the book of Taylor, "Introduction to error analysis" I can easily ...
2k views

### R2 cross validation metric vs linear regression R2

I've been using sklearn for a project I'm working on. I'm following this example for analyzing my data. I believe I have an overfitting problem, but before I try to address it I want to make sure I'm ...
63 views

### Relationship between medication and severity of symptoms

I'm looking to determine whether there is a relationship between the type of medication used by an individual and their self-reported severity of symptoms. Independent Variable: Medication Type (...
537 views

### Studying fluctuations in time series

I have some time series to analyze. Given the domain the data is coming from - Time series is supposed to have some fluctuations. A regular periodicity might not be present at all in some cases. ...
2k views

### Align two datasets in Python

I want to develop some python code to align datasets obtained by different instruments recording the same event. As an example, say I have two sets of measurements: ...
174 views

### How do I filter insignificant modes from a kernel density estimation

How do I detect the number of significant modes in a set of data? I have a set of 1 dimensional data where the number of modes are unknown. I'm familiar with python and scikit learn so I use the ...
34 views

### Hypothesis Testing on German Credit Data

I am trying to perform hypothesis testing on German Credit Data ...
17 views

### Determining shape parameter for Generalized Pareto Distribution Scipy

I have a set of values to which I want to fit a Generalized Pareto Distribution. Scipy provides functions for doing so: https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats....
7 views

### What is the purpose of rotating filters while building convolutions with scipy signal?

I recently came across a bit of python code (here) which does 2d convolution with scipy signal. But before the convolve2d operation, the filter was rotated. What is the purpose of that?
19 views

### Correlation one-sided p-value

I want to test a directional correlational hypothesis in the form of: H0: There is a positive correlation (𝜌) between X and Y: 𝜌 > 0. Context: The ...
16 views

### Beta rectangular distribution

I need to sample data from a beta rectangular distribution. As far as I know, this distribution is a mixture of the beta and uniform distribution. I am using Python and in particular SciPy. There is a ...
32 views

### Am I doing it right conducting Mann–Whitney test in scipy?

I have a group of users. I comapre their behaviour before and after certain event. Here are distributions of the metric before and after event. Here are distrubutions ...
73 views

### Trouble optimizing Five Parameter Logistic (5PL) Standard Curve for ELISA data using Python

Cross-posted from Bioinformatics SE here. The program is used for regression of a bioassay standard curve, but the question I'm asking is statistical in nature. I am attempting to write a tool to be ...
34 views

### mismatch in sampling between t distribution and normal-inverse-gamma distribution

I am looking at equivalence of sampling between t distribution and normal-inverse-gamma (NIG) distribution in python. The results don't match, and I want to see if there's a mistake in how I am ...