Questions tagged [scipy]

SciPy is a Python-based ecosystem of open-source software for mathematics, science, and engineering.

77 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6
votes
0answers
1k views
6
votes
0answers
598 views

Goodness-of-fit for Discrete Distributions

I've been doing some data analysis with Scipy. So far I accomplished this with continuous distributions: I can fit a probability distribution to a set of data points using a maximum likelihood fit. ...
5
votes
0answers
5k views

Assessing A/B test results using Python

Problem's setting Assume having results of an A/B test. You let you users experience two variants of your website and you counted how many converted: ...
3
votes
0answers
91 views

Why is it much quicker to compute ridge regression than regular linear regression?

By my understanding, for a matrix with n samples and p features: Ridge regression using cholesky takes O(p^3) time Ordinary linear regression takes O(p^3) time Singular value decomposition if u, v ...
3
votes
0answers
5k views

MANOVA with SciPy

Is it possible to conduct a multivariate ANOVA with Python and SciPy? I would prefer to stick to the SciPy stack (NumPy, Statsmodels, Scikit-Learn, Matplotlib, etc.)
3
votes
0answers
235 views

Best method to detect structure in the residuals?

I need to fit an expression to a large number of datasets. In most, the fit is good; but for some, it is not (residuals have a clear structure). Variance in the data differs between the datasets, so ...
3
votes
0answers
207 views

Estimating the non- parametric conditional probability

I have a set of observation from two parameters, let say $x$ and $y$ and then I want to make the conditional probability of $x$ for the given $y$, $p(x|y)$. So first I use ...
3
votes
0answers
101 views

Logistic Regression not quite working

I'm playing around with logistic regression (using scikit-learn, which uses liblinear) I created some example data sets, and it often works well. Since I created the data sets, I already know the ...
2
votes
0answers
117 views

Fitting two parallel lines with mutually excluding samples

I have a set of (slightly noisy) data taken from two (or more) curves, but taken either from one or the other curve. As an example, consider this code (in reality, data is noisier, especially in how ...
2
votes
0answers
707 views

How to consider “pratt” zero method in Wilcoxon Test of R, which is available in Python Scipy?

I have a dataset of two columns which represent rank data. These ranks are ranged from 1, 2, and 3. To test the difference in medians between these two columns, I am considering the Wilcoxon rank-sum ...
2
votes
0answers
582 views

Standard errors different from sqrt of covariance matrix diagonal elements in scipy.odr

I'm writing a code to fit some data (with x_errors and y_errors) to a model with three parameters a, b and c, using the ODR method with scipy.odr. The fitting ...
2
votes
0answers
1k views

how to generate sliding window features in sci kit learn?

I have a data matrix where each row is measurement in time and each column is a feature. ...
2
votes
0answers
326 views

Calculating standard error in linear regression

I have seen in a couple of places that the following expression gives the standard error in linear regression: $$se = \sqrt{\frac{1-r^2}{N-2}}$$ where $r$ is the correlation coefficient. I was ...
2
votes
0answers
646 views

Selecting correct settings for the order of Minkowski distance

I am looking to compute the distance between vectors of word frequencies (and I am new to this). I am trying out the Minkowski distance as implemented in Scipy. The documentation asks me to specify a "...
2
votes
0answers
326 views

finding the interpolated values of an empirical probability distribution

I have an empirical probability function $p(z)$. In the first column $z$ and the second column contains $p(z)$ values. The data is given as following : data.cat ...
2
votes
0answers
120 views

Visualise significant differences in a contingency table?

I'm working with pandas and scipy, and I would like to find a quick way to visualise unusual values in a contingency table. Say this is my contingency table: ...
2
votes
0answers
218 views

Sample data from combination of two probability distributions

I want to generate a mock catalogue. I have access to two independent sets of real data and I want to use their properties to make the mock catalogue: The first one contains the information from ...
1
vote
1answer
23 views

Multi-variable nonlinear scipy curve_fit

I have been trying to fit my data to a custom equation.which is the following y=(a1/x)+a2*x2+b with curve fit i used curve fit with 1 independant variable it works perfectly but i cannot figure out ...
1
vote
0answers
61 views

Fitting mixture of Gamma variate functions at once (with python)

I am trying to automate the fitting of a signal composed of several Gamma variate functions with some added noise. However, I face some troubles and I do not know how to deal with it. First I do not ...
1
vote
0answers
24 views

What is the name of the significance test scipy.stats.pearsonr is using?

The Notes section in the docs describes it in detail but does not name it. I would like to avoid guessing. They use a beta distribution in this way: ...
1
vote
1answer
2k views

Goodness of fit measurement in Python

I have a (pretty noisy) 1D array of data in Python that I fit a linear function to -- see plot below. The x-axis is a wavelength scale. I would like to measure the "goodness of fit" of this new line ...
1
vote
0answers
166 views

How to improve Pairwise Euclidean Distance for Similarity Measure

I am trying to identify the most similar stations between two DataFrames like below: ...
1
vote
0answers
68 views

Can't understand the Beta distribution as described in this paper

I'm reading through the following paper (1). On page 2, section 2.1, there's a description of the selection of a beta distribution used: Since both the choice of s and the strategy for handling the ...
1
vote
1answer
224 views

How to estimate the leafsize of the kd-tree?

The kd-tree implementation proposed by the scipy python libray asks for the value of the leafsize parameter that is to say the maximum number of points a node can hold. It is by default set to 10. ...
1
vote
0answers
15 views

Constructing an appropriate null hypothesis?

I have a collection of 20 2D surfaces embedded in 3D. The mean curvature of each surface has been sampled. The number of samples per surface varies from ~700,000 to ~20,000,000. The choice of ...
1
vote
0answers
687 views

Calculating P-value for multivariate normal distributions?

I'm given a set of 500 ($\sigma_i, \mu_i$) that define a 500-dimensional multivariate normal distribution, where each dimension is effectively independent from all the others. Given a 500-...
1
vote
0answers
26 views

3 Layer neural network (feedforward and backpropagation) - Desired error not necessarily achieved due to precision loss

I am trying to implement a 3 layer neural network with feedforward and backpropagation. I have tested my cost function and it is working fine. My gradient function also seems ok. but when I try to ...
1
vote
0answers
27 views

Mode of function kstest in module scipy.stats.stats

In the function kstest in module scipy.stats.stats, there are two options for the mode, described in the help file: ...
1
vote
0answers
33 views

How to make chisquare goodness of fit not vulnerable to a single high values?

There is an observation of a process, that I suppose to be a Poisson process. And I'm trying to make a test, to check goodness of fit of my data with Poisson distribution. ...
1
vote
0answers
1k views

χ² (chi-squared) statistic of scipy.stats.chi2_contingency vs sklearn.feature_selection.chi2

It appears from reading Scikit-learn χ² (chi-squared) statistic and corresponding contingency table that sklearn does not perform a standard contingency table analysis when calculating the χ² ...
1
vote
0answers
631 views

Computing the correlation / covariance of time series of multiple length and multiple variables

I have N time series of DIFFERENT lengths with i number of data points in each observation. I would like to compute the similarity of each time series and generate M number of clusters. I have tried ...
1
vote
0answers
224 views

How can I shrink/compress a gaussian survival function based on two or more datapoints?

I have some data that I have fitted with a normal distribution using scipy. Here's how the fit looks like (whether it is a good fit or not is not a problem here) ...
1
vote
1answer
226 views

Recommendation for fitting mainly linear time-series data to a curve

Most of the time-series data I'll be looking at is linear and uniform - a straightish light parallel to the x-axis. The exceptions I need to find are those that deviate recently, the last one, or two ...
1
vote
0answers
617 views

Discrepancy Chi-Squared test of independence in scipy

I don't understand the value scipy.stats.chi2_contingency is giving me. The expected values seem fine, but the $\chi^2$ value is different from what manual ...
1
vote
0answers
409 views

Interpretation and use of the correlation matrix and Python and/or Matlab available codes for errors calculations

I am writing here because I would like you to help me understanding how to interpret and correctly use the correlation matrix. From the book of Taylor, "Introduction to error analysis" I can easily ...
1
vote
0answers
2k views

R2 cross validation metric vs linear regression R2

I've been using sklearn for a project I'm working on. I'm following this example for analyzing my data. I believe I have an overfitting problem, but before I try to address it I want to make sure I'm ...
1
vote
0answers
63 views

Relationship between medication and severity of symptoms

I'm looking to determine whether there is a relationship between the type of medication used by an individual and their self-reported severity of symptoms. Independent Variable: Medication Type (...
1
vote
0answers
537 views

Studying fluctuations in time series

I have some time series to analyze. Given the domain the data is coming from - Time series is supposed to have some fluctuations. A regular periodicity might not be present at all in some cases. ...
1
vote
0answers
2k views

Align two datasets in Python

I want to develop some python code to align datasets obtained by different instruments recording the same event. As an example, say I have two sets of measurements: ...
1
vote
0answers
174 views

How do I filter insignificant modes from a kernel density estimation

How do I detect the number of significant modes in a set of data? I have a set of 1 dimensional data where the number of modes are unknown. I'm familiar with python and scikit learn so I use the ...
0
votes
0answers
34 views

Hypothesis Testing on German Credit Data

I am trying to perform hypothesis testing on German Credit Data ...
0
votes
1answer
17 views

Determining shape parameter for Generalized Pareto Distribution Scipy

I have a set of values to which I want to fit a Generalized Pareto Distribution. Scipy provides functions for doing so: https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats....
0
votes
0answers
7 views

What is the purpose of rotating filters while building convolutions with scipy signal?

I recently came across a bit of python code (here) which does 2d convolution with scipy signal. But before the convolve2d operation, the filter was rotated. What is the purpose of that?
0
votes
1answer
19 views

Correlation one-sided p-value

I want to test a directional correlational hypothesis in the form of: H0: There is a positive correlation (𝜌) between X and Y: 𝜌 > 0. Context: The ...
0
votes
0answers
16 views

Beta rectangular distribution

I need to sample data from a beta rectangular distribution. As far as I know, this distribution is a mixture of the beta and uniform distribution. I am using Python and in particular SciPy. There is a ...
0
votes
0answers
32 views

Am I doing it right conducting Mann–Whitney test in scipy?

I have a group of users. I comapre their behaviour before and after certain event. Here are distributions of the metric before and after event. Here are distrubutions ...
0
votes
0answers
73 views

Trouble optimizing Five Parameter Logistic (5PL) Standard Curve for ELISA data using Python

Cross-posted from Bioinformatics SE here. The program is used for regression of a bioassay standard curve, but the question I'm asking is statistical in nature. I am attempting to write a tool to be ...
0
votes
1answer
34 views

mismatch in sampling between t distribution and normal-inverse-gamma distribution

I am looking at equivalence of sampling between t distribution and normal-inverse-gamma (NIG) distribution in python. The results don't match, and I want to see if there's a mistake in how I am ...
0
votes
0answers
45 views

Does scipy.stats.pearsonr() have assumption of continuous value?

Problem I have to calculate correlation between two variables, let's say, variable A and B. Variable A contains discrete values - "How many times a student has done a task?". Variable B contains ...
0
votes
0answers
28 views

What are the various ways I can use to find out correlations between multiple columns

I'm writing a program to find the relationship between Bitcoin and various Altcoins (ETH, XRP, LTC, TRON). I want to find out the best Altcoin or the best subset of Altcoins which can give a ...