Questions tagged [scipy]
SciPy is a Python-based ecosystem of open-source software for mathematics, science, and engineering.
203
questions
0
votes
1answer
4 views
What statistical significance test should I use in this situation with two variables?
I have two variables: Wind speed (MPH) and passing yards in the NFL. I want to test if wind speed has a statistically significant effect on a player's passing yards.
I used independent t-test from ...
0
votes
1answer
17 views
SciPy Documentations outdated?
On web page:
scipy.stats.skew
https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.skew.html
But when I try it:
0
votes
0answers
16 views
Beta rectangular distribution
I need to sample data from a beta rectangular distribution. As far as I know, this distribution is a mixture of the beta and uniform distribution. I am using Python and in particular SciPy. There is a ...
0
votes
1answer
23 views
How does scipy's Kruskal-Wallis compute its H-value?
My textbook states the following formula to calculate the H-value of the Kruskal-Wallis test (without tie ranks):
Let's use this formula to compute H for a really simple example dataset:
group 1 ...
0
votes
0answers
32 views
Am I doing it right conducting Mann–Whitney test in scipy?
I have a group of users. I comapre their behaviour before and after certain event.
Here are distributions of the metric before and after event.
Here are distrubutions ...
0
votes
0answers
30 views
Trouble optimizing Five Parameter Logistic (5PL) Standard Curve for ELISA data using Python
Cross-posted from Bioinformatics SE here. The program is used for regression of a bioassay standard curve, but the question I'm asking is statistical in nature.
I am attempting to write a tool to be ...
1
vote
0answers
45 views
Fitting mixture of Gamma variate functions at once (with python)
I am trying to automate the fitting of a signal composed of several Gamma variate functions with some added noise. However, I face some troubles and I do not know how to deal with it. First I do not ...
0
votes
1answer
32 views
mismatch in sampling between t distribution and normal-inverse-gamma distribution
I am looking at equivalence of sampling between t distribution and normal-inverse-gamma (NIG) distribution in python. The results don't match, and I want to see if there's a mistake in how I am ...
1
vote
1answer
21 views
Scipy Kolmogorov Smirnov test
I wanted to know what is significance level (alpha) in scipy's Kolmogorov test.
Here is a link to documentation page.
I couldn't find anything in its documentation.
1
vote
1answer
16 views
t test and kde plot does not match
I cannot understand the results of scipy independent two samples tests on my my dataset. the results of the test as I understand it suggest there is a significant difference between the means of the ...
0
votes
0answers
38 views
Does scipy.stats.pearsonr() have assumption of continuous value?
Problem
I have to calculate correlation between two variables, let's say, variable A and B. Variable A contains discrete values - "How many times a student has done a task?". Variable B contains ...
0
votes
0answers
27 views
What are the various ways I can use to find out correlations between multiple columns
I'm writing a program to find the relationship between Bitcoin and various Altcoins (ETH, XRP, LTC, TRON). I want to find out the best Altcoin or the best subset of Altcoins which can give a ...
0
votes
2answers
46 views
Two-sample KS test results vs distributions
WARNING : MAJOR UPDATE BELOW, I've made a mistake
I am currently working on validating a predictive model that was made some years ago. In doing so, I am comparing the distribution of the Target ...
1
vote
0answers
54 views
Weird sample variance when creating a lognromal sample with pre-defined mean and variance [duplicate]
I have been working with lognormal distributions as a proposal distribution for some MCMC routines which require the proposal distribution to have some pre-defined sample mean and variance, $m$ and $v$...
1
vote
1answer
34 views
What is the reasoning behind the default eps value for the 'SLSQP' method in SciPy's minimize function? [closed]
From here:
https://docs.scipy.org/doc/scipy/reference/optimize.minimize-slsqp.html#optimize-minimize-slsqp
It says that the default value for the eps option is <...
0
votes
1answer
57 views
Generating synthetic data based on mean, skewness, standard deviation and autocorrelation
Given that I have the mean, standard deviation, skewness and autocorrelation,
How do I generate 1000 years of random data based on the above parameters in python or Matlab?
I know for example I can ...
0
votes
0answers
17 views
linear correlation in N-Dimensional sample
I've an N-dimensional sample linearly correlated to each other in pairs. That is, every event includes N variables, and every variable is a predictor to all the other (or, in other words, putting the ...
3
votes
1answer
172 views
Kolmogorov-Smirnov (ks_2samp) p-value not as expected - Wrong test or understanding?
Context
I am using scipy's ks_samp in order to apply the Kolmogorov-Smirnov-test.
The data I use is twofold:
I have a dataset ...
1
vote
1answer
29 views
How do I statistically compare the readings from two different sensors? [duplicate]
I have temperature and humidity readings from two different sensors (located in the same geographical area). The data is logged every hour and I have data for more than a year, logged this way 2/1/...
0
votes
1answer
33 views
Comparing sensor data across multiple experiments
I have some questions about doing statistics on a series of sensor measurements.
I have 12 series of data of sensor measurements (a force sensor, measuring at 10 Hz for 30 minutes). Each series ...
0
votes
0answers
19 views
Location and scale estimation for log-logistic in scipy
I have to estimate the location and scale parameters for a log-logistic or fisk distribution using scipy. The shape parameter is already defined and I would like to estimate loc and scale parameter ...
0
votes
1answer
120 views
Why does the mean for a beta distribution when input into the scipy beta cdf not return 0.5? [duplicate]
Very likely I am missing some fundamentals here, or my python coding is not up to scratch.
When I use the standard formula to calculate the mean for a beta distribution ~ $B(\alpha,\beta)$
$$ \mu = \...
1
vote
0answers
19 views
What is the name of the significance test scipy.stats.pearsonr is using?
The Notes section in the docs describes it in detail but does not name it. I would like to avoid guessing.
They use a beta distribution in this way:
...
0
votes
0answers
24 views
Is Fisher Kurtosis the same as Excess Kurtosis?
The documentation for scipy.stats.describe() defines the returned kurtosis like so:
...
0
votes
0answers
22 views
Fitting data to a distribution that appears lognormal without right-hump
I will like to fit a distribution and extract its moments in Python, given Length of Stay (LOS) data in hospitals.
Asking out of curiosity, suppose the plot of the data looks like:
The reason why I ...
1
vote
1answer
297 views
Fitting Beta Distributions to Data
I am trying to reproduce some beta distribution parameters found in this published paper. I have two data sets, y1 and y2, that ...
1
vote
0answers
111 views
Calculating standard errors of non-OLS regressions in python and scipy [closed]
Scipy/statsmodels provides the ability to get a number of metrics from the output of an OLS regression model, a few examples of which (from statsmodels) are the following:
...
1
vote
1answer
221 views
Weighted Minkowski distance for DBSCAN
I'm trying to make clustering of image's pixels with DBSCAN, using RGB values and pixel's coordinates as features. It works well with just RGB values as features, but I want pixels with the same color,...
0
votes
0answers
151 views
Gamma Distribution CDF Intuition; Are My Parameters Correct?
I'm learning about gamma distributions and I want to make sure my reasoning is correct for an example I found here.
Suppose you are fishing and you expect to get a fish once every 1/2
hour. ...
1
vote
1answer
1k views
Goodness of fit measurement in Python
I have a (pretty noisy) 1D array of data in Python that I fit a linear function to -- see plot below. The x-axis is a wavelength scale.
I would like to measure the "goodness of fit" of this new line ...
1
vote
0answers
36 views
Optimizing a piecewise-linear, convex function [closed]
I want to find the matrix $U$ that minimizes the following loss function:
$$ \min_U( \max_{0 \le j \le t} (\operatorname{tr}(W_jU) + b_j) + \operatorname{tr}(U^TU)) $$
The given loss function is ...
0
votes
0answers
66 views
Truncate a Hierarchical Clustering tree in order to get the cophenetic coefficient
I've gone ahead and clustered a dataset using a Euclidian Hierarchical Clustering algorithm:
...
0
votes
1answer
228 views
Chi Square Analysis Throws Error - The internally computed table of expected frequencies has a zero element at (0,)
I am trying to see the association between two variables. I used Chi-Square analysis in Scipy package in Python.
Here is the crosstab result of the two variables:
...
1
vote
1answer
102 views
Difference between p-value of chi-squared calculation, and p-value in “critical value” lookup table
I am learning about chi-squared. It has two steps; a) chi-squared calculation b) critical value
When I calculate chi-squared, the output includes a) chi-squared statistic (220.5) and b) a p-value (...
0
votes
1answer
37 views
least square with equality constraint and singular matrix
I have the following problem to solve:
Ax = b where A is singular.
To resolve this I introduce a condition Cx = d
Even after this, I am not able to solve it using scipy.optimize.minimize as it ...
1
vote
0answers
162 views
How to improve Pairwise Euclidean Distance for Similarity Measure
I am trying to identify the most similar stations between two DataFrames like below:
...
1
vote
0answers
65 views
Can't understand the Beta distribution as described in this paper
I'm reading through the following paper (1). On page 2, section 2.1, there's a description of the selection of a beta distribution used:
Since both the choice of s and the strategy for handling the
...
0
votes
1answer
330 views
Fixing the maximum distance within a cluster
I am trying to cluster geographical locations in such a way that all the locations inside each cluster are at max within 25 miles of each other. For this, I am using Agglomerative clustering. I am ...
3
votes
1answer
468 views
Why is computing ridge regression with a Cholesky decomposition much quicker than using SVD?
By my understanding, for a matrix with n samples and p features:
Ridge regression using Cholesky decomposition takes O(p^3) time
Ridge regression using SVD takes O(p^3) time
Computing SVD when only ...
3
votes
0answers
75 views
Why is it much quicker to compute ridge regression than regular linear regression?
By my understanding, for a matrix with n samples and p features:
Ridge regression using cholesky takes O(p^3) time
Ordinary linear regression takes O(p^3) time
Singular value decomposition if u, v ...
0
votes
0answers
61 views
How to stop fmin_slqsp from converging to local minimum?
I want to run linear regression with inequality constraints and non-negative coefficients.
My data looks roughly like this:
...
0
votes
0answers
710 views
Proper way to plot confidence interval under normal distribution curve?
I write a small program which calculates a confidence interval for a given norm value. The output is then visualized by plotting the confidence interval and the corresponding hypothetical norm value ...
0
votes
0answers
48 views
Numerical approximation of sandwich estimator
I have found the MLE of a model using scipy.optimize.minimize with the BFGS algorithm. As is commonly done, I intend to estimate the variance from
$$ Var(\theta_0) ...
0
votes
1answer
28 views
Using ANOVA to judge Yes/Sometimes/No questions ability to significantly predict a continuous variable?
I have a dataset that looks something like this:
...
1
vote
1answer
187 views
How to estimate the leafsize of the kd-tree?
The kd-tree implementation proposed by the scipy python libray asks for the value of the leafsize parameter that is to say the maximum number of points a node can hold. It is by default set to 10.
...
1
vote
0answers
15 views
Constructing an appropriate null hypothesis?
I have a collection of 20 2D surfaces embedded in 3D. The mean curvature of each surface has been sampled.
The number of samples per surface varies from ~700,000 to ~20,000,000. The choice of ...
1
vote
0answers
614 views
Calculating P-value for multivariate normal distributions?
I'm given a set of 500 ($\sigma_i, \mu_i$) that define a 500-dimensional multivariate normal distribution, where each dimension is effectively independent from all the others.
Given a 500-...
2
votes
1answer
396 views
How does scipy calculate the binomial CDF so fast?
I ran scipy.stats.binom.cdf(500006, 1000000, 0.5) and it took less than a milisecond. This is crazy as binomial CDF involves summing up a bunch of binomial coefficients. What approximation algorithm ...
0
votes
1answer
330 views
How to perform batch training using L-BFGS?
I want to train a neural network for regression.
The neural network is actually composed of 4 separate child neural networks, each child neural network has a layer structure of
...
2
votes
1answer
48 views
