Questions tagged [scipy]
SciPy is a Python-based ecosystem of open-source software for mathematics, science, and engineering.
0
votes
0answers
11 views
What can I use to create an infographic using data to scale? [closed]
Very simply, what can be used to make images like these? I want to create a very similar infographic with data that needs to scaled i.e. I can't use PowerPoint or something for this because the points ...
1
vote
1answer
14 views
Weighted Minkowski distance for DBSCAN
I'm trying to make clustering of image's pixels with DBSCAN, using RGB values and pixel's coordinates as features. It works well with just RGB values as features, but I want pixels with the same color,...
0
votes
0answers
59 views
Gamma Distribution CDF Intuition; Are My Parameters Correct?
I'm learning about gamma distributions and I want to make sure my reasoning is correct for an example I found here.
Suppose you are fishing and you expect to get a fish once every 1/2
hour. ...
1
vote
1answer
26 views
Goodness of fit measurement in Python
I have a (pretty noisy) 1D array of data in Python that I fit a linear function to -- see plot below. The x-axis is a wavelength scale.
I would like to measure the "goodness of fit" of this new line ...
1
vote
0answers
17 views
Optimizing a piecewise-linear, convex function [closed]
I want to find the matrix $U$ that minimizes the following loss function:
$$ \min_U( \max_{0 \le j \le t} (\operatorname{tr}(W_jU) + b_j) + \operatorname{tr}(U^TU)) $$
The given loss function is ...
0
votes
0answers
5 views
Truncate a Hierarchical Clustering tree in order to get the cophenetic coefficient
I've gone ahead and clustered a dataset using a Euclidian Hierarchical Clustering algorithm:
...
0
votes
1answer
46 views
Chi Square Analysis Throws Error - The internally computed table of expected frequencies has a zero element at (0,)
I am trying to see the association between two variables. I used Chi-Square analysis in Scipy package in Python.
Here is the crosstab result of the two variables:
...
1
vote
1answer
28 views
Difference between p-value of chi-squared calculation, and p-value in “critical value” lookup table
I am learning about chi-squared. It has two steps; a) chi-squared calculation b) critical value
When I calculate chi-squared, the output includes a) chi-squared statistic (220.5) and b) a p-value (...
0
votes
1answer
23 views
least square with equality constraint and singular matrix
I have the following problem to solve:
Ax = b where A is singular.
To resolve this I introduce a condition Cx = d
Even after this, I am not able to solve it using scipy.optimize.minimize as it ...
1
vote
0answers
74 views
How to improve Pairwise Euclidean Distance for Similarity Measure
I am trying to identify the most similar stations between two DataFrames like below:
...
1
vote
0answers
51 views
Can't understand the Beta distribution as described in this paper
I'm reading through the following paper (1). On page 2, section 2.1, there's a description of the selection of a beta distribution used:
Since both the choice of s and the strategy for handling the
...
0
votes
1answer
39 views
Fixing the maximum distance within a cluster
I am trying to cluster geographical locations in such a way that all the locations inside each cluster are at max within 25 miles of each other. For this, I am using Agglomerative clustering. I am ...
3
votes
1answer
302 views
Why is computing ridge regression with a Cholesky decomposition much quicker than using SVD?
By my understanding, for a matrix with n samples and p features:
Ridge regression using Cholesky decomposition takes O(p^3) time
Ridge regression using SVD takes O(p^3) time
Computing SVD when only ...
3
votes
0answers
43 views
Why is it much quicker to compute ridge regression than regular linear regression?
By my understanding, for a matrix with n samples and p features:
Ridge regression using cholesky takes O(p^3) time
Ordinary linear regression takes O(p^3) time
Singular value decomposition if u, v ...
0
votes
0answers
12 views
How to stop fmin_slqsp from converging to local minimum?
I want to run linear regression with inequality constraints and non-negative coefficients.
My data looks roughly like this:
...
0
votes
0answers
130 views
Proper way to plot confidence interval under normal distribution curve?
I write a small program which calculates a confidence interval for a given norm value. The output is then visualized by plotting the confidence interval and the corresponding hypothetical norm value ...
0
votes
0answers
31 views
Numerical approximation of sandwich estimator
I have found the MLE of a model using scipy.optimize.minimize with the BFGS algorithm. As is commonly done, I intend to estimate the variance from
$$ Var(\theta_0) ...
0
votes
1answer
23 views
Using ANOVA to judge Yes/Sometimes/No questions ability to significantly predict a continuous variable?
I have a dataset that looks something like this:
...
1
vote
1answer
21 views
How to estimate the leafsize of the kd-tree?
The kd-tree implementation proposed by the scipy python libray asks for the value of the leafsize parameter that is to say the maximum number of points a node can hold. It is by default set to 10.
...
1
vote
0answers
12 views
Constructing an appropriate null hypothesis?
I have a collection of 20 2D surfaces embedded in 3D. The mean curvature of each surface has been sampled.
The number of samples per surface varies from ~700,000 to ~20,000,000. The choice of ...
1
vote
0answers
177 views
Calculating P-value for multivariate normal distributions?
I'm given a set of 500 ($\sigma_i, \mu_i$) that define a 500-dimensional multivariate normal distribution, where each dimension is effectively independent from all the others.
Given a 500-...
2
votes
1answer
342 views
How does scipy calculate the binomial CDF so fast?
I ran scipy.stats.binom.cdf(500006, 1000000, 0.5) and it took less than a milisecond. This is crazy as binomial CDF involves summing up a bunch of binomial coefficients. What approximation algorithm ...
0
votes
1answer
111 views
How to perform batch training using L-BFGS?
I want to train a neural network for regression.
The neural network is actually composed of 4 separate child neural networks, each child neural network has a layer structure of
...
2
votes
1answer
35 views
1
vote
3answers
139 views
How to calculate the p-value of a test, that checked for a binary property?
16081 subjects were randomly assigned into two groups:
test group: 7916 subjects
control group: 8165 subjects
Only the test group was exposed to something. During the test period
10 subjects of ...
0
votes
1answer
150 views
Normal distribution test ( D'Agostino and Pearson's) in Scipy - results don't make sense
I want to test if an array of numbers is approximately normally distributed.
...
1
vote
0answers
19 views
3 Layer neural network (feedforward and backpropagation) - Desired error not necessarily achieved due to precision loss
I am trying to implement a 3 layer neural network with feedforward and backpropagation.
I have tested my cost function and it is working fine. My gradient function also seems ok.
but when I try to ...
0
votes
1answer
29 views
Remove individual points and find slope
I am trying to delete one pair of x and y coordinates from a set of 10 data points and get the slope for the other 9 points.
How do I go about this issue?
Attached herewith is an image of what I am ...
1
vote
0answers
221 views
Update R2 and slope by removing outliers-python [closed]
I am using the following code to find out Linregress parameters:
...
0
votes
1answer
345 views
Statsmodels' Negative Binomial: after .fit_regularized(), how to turn PMF into PPF to get the discrete values?
I used the package statsmodels to fit a Negative Binomial to my data. This data contains ~1500 samples with 21 covariates. Since I have overdispersion in my data ...
0
votes
1answer
17 views
Can findings based on low values be considered valid?
Lets say i do have a TimeSeries with N > 300 but values like [0, 1, 2, 1, 2, 0, 2, ...] representing the visitors count of a website per day.
Since there are only few visitors and each of them can be ...
0
votes
0answers
77 views
Problem with U Mann Whitney test in scipy for nominal values
When I am using scipy.stats.mannwhitneyu sometimes I got this error:
TypeError: '<' not supported between instances of 'int' and 'str'
Should not it work? I have noticed that scipy.stats....
0
votes
0answers
101 views
Periodogram interpretaion to get main cycles for time serie
I try to use the periodogram in order to get main cycle on my time series. My time series is a the result of measure every hour of one variable related with the atmosphere. So in one day I have 24 ...
1
vote
1answer
441 views
How to know if a parameter is statistically significant in a “curve_fit” estimation? [closed]
I use curve_fit from scipy to estimate parameter values from a specific function.
...
0
votes
1answer
97 views
nmf in scipy returns components with all zero weights
I'm trying to understand whether this behavior is a bug or a feature.
Essentially, I have a dataset of ten thousand short pieces of text. I have used the CountVectorizer function to turn this into a ...
1
vote
0answers
22 views
Mode of function kstest in module scipy.stats.stats
In the function kstest in module scipy.stats.stats, there are two options for the mode, described in the help file:
...
2
votes
1answer
312 views
Gaussian Mixture Model
with the following code I fit a Gaussian Mixture Model to arbitrarily created data. The code is working. The only thing I encounter is that during the calculation of the multivariate_normal I ...
0
votes
1answer
533 views
Bandwidth parameters in multivariate KDE using scipy.stats.gaussian_kde
I am working on a project which involves implementing in Python two different density estimation functions over multivariate data; one using N-d histograms and the other using kernel density ...
0
votes
1answer
126 views
0
votes
1answer
169 views
Interpreting p value in SciPy Welch's T-Test [duplicate]
Can someone help me understand the meaning of the p-value of the Whelch's T Test in SciPy?
test = stats.ttest_ind(x, y, equal_var=False)
gives me
...
3
votes
1answer
3k views
How to Compare the Data Distribution of 2 datasets?
I'm having trouble to understand how to compare 2 sets of data by their distribution .
For Example,
how can I understand that column X100 has the same distribution as column Y1?
Also, is there a ...
1
vote
2answers
128 views
Normality test giving confusing results
So I was transforming data for machine learning purposes and checking whether I should use the data or log-transform it. In addition to creating histograms I decided to test for normality using ...
0
votes
0answers
331 views
Tuning skopt to minimise correctly
I have a problem where I need to fit a pair of parameters in a model to best fit observed data for hundreds of instances. I need to determine these parameters to high accuracy, rather than just ...
2
votes
1answer
623 views
Understanding hypothesis testing (Kruskal Wallis, Mann-Whitney) when there are tied values
tldr: In Python's Scipy stats modules, the kruskal and mannwhitneyu functions will both raise an error if called with only ...
2
votes
0answers
81 views
Fitting two parallel lines with mutually excluding samples
I have a set of (slightly noisy) data taken from two (or more) curves, but taken either from one or the other curve. As an example, consider this code (in reality, data is noisier, especially in how ...
1
vote
2answers
328 views
Time Series: use of Box-Cox to reduce the “noise”
I am researching the best method to use with time series. FBprophet (Python) seems like a strong option.
To prepare time series for Prophet I am thinking about using boxcox and inv_boxcox at the end
...
0
votes
0answers
132 views
scipy.optimize.minimize How to manage different units?
I am wondering how spicy.optimize.minimize manages different units in the searched vector. My vector is [x,y,z,yaw,pitch,roll]. The first three are translations in <...
2
votes
0answers
435 views
How to consider “pratt” zero method in Wilcoxon Test of R, which is available in Python Scipy?
I have a dataset of two columns which represent rank data. These ranks are ranged from 1, 2, and 3. To test the difference in medians between these two columns, I am considering the Wilcoxon rank-sum ...
2
votes
2answers
166 views
Database-friendly random projections with Numpy
In his well known paper [1], Achlioptas showed that Random Projections could be performed with a sparse projection matrix, whose nonzero entries are either $1$ or $-1$.
I have noticed that scikit-...
1
vote
0answers
89 views
Working between sklearn and scipy for convex optimization [closed]
I have a method for solving convex programs using an alternative method of iterative procedures and would like to compare this to current solvers methods.
My approach relies upon randomly projecting ...
