# Questions tagged [scipy]

SciPy is a Python-based ecosystem of open-source software for mathematics, science, and engineering.

211 questions
Filter by
Sorted by
Tagged with
51 views

### Numerical approximation of sandwich estimator

I have found the MLE of a model using scipy.optimize.minimize with the BFGS algorithm. As is commonly done, I intend to estimate the variance from  Var(\theta_0) ...
28 views

### Using ANOVA to judge Yes/Sometimes/No questions ability to significantly predict a continuous variable?

I have a dataset that looks something like this: ...
221 views

### How to estimate the leafsize of the kd-tree?

The kd-tree implementation proposed by the scipy python libray asks for the value of the leafsize parameter that is to say the maximum number of points a node can hold. It is by default set to 10. ...
15 views

### Constructing an appropriate null hypothesis?

I have a collection of 20 2D surfaces embedded in 3D. The mean curvature of each surface has been sampled. The number of samples per surface varies from ~700,000 to ~20,000,000. The choice of ...
681 views

### Calculating P-value for multivariate normal distributions?

I'm given a set of 500 ($\sigma_i, \mu_i$) that define a 500-dimensional multivariate normal distribution, where each dimension is effectively independent from all the others. Given a 500-...
401 views

### How does scipy calculate the binomial CDF so fast?

I ran scipy.stats.binom.cdf(500006, 1000000, 0.5) and it took less than a milisecond. This is crazy as binomial CDF involves summing up a bunch of binomial coefficients. What approximation algorithm ...
366 views

### How to perform batch training using L-BFGS?

I want to train a neural network for regression. The neural network is actually composed of 4 separate child neural networks, each child neural network has a layer structure of ...
53 views

...
1k views

### How to calculate the p-value of a test, that checked for a binary property?

16081 subjects were randomly assigned into two groups: test group: 7916 subjects control group: 8165 subjects Only the test group was exposed to something. During the test period 10 subjects of ...
304 views

### Normal distribution test ( D'Agostino and Pearson's) in Scipy - results don't make sense

I want to test if an array of numbers is approximately normally distributed. ...
26 views

### 3 Layer neural network (feedforward and backpropagation) - Desired error not necessarily achieved due to precision loss

I am trying to implement a 3 layer neural network with feedforward and backpropagation. I have tested my cost function and it is working fine. My gradient function also seems ok. but when I try to ...
34 views

### Remove individual points and find slope

I am trying to delete one pair of x and y coordinates from a set of 10 data points and get the slope for the other 9 points. How do I go about this issue? Attached herewith is an image of what I am ...
932 views

### Statsmodels' Negative Binomial: after .fit_regularized(), how to turn PMF into PPF to get the discrete values?

I used the package statsmodels to fit a Negative Binomial to my data. This data contains ~1500 samples with 21 covariates. Since I have overdispersion in my data ...
17 views

### Can findings based on low values be considered valid?

Lets say i do have a TimeSeries with N > 300 but values like [0, 1, 2, 1, 2, 0, 2, ...] representing the visitors count of a website per day. Since there are only few visitors and each of them can be ...
840 views

### How to know if a parameter is statistically significant in a “curve_fit” estimation? [closed]

I use curve_fit from scipy to estimate parameter values from a specific function. ...
184 views

### nmf in scipy returns components with all zero weights

I'm trying to understand whether this behavior is a bug or a feature. Essentially, I have a dataset of ten thousand short pieces of text. I have used the CountVectorizer function to turn this into a ...
27 views

### Mode of function kstest in module scipy.stats.stats

In the function kstest in module scipy.stats.stats, there are two options for the mode, described in the help file: ...
542 views

### Gaussian Mixture Model

with the following code I fit a Gaussian Mixture Model to arbitrarily created data. The code is working. The only thing I encounter is that during the calculation of the multivariate_normal I ...
930 views

### Bandwidth parameters in multivariate KDE using scipy.stats.gaussian_kde

I am working on a project which involves implementing in Python two different density estimation functions over multivariate data; one using N-d histograms and the other using kernel density ...
221 views

### Scipy ttest_ind versus ks_2samp. When to use which test

From the docs ...
246 views

### Interpreting p value in SciPy Welch's T-Test [duplicate]

Can someone help me understand the meaning of the p-value of the Whelch's T Test in SciPy? test = stats.ttest_ind(x, y, equal_var=False) gives me ...
9k views

### How to Compare the Data Distribution of 2 datasets?

I'm having trouble to understand how to compare 2 sets of data by their distribution . For Example, how can I understand that column X100 has the same distribution as column Y1? Also, is there a ...
152 views

### Normality test giving confusing results

So I was transforming data for machine learning purposes and checking whether I should use the data or log-transform it. In addition to creating histograms I decided to test for normality using ...
632 views

### Tuning skopt to minimise correctly

I have a problem where I need to fit a pair of parameters in a model to best fit observed data for hundreds of instances. I need to determine these parameters to high accuracy, rather than just ...
1k views

### Understanding hypothesis testing (Kruskal Wallis, Mann-Whitney) when there are tied values

tldr: In Python's Scipy stats modules, the kruskal and mannwhitneyu functions will both raise an error if called with only ...
117 views

### Fitting two parallel lines with mutually excluding samples

I have a set of (slightly noisy) data taken from two (or more) curves, but taken either from one or the other curve. As an example, consider this code (in reality, data is noisier, especially in how ...
691 views

### Time Series: use of Box-Cox to reduce the “noise”

I am researching the best method to use with time series. FBprophet (Python) seems like a strong option. To prepare time series for Prophet I am thinking about using boxcox and inv_boxcox at the end ...
147 views

### scipy.optimize.minimize How to manage different units?

I am wondering how spicy.optimize.minimize manages different units in the searched vector. My vector is [x,y,z,yaw,pitch,roll]. The first three are translations in <...
703 views

### How to consider “pratt” zero method in Wilcoxon Test of R, which is available in Python Scipy?

I have a dataset of two columns which represent rank data. These ranks are ranged from 1, 2, and 3. To test the difference in medians between these two columns, I am considering the Wilcoxon rank-sum ...
202 views

### Database-friendly random projections with Numpy

In his well known paper , Achlioptas showed that Random Projections could be performed with a sparse projection matrix, whose nonzero entries are either $1$ or $-1$. I have noticed that scikit-...
2k views

### can't make sense of scipy.lognorm [duplicate]

I have been trying to work with random variables in Python and more denerally. My model works with large numbers and lohnormal distributions, but it is very hard to get a clear answer anywhere on the ...
166 views

### How to compare two classifiers performance using scipy.stats.ttest_rel?

I want to compare two classifier performance using T-Test. Using the scipy.stats.ttest_rel: ...
77 views

### Mahalanobis distance fixed at root 2

I have this data set of 12 input variables with a size of over 9000. Wanted to find any outliers using mahalanobis distance. As a central point, I decided to take mean and median of all observations, ...
5k views

### Assessing A/B test results using Python

Problem's setting Assume having results of an A/B test. You let you users experience two variants of your website and you counted how many converted: ...
987 views

### ODR residual variance and reduced chi^2 - do the beta uncertainties represent confidence intervals?

I'm looking for the relationship between residual variance of an Orthogonal Distance Regression (ODR) fit (as implemented in scipy.odr) and reduced $\chi^2$. I'm ...
33 views

### How to make chisquare goodness of fit not vulnerable to a single high values?

There is an observation of a process, that I suppose to be a Poisson process. And I'm trying to make a test, to check goodness of fit of my data with Poisson distribution. ...
947 views

### SVD decomposition and original matrix are not equal [closed]

When l compute the SVD of my matrix x as defined in kernel_hist_to_SVD(). The resulted decomposition is not equal (approximatively) to the original matrix k. Here ...
1k views

### χ² (chi-squared) statistic of scipy.stats.chi2_contingency vs sklearn.feature_selection.chi2

It appears from reading Scikit-learn χ² (chi-squared) statistic and corresponding contingency table that sklearn does not perform a standard contingency table analysis when calculating the χ² ...
630 views

### Computing the correlation / covariance of time series of multiple length and multiple variables

I have N time series of DIFFERENT lengths with i number of data points in each observation. I would like to compute the similarity of each time series and generate M number of clusters. I have tried ...
916 views

### Adjusting Sample with Propensity Score Weighting and ATT

I have a retrospective sample that contains a treatment and non-treatment group with >10 covariates comprised of both categorical and continuous variables. I used the chi-squared and Mann-Whitney U ...
205 views

### Nearest neighbor with lower value

I have a collection of p points in n-space, and a p-vector of scalar values corresponding to each point. In this example, p is much larger than n. Is it possible to build an R-tree (or some other ...
538 views

### using skimage shannon-entropy to calculate image entropy

I'm using shannon-entropy function in skimage to calculate entropy of grey-scale image. After examine the its source code: ...
523 views

### ks_2samp test in Python scipy - low D statistic, low p-value? [closed]

As the heading says, I'm getting both D statistic and p-value to be low in ks_2samp test. More specificaly: ...
505 views

### The special case of the negative binomial, the geometric and calculation with scipy

A particular case? One might consider the geometric distribution is simply a special case of the negative binomial with $N=1$. We know the geometric provides: "The probability the first occurrence ...
3k views

### How to interpret Wilcoxon rank sum result

I have 2 algorithms, A and B, and 9 data-sets. I ran both algorithms on the data-sets and got 9 pair of results. I have to prove that algorithm A is better than algorithm B by comparing the results. I ...
2k views

### Uniform distribution testing

I'm trying to determine the best way to check if a sample of angles is uniformally distributed. As such I've built two toy functions to see what type of sampling error to expect from actually uniform ...
4k views

### How are p-value and odds ratio confidence interval in fisher.test are related?

I aim to conduct Fisher exact test on 2x2 matrix. It took a while to understand difference in R and Python scipy.stats implementation. Now I see that p-values are ...
340 views

### Linkage - Create Even Sized Clusters?

I want to cluster some data points into a given number of clusters according to their nearest neighbour, but with a preference towards even sized clusters when there are multiple pairs with the same ...