SciPy is a Python-based ecosystem of open-source software for mathematics, science, and engineering.
1
vote
1answer
27 views
0
votes
0answers
10 views
scipy L-BFGS-B optimizer with different step size per dimension [migrated]
How can I adjust the optimizer to use a different step size for each DOF? When I print the parameters the step size seems to the the same per dimension.
Any other alternative optimizer that can ...
1
vote
3answers
34 views
How to calculate the p-value of a test, that checked for a binary property?
16081 subjects were randomly assigned into two groups:
test group: 7916 subjects
control group: 8165 subjects
Only the test group was exposed to something. During the test period
10 subjects of ...
0
votes
1answer
28 views
Normal distribution test ( D'Agostino and Pearson's) in Scipy - results don't make sense
I want to test if an array of numbers is approximately normally distributed.
...
0
votes
0answers
20 views
How to express the joint probability of two multivariate distributions over the same domain
I am working with multivariate normal distributions in scipy, but I have a question which is more statistics oriented:
Say I have a multivariate distribution from two discrete random variables $X1$ ...
1
vote
0answers
9 views
3 Layer neural network (feedforward and backpropagation) - Desired error not necessarily achieved due to precision loss
I am trying to implement a 3 layer neural network with feedforward and backpropagation.
I have tested my cost function and it is working fine. My gradient function also seems ok.
but when I try to ...
0
votes
0answers
21 views
Calculate horizontal asymptote for data set with python
I have a data set comprised of X files. Each file contains Y 60-bit binary numbers that are not random but I don't have access to the rules that would tell me the allowed bit patterns and allow me to ...
0
votes
1answer
24 views
Remove individual points and find slope
I am trying to delete one pair of x and y coordinates from a set of 10 data points and get the slope for the other 9 points.
How do I go about this issue?
Attached herewith is an image of what I am ...
0
votes
0answers
39 views
RBF Interpolation Singular Matrix Error
I am attempting to RBF interpolate a set of data, however the interpolation fails when numpy reaches a singular matrix it cannot handle. I saw this post and thought ...
1
vote
0answers
73 views
Update R2 and slope by removing outliers-python
I am using the following code to find out Linregress parameters:
...
0
votes
1answer
57 views
Statsmodels' Negative Binomial: after .fit_regularized(), how to turn PMF into PPF to get the discrete values?
I used the package statsmodels to fit a Negative Binomial to my data. This data contains ~1500 samples with 21 covariates. Since I have overdispersion in my data ...
0
votes
1answer
13 views
Can findings based on low values be considered valid?
Lets say i do have a TimeSeries with N > 300 but values like [0, 1, 2, 1, 2, 0, 2, ...] representing the visitors count of a website per day.
Since there are only few visitors and each of them can be ...
0
votes
0answers
28 views
Problem with U Mann Whitney test in scipy for nominal values
When I am using scipy.stats.mannwhitneyu sometimes I got this error:
TypeError: '<' not supported between instances of 'int' and 'str'
Should not it work? I have noticed that scipy.stats....
0
votes
0answers
32 views
Periodogram interpretaion to get main cycles for time serie
I try to use the periodogram in order to get main cycle on my time series. My time series is a the result of measure every hour of one variable related with the atmosphere. So in one day I have 24 ...
1
vote
1answer
156 views
How to know if a parameter is statistically significant in a “curve_fit” estimation? [closed]
I use curve_fit from scipy to estimate parameter values from a specific function.
...
0
votes
1answer
34 views
nmf in scipy returns components with all zero weights
I'm trying to understand whether this behavior is a bug or a feature.
Essentially, I have a dataset of ten thousand short pieces of text. I have used the CountVectorizer function to turn this into a ...
2
votes
1answer
127 views
Gaussian Mixture Model
with the following code I fit a Gaussian Mixture Model to arbitrarily created data. The code is working. The only thing I encounter is that during the calculation of the multivariate_normal I ...
0
votes
1answer
185 views
Bandwidth parameters in multivariate KDE using scipy.stats.gaussian_kde
I am working on a project which involves implementing in Python two different density estimation functions over multivariate data; one using N-d histograms and the other using kernel density ...
0
votes
1answer
41 views
0
votes
1answer
56 views
Interpreting p value in SciPy Welch's T-Test [duplicate]
Can someone help me understand the meaning of the p-value of the Whelch's T Test in SciPy?
test = stats.ttest_ind(x, y, equal_var=False)
gives me
...
1
vote
1answer
489 views
How to Compare the Data Distribution of 2 datasets?
I'm having trouble to understand how to compare 2 sets of data by their distribution .
For Example,
how can I understand that column X100 has the same distribution as column Y1?
Also, is there a ...
1
vote
2answers
117 views
Normality test giving confusing results
So I was transforming data for machine learning purposes and checking whether I should use the data or log-transform it. In addition to creating histograms I decided to test for normality using ...
0
votes
0answers
99 views
Tuning skopt to minimise correctly
I have a problem where I need to fit a pair of parameters in a model to best fit observed data for hundreds of instances. I need to determine these parameters to high accuracy, rather than just ...
2
votes
1answer
204 views
Understanding hypothesis testing (Kruskal Wallis, Mann-Whitney) when there are tied values
tldr: In Python's Scipy stats modules, the kruskal and mannwhitneyu functions will both raise an error if called with only ...
2
votes
0answers
31 views
Fitting two parallel lines with mutually excluding samples
I have a set of (slightly noisy) data taken from two (or more) curves, but taken either from one or the other curve. As an example, consider this code (in reality, data is noisier, especially in how ...
1
vote
2answers
145 views
Time Series: use of Box-Cox to reduce the “noise”
I am researching the best method to use with time series. FBprophet (Python) seems like a strong option.
To prepare time series for Prophet I am thinking about using boxcox and inv_boxcox at the end
...
0
votes
0answers
95 views
scipy.optimize.minimize How to manage different units?
I am wondering how spicy.optimize.minimize manages different units in the searched vector. My vector is [x,y,z,yaw,pitch,roll]. The first three are translations in <...
1
vote
0answers
198 views
How to consider “pratt” zero method in Wilcoxon Test of R, which is available in Python Scipy?
I have a dataset of two columns which represent rank data. These ranks are ranged from 1, 2, and 3. To test the difference in medians between these two columns, I am considering the Wilcoxon rank-sum ...
1
vote
2answers
124 views
Database-friendly random projections with Numpy
In his well known paper [1], Achlioptas showed that Random Projections could be performed with a sparse projection matrix, whose nonzero entries are either $1$ or $-1$.
I have noticed that scikit-...
1
vote
0answers
54 views
Working between sklearn and scipy for convex optimization [closed]
I have a method for solving convex programs using an alternative method of iterative procedures and would like to compare this to current solvers methods.
My approach relies upon randomly projecting ...
0
votes
0answers
49 views
Methods for selecting the required n_components for TruncatedSVD?
Methods for selecting the required n_components for TruncatedSVD?
I found this (https://chrisalbon.com/machine_learning/feature_engineering/...
-1
votes
1answer
292 views
can't make sense of scipy.lognorm [duplicate]
I have been trying to work with random variables in Python and more denerally. My model works with large numbers and lohnormal distributions, but it is very hard to get a clear answer anywhere on the ...
0
votes
0answers
14 views
Finding p-value and t-value for non-normal distributed datasets (multiple dataframes)
I'm currently working on a project that involves using data from 9 different dataframes. Each dataframe contains 58 rows and 8 columns, and they each contain different data (i.e. Unemployment Rate, ...
0
votes
0answers
12 views
Check whether two tables come from the same population
I have two data frames and I want to check whether they come from the same population.
Essentially, I want to go through the data columns one by one and, depending on column type and how many ...
0
votes
0answers
76 views
How to compare two classifiers performance using `scipy.stats.ttest_rel`?
I want to compare two classifier performance using T-Test. Using the scipy.stats.ttest_rel:
...
2
votes
1answer
50 views
Mahalanobis distance fixed at root 2
I have this data set of 12 input variables with a size of over 9000. Wanted to find any outliers using mahalanobis distance. As a central point, I decided to take mean and median of all observations, ...
4
votes
0answers
2k views
Assessing A/B test results using Python
Problem's setting
Assume having results of an A/B test.
You let you users experience two variants of your website and you counted how many converted:
...
0
votes
0answers
15 views
Analysis of power after filtering but before calculating t-statistic
Suppose I want to calculate many t-statistics from some arrays of data. The length n of these arrays changes depending on a filtering step I perform previously. ...
1
vote
1answer
269 views
ODR residual variance and reduced chi^2 - do the beta uncertainties represent confidence intervals?
I'm looking for the relationship between residual variance of an Orthogonal Distance Regression (ODR) fit (as implemented in scipy.odr) and reduced $\chi^2$.
I'm ...
1
vote
0answers
23 views
How to make chisquare goodness of fit not vulnerable to a single high values?
There is an observation of a process, that I suppose to be a Poisson process.
And I'm trying to make a test, to check goodness of fit of my data with Poisson distribution.
...
1
vote
1answer
502 views
SVD decomposition and original matrix are not equal [closed]
When l compute the SVD of my matrix x as defined in kernel_hist_to_SVD(). The resulted decomposition is not equal (approximatively) to the original matrix k. Here ...
0
votes
0answers
112 views
Interpreting p value of Pearsons Chi² Test as goodness of fit
at the moment I want to create a model for continous, empirical data. I derived a histogram, and fitted an model distribution using a least mean square fit.
To test the goodness of fit, I decided to ...
0
votes
0answers
611 views
χ² (chi-squared) statistic of scipy.stats.chi2_contingency vs sklearn.feature_selection.chi2
It appears from reading Scikit-learn χ² (chi-squared) statistic and corresponding contingency table that sklearn does not perform a standard contingency table analysis when calculating the χ² ...
1
vote
0answers
392 views
Computing the correlation / covariance of time series of multiple length and multiple variables
I have N time series of DIFFERENT lengths with i number of data points in each observation.
I would like to compute the similarity of each time series and generate M number of clusters.
I have tried ...
1
vote
1answer
436 views
Adjusting Sample with Propensity Score Weighting and ATT
I have a retrospective sample that contains a treatment and non-treatment group with >10 covariates comprised of both categorical and continuous variables. I used the chi-squared and Mann-Whitney U ...
2
votes
1answer
146 views
Nearest neighbor with lower value
I have a collection of p points in n-space, and a p-vector of scalar values corresponding to each point. In this example, p is much larger than n.
Is it possible to build an R-tree (or some other ...
0
votes
0answers
251 views
using skimage shannon-entropy to calculate image entropy
I'm using shannon-entropy function in skimage to calculate entropy of grey-scale image.
After examine the its source code:
...
2
votes
0answers
344 views
ks_2samp test in Python scipy - low D statistic, low p-value?
As the heading says, I'm getting both D statistic and p-value to be low in ks_2samp test. More specificaly:
...
1
vote
1answer
177 views
The special case of the negative binomial, the geometric and calculation with scipy
A particular case?
One might consider the geometric distribution is simply a special case of the negative binomial with $N=1$.
We know the geometric provides: "The probability the first occurrence ...
1
vote
1answer
1k views
How to interpret Wilcoxon rank sum result
I have 2 algorithms, A and B, and 9 data-sets. I ran both algorithms on the data-sets and got 9 pair of results. I have to prove that algorithm A is better than algorithm B by comparing the results. I ...
