Questions tagged [numpy]
NumPy is the fundamental package for scientific computing with Python.
107
questions
1
vote
1answer
26 views
Why when I increase the standard deviation in the pdf function the graph looks squeezed instead of more dispersed?
So what I am doing is this:
...
0
votes
0answers
17 views
Combine TFIDF with non-textual features
I am dealing with an email classification problem in which I have email requests coming from different groups of people. I am building a classifier to classify these emails based on historical email ...
1
vote
0answers
21 views
Estimate joint probability of two dependent variable
I've a dataset created in the following way:
The input of the system is a 8 bit binary number ranging from $x_1$= 0000
0000 to $x_N$ = 1111 1111
For every input i've read the output of the system (...
0
votes
0answers
19 views
Generate random values to mimic skewness
I have a actual set of data where the variables are heavily skewed, both positively and negatively.
I need to generate random sample data for the values going forward. The data needs to be similarly ...
0
votes
0answers
14 views
Q-Exponential distribution in Numpy
I am studying a phenomena, which I know is characterised by the q-exponential distribution, with parameters $c=1$, $q = 1.355$ and $b = 0.524$ in the equation
$$e_{q,b,c}(t) = c(1 + b(q-1)t)^{\frac{1}{...
0
votes
0answers
15 views
Adding features lowered accuracy score Gaussian Naive Bayes algorithm, Python
I am in third year of university, this simple py program is meant to use the Gaussian Naive Bayes algorithm to create a model and evaluate it.
...
0
votes
1answer
13 views
Feeding multiple rows of data into ANN [closed]
I've built an ANN from scratch, that works with one row of data with any number of neurons and hidden layers. For the setup I am using 2 hidden layers, 5 neurons (just while building). The network ...
1
vote
0answers
27 views
Infinitesimal generator of Markov chain using numpy
I am computing the infinitesimal generator of a continuous-time Markov chain from the transition probabilities p. I am following the methodology described here, ...
-1
votes
2answers
76 views
Transforming data for normality with negative values using python
I am working with a data set in a machine learning project, which has lots to negative values. I want to transform the distribution of my data to normal. I tried using numpy.log, but since log is only ...
0
votes
0answers
10 views
Numpy corrcoef giving different result from manual calculation
I'm trying to calculate a partial correlation matrix for a high dimensional problem. I'm using this paper as a guide. I'm also referencing this function from Pingouin.
Starting from the inverse ...
2
votes
0answers
16 views
Can't Recreate Values for U, S, V from SVD in numpy [duplicate]
To better understand SVD, I'm trying to recreate the values for U, S, and V using straight ...
2
votes
1answer
30 views
QR Factorization to Solve Least Squares Without Using an Inverse
I'm playing around with different ways to solve least squares, and am using numpy to derive values for $\beta$ in a regression problem.
I know that if you do a $QR$ factorization of $X$ such that $...
1
vote
0answers
27 views
Dataframe containing value 0 should be removed or replaced?
I have a question that mostly I get stuck at. I was looking at the data for diabetes patients and found that most of the rows have 0 values under most of their columns. Reference for the url
https://...
0
votes
0answers
10 views
How to find optimum matrix set based on determinant values using python
I am new at programming, so I want to find the optimum set of row values based on maximum determinant logic.
1) Set the 1st Column 'Serial_no' as index.
2) Take first 'N' row values as user input ...
0
votes
1answer
140 views
KL Divergence of two standard normal arrays
I generated two 9000,1 np arrays with
a = np.random.standard_normal(9000)
b = np.random.standard_normal(9000)
Then I check the KL Divergence with
...
2
votes
0answers
80 views
James Stein Estimator for more than one Sample
I have a hard time understanding the James-Stein Estimator. I show you how I tried to comprehend it by using a python example. I take a normally distributed random vector with mean $(0.1, 0.2, 0.3, 0....
1
vote
1answer
44 views
Outliers of small dataset
I have a python function that takes a list of smaller images boxes (represented as float arrays) and the whole image img in as a ...
0
votes
0answers
28 views
Different number of Eigen/Singular values from PCA and SVD
My understanding is that a SVD done on a raw data matrix M and a PCA done on its covariance matrix C should return the same eigen/singular values.
I have a 2736 x 356 data matrix and am using the ...
1
vote
1answer
304 views
Fitting Beta Distributions to Data
I am trying to reproduce some beta distribution parameters found in this published paper. I have two data sets, y1 and y2, that ...
0
votes
0answers
19 views
Proper way of calculating statistics for samples probabilites [closed]
I have 64 samples and for each one of them, I have their probability. I was wondering which is the correct way to find a mean and std for the above-mentioned population. If I do something like:
<...
0
votes
0answers
5 views
Correlation fails on high sampling rate vector
I am trying to find the time shift between two vectors.
My function succeeds on randomly generated vectors, but fails on small shifted, high sampling rate vectors.
my code:
...
1
vote
0answers
111 views
Gradient Descent Vectorization with Numpy (1D transpose confusion) [closed]
I'm working through Andrew Ng's original Stanford course and ran into some numpy confusion. Basically, my main question is, if we dot product a 1D array with a 2D array in numpy (and the dimensions ...
0
votes
0answers
25 views
is it reasonable to consider a plot of 2 separate normal random variables as the geometric representation of a joint probability distribution?
I am plotting 2 normal distribution in Python. if anyone could provide an R version to demonstrate and explain, that would be grateful.
...
0
votes
1answer
1k views
Array of samples from multivariate gaussian distribution Python
I am trying to build in Python the scatter plot in part 2 of Elements of Statistical Learning. First it is said to generate
10 means mk
from a bivariate Gaussian distribution N((1,0)T,I) and ...
2
votes
1answer
183 views
Efficiently Computing The Beta CDF [duplicate]
I am using numba to JIT compile some looped python functions as part of a larger application. Ideally, everything will run in numba's "no python" mode, such that the loop can be parallelised.
One of ...
2
votes
1answer
726 views
Ridge Regression with Gradient Descent Converges to OLS estimates
I'm implementing a homespun version of Ridge Regression with gradient descent, and to my surprise it always converges to the same answers as OLS, not the closed form of Ridge Regression.
This is ...
2
votes
0answers
396 views
Implementing Lasso Regression in Numpy
I'm doing a little self study project, and am trying to implement OLS, Ridge, and Lasso regression from scratch using just Numpy, and am having problems getting this to work with Lasso regression.
...
0
votes
0answers
42 views
Rank 1 SVD with constraint on U
I need to perform a particular rank 1 decomposition of a sparse matrix $\mathbf{A} \in \mathbb{R}^{n\times n}$.
In particular I am looking for the positive vector $\mathbf{u} \in \mathbb{R}^{+n}$ ...
0
votes
1answer
58 views
What is the formula for calculation of `R_ij` in `numpy.corrcoef(x, y, rowvar = False)`?
The manual does not provide the formula if we pass x and y. I do not understand the matrix I get. Here is my code:
...
0
votes
0answers
26 views
Using numpy SVD to calculate factor loadings [duplicate]
I'm doing PCA (Principal Component Analysis) in Python using the numpys Singular Value Decomposition. Effectively extracting the principal components like so:
...
0
votes
0answers
23 views
Implement an Intercept T-Test in NumPy
Quick statistical question from an university econ student.
In Stata, when you run a linear regression, they perform a t-test of the intercept coefficient to see if it is statistically different from ...
1
vote
1answer
22 views
Memory considerations with sklearn classfier
I am trying to fit a sklearn.ensemble.RandomForestClassifier. The [docs] explain that a matrix (rows - observations, columns - features). My observations are 700 ...
0
votes
0answers
19 views
Generating correlated data using numpy while controlling multicollinearity
I am using the following code (adopted from the code in this post). I have no problems with the code. My question is that if with this code I can create or prevent multicollinearity among the ...
1
vote
1answer
1k views
PCA principal components in sklearn not matching eigen-vectors of covariance calculated by numpy
I was trying to replicate PCA in sklearn's PCA API using numpy using PCA in numpy and sklearn produces different results.
I noticed that:
eigenvalues are same as the PCA object's explained_variance_ ...
1
vote
0answers
15 views
Constructing an appropriate null hypothesis?
I have a collection of 20 2D surfaces embedded in 3D. The mean curvature of each surface has been sampled.
The number of samples per surface varies from ~700,000 to ~20,000,000. The choice of ...
2
votes
1answer
418 views
Normalised and “Normal” Cross Correlation giving different lag positions
Using numpy's np.correlate() am trying to find the lag position of two data sets of different length.
When I use this operation by its own I find a lag position ...
0
votes
1answer
59 views
Shifting Data So That When Cross Correlated Lag position is 0
Using Numpy, I am cross correlating two large data sets (of different lengths), as part of a method to compare the similarity of the data. However to take the data onto the next step of the comparison ...
0
votes
1answer
17 views
Matplotlib: Why would a KNN regression model draw a line through ALL points regardless of K?
Having difficulty doing something very basic: create a random dataset taking on values between -1,1, plot them, and also fit a KNN model to the data and fit it over the data. Using numpy to create the ...
1
vote
1answer
585 views
Finding the appropriate polynomial fit in Python
Is there a function or library in Python to automatically compute the best polynomial fit for a set of data points? I am not really interested in the ML use case of generalizing to a set of new data, ...
0
votes
0answers
328 views
How to interpret e.g. pcov returned by numpy.optimize.curve_fit
When doing parameter fits with mathematics frameworks as e.g. numpy, often a covariance matrix is returned. I wonder how to interpret these and if the following is right:
The entries of the ...
1
vote
1answer
206 views
What implicit function used for gradient descent in numpy gradient?
TL;DR numpy.gradient calculates the gradient of an ndarray, but I am not clear as to what it is with respect to what original function.
An example, (although I ...
0
votes
1answer
82 views
How to compute the F1 score?
Here is my code:
score = metrics.f1_score(y_test[0:], y_pred, pos_label=list(set(y_test)))
And here are my dimensions/shapes, which I print before executing the ...
1
vote
0answers
38 views
Why my perceptron doesn't train well and produces bad results on test data?
I am a newbie in Machine learning and I am writing a small code for Perceptron. This is the first time I am writing code in Python.
I have four training data points (X). As they are used for ...
0
votes
1answer
33 views
Remove individual points and find slope
I am trying to delete one pair of x and y coordinates from a set of 10 data points and get the slope for the other 9 points.
How do I go about this issue?
Attached herewith is an image of what I am ...
4
votes
2answers
825 views
python computing likelihood causing exp overflow
I am using numpy to compute the likelihood of a variable $Z$ using numpy. $Z$ is a Bernoulli random variable which has two outcomes $[0,1]$. I compute the log likelihood of observing $Z$ given the ...
0
votes
1answer
320 views
Discrete Fourier transform of an exponential decay
I have a vector with an exponential decay signal, using Numpy:
t=np.arange(128)
a=0.1
decay=np.exp(-a*t)
I would like to compute the discrete Fourier transform (...
2
votes
0answers
868 views
Normalization of convolution kernel
I am trying to smooth a noisy one-dimensional physical signal, y, while retaining correspondence between the signal's amplitude and its units. I'm applying a ...
2
votes
0answers
114 views
Fitting two parallel lines with mutually excluding samples
I have a set of (slightly noisy) data taken from two (or more) curves, but taken either from one or the other curve. As an example, consider this code (in reality, data is noisier, especially in how ...
2
votes
2answers
200 views
Database-friendly random projections with Numpy
In his well known paper [1], Achlioptas showed that Random Projections could be performed with a sparse projection matrix, whose nonzero entries are either $1$ or $-1$.
I have noticed that scikit-...
0
votes
0answers
269 views
Get rotation of noisy rectangular 2d point cloud with pca
I have a set of point clouds in nd space. The clouds tend to be rectangular in nature in 2d space but can easily have outliers or look slighly L shaped.
I would like to rotate the point clouds so that ...
