Questions tagged [numpy]
NumPy is the fundamental package for scientific computing with Python.
0
votes
0answers
7 views
How to convert a TensorFlow tensor to a NumPy array within tf.Dataset.map() while multithreading/multiprocessing? [on hold]
I use TensorFlow 1.12 in eager execution. I have a very expensive function which I map onto this dataset using tf.Dataset.map(). Inside this function — which I ...
0
votes
0answers
25 views
is it reasonable to consider a plot of 2 separate normal random variables as the geometric representation of a joint probability distribution?
I am plotting 2 normal distribution in Python. if anyone could provide an R version to demonstrate and explain, that would be grateful.
...
0
votes
1answer
15 views
Array of samples from multivariate gaussian distribution Python
I am trying to build in Python the scatter plot in part 2 of Elements of Statistical Learning. First it is said to generate
10 means mk
from a bivariate Gaussian distribution N((1,0)T,I) and ...
2
votes
1answer
36 views
Efficiently Computing The Beta CDF [duplicate]
I am using numba to JIT compile some looped python functions as part of a larger application. Ideally, everything will run in numba's "no python" mode, such that the loop can be parallelised.
One of ...
2
votes
1answer
212 views
Ridge Regression with Gradient Descent Converges to OLS estimates
I'm implementing a homespun version of Ridge Regression with gradient descent, and to my surprise it always converges to the same answers as OLS, not the closed form of Ridge Regression.
This is ...
2
votes
0answers
94 views
Implementing Lasso Regression in Numpy
I'm doing a little self study project, and am trying to implement OLS, Ridge, and Lasso regression from scratch using just Numpy, and am having problems getting this to work with Lasso regression.
...
0
votes
0answers
31 views
Rank 1 SVD with constraint on U
I need to perform a particular rank 1 decomposition of a sparse matrix $\mathbf{A} \in \mathbb{R}^{n\times n}$.
In particular I am looking for the positive vector $\mathbf{u} \in \mathbb{R}^{+n}$ ...
0
votes
1answer
25 views
What is the formula for calculation of `R_ij` in `numpy.corrcoef(x, y, rowvar = False)`?
The manual does not provide the formula if we pass x and y. I do not understand the matrix I get. Here is my code:
...
0
votes
0answers
26 views
Using numpy SVD to calculate factor loadings [duplicate]
I'm doing PCA (Principal Component Analysis) in Python using the numpys Singular Value Decomposition. Effectively extracting the principal components like so:
...
0
votes
0answers
13 views
Implement an Intercept T-Test in NumPy
Quick statistical question from an university econ student.
In Stata, when you run a linear regression, they perform a t-test of the intercept coefficient to see if it is statistically different from ...
1
vote
1answer
21 views
Memory considerations with sklearn classfier
I am trying to fit a sklearn.ensemble.RandomForestClassifier. The [docs] explain that a matrix (rows - observations, columns - features). My observations are 700 ...
0
votes
0answers
15 views
Generating correlated data using numpy while controlling multicollinearity
I am using the following code (adopted from the code in this post). I have no problems with the code. My question is that if with this code I can create or prevent multicollinearity among the ...
0
votes
1answer
239 views
PCA principal components in sklearn not matching eigen-vectors of covariance calculated by numpy
I was trying to replicate PCA in sklearn's PCA API using numpy using PCA in numpy and sklearn produces different results.
I noticed that:
eigenvalues are same as the PCA object's explained_variance_ ...
1
vote
0answers
12 views
Constructing an appropriate null hypothesis?
I have a collection of 20 2D surfaces embedded in 3D. The mean curvature of each surface has been sampled.
The number of samples per surface varies from ~700,000 to ~20,000,000. The choice of ...
2
votes
1answer
106 views
Normalised and “Normal” Cross Correlation giving different lag positions
Using numpy's np.correlate() am trying to find the lag position of two data sets of different length.
When I use this operation by its own I find a lag position ...
0
votes
1answer
28 views
Shifting Data So That When Cross Correlated Lag position is 0
Using Numpy, I am cross correlating two large data sets (of different lengths), as part of a method to compare the similarity of the data. However to take the data onto the next step of the comparison ...
0
votes
1answer
15 views
Matplotlib: Why would a KNN regression model draw a line through ALL points regardless of K?
Having difficulty doing something very basic: create a random dataset taking on values between -1,1, plot them, and also fit a KNN model to the data and fit it over the data. Using numpy to create the ...
0
votes
0answers
6 views
Understanding percentile computation [duplicate]
I understand percentile in the context of test scores with many examples (eg. you SAT score falls in the 99th percentile), but I am not sure I understand percentile in the following context and what ...
1
vote
1answer
308 views
Finding the appropriate polynomial fit in Python
Is there a function or library in Python to automatically compute the best polynomial fit for a set of data points? I am not really interested in the ML use case of generalizing to a set of new data, ...
0
votes
0answers
121 views
How to interpret e.g. pcov returned by numpy.optimize.curve_fit
When doing parameter fits with mathematics frameworks as e.g. numpy, often a covariance matrix is returned. I wonder how to interpret these and if the following is right:
The entries of the ...
1
vote
1answer
85 views
What implicit function used for gradient descent in numpy gradient?
TL;DR numpy.gradient calculates the gradient of an ndarray, but I am not clear as to what it is with respect to what original function.
An example, (although I ...
0
votes
1answer
35 views
How to compute the F1 score?
Here is my code:
score = metrics.f1_score(y_test[0:], y_pred, pos_label=list(set(y_test)))
And here are my dimensions/shapes, which I print before executing the ...
1
vote
0answers
35 views
Why the sign is *plus* in neural network [closed]
REFERENCE
GITHUB GIST
I wanted to implement Neural Network with Numpy in Python. Then I have two question. The first one is about the sign
...
1
vote
0answers
38 views
Why my perceptron doesn't train well and produces bad results on test data?
I am a newbie in Machine learning and I am writing a small code for Perceptron. This is the first time I am writing code in Python.
I have four training data points (X). As they are used for ...
0
votes
0answers
123 views
Calculate corr/cov matrix for large Pandas dataframe/Numpy matrix
I have a large dataframe (gigs), and I would like to compute the corr matrix. What is the most efficient way of doing this in parallel. I have access to many machines and I'm open to using any ...
1
vote
0answers
139 views
PCA = Eigen decomposition of Covariance Matrix is Not True? [closed]
I have a dataset with 400 features.
What I did:
...
0
votes
1answer
29 views
Remove individual points and find slope
I am trying to delete one pair of x and y coordinates from a set of 10 data points and get the slope for the other 9 points.
How do I go about this issue?
Attached herewith is an image of what I am ...
1
vote
0answers
220 views
Update R2 and slope by removing outliers-python [closed]
I am using the following code to find out Linregress parameters:
...
4
votes
2answers
447 views
python computing likelihood causing exp overflow
I am using numpy to compute the likelihood of a variable $Z$ using numpy. $Z$ is a Bernoulli random variable which has two outcomes $[0,1]$. I compute the log likelihood of observing $Z$ given the ...
0
votes
1answer
253 views
Discrete Fourier transform of an exponential decay
I have a vector with an exponential decay signal, using Numpy:
t=np.arange(128)
a=0.1
decay=np.exp(-a*t)
I would like to compute the discrete Fourier transform (...
2
votes
0answers
499 views
Normalization of convolution kernel
I am trying to smooth a noisy one-dimensional physical signal, y, while retaining correspondence between the signal's amplitude and its units. I'm applying a ...
2
votes
0answers
80 views
Fitting two parallel lines with mutually excluding samples
I have a set of (slightly noisy) data taken from two (or more) curves, but taken either from one or the other curve. As an example, consider this code (in reality, data is noisier, especially in how ...
1
vote
0answers
34 views
Neural Network Accuracy doesn't improve [closed]
I am writing code for a neural network with 2 hidden layers and trying to evaluate its performance on MNIST. Here is my code snippet:
...
0
votes
0answers
7 views
Is it possible to train an kmeans clustering algorithm using the data from sift(open cv image descriptor)?
I am trying to build an object classifier (lets say red apple for example).
I am trying to build a model which can use the descriptors of sift feature extraction to train itself. The descriptor ...
2
votes
2answers
165 views
Database-friendly random projections with Numpy
In his well known paper [1], Achlioptas showed that Random Projections could be performed with a sparse projection matrix, whose nonzero entries are either $1$ or $-1$.
I have noticed that scikit-...
0
votes
0answers
194 views
Get rotation of noisy rectangular 2d point cloud with pca
I have a set of point clouds in nd space. The clouds tend to be rectangular in nature in 2d space but can easily have outliers or look slighly L shaped.
I would like to rotate the point clouds so that ...
2
votes
1answer
444 views
PCA - Reconstruction from a “clean” set of eigenvectors?
This is a question related to the explanation here on how to reconstruct data from PCs found here:
How to reverse PCA and reconstruct original variables from several principal components?
I have two ...
1
vote
0answers
57 views
Understanding Diagonal Matrix of SVD [closed]
I have a matrix A with dimensions 4x3. I performed SVD on the matrix using numpy (np.linalg.svd) on matrix A. The output dimensions of U, V, and S are (4x4),(3x3),(3,). So V comprise orthonormal ...
1
vote
0answers
650 views
How to produce a normalized cumulative histogram?
I am having trouble understanding the proper method to calculate specific histograms, specifically with regard to cumulative and normalized histograms. If I want to calculate a normalized cumulative ...
0
votes
1answer
109 views
How can I train a CNN on raw numbers?
I am trying to make a binary classification using Keras' Conv2D classes based on this blog. I have many files with a matrix of floating point numbers in each one (these matrices are not pixels values)....
2
votes
1answer
179 views
Nearest neighbor with lower value
I have a collection of p points in n-space, and a p-vector of scalar values corresponding to each point. In this example, p is much larger than n.
Is it possible to build an R-tree (or some other ...
1
vote
0answers
275 views
Linear Algebra and NumPy - How array broadcast translates to matrix operations?
I am familiar with Python's NumPy library and its broadcasting feature that allows to perform operations with vectors and matrices of different sizes. I might be missing the mathematical connection as ...
2
votes
1answer
754 views
Log Transformation Instead of Z-Score Normalizatrion For Machine Learning
I almost always used Numpy's StandardScaler to normalize my data for machine learning. I noticed however that simply taking the log of the variables that I wanted to normalize often resulted in better ...
2
votes
1answer
161 views
Why simulated and calculated t-distribution pdf at degree-of-freedom=1 doesn't match
See the attached graph, I am simulating the pdf of t-distribution at different degrees of freedom (dof), when dof is low (e.g. dof=1), why don't they match? Is it because my experiment wasn't done ...
0
votes
1answer
82 views
Finding change in spending habits
I have a numpy array full of customer spending data:
x = np.array([5000,5500,6250,4800,3950,5800,5500,800,1200,900,500,400,300,200,3100])
Above, you can see ...
3
votes
1answer
12k views
Are 1-dimensional numpy arrays equivalent to vectors? [closed]
I'm new to both linear algebra and numpy, so please bear with me. I'm taking a course on linear regression, where I learned that we can express our hypothesis as $\theta^TX$ where $\theta$ is our ...
1
vote
0answers
48 views
Comparing curves to find anomalies in python
I am currently working on a Data Mining project in an assembly line. A lot of parts are produced in this assembly line and each part gives me a curve (it's the energy use of the part over the process ...
0
votes
1answer
123 views
How to transform a dataset so that it has a given mean and standard deviation? [duplicate]
Suppose I have the following dataset:
...
0
votes
1answer
195 views
What's wrong with my algorithm for implementing the Storkey learning rule for Hopfield nets?
Trying to implement the Storkey rule...
I can use the below algorithm to train an initial pattern -- because it goes into the if statement and simply uses the hebbian learning/outer-product method. ...
0
votes
3answers
392 views
What's a simple way to generate a random sample of a continuous distribution given as a series of trapezoids using scipy? [duplicate]
I currently have a continuous distribution that's described as a series of trapezoids in two arrays xs and ys, which integrate ...
