# Questions tagged [numpy]

NumPy is the fundamental package for scientific computing with Python.

111 questions
Filter by
Sorted by
Tagged with
916 views

### Normalization of convolution kernel

I am trying to smooth a noisy one-dimensional physical signal, y, while retaining correspondence between the signal's amplitude and its units. I'm applying a ...
117 views

### Fitting two parallel lines with mutually excluding samples

I have a set of (slightly noisy) data taken from two (or more) curves, but taken either from one or the other curve. As an example, consider this code (in reality, data is noisier, especially in how ...
203 views

### Database-friendly random projections with Numpy

In his well known paper , Achlioptas showed that Random Projections could be performed with a sparse projection matrix, whose nonzero entries are either $1$ or $-1$. I have noticed that scikit-...
277 views

### Get rotation of noisy rectangular 2d point cloud with pca

I have a set of point clouds in nd space. The clouds tend to be rectangular in nature in 2d space but can easily have outliers or look slighly L shaped. I would like to rotate the point clouds so that ...
735 views

### PCA - Reconstruction from a “clean” set of eigenvectors?

This is a question related to the explanation here on how to reconstruct data from PCs found here: How to reverse PCA and reconstruct original variables from several principal components? I have two ...
1k views

### How to produce a normalized cumulative histogram?

I am having trouble understanding the proper method to calculate specific histograms, specifically with regard to cumulative and normalized histograms. If I want to calculate a normalized cumulative ...
207 views

### How can I train a CNN on raw numbers?

I am trying to make a binary classification using Keras' Conv2D classes based on this blog. I have many files with a matrix of floating point numbers in each one (these matrices are not pixels values)....
206 views

### Nearest neighbor with lower value

I have a collection of p points in n-space, and a p-vector of scalar values corresponding to each point. In this example, p is much larger than n. Is it possible to build an R-tree (or some other ...
338 views

### Linear Algebra and NumPy - How array broadcast translates to matrix operations?

I am familiar with Python's NumPy library and its broadcasting feature that allows to perform operations with vectors and matrices of different sizes. I might be missing the mathematical connection as ...
1k views

### Log Transformation Instead of Z-Score Normalizatrion For Machine Learning

I almost always used Numpy's StandardScaler to normalize my data for machine learning. I noticed however that simply taking the log of the variables that I wanted to normalize often resulted in better ...
231 views

### Why simulated and calculated t-distribution pdf at degree-of-freedom=1 doesn't match

See the attached graph, I am simulating the pdf of t-distribution at different degrees of freedom (dof), when dof is low (e.g. dof=1), why don't they match? Is it because my experiment wasn't done ...
95 views

### Finding change in spending habits

I have a numpy array full of customer spending data: x = np.array([5000,5500,6250,4800,3950,5800,5500,800,1200,900,500,400,300,200,3100]) Above, you can see ...
19k views

### Are 1-dimensional numpy arrays equivalent to vectors? [closed]

I'm new to both linear algebra and numpy, so please bear with me. I'm taking a course on linear regression, where I learned that we can express our hypothesis as $\theta^TX$ where $\theta$ is our ...
58 views

### Comparing curves to find anomalies in python

I am currently working on a Data Mining project in an assembly line. A lot of parts are produced in this assembly line and each part gives me a curve (it's the energy use of the part over the process ...
156 views

### How to transform a dataset so that it has a given mean and standard deviation? [duplicate]

Suppose I have the following dataset: ...
234 views

### What's wrong with my algorithm for implementing the Storkey learning rule for Hopfield nets?

Trying to implement the Storkey rule... I can use the below algorithm to train an initial pattern -- because it goes into the if statement and simply uses the hebbian learning/outer-product method. ...
759 views

### What's a simple way to generate a random sample of a continuous distribution given as a series of trapezoids using scipy? [duplicate]

I currently have a continuous distribution that's described as a series of trapezoids in two arrays xs and ys, which integrate ...
552 views

### Iman-Conover implementation for correlated randoms in python with numpy/scipy

I've been trying to implement the iman-conover method in python so I could generate correlated random numbers from distributions other than normal (I use a normal & uniform in my example below). ...
553 views

### Built-in var() function in R computes the quasi-variance

I have recently noticed that the var() function in R returns a different value from the analogous function in the python package numpy. In particular, R's ...
323 views

### Is it right to use normal distribution to compute confidence intervals of a point in a zipf distribution?

I have a list of datapoints that falls within some sort of zipfian distribution or gamma distribution: ...
32 views

### Statistical classification of multiple arrays

I am working on an algorithm to mask uninteresting parts of images based on entropy. Based on samples saved as numpy arrays I am able to calculate statistics and do things like creating boxplots. My ...
4k views

### Keras Numpy Error: Why does accuracy go to zero?

I am trying to follow this tutorial. Everything works when I copy and paste the data, but when I try to replace the dataset with a random numpy matrix, my accuracy goes to zero. It shouldn't go to ...
73 views

### Logistic regression getting worse, what are the common causes?

I am making a naive logistic regression implementation to better understand how it works. I am using a very simple dataset provided by Andrew Ng's course on Coursera (pastebin link with the rest of ...
6k views

### Numpy.corrcoef() for twice one object does not return matrix of 1's

I'm trying to compare if two pictures are similar or close to similar. The picture is available as numpy.ndarray and contains of 28x28 pixels. It might look like the one below: When I get the image ...
912 views

### Correlation coefficient between two data sets having points of different dimensions

I have a dataset in which one side there are vectors of dimension 3 and other side is a real number corresponding to each vector. Now I need to find the correlation coefficient between the two set. ...
7k views

### How does NumPy solve least squares for underdetermined systems?

Let's say that we have X of shape (2, 5) and y of shape (2,) This works: np.linalg.lstsq(X, y) We would expect this to work only if X was of shape (N,5) where N>=...
1k views

### how to generate sliding window features in sci kit learn?

I have a data matrix where each row is measurement in time and each column is a feature. ...
244 views

### ROC curve end point confusion

Here is the code and output, I think from the output, it mean when fpr is 0, tpr is 0, this is correct as the prediction results marks everything to be 0. But the output also said, when fpr is 1, tpr ...
5k views

### Best way to seed N independent random number generators from 1 value

In my program I need to run N separate threads each with their own RNG which is used to sample a large dataset. I need to be able to seed this entire process with a single value so I can reproduce ...
57 views

### Chi2 test on big uniformly random sample [duplicate]

I expect that, in Python with numpy and scipy, scipy.stats.chisquare(numpy.bincount(numpy.random.randint(100, size=1000000))) will return P-value which is very ...
2k views

### How to cluster an 1-D array by K-means or any other algorithm using scikit-learn? [closed]

I have an one dimensional toy array X. I want to cluster the data into some numbers of clusters.But when I try to fit my data in scikit-learn K-Means function it shows ValueError: n_samples=1 ...
65 views

### mvBacon from R to Python [closed]

I use package "robustX" in R in order to compute the Mahalanobis distance in a dataset to detect outliers, using more than one variable. Specifically the code I am using is: ...
1k views

### Goodness of fit for long-tailed distributed data

I'm trying to fit a distribution to a set of data (the elevation of all land areas in the world). The histogram shows a long-tailed distribution, and I'd like to see which of the long-tailed models ...
407 views

### My neural network is good learning products but fails for sums

I implemented a simple feed-forward neural network with biases trained with backpropagation and stochastic gradient decent. What surprises me is how differently is behaves on two datasets I generated. ...
265 views

### Using a distance matrix *with errors* to find the coordinates of points

(I asked this same question in stackoverflow, without getting any answer, but maybe this is a more appropriate forum.) I would like to find the coordinates of a set of points in 3D from a distance ...
581 views

### Box Muller vs Numpy?

I used the Box Muller transform and pythons uniform random number generator to sample random numbers in a given interval [a,b]. Here's the approach I used: Note: I know the average and the standard ...
1k views

### How to implement Kernel density estimation in multivariate/3D

I have dataset like the following fromat and im trying to find out the Kernel density estimation with optimal bandwidth. ...
47 views

### What to do with paired data when a minority of pairs have more or less than two members?

Let's say I have two sets of measurements that go on like this: ...
327 views

### finding the interpolated values of an empirical probability distribution

I have an empirical probability function $p(z)$. In the first column $z$ and the second column contains $p(z)$ values. The data is given as following : data.cat ...
258 views

### Which model fit best for semi sinusoidal data? [duplicate]

I have a record containing the maximum and the minimum monthly temperatures at a particular station. The record shows information for each month from January 1908 to March 2012. However, some of the ...
4k views

### rnorm vs numpy.random.randn

For a regression example, I constructed some artificial data and ran ols, ...
127 views

### What are the good study materials on Association Rules?

I am looking to learn Association Rules, from basic level. I was looking for some good web based materials to start with. My objectives in the materials is to: (a) learn the aspect nicely from basic;...
124 views

### Recommendation Engine With Physical Distance Cutoff

I am looking to develop a recommendation engine for local stores to users. There are approximately 1 million stores in the database and around 1 million users. The 1Mx1M matrix for a user-based ...
1k views

### Reproduce Cross Correlation results in Python

I'm trying and failing to reproduce the results found in this paper, where the cross correlation between two distributions is calculated: ...
2k views

### Align two datasets in Python

I want to develop some python code to align datasets obtained by different instruments recording the same event. As an example, say I have two sets of measurements: ...
175 views

### How do I filter insignificant modes from a kernel density estimation

How do I detect the number of significant modes in a set of data? I have a set of 1 dimensional data where the number of modes are unknown. I'm familiar with python and scikit learn so I use the ...
153 views

### Extraction of a decision boundary (LDA) after a systematic querying of the feature space and convolution with Sobel filter (examples in numpy) [duplicate]

I am doing some experiments with LDA (Linear Discriminant Analysis), in python. Now I am at the point in which I would like to display the separation planes in the 3-dimensional feature space. I ...
6k views

### How is the poisson distribution a distribution? It seems more like a formula

I just watched this video: https://www.youtube.com/watch?v=Fk02TW6reiA It shows a formula to calculate an answer for the following problem: There are 2 customers expected every 3 minutes in a store ...
I have a set of observation from two parameters, let say $x$ and $y$ and then I want to make the conditional probability of $x$ for the given $y$, $p(x|y)$. So first I use ...