Questions tagged [numpy]

NumPy is the fundamental package for scientific computing with Python.

Filter by
Sorted by
Tagged with
2
votes
0answers
916 views

Normalization of convolution kernel

I am trying to smooth a noisy one-dimensional physical signal, y, while retaining correspondence between the signal's amplitude and its units. I'm applying a ...
2
votes
0answers
117 views

Fitting two parallel lines with mutually excluding samples

I have a set of (slightly noisy) data taken from two (or more) curves, but taken either from one or the other curve. As an example, consider this code (in reality, data is noisier, especially in how ...
2
votes
2answers
203 views

Database-friendly random projections with Numpy

In his well known paper [1], Achlioptas showed that Random Projections could be performed with a sparse projection matrix, whose nonzero entries are either $1$ or $-1$. I have noticed that scikit-...
0
votes
0answers
277 views

Get rotation of noisy rectangular 2d point cloud with pca

I have a set of point clouds in nd space. The clouds tend to be rectangular in nature in 2d space but can easily have outliers or look slighly L shaped. I would like to rotate the point clouds so that ...
3
votes
2answers
735 views

PCA - Reconstruction from a “clean” set of eigenvectors?

This is a question related to the explanation here on how to reconstruct data from PCs found here: How to reverse PCA and reconstruct original variables from several principal components? I have two ...
1
vote
0answers
1k views

How to produce a normalized cumulative histogram?

I am having trouble understanding the proper method to calculate specific histograms, specifically with regard to cumulative and normalized histograms. If I want to calculate a normalized cumulative ...
0
votes
1answer
207 views

How can I train a CNN on raw numbers?

I am trying to make a binary classification using Keras' Conv2D classes based on this blog. I have many files with a matrix of floating point numbers in each one (these matrices are not pixels values)....
2
votes
1answer
206 views

Nearest neighbor with lower value

I have a collection of p points in n-space, and a p-vector of scalar values corresponding to each point. In this example, p is much larger than n. Is it possible to build an R-tree (or some other ...
1
vote
0answers
338 views

Linear Algebra and NumPy - How array broadcast translates to matrix operations?

I am familiar with Python's NumPy library and its broadcasting feature that allows to perform operations with vectors and matrices of different sizes. I might be missing the mathematical connection as ...
3
votes
1answer
1k views

Log Transformation Instead of Z-Score Normalizatrion For Machine Learning

I almost always used Numpy's StandardScaler to normalize my data for machine learning. I noticed however that simply taking the log of the variables that I wanted to normalize often resulted in better ...
2
votes
1answer
231 views

Why simulated and calculated t-distribution pdf at degree-of-freedom=1 doesn't match

See the attached graph, I am simulating the pdf of t-distribution at different degrees of freedom (dof), when dof is low (e.g. dof=1), why don't they match? Is it because my experiment wasn't done ...
0
votes
1answer
95 views

Finding change in spending habits

I have a numpy array full of customer spending data: x = np.array([5000,5500,6250,4800,3950,5800,5500,800,1200,900,500,400,300,200,3100]) Above, you can see ...
3
votes
1answer
19k views

Are 1-dimensional numpy arrays equivalent to vectors? [closed]

I'm new to both linear algebra and numpy, so please bear with me. I'm taking a course on linear regression, where I learned that we can express our hypothesis as $\theta^TX$ where $\theta$ is our ...
1
vote
0answers
58 views

Comparing curves to find anomalies in python

I am currently working on a Data Mining project in an assembly line. A lot of parts are produced in this assembly line and each part gives me a curve (it's the energy use of the part over the process ...
0
votes
1answer
156 views
0
votes
1answer
234 views

What's wrong with my algorithm for implementing the Storkey learning rule for Hopfield nets?

Trying to implement the Storkey rule... I can use the below algorithm to train an initial pattern -- because it goes into the if statement and simply uses the hebbian learning/outer-product method. ...
0
votes
3answers
759 views

What's a simple way to generate a random sample of a continuous distribution given as a series of trapezoids using scipy? [duplicate]

I currently have a continuous distribution that's described as a series of trapezoids in two arrays xs and ys, which integrate ...
1
vote
2answers
552 views

Iman-Conover implementation for correlated randoms in python with numpy/scipy

I've been trying to implement the iman-conover method in python so I could generate correlated random numbers from distributions other than normal (I use a normal & uniform in my example below). ...
2
votes
1answer
553 views

Built-in var() function in R computes the quasi-variance

I have recently noticed that the var() function in R returns a different value from the analogous function in the python package numpy. In particular, R's ...
0
votes
0answers
323 views

Is it right to use normal distribution to compute confidence intervals of a point in a zipf distribution?

I have a list of datapoints that falls within some sort of zipfian distribution or gamma distribution: ...
0
votes
1answer
32 views

Statistical classification of multiple arrays

I am working on an algorithm to mask uninteresting parts of images based on entropy. Based on samples saved as numpy arrays I am able to calculate statistics and do things like creating boxplots. My ...
1
vote
1answer
4k views

Keras Numpy Error: Why does accuracy go to zero?

I am trying to follow this tutorial. Everything works when I copy and paste the data, but when I try to replace the dataset with a random numpy matrix, my accuracy goes to zero. It shouldn't go to ...
1
vote
0answers
73 views

Logistic regression getting worse, what are the common causes?

I am making a naive logistic regression implementation to better understand how it works. I am using a very simple dataset provided by Andrew Ng's course on Coursera (pastebin link with the rest of ...
0
votes
1answer
6k views

Numpy.corrcoef() for twice one object does not return matrix of 1's

I'm trying to compare if two pictures are similar or close to similar. The picture is available as numpy.ndarray and contains of 28x28 pixels. It might look like the one below: When I get the image ...
2
votes
1answer
912 views

Correlation coefficient between two data sets having points of different dimensions

I have a dataset in which one side there are vectors of dimension 3 and other side is a real number corresponding to each vector. Now I need to find the correlation coefficient between the two set. ...
14
votes
1answer
7k views

How does NumPy solve least squares for underdetermined systems?

Let's say that we have X of shape (2, 5) and y of shape (2,) This works: np.linalg.lstsq(X, y) We would expect this to work only if X was of shape (N,5) where N>=...
2
votes
0answers
1k views

how to generate sliding window features in sci kit learn?

I have a data matrix where each row is measurement in time and each column is a feature. ...
0
votes
1answer
244 views

ROC curve end point confusion

Here is the code and output, I think from the output, it mean when fpr is 0, tpr is 0, this is correct as the prediction results marks everything to be 0. But the output also said, when fpr is 1, tpr ...
10
votes
4answers
5k views

Best way to seed N independent random number generators from 1 value

In my program I need to run N separate threads each with their own RNG which is used to sample a large dataset. I need to be able to seed this entire process with a single value so I can reproduce ...
2
votes
0answers
57 views

Chi2 test on big uniformly random sample [duplicate]

I expect that, in Python with numpy and scipy, scipy.stats.chisquare(numpy.bincount(numpy.random.randint(100, size=1000000))) will return P-value which is very ...
0
votes
1answer
2k views

How to cluster an 1-D array by K-means or any other algorithm using scikit-learn? [closed]

I have an one dimensional toy array X. I want to cluster the data into some numbers of clusters.But when I try to fit my data in scikit-learn K-Means function it shows ValueError: n_samples=1 ...
1
vote
1answer
65 views

mvBacon from R to Python [closed]

I use package "robustX" in R in order to compute the Mahalanobis distance in a dataset to detect outliers, using more than one variable. Specifically the code I am using is: ...
1
vote
2answers
1k views

Goodness of fit for long-tailed distributed data

I'm trying to fit a distribution to a set of data (the elevation of all land areas in the world). The histogram shows a long-tailed distribution, and I'd like to see which of the long-tailed models ...
0
votes
1answer
407 views

My neural network is good learning products but fails for sums

I implemented a simple feed-forward neural network with biases trained with backpropagation and stochastic gradient decent. What surprises me is how differently is behaves on two datasets I generated. ...
3
votes
1answer
265 views

Using a distance matrix *with errors* to find the coordinates of points

(I asked this same question in stackoverflow, without getting any answer, but maybe this is a more appropriate forum.) I would like to find the coordinates of a set of points in 3D from a distance ...
1
vote
0answers
581 views

Box Muller vs Numpy?

I used the Box Muller transform and pythons uniform random number generator to sample random numbers in a given interval [a,b]. Here's the approach I used: Note: I know the average and the standard ...
2
votes
0answers
1k views

How to implement Kernel density estimation in multivariate/3D

I have dataset like the following fromat and im trying to find out the Kernel density estimation with optimal bandwidth. ...
1
vote
1answer
47 views

What to do with paired data when a minority of pairs have more or less than two members?

Let's say I have two sets of measurements that go on like this: ...
2
votes
0answers
327 views

finding the interpolated values of an empirical probability distribution

I have an empirical probability function $p(z)$. In the first column $z$ and the second column contains $p(z)$ values. The data is given as following : data.cat ...
0
votes
2answers
258 views

Which model fit best for semi sinusoidal data? [duplicate]

I have a record containing the maximum and the minimum monthly temperatures at a particular station. The record shows information for each month from January 1908 to March 2012. However, some of the ...
1
vote
0answers
4k views

rnorm vs numpy.random.randn

For a regression example, I constructed some artificial data and ran ols, ...
0
votes
1answer
127 views

What are the good study materials on Association Rules?

I am looking to learn Association Rules, from basic level. I was looking for some good web based materials to start with. My objectives in the materials is to: (a) learn the aspect nicely from basic;...
1
vote
1answer
124 views

Recommendation Engine With Physical Distance Cutoff

I am looking to develop a recommendation engine for local stores to users. There are approximately 1 million stores in the database and around 1 million users. The 1Mx1M matrix for a user-based ...
2
votes
2answers
1k views

Reproduce Cross Correlation results in Python

I'm trying and failing to reproduce the results found in this paper, where the cross correlation between two distributions is calculated: ...
1
vote
0answers
2k views

Align two datasets in Python

I want to develop some python code to align datasets obtained by different instruments recording the same event. As an example, say I have two sets of measurements: ...
1
vote
0answers
175 views

How do I filter insignificant modes from a kernel density estimation

How do I detect the number of significant modes in a set of data? I have a set of 1 dimensional data where the number of modes are unknown. I'm familiar with python and scikit learn so I use the ...
0
votes
0answers
153 views

Extraction of a decision boundary (LDA) after a systematic querying of the feature space and convolution with Sobel filter (examples in numpy) [duplicate]

I am doing some experiments with LDA (Linear Discriminant Analysis), in python. Now I am at the point in which I would like to display the separation planes in the 3-dimensional feature space. I ...
5
votes
3answers
6k views

How is the poisson distribution a distribution? It seems more like a formula

I just watched this video: https://www.youtube.com/watch?v=Fk02TW6reiA It shows a formula to calculate an answer for the following problem: There are 2 customers expected every 3 minutes in a store ...
3
votes
0answers
207 views

Estimating the non- parametric conditional probability

I have a set of observation from two parameters, let say $x$ and $y$ and then I want to make the conditional probability of $x$ for the given $y$, $p(x|y)$. So first I use ...
2
votes
0answers
218 views

Sample data from combination of two probability distributions

I want to generate a mock catalogue. I have access to two independent sets of real data and I want to use their properties to make the mock catalogue: The first one contains the information from ...