Questions tagged [numpy]
NumPy is the fundamental package for scientific computing with Python.
124
questions
0
votes
0answers
27 views
How to approximate a function using Fast Fourier Transform? [closed]
I'm trying to approximate a complex function (I have millions of points to sample) using Fourier transform. I used Legendre and Chebyshev polynomials to approximate it before, and it worked pretty ...
0
votes
0answers
12 views
time complexity of sampling from multivariate hypergeometric distribuiton
numpy has an implementation and the doc is here. It says it is "roughly" equivalent to:
...
1
vote
2answers
35 views
Violin plot of 2 numpy arrays with seaborn
I would like to compare the distribution of 2 numpy arrays using a violin plot made with seaborn.
The maximal value in both arrays is 1.
The plot suggests a higher maximum.
Am I misunderstanding the ...
0
votes
0answers
33 views
Neural network from scratch: only predicts training inputs correctly [duplicate]
Here is a neural network I've been working on. It takes in an array of four zeros or ones and predicts whether that pattern of zeros and ones is a backslash.
...
0
votes
0answers
22 views
Predicting new values after feature scaling
I am trying to do linear regression with one feature only: predicting height with weights. Gradient descent took too many epochs so I used a min max scaler and it converged to the optimum point pretty ...
2
votes
0answers
64 views
Basic RNN sequence classifier diagram?
I'd like to build an RNN in numpy from scratch to really get come comfortable with backpropagation through time (BPTT.) In the below diagram and LaTeX, I show two neurons, each with a non-linearity, N(...
2
votes
2answers
50 views
Why sampling different random variables sequencially using the same PRNG alters the sequence that would be obtained if only one was sampled?
When using random variables in most programming languages the usual process is based on instatiating a RandomGenerator which will output an stream of pseudo-random numbers and with this stream the ...
0
votes
0answers
21 views
entropically regularized optimal transport on matrix with zero's on diagonal
I am a student reading an article where they are using the sinkhorn function from the ot Pyhton package to calculate the ...
0
votes
0answers
5 views
How to check the level against target encoding in my numpy array for the target feature?
I have a target variable called population that i first discretized using qcut with equal width binning ,
This gave me a single column df with 235 rows for the various countries
...
4
votes
5answers
22 views
Neural network based on twitter followers, what would be my features?
I was thinking of training a neural network that would be able to classify twitter users according to their followers. For example, I would like to know if a user is "gamer" or not by the people they ...
1
vote
1answer
106 views
representing quantile like quartile in form of normal distribution curve
I learned in statistics the first quartile, 2nd quartile, and 3rd quartile can be represented in the figure1 below
I came across this part of the article Step 4 - Feature Engineering.In this portion ...
0
votes
1answer
78 views
How to plot the prior, posterior and likelihood function from given data in python [closed]
I wrote a simple bayesian program which calculates prior, posterior and likelihood in python.
...
2
votes
1answer
30 views
Does normalization also help to prevend the vanish/exploding gradients?
I am implementing my own neural network from scratch using numpy. I tested my code with the MNIST dataset and I forgot to normalize the images and my code did not work, because I got an error about a ...
0
votes
0answers
23 views
What is the difference between np.linalg.norm(x-y,axis=1) and np.linalg.norm(x-y)?
I'm creating a K-Medoids algorithm from scratch in Python using numpy, and I'm in the process of using a distance function to determine the cluster center. I want the center to be the point in the ...
0
votes
1answer
32 views
How to make predictions with libmf using python? [closed]
I'm trying to implement libmf library in python. I tried with the following example:
...
0
votes
1answer
14 views
Stnadard deviation of values containing different ranges
Taking the $\sigma$ of these values :
np.std([55,50,40,45])
returns:
5.59
If I take the $\sigma$ of these values:
...
0
votes
0answers
13 views
Simple ANN model converges with tanh(x) as the activation function, but it doesn't with leaky ReLu
I'm training a simple ANN model (MLP) using as the activation function tanh(x) and, after some interactions, it converges with error equal to 10^-5, here's my full ...
0
votes
0answers
15 views
Weights between the Last 2 layers keep getting negative
TL;DR weights between the last 2 layers keep getting negative to the point that the softmax(z) of the output layer can't divide by zero ( e^-750 ~= 0 thus deciding by 0)
I am making a Neural Network ...
0
votes
1answer
74 views
How to get the variance between two arrays?
import numpy as np
a = np.array([[1,2,4],[2,4,8]])
np.var(a)
output: 5.25
Can anyone enlight me what's the calculation process to get variance = 5.25?
0
votes
0answers
32 views
Finding weight value for smooth constrained least squares that comes closest with a priori solution?
I need to solve a system of $n.k$ equations of form $t_{ij} = a_i + b_j + x_{ij}/v$, with $i = 1, ..., n$ and $j = 1, ..., k$.
A least squares approach can be used, as in $d = Gm$. My data is $t_{ij}$...
1
vote
0answers
13 views
Gaussian mixture models for image matrix not determining E step
I want to calculate responsibility for each of the data points, for the given MU, SIGMA and PI.
...
2
votes
1answer
124 views
Interpreting scipy.stats: ks_2samp and mannwhitneyu give conflicting results
I've encountered conflicting results between ks_2samp and mannwhitneyu while trying to compare two empirical distributions, ...
1
vote
1answer
27 views
Why when I increase the standard deviation in the pdf function the graph looks squeezed instead of more dispersed?
So what I am doing is this:
...
0
votes
0answers
72 views
Combine TFIDF with non-textual features
I am dealing with an email classification problem in which I have email requests coming from different groups of people. I am building a classifier to classify these emails based on historical email ...
1
vote
0answers
37 views
Estimate joint probability of two dependent variable
I've a dataset created in the following way:
The input of the system is a 8 bit binary number ranging from $x_1$= 0000
0000 to $x_N$ = 1111 1111
For every input i've read the output of the system (...
0
votes
0answers
77 views
Generate random values to mimic skewness
I have a actual set of data where the variables are heavily skewed, both positively and negatively.
I need to generate random sample data for the values going forward. The data needs to be similarly ...
0
votes
0answers
54 views
Q-Exponential distribution in Numpy
I am studying a phenomena, which I know is characterised by the q-exponential distribution, with parameters $c=1$, $q = 1.355$ and $b = 0.524$ in the equation
$$e_{q,b,c}(t) = c(1 + b(q-1)t)^{\frac{1}{...
0
votes
0answers
18 views
Adding features lowered accuracy score Gaussian Naive Bayes algorithm, Python
I am in third year of university, this simple py program is meant to use the Gaussian Naive Bayes algorithm to create a model and evaluate it.
...
0
votes
1answer
22 views
Feeding multiple rows of data into ANN [closed]
I've built an ANN from scratch, that works with one row of data with any number of neurons and hidden layers. For the setup I am using 2 hidden layers, 5 neurons (just while building). The network ...
1
vote
0answers
72 views
Infinitesimal generator of Markov chain using numpy
I am computing the infinitesimal generator of a continuous-time Markov chain from the transition probabilities p. I am following the methodology described here, ...
-1
votes
2answers
322 views
Transforming data for normality with negative values using python
I am working with a data set in a machine learning project, which has lots to negative values. I want to transform the distribution of my data to normal. I tried using numpy.log, but since log is only ...
0
votes
0answers
45 views
Numpy corrcoef giving different result from manual calculation
I'm trying to calculate a partial correlation matrix for a high dimensional problem. I'm using this paper as a guide. I'm also referencing this function from Pingouin.
Starting from the inverse ...
2
votes
0answers
16 views
Can't Recreate Values for U, S, V from SVD in numpy [duplicate]
To better understand SVD, I'm trying to recreate the values for U, S, and V using straight ...
4
votes
1answer
208 views
QR Factorization to Solve Least Squares Without Using an Inverse
I'm playing around with different ways to solve least squares, and am using numpy to derive values for $\beta$ in a regression problem.
I know that if you do a $QR$ factorization of $X$ such that $...
1
vote
0answers
27 views
Dataframe containing value 0 should be removed or replaced?
I have a question that mostly I get stuck at. I was looking at the data for diabetes patients and found that most of the rows have 0 values under most of their columns. Reference for the url
https://...
0
votes
1answer
274 views
KL Divergence of two standard normal arrays
I generated two 9000,1 np arrays with
a = np.random.standard_normal(9000)
b = np.random.standard_normal(9000)
Then I check the KL Divergence with
...
3
votes
0answers
181 views
James Stein Estimator for more than one Sample
I have a hard time understanding the James-Stein Estimator. I show you how I tried to comprehend it by using a python example. I take a normally distributed random vector with mean $(0.1, 0.2, 0.3, 0....
1
vote
1answer
106 views
Outliers of small dataset
I have a python function that takes a list of smaller images boxes (represented as float arrays) and the whole image img in as a ...
1
vote
1answer
654 views
Fitting Beta Distributions to Data
I am trying to reproduce some beta distribution parameters found in this published paper. I have two data sets, y1 and y2, that ...
0
votes
0answers
20 views
Proper way of calculating statistics for samples probabilites [closed]
I have 64 samples and for each one of them, I have their probability. I was wondering which is the correct way to find a mean and std for the above-mentioned population. If I do something like:
<...
1
vote
0answers
174 views
Gradient Descent Vectorization with Numpy (1D transpose confusion) [closed]
I'm working through Andrew Ng's original Stanford course and ran into some numpy confusion. Basically, my main question is, if we dot product a 1D array with a 2D array in numpy (and the dimensions ...
0
votes
0answers
26 views
is it reasonable to consider a plot of 2 separate normal random variables as the geometric representation of a joint probability distribution?
I am plotting 2 normal distribution in Python. if anyone could provide an R version to demonstrate and explain, that would be grateful.
...
0
votes
1answer
3k views
Array of samples from multivariate gaussian distribution Python
I am trying to build in Python the scatter plot in part 2 of Elements of Statistical Learning. First it is said to generate
10 means mk
from a bivariate Gaussian distribution N((1,0)T,I) and ...
2
votes
1answer
312 views
Efficiently Computing The Beta CDF [duplicate]
I am using numba to JIT compile some looped python functions as part of a larger application. Ideally, everything will run in numba's "no python" mode, such that the loop can be parallelised.
One of ...
2
votes
1answer
1k views
Ridge Regression with Gradient Descent Converges to OLS estimates
I'm implementing a homespun version of Ridge Regression with gradient descent, and to my surprise it always converges to the same answers as OLS, not the closed form of Ridge Regression.
This is ...
3
votes
1answer
785 views
Implementing Lasso Regression in Numpy
I'm doing a little self study project, and am trying to implement OLS, Ridge, and Lasso regression from scratch using just Numpy, and am having problems getting this to work with Lasso regression.
...
0
votes
0answers
47 views
Rank 1 SVD with constraint on U
I need to perform a particular rank 1 decomposition of a sparse matrix $\mathbf{A} \in \mathbb{R}^{n\times n}$.
In particular I am looking for the positive vector $\mathbf{u} \in \mathbb{R}^{+n}$ ...
0
votes
1answer
76 views
What is the formula for calculation of `R_ij` in `numpy.corrcoef(x, y, rowvar = False)`?
The manual does not provide the formula if we pass x and y. I do not understand the matrix I get. Here is my code:
...
1
vote
1answer
23 views
Memory considerations with sklearn classfier
I am trying to fit a sklearn.ensemble.RandomForestClassifier. The [docs] explain that a matrix (rows - observations, columns - features). My observations are 700 ...
3
votes
1answer
3k views
PCA principal components in sklearn not matching eigen-vectors of covariance calculated by numpy
I was trying to replicate PCA in sklearn's PCA API using numpy using PCA in numpy and sklearn produces different results.
I noticed that:
eigenvalues are same as the PCA object's explained_variance_ ...
