Questions tagged [numpy]
NumPy is the fundamental package for scientific computing with Python.
118
questions
0
votes
0answers
4 views
How to check the level against target encoding in my numpy array for the target feature?
I have a target variable called population that i first discretized using qcut with equal width binning ,
This gave me a single column df with 235 rows for the various countries
...
1
vote
1answer
31 views
representing quantile like quartile in form of normal distribution curve
I learned in statistics the first quartile, 2nd quartile, and 3rd quartile can be represented in the figure1 below
I came across this part of the article Step 4 - Feature Engineering.In this portion ...
0
votes
0answers
5 views
How to read complex number matrix from excel/csv using python? [closed]
I am trying to read complex numbers matrix saved in excel xlsx format.
I used pd.read_excel('data.xlsx') to read using pandas. But after reading when I convert it to NumPy array using .to_numpy(), ...
0
votes
1answer
14 views
How to plot the prior, posterior and likelihood function from given data in python [closed]
I wrote a simple bayesian program which calculates prior, posterior and likelihood in python.
...
2
votes
1answer
20 views
Does normalization also help to prevend the vanish/exploding gradients?
I am implementing my own neural network from scratch using numpy. I tested my code with the MNIST dataset and I forgot to normalize the images and my code did not work, because I got an error about a ...
0
votes
0answers
11 views
What is the difference between np.linalg.norm(x-y,axis=1) and np.linalg.norm(x-y)?
I'm creating a K-Medoids algorithm from scratch in Python using numpy, and I'm in the process of using a distance function to determine the cluster center. I want the center to be the point in the ...
0
votes
1answer
27 views
How to make predictions with libmf using python? [closed]
I'm trying to implement libmf library in python. I tried with the following example:
...
0
votes
1answer
13 views
Stnadard deviation of values containing different ranges
Taking the $\sigma$ of these values :
np.std([55,50,40,45])
returns:
5.59
If I take the $\sigma$ of these values:
...
0
votes
0answers
12 views
Simple ANN model converges with tanh(x) as the activation function, but it doesn't with leaky ReLu
I'm training a simple ANN model (MLP) using as the activation function tanh(x) and, after some interactions, it converges with error equal to 10^-5, here's my full ...
0
votes
0answers
12 views
Weights between the Last 2 layers keep getting negative
TL;DR weights between the last 2 layers keep getting negative to the point that the softmax(z) of the output layer can't divide by zero ( e^-750 ~= 0 thus deciding by 0)
I am making a Neural Network ...
0
votes
1answer
28 views
How to get the variance between two arrays?
import numpy as np
a = np.array([[1,2,4],[2,4,8]])
np.var(a)
output: 5.25
Can anyone enlight me what's the calculation process to get variance = 5.25?
0
votes
0answers
29 views
Finding weight value for smooth constrained least squares that comes closest with a priori solution?
I need to solve a system of $n.k$ equations of form $t_{ij} = a_i + b_j + x_{ij}/v$, with $i = 1, ..., n$ and $j = 1, ..., k$.
A least squares approach can be used, as in $d = Gm$. My data is $t_{ij}$...
1
vote
0answers
13 views
Gaussian mixture models for image matrix not determining E step
I want to calculate responsibility for each of the data points, for the given MU, SIGMA and PI.
...
2
votes
1answer
55 views
Interpreting scipy.stats: ks_2samp and mannwhitneyu give conflicting results
I've encountered conflicting results between ks_2samp and mannwhitneyu while trying to compare two empirical distributions, ...
1
vote
1answer
27 views
Why when I increase the standard deviation in the pdf function the graph looks squeezed instead of more dispersed?
So what I am doing is this:
...
0
votes
0answers
32 views
Combine TFIDF with non-textual features
I am dealing with an email classification problem in which I have email requests coming from different groups of people. I am building a classifier to classify these emails based on historical email ...
1
vote
0answers
31 views
Estimate joint probability of two dependent variable
I've a dataset created in the following way:
The input of the system is a 8 bit binary number ranging from $x_1$= 0000
0000 to $x_N$ = 1111 1111
For every input i've read the output of the system (...
0
votes
0answers
55 views
Generate random values to mimic skewness
I have a actual set of data where the variables are heavily skewed, both positively and negatively.
I need to generate random sample data for the values going forward. The data needs to be similarly ...
0
votes
0answers
43 views
Q-Exponential distribution in Numpy
I am studying a phenomena, which I know is characterised by the q-exponential distribution, with parameters $c=1$, $q = 1.355$ and $b = 0.524$ in the equation
$$e_{q,b,c}(t) = c(1 + b(q-1)t)^{\frac{1}{...
0
votes
0answers
17 views
Adding features lowered accuracy score Gaussian Naive Bayes algorithm, Python
I am in third year of university, this simple py program is meant to use the Gaussian Naive Bayes algorithm to create a model and evaluate it.
...
0
votes
1answer
14 views
Feeding multiple rows of data into ANN [closed]
I've built an ANN from scratch, that works with one row of data with any number of neurons and hidden layers. For the setup I am using 2 hidden layers, 5 neurons (just while building). The network ...
1
vote
0answers
52 views
Infinitesimal generator of Markov chain using numpy
I am computing the infinitesimal generator of a continuous-time Markov chain from the transition probabilities p. I am following the methodology described here, ...
-1
votes
2answers
200 views
Transforming data for normality with negative values using python
I am working with a data set in a machine learning project, which has lots to negative values. I want to transform the distribution of my data to normal. I tried using numpy.log, but since log is only ...
0
votes
0answers
29 views
Numpy corrcoef giving different result from manual calculation
I'm trying to calculate a partial correlation matrix for a high dimensional problem. I'm using this paper as a guide. I'm also referencing this function from Pingouin.
Starting from the inverse ...
2
votes
0answers
16 views
Can't Recreate Values for U, S, V from SVD in numpy [duplicate]
To better understand SVD, I'm trying to recreate the values for U, S, and V using straight ...
3
votes
1answer
117 views
QR Factorization to Solve Least Squares Without Using an Inverse
I'm playing around with different ways to solve least squares, and am using numpy to derive values for $\beta$ in a regression problem.
I know that if you do a $QR$ factorization of $X$ such that $...
1
vote
0answers
27 views
Dataframe containing value 0 should be removed or replaced?
I have a question that mostly I get stuck at. I was looking at the data for diabetes patients and found that most of the rows have 0 values under most of their columns. Reference for the url
https://...
0
votes
0answers
13 views
How to find optimum matrix set based on determinant values using python
I am new at programming, so I want to find the optimum set of row values based on maximum determinant logic.
1) Set the 1st Column 'Serial_no' as index.
2) Take first 'N' row values as user input ...
0
votes
1answer
221 views
KL Divergence of two standard normal arrays
I generated two 9000,1 np arrays with
a = np.random.standard_normal(9000)
b = np.random.standard_normal(9000)
Then I check the KL Divergence with
...
3
votes
0answers
120 views
James Stein Estimator for more than one Sample
I have a hard time understanding the James-Stein Estimator. I show you how I tried to comprehend it by using a python example. I take a normally distributed random vector with mean $(0.1, 0.2, 0.3, 0....
1
vote
1answer
81 views
Outliers of small dataset
I have a python function that takes a list of smaller images boxes (represented as float arrays) and the whole image img in as a ...
0
votes
0answers
41 views
Different number of Eigen/Singular values from PCA and SVD
My understanding is that a SVD done on a raw data matrix M and a PCA done on its covariance matrix C should return the same eigen/singular values.
I have a 2736 x 356 data matrix and am using the ...
1
vote
1answer
513 views
Fitting Beta Distributions to Data
I am trying to reproduce some beta distribution parameters found in this published paper. I have two data sets, y1 and y2, that ...
0
votes
0answers
20 views
Proper way of calculating statistics for samples probabilites [closed]
I have 64 samples and for each one of them, I have their probability. I was wondering which is the correct way to find a mean and std for the above-mentioned population. If I do something like:
<...
0
votes
0answers
5 views
Correlation fails on high sampling rate vector
I am trying to find the time shift between two vectors.
My function succeeds on randomly generated vectors, but fails on small shifted, high sampling rate vectors.
my code:
...
1
vote
0answers
146 views
Gradient Descent Vectorization with Numpy (1D transpose confusion) [closed]
I'm working through Andrew Ng's original Stanford course and ran into some numpy confusion. Basically, my main question is, if we dot product a 1D array with a 2D array in numpy (and the dimensions ...
0
votes
0answers
26 views
is it reasonable to consider a plot of 2 separate normal random variables as the geometric representation of a joint probability distribution?
I am plotting 2 normal distribution in Python. if anyone could provide an R version to demonstrate and explain, that would be grateful.
...
0
votes
1answer
2k views
Array of samples from multivariate gaussian distribution Python
I am trying to build in Python the scatter plot in part 2 of Elements of Statistical Learning. First it is said to generate
10 means mk
from a bivariate Gaussian distribution N((1,0)T,I) and ...
2
votes
1answer
240 views
Efficiently Computing The Beta CDF [duplicate]
I am using numba to JIT compile some looped python functions as part of a larger application. Ideally, everything will run in numba's "no python" mode, such that the loop can be parallelised.
One of ...
2
votes
1answer
1k views
Ridge Regression with Gradient Descent Converges to OLS estimates
I'm implementing a homespun version of Ridge Regression with gradient descent, and to my surprise it always converges to the same answers as OLS, not the closed form of Ridge Regression.
This is ...
2
votes
0answers
650 views
Implementing Lasso Regression in Numpy
I'm doing a little self study project, and am trying to implement OLS, Ridge, and Lasso regression from scratch using just Numpy, and am having problems getting this to work with Lasso regression.
...
0
votes
0answers
45 views
Rank 1 SVD with constraint on U
I need to perform a particular rank 1 decomposition of a sparse matrix $\mathbf{A} \in \mathbb{R}^{n\times n}$.
In particular I am looking for the positive vector $\mathbf{u} \in \mathbb{R}^{+n}$ ...
0
votes
1answer
72 views
What is the formula for calculation of `R_ij` in `numpy.corrcoef(x, y, rowvar = False)`?
The manual does not provide the formula if we pass x and y. I do not understand the matrix I get. Here is my code:
...
1
vote
1answer
22 views
Memory considerations with sklearn classfier
I am trying to fit a sklearn.ensemble.RandomForestClassifier. The [docs] explain that a matrix (rows - observations, columns - features). My observations are 700 ...
3
votes
1answer
2k views
PCA principal components in sklearn not matching eigen-vectors of covariance calculated by numpy
I was trying to replicate PCA in sklearn's PCA API using numpy using PCA in numpy and sklearn produces different results.
I noticed that:
eigenvalues are same as the PCA object's explained_variance_ ...
1
vote
0answers
16 views
Constructing an appropriate null hypothesis?
I have a collection of 20 2D surfaces embedded in 3D. The mean curvature of each surface has been sampled.
The number of samples per surface varies from ~700,000 to ~20,000,000. The choice of ...
2
votes
1answer
471 views
Normalised and “Normal” Cross Correlation giving different lag positions
Using numpy's np.correlate() am trying to find the lag position of two data sets of different length.
When I use this operation by its own I find a lag position ...
0
votes
1answer
75 views
Shifting Data So That When Cross Correlated Lag position is 0
Using Numpy, I am cross correlating two large data sets (of different lengths), as part of a method to compare the similarity of the data. However to take the data onto the next step of the comparison ...
0
votes
1answer
17 views
Matplotlib: Why would a KNN regression model draw a line through ALL points regardless of K?
Having difficulty doing something very basic: create a random dataset taking on values between -1,1, plot them, and also fit a KNN model to the data and fit it over the data. Using numpy to create the ...
1
vote
1answer
733 views
Finding the appropriate polynomial fit in Python
Is there a function or library in Python to automatically compute the best polynomial fit for a set of data points? I am not really interested in the ML use case of generalizing to a set of new data, ...
