Questions tagged [numpy]
NumPy is the fundamental package for scientific computing with Python.
150
questions
0
votes
0
answers
21
views
Reinforcement learning on pandas dataframe. Problems regarding space and observations
I am trying to create a reinforcement learning algorithm which optimizes Pulltimes (timestamp) on a flight schedule, this happens by the agent substracting a number between 30-60 from the current STD (...
- 1
1
vote
2
answers
122
views
Python/numpy - conditional sampling of variables, distribution of subsequent value is based on result of previous value
I am trying to generate a random sample of multiple variables which are loosely related to each other. Meaning that "allowed" values of some variables depend on the value which is set for ...
1
vote
0
answers
56
views
How to calculate the equilibrium (initial probability) for second order HMM in python?
I have a transition matrix looks like:
...
- 11
0
votes
0
answers
83
views
Generating multivariate random variable with normal and exponential marginals
I have a collection of data points of the form $[U, V, X, Y]$, where $U$ ~ $N(\mu_1, \sigma_1)$; $V$ ~ $N(\mu_2, \sigma_2)$; $X$ ~ $exp(\lambda_1)$; and $Y$ ~ $exp(\lambda_2)$, and I am looking to ...
- 123
1
vote
0
answers
59
views
Logistic regression with SGD getting low error with high learning rate
The thing is that I am implementing LR with SGD.
Here is my cross error function:
...
- 61
0
votes
0
answers
81
views
Regression analysis with max value
I am trying to find the best curve that will describe my data. my data are stored in numpy arrays of t and dur they are both in ...
7
votes
8
answers
1k
views
Statistically compare two large continuous datasets
I have multiple images with 1 band/channel. This of an RGB image, then I only have the blue band/channel. In other words, multiple 1D datasets, or multiple 1D arrays.
I would like to statistically ...
- 227
1
vote
1
answer
670
views
Gradient of a multivariate function numpy
I'm trying to calculate the gradient of multivariate function g using NumPy.
g = lambda w: -np.sin(np.pi*np.sum(w**2)) + np.log(np.sum(w**2))
...
- 13
1
vote
0
answers
69
views
Positive semi definite matrix with negative eigenvalues?
From what I know, for any square real matrix A, a matrix generated with the following should be a positive semidefinite (PSD) matrix:
...
- 11
1
vote
0
answers
29
views
Working with sparse data in numpy and sklearn [closed]
I have a time series dataset geenrated from some electrophysiological data. I have a frequncy dataset and the matrix is quite sparse but huge, like it contains 0.005 s time bins for 2000+ neurons ...
- 329
0
votes
0
answers
222
views
RMSE, MAE and MAPE reduced but RMSPE increased after removing outliers in linear regression
I am using a linear regression model with 465 datapoints to predict crop yield of wheat. The data points are spilt into training and testing sets in 80:20 ratio. Initially, I tried linear regression ...
0
votes
0
answers
37
views
Sklearn PCA projection differences while reproducing projection sample by sample
I've been stuck on this issue due to inability to reproduce the performance of PCA-classification pipeline on training data, but in case the pipeline receives one sample at a time. This seems to be ...
0
votes
0
answers
15
views
Preprocessing protocol to derive wavelength from slanted wave profile
I am trying to derive a wavelength from sediment wave profiles. In some instances with the data, the sediment waves occur on a slope. This slope can be straight or curved depending on each instance.
...
1
vote
0
answers
67
views
How do I get the stationary distribution of a Markov chain matrix from SVD?
I have a matrix that represents a Markov chain.
...
- 1,327
1
vote
0
answers
45
views
Probability of drawing N elements from M elements with per element draw probability without replacement
I have a list of M elements with per element draw probability, summing to 1.0.
Example (M=10): L = [0.3, 0.2, 0.01, 0.05, 0.02, 0.02, 0.2, 0.05, 0.05, 0.1]
Now I ...
-1
votes
1
answer
164
views
Implement permutation test for difference of means (procedure described in linked book) in Python
Asked also on Stack Overflow. Question summary: I've implemented method described in a book and got results different than expected. What am I doing wrong?
Consider the following problem: farmer is ...
- 71
0
votes
1
answer
393
views
How to sample from non-gaussian dataset [duplicate]
Let's say I have a small dataset (normalized), that doesn't necessarily follow a gaussian distribution.
We can see that by plotting it on a simple histogram. There is clearly a bias. There is also a ...
K Split X
0
votes
2
answers
88
views
I'm performing a multi linear regression on the The Oxford Covid-19 Government Response Tracker. I have a couple of doubts about the process
I hope this is the right place to ask this.
I'm currently working my way through a dataset and performing Multiple Linear Regression on it. The data is for Oxford Governement Response Tracker for the ...
1
vote
1
answer
145
views
Behaviour of intralayer (lateral) connections in neural network during training
I am interested in experimenting with some very simple neural network architectures by building them in numpy.
I have followed a course in how to build a feed-forward multilayer neural network in ...
- 23
0
votes
1
answer
2k
views
How can i get the phase difference between two frequencies?
I have two signals of the same frequency. Both have frequencies of 13.56Mhz, and I want to find the phase difference between these two frequencies.
The function generator generates a sinusoidal ...
이신행
2
votes
2
answers
1k
views
How to sample from truncated distributions using scipy?
I have min and max values for certain variables such as:
Expenses
Loss
Growth
I'd like to add a distribution around them and plot a histogram in python. Distribution could be beta, gamma right type ...
- 488
0
votes
0
answers
251
views
How to create and solve a underdetermined nonlinear system of equations using python
as i already mentioned in the header my plan is to create and solve a system of nonlinear equations in python. As I also mentioned, the equations system will be underdetermined.
I will try to give an ...
- 1
0
votes
1
answer
34
views
Standard deviation cap crosses zero
Let's say I have some reference data and some predicted data and I want to show the standard deviation of the mean absolute error. When I create error bar using numpy and matplotlib, errorbar crosses ...
- 111
0
votes
1
answer
692
views
In Python, can I use mean, median, minimum, maximum, standard deviation, population size, and single sample to generate statistically identical data?
Example situation
After each exam, the professor provides the following information.
Minimum Score
[Arithmetic] Mean
Median
Maximum Score
Standard Deviation
I also know what my score was as well as ...
- 1
0
votes
0
answers
107
views
Visualize PCA, train or test?
I am new to PCA and have a question about visualization when it comes to fitting and transposing. I have two data, which are train and test. Here are four methods:
...
- 9
0
votes
0
answers
32
views
What I have this rare output value of variance and standart deviation functions in numpy? [duplicate]
Hell i have nonsense values of variance and standart deviation in numpy.
this documentation no make sense with this sample.
https://numpy.org/doc/stable/reference/generated/numpy.std.html
youtube ...
- 107
0
votes
1
answer
1k
views
When does MAPE (Mean Absolute Percentage Error) fail?
I have a multioutput regression model that predicts float values. When using MAPE to evaluate regression model performance (using either built in libraries or implementing a function for it) I am ...
- 101
1
vote
1
answer
612
views
Numpy.cov() in Python returning a matrix of NaN's instead of 0's [closed]
The cov() method from the numpy library returns a covariance matrix where columns represent different features and rows represent separate instances of the same feature. Therefore, passing a nx1 ...
- 121
1
vote
0
answers
584
views
Numpy and Statmodel issue - LinAlgError - Singular Matrix
I have seen other topics on this, but I couldn't get their solutions working for me. Perhaps someone else will know why.
namely:
statsmodels: error in kde on a list of repeated values
However, my ...
- 111
0
votes
1
answer
93
views
Stats Model contingency table, unable to reshape data and run test for nd array 2 x 2 x k
Consider below list of 2×2 tables and CMH (Cochran–Mantel–Haenszel) test results. We are trying to determine if each specific centre was associated with the success of the treatment [Data from Agresti,...
- 125
0
votes
0
answers
224
views
How to find out the interval within which probability of sample mean lying within $\pm a$ units of population mean is $0.95$ using Python?
I am a beginner in the field of statistics and data science, so request you to treat me kindly.
I have to solve the following problem in python:
Suppose the population variable $X$ is $\text{N}(3, 0....
1
vote
2
answers
1k
views
Proper shape of LSTM dataset for keras
I understand that similar questions have been asked before, but they are all based on specific examples. I want to consider a very simple example: we have a sequence of 1000 numbers, and want an LSTM ...
- 1,215
0
votes
1
answer
498
views
How to find the percentage of values within a given range?
I am a beginner in the field of statistics and data science, so request you to treat me kindly.
Problem statement - Variable X has a mean of 15 and a standard deviation of 2.
What is the minimum ...
0
votes
1
answer
245
views
Bounds for samples from the normal distribution using numpy.random.randn
I am new to both stats and scientific Python, so apologies if this question breaches any guidelines.
I want to know what x = np.random.randn(500) actually does, in ...
- 255
1
vote
1
answer
64
views
Probability of a point taken from a certain normal distribution will be less than or equal to than a point taken from another?
Suppose we have two independent normal distributions
How do I calculate the probability of a certain point taken from distribution X1 being less than or equal to a ...
- 123
3
votes
1
answer
3k
views
How does numpy generate samples from a beta distribution?
numpy lets you generate random samples from a beta distribution (or any other arbitrary distribution) with this API:
...
- 153
1
vote
1
answer
37
views
Clarification needed on gradients in backpropagation
I was going through this book "Practical Convolutional Neural Networks" and there under the backpropagation section, it demonstrates calculating the gradient for ...
0
votes
2
answers
303
views
Generating statistical distributions and sorting them
I am creating two distributions as shown below.
...
- 153
1
vote
0
answers
31
views
Generating correlated samples from bernoulli variables [duplicate]
I am trying to implement a program that samples from two identically distributed Bernoulli random variables, $X_1$ and $X_2$, such that they have a specific correlation coefficient, $\rho$.
I found a ...
- 199
2
votes
2
answers
3k
views
Violin plot of 2 numpy arrays with seaborn
I would like to compare the distribution of 2 numpy arrays using a violin plot made with seaborn.
The maximal value in both arrays is 1.
The plot suggests a higher maximum.
Am I misunderstanding the ...
0
votes
0
answers
33
views
Neural network from scratch: only predicts training inputs correctly [duplicate]
Here is a neural network I've been working on. It takes in an array of four zeros or ones and predicts whether that pattern of zeros and ones is a backslash.
...
- 1
0
votes
0
answers
130
views
Predicting new values after feature scaling
I am trying to do linear regression with one feature only: predicting height with weights. Gradient descent took too many epochs so I used a min max scaler and it converged to the optimum point pretty ...
2
votes
0
answers
75
views
Basic RNN sequence classifier diagram?
I'd like to build an RNN in numpy from scratch to really get come comfortable with backpropagation through time (BPTT.) In the below diagram and LaTeX, I show two neurons, each with a non-linearity, N(...
- 1,728
2
votes
2
answers
111
views
Why sampling different random variables sequencially using the same PRNG alters the sequence that would be obtained if only one was sampled?
When using random variables in most programming languages the usual process is based on instatiating a RandomGenerator which will output an stream of pseudo-random numbers and with this stream the ...
- 187
4
votes
5
answers
103
views
Neural network based on twitter followers, what would be my features?
I was thinking of training a neural network that would be able to classify twitter users according to their followers. For example, I would like to know if a user is "gamer" or not by the people they ...
Sharki
2
votes
1
answer
3k
views
representing quantile like quartile in form of normal distribution curve
I learned in statistics the first quartile, 2nd quartile, and 3rd quartile can be represented in the figure1 below
I came across this part of the article Step 4 - Feature Engineering.In this portion ...
- 121
0
votes
1
answer
3k
views
How to plot the prior, posterior and likelihood function from given data in python [closed]
I wrote a simple bayesian program which calculates prior, posterior and likelihood in python.
...
- 2,041
4
votes
1
answer
693
views
Does normalization also help to prevend the vanish/exploding gradients?
I am implementing my own neural network from scratch using numpy. I tested my code with the MNIST dataset and I forgot to normalize the images and my code did not work, because I got an error about a ...
- 173
1
vote
1
answer
136
views
How to make predictions with libmf using python? [closed]
I'm trying to implement libmf library in python. I tried with the following example:
...
- 197
1
vote
1
answer
17
views
Stnadard deviation of values containing different ranges
Taking the $\sigma$ of these values :
np.std([55,50,40,45])
returns:
5.59
If I take the $\sigma$ of these values:
...
- 617