Skip to main content

Questions tagged [python]

Python is a programming language commonly used for machine learning. Use this tag for any *on-topic* question that (a) involves `Python` either as a critical part of the question or expected answer, & (b) is not *just* about how to use `Python`.

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

country-year dummies in Python [closed]

I am currently using the linearmodels 6.0 package to perform econometric analyses. I included region and time fixed-effects using ...
fernand's user avatar
0 votes
0 answers
10 views

How to determine the ideal number of components for PLSR using RMSE?

I would like to determine a non-visual, numeric based approach to determine the ideal number of components for my PLSR model. There are 10 components in the model for 1 target variable. If I simply ...
tds's user avatar
  • 1
0 votes
0 answers
12 views

Got numerical difference between two implementations

I've been working around RetNet (Paper: https://arxiv.org/pdf/2307.08621, PyTorch implementation: https://github.com/Jamie-Stirling/RetNet/). I rewrote the some of the code with TensorFlow: ...
UndefinedCpp's user avatar
0 votes
0 answers
9 views

CNN matrices shape for time series data processing

I would like to ask you for advice regarding CNNs for analysing 60000x16 data (single input) - time series records from 16 channels. I did some research on this and my initial idea was to use CNN with ...
kalmary's user avatar
0 votes
0 answers
17 views

Reproducing PCA results of pca.fit_transform() using pca.fit()? [duplicate]

I have a data frame called data_principal_components with dimensions (306x21154), so 306 observations and 21154 features. Using PCA, I want to project the data into 10 dimensions. As far as I ...
george1994's user avatar
0 votes
0 answers
20 views

What dimensions to expect of Principal Components Analysis? [duplicate]

In both Python and R, the matrix of eigenvectors of Principal Component Analysis (PCA) is a matrix of principal components with dimensions (Number of Observations x Number of Principal Components). ...
george1994's user avatar
0 votes
0 answers
38 views

How to fit data to a parametric curve/model (x(t), y(t)?

I've got data of x and y pairs and I'd like to fit it to a model that is parametrized as f = (x(t), y(t)). Unfortunately, there is no way for me to analytically solve for t and get a direct ...
StatAnomaly123's user avatar
1 vote
1 answer
65 views

Errors on cubic fits are relative to the order of magnitude of the y-data, not fit quality

I am writing some mass spec data reduction software (specifically, residual gas analysis mass spectrometry). I recently implemented a cubic fitting algorithm as an optional feature, however, some of ...
ohshitgorillas's user avatar
0 votes
0 answers
12 views

Why does the forecast for some series degrade when using a VARMA model comparing to independent ARMA models?

I am working with multiple time series that I suspect are correlated, and I have assumed that using a VARMA model would at least not degrade the forecasts of each series, if not improve them. However, ...
Rocio's user avatar
  • 1
0 votes
0 answers
17 views

R implementation of Bivariate Test by Maronna

Does anyone know an R package that implements the Bivariate Test by Maronna and Yohai, 1978? Or maybe a derivate like the adaption from Potter, 1981 ? Or a baysian adaption ? Was not able to find an ...
Dirk's user avatar
  • 3
0 votes
0 answers
16 views

Which type of ANOVA is the right choice?

I have a data set as follows and I'm wondering how: (i) this should be organised for an ANOVA test and (ii) which ANOVA (two-way mixed, two-way repeated measures) test is the correct one. ...
Ryan's user avatar
  • 1
1 vote
0 answers
11 views

Fixed-effect trained model inspection in mixed-effects random forest (MERF)

I have run a Mixed-Effects Random Forest (MERF) using the python merf module, see therein example use here (see also blog post). I have read the above and also Hajjem et al's paper, to get an idea of ...
Emma Wiik's user avatar
2 votes
2 answers
68 views

Python Fitted Sigmoid extreme values barely being used

I was initially dealing with huge sets of couple of values. I used a custom heuristic to compute a score from each couple of values and turn the set into an array of values. I sorted it and assigned ...
Axel Carré's user avatar
6 votes
2 answers
573 views

How can I reconstruct a normal distribution from a set of percentiles?

I have the 3rd, 10th, 50th, 90th, and 97th percentile values of a normally distributed variable, and I wish to generate a dataset that will allow me to query for other percentile values (say, the 67th....
Omroth's user avatar
  • 195
0 votes
0 answers
18 views

Why is my accuracy fluctuating for a while and then stuck? [duplicate]

I am building a cnn classifying model to predict images over 3 classes. The data is balanced, with 10.5k images for train ( 3.5k for each ), 3k validation images ( ...
Dragos123's user avatar
0 votes
0 answers
10 views

2SLS: does it just rescale the instrument variable?

I am investigating using instrument variables to correct for some known sources of measurement error in my data (measures that reflect mistakes/inconsistencies that we think directly affect our main ...
Maks Hall's user avatar
2 votes
2 answers
116 views

Why is there such is difference in sample size estimation between Lehr's method and Statsmodels [closed]

I have been using Lehr's rule to estimate the required sample sizes for some experiments I have to run. Lehr's rule states that we need: $$ n = \frac{ 2\left(z(1-\alpha) + z(1-\beta)\right)^2 \sigma^2}...
Tom Kealy's user avatar
  • 149
2 votes
1 answer
52 views

How to Predict a Growing Time Series with Changing Slopes in Python?

I have a univariate time series dataset that represents a continuously growing trend, but the slope changes at different intervals. I want to predict future values of this time series. Here are some ...
Amine Boutaleb's user avatar
4 votes
1 answer
86 views

Help fitting doube-exponential curve to raw mass spec data, time

I am writing mass spec data reduction software in Python for a helium measurement system and could use a hand getting a double-exponential function to fit my data. Basically, the gas in the mass spec ...
ohshitgorillas's user avatar
0 votes
0 answers
14 views

How to use Conv2D for make predictions on spatio-temporal data (non-image)?

I have multivariate time series data consists of 4 independent variables, 1 dependent variable (target variable), and spatial data (latitude and longitude). The data is taken from 5 different cities, ...
Riri Ana's user avatar
0 votes
1 answer
41 views

statsmodels: Update OLS' degrees of freedom when absorbing 3+ fixed effects

I want to run an OLS regression with 3+ fixed effects. (Whether this is a good idea is out of the scope of this question). I can do it using Stata: ...
ebosi's user avatar
  • 138
0 votes
0 answers
25 views

How to assess the goodness-of-fit of an extended Cox model?

I'm trying to build an extended Cox hazard model on a dataset with time-varying covariates, using Python's Lifeline package. However, I've a hard time trying to find out how I can assess the goodness-...
wanderingcatto's user avatar
0 votes
0 answers
14 views

GridSearchCV performs worse than baseline

I'm working on a binary classification problem using scikit-learn. One of the models I've tested is KNeighborsClassifier, for ...
AndreaTerenz's user avatar
2 votes
0 answers
19 views

binary timeserie to binary timeserie regression [closed]

I'm looking for a way to determine which of my predictive variables (binary time series) best explains my response variable (also a binary time series). Basically, I ask people to listen to audio ...
laure's user avatar
  • 21
2 votes
1 answer
104 views

Numpy implementation of the Quantile function [duplicate]

I am trying to wrap my head around the implementation of the quantile function for finite samples and, specifically, in numpy (main reason to do this: I am working ...
non87's user avatar
  • 33
1 vote
0 answers
16 views

Method to compare two latency measurement groups

I have two measurement (or rather two or more sets of measurements) of server software (DNS). The measurement results does not have normal distribution, they are very skewed towards 0-1 milisecond ...
oerdnj's user avatar
  • 111
6 votes
1 answer
154 views

Estimate the probability that a crack length exceeds a threshold value after N cycles

The length $a$ of a crack after $N$ fatigue cycles $N$ is $$a(N;C,m) = \left(a_0^{\left(1-\frac m 2\right)}+ C\left(1-\frac m 2\right)B^mN\right)^{\frac{2}{2-m}}$$ where $a_0$ is the initial crack ...
DeltaIV's user avatar
  • 18.2k
4 votes
2 answers
86 views

Finding the corners of noisy polygons

I have some polygons that look for example like this: If I zoom in very close on one side, you can see the noise. The data is a list of x coordinates and a corresponding list of y coordinates. I ...
sav's user avatar
  • 239
0 votes
0 answers
8 views

How to cross-correlate two data sequences with considerable differences in size?

I need to cross-correlate two sequences of frames that represent a video signal. However, the videos were compressed in different ways. The first data (dados) array has 17167 frames and the second (...
user avatar
1 vote
1 answer
37 views

Reason for high MSE and negative R square value

I am getting really high MSE and negative R square value. Dataset: https://docs.google.com/spreadsheets/d/1moTZS_LgOn6d74NC44i9lVcWchj-abVx/edit?usp=sharing&ouid=100514649347129021200&rtpof=...
user avatar
0 votes
0 answers
26 views

Is it possible to train a neural network to feed into a Random Forest Classifier or any other type of classifier like XGBoost or Decision Tree?

I want to create a model architecture to predict future stock price movement as such: The Goal of this model is to predict if the price will go UP or DOWN within the next 3 months. I have tried a few ...
Evank's user avatar
  • 1
0 votes
0 answers
54 views

Is my approach to compute Granger causality valid?

I have two time-series, let us call them A (colored in red) and B (colored in blue). There are ~770 data points per time-series. Note: Both time-series are in fact not the recorded raw signals, but ...
Philipp's user avatar
  • 297
0 votes
0 answers
20 views

statsmodels.stats.multitest.multipletests: 1.0 pvalues [duplicate]

I am running a series of statistical tests between many lists of values all sampled from the same population. After computing pvalues, I thought to apply an adjustment with statsmodels.stats.multitest....
Indefeasible's user avatar
0 votes
0 answers
22 views

How to obtain the similarity ratio between graphs hand-drawn on paper?

To explain with an example: In one class, I showed students a plot that I wanted them to draw. They saw it as too short and tried to draw the same. Some were very similar, some were less similar, and ...
doqukan's user avatar
0 votes
0 answers
6 views

Sequential Approach lower error than Simultaneous approach

I'm looking for a situation/code example and dataset (preferably 2 datasets), where the greedy/sequential approach has a lower cross-validation error than a simultaneous approach (where comparison is ...
matthew George's user avatar
1 vote
0 answers
39 views

Calculate the probability of a peak in cyclical data using historical observations? [closed]

I find that using a histogram does not work for a sine wave because it has a lot of data points close to its peak. So I included to measure the distance traveled of values higher than our current ...
litmus's user avatar
  • 83
0 votes
0 answers
34 views

Different P-Value and AIC before/after standardization [Python - Statsmodels]

I am investigating the correlation between environmental variables (15 continuous variables grouped as 'DHIs' in the code below) and fox occurrence (binary), using logistic regression / Python ...
Andrew Norfield's user avatar
0 votes
0 answers
26 views

Linear regression results do not match expected ones [duplicate]

I am fitting a simple linear model. First I sample $X\sim\mathcal{N}(0,1)$ and $Y\sim\mathcal{N}(12,3)$. Then I impose the following linear model for $Z = 2+2X-\frac{1}{4}Y+\mathcal{N}(0,\frac{1}{2})$....
xcesc's user avatar
  • 90
0 votes
0 answers
37 views

Multinomial one-sided confidence intervals

Building on this question: Confidence interval and sample size multinomial probabilities In a binomial confidence interval, a 90% two-sided CI corresponds to a 95% one-sided CI. The question above ...
user11629's user avatar
  • 101
2 votes
0 answers
22 views

How to incorporate sources of observational error in state space model?

I’m learning about state space models. I understand the concept of a latent process that is unobserved, and a noisy set of observed data that we can use to estimate the latent process. I am trying to ...
Jared's user avatar
  • 21
1 vote
0 answers
17 views

Can I use apriori / association rules on aggregate data?

I'm trying to find patterns for cross selling opportunities for a professional services company. I'm using apriori to do this. My company has different project types. We might work for a client on one ...
jabs's user avatar
  • 209
2 votes
0 answers
18 views

Implementing a SEM with lasso regression [closed]

I have the following structural equation model (SEM) structure in mind: In reality there are more than 3,2, resp. 3 variables, so I want to use lasso regression for the arrows. Is it possible to ...
Hendrik's user avatar
  • 21
2 votes
1 answer
63 views

Individual fixed effects for Cox Proportional Hazard model

I have a panel data set with right censoring. So I have individuals i with several time varying covariates Xit. For some individuals we observe event E (canceling their contract) and others we don't. ...
TiTo's user avatar
  • 249
0 votes
0 answers
26 views

Predicting Milestones

I have data detailing specific projects going on and when they achieved certain milestones. I'm trying to predict the end date of the project based on these milestone dates. An example of the dates ...
user22484888's user avatar
1 vote
1 answer
29 views

Where am I making a mistake with this hypothesis test in Python? [closed]

I have the following measurements for body temperature: 99.6 97.8 98.7 99.2 99.1 96.9 99.9 98.7 97.6 98.7 99.2 97.4 99 98 98.8 98.1 99 98.2 97.9 99.5 97.7 97.8 98.3 98 98.6 96.9 98.6 98.4 98.3 98 97....
jerH's user avatar
  • 235
1 vote
0 answers
39 views

Can the One-Sided Mann-Whitney U Test Have a Composite Null Hypothesis?

Let's say there's an i.i.d. sample $X_1, X_2, \ldots, X_{n_1}$ from a distribution with the CDF $F$ and $Y_1, Y_2, \ldots, Y_n$ from a distribution whose CDF is $G$. As far as I know, the null ...
Milos's user avatar
  • 1,099
0 votes
0 answers
21 views

Reconstruct Kernel from sampling a Gaussian Process

I am generating 50K draws from Gaussian Process with GPy in python. The Gaussian Process has a RBF kernel with length scale = 10....
apt45's user avatar
  • 273
0 votes
0 answers
17 views

Predicting Millstones [duplicate]

I have data detailing specific projects going on and when they achieved certain milestones. I'm trying to predict the end date of the project based on these milestone dates. An example of the dates ...
user22484888's user avatar
1 vote
1 answer
64 views

Two-sample Kolmogorov-Smirnoff test in R and/or Python, how to find degrees of freedom?

I am trying to run a Kolmogorov-Smirnov/K-S test in R using the ks.test() function, and I am trying to find the degrees of freedom when comparing across two different groups. I am also trying in ...
Rachel's user avatar
  • 13
0 votes
0 answers
6 views

Expectation of Changes in Top K Elements Amongst Randomly Generated Numbers

I am conducting a Monte Carlo simulation where: I generate n random numbers uniformly. Select the top k of these numbers. Then regenerate c of the n numbers at random. I aim to compute the mean ...
Daniel's user avatar
  • 1

1
2 3 4 5
97