Questions tagged [python]
Python is a programming language commonly used for machine learning. Use this tag for any *on-topic* question that (a) involves `Python` either as a critical part of the question or expected answer, & (b) is not *just* about how to use `Python`.
4,840
questions
0
votes
0
answers
15
views
PanelOLS - R Squared
I'm using the PanelOLS package in Python to estimate a fixed-effects model and have noticed that it provides four different R-squared values: rsquared_within, ...
- 351
0
votes
0
answers
10
views
Seasonal decompose with statsmodels.tsa.seasonal
Could you help me understand the result of a seasonal decompose with
statsmodels.tsa.seasonal import MSTL?
I'm playing around with this for the first time after ...
- 101
0
votes
0
answers
11
views
Are x-means' BIC and likelihood scores comparable between clusters of different dimensionality?
I'm currently working on implementing an x-means variant and ran into an issue when computing the Bayesian Information Criterion's log-likelihood (via scipy's multivariate normal logpdf) which fails ...
1
vote
1
answer
13
views
Title: Issues with Fitting a Quasi-Periodic Function with a Trend Term in Python
I'm trying to fit a quasi-periodic function with a trend term to my time series data using Python. The data consists of monthly observations from 1963 to 1976. My goal is to model the data with a ...
- 369
1
vote
0
answers
23
views
Model still overfits after hyperparameter tuning, dataset balancing and convolution layering
I am trying to classify either an image of 25x25 px stacked together as 50x25 px is the same(1) or different(0). I am using keras to create the NN layers. There are 10,000 instances of both 1s and 0s ...
- 111
0
votes
0
answers
14
views
Clustering Mixed Data Types: Algorithm Selection, Distance Measurement, and Feature Weighting
I have a database of 74,000 records with 29 features. Fourteen of these features are categorical and are either 0 or 1, while the other 15 features are continuous and have been normalized and scaled ...
4
votes
1
answer
160
views
Can probabilistic predictions be obtained from gradient boosting models CatBoost and XGBoost?
I'm looking for probabilistic predictions ($\Pr(Y\mid X=x)$) using CatBoost or XGBoost for a continuous target variable that is in [0, 1] (i.e., ratio). Can I use the official library to generate ...
- 145
1
vote
1
answer
38
views
Time-series forecasting problem in Python
I am working on a Python project where I have to predict the energy consumption in individual households. My dataset consists of several thousands of households each having a monthly value of the ...
- 11
0
votes
0
answers
9
views
Feature based anomaly detection
I am working on a concept that is new to me in the field of anomaly detection in multivariate time series. A motor is going through different phases (3 phases in total) and in each phase a certain ...
0
votes
0
answers
5
views
Partial dependence plot from scratch - XgBoost depth 1
There are several packages that introduces enhanced explainability to a ML model like XGBoost, for example PiML. And there is a lot of other resources online about different aspects of ...
- 243
1
vote
0
answers
15
views
Cohort saliency analysis in Python
I trained a CNN model to predict clinically relevant parameters by using images as inputs. Now I want to do a cohort GradCAM analysis on the images that were used in the test set to see where the ...
1
vote
0
answers
12
views
How to train a neural network to identify multiple features in one image, where the order of predicted features doesn't matter
I recently created a toy dataset for myself, which I call "multi-color MNIST", where multiple digits with different colors appear on a single RGB image. See the image I attached. I am using ...
0
votes
1
answer
26
views
How to adjust a variable by age, sex and BMI
I'm working on a database with almost 300 patients, of which I have their age, sex, BMI and their level of diabetes (which, for the purpose of the study, is stratified into mild, intermediate or ...
8
votes
1
answer
462
views
python stats.spearmanr and R cor.test(method='spearman') don't return the same p-value?
I am trying to calculate spearman's rho value for some data in both Python and R but I get different P-value results, what am I doing wrong with python?
...
- 183
0
votes
0
answers
6
views
Analysing experiment data with multiple changing parameters [closed]
I ran some LLM workload experiments and want to evaluate the effect of Tensor Parallelism and npu_memory_util and parallel number on throughput and latency by plotting various plots. I managed to plot ...
- 123
0
votes
0
answers
32
views
Why does my Bayesian linear regression with survival analysis perform worse than standard linear regression?
Experts in statistics wanted here.
I am working on a Bayesian linear regression function in Python that considers asymmetric uncertainties and censored data (upper/lower limits) in both x and y. The ...
- 133
3
votes
1
answer
47
views
Solving logistic regression using CVXPY
I am trying to code a logistic regression model using the CVXPY library. The code I have written so far "works" in the sense that it can be executed, it does not yield any error message and ...
- 1,348
0
votes
0
answers
10
views
Cross-correlation function with null values
I am working on an analysis to look at features correlated with an uptrend in another feature. In order to remove noise, I've removed all dates besides where my dependent variable y is trending ...
- 399
0
votes
0
answers
5
views
find k nearest neighbors of a given data point x in KNN_classifier [migrated]
in python, the knn algorithm can be applied on training data with the code like below:
...
- 133
0
votes
0
answers
16
views
Time series seasonality analysis - scoring method
I am working on the seasonal search in a time series and would like to automate the process with notations from the series, without having to manually inspect the curves. The three scores I propose to ...
0
votes
0
answers
20
views
R VGAM (vglm) Some elements in the working weights variable 'wz' are not finite
I'm trying to fit a beta-binomial function with vglm from the VGAM library, calling it from rpy2:
...
1
vote
1
answer
63
views
Using Friedman Test for Time-of-Day Analysis on Vaccine Fridge Data?
I worked on a project monitoring vaccine fridge temperatures, collecting data every 30 seconds over two years. I want to compare high temperature anomalies (above 8°C) across different times of the ...
- 23
1
vote
1
answer
44
views
How do I estimate the standard deviation of the measured variables in linear regression?
Assuming a linear model:
$$
y = \alpha x + \beta
$$
I make n observations of $(x_i,y_i)$.
Each observation is subject to a measure uncertainty that I assume to be normally distributed with mean = 0 ...
- 601
0
votes
1
answer
31
views
How does SKLearn derive LASSO coefficients?
I am trying to derive SKLearn's LASSO coefficients using SciPy optimize, just to get an idea of how SKLearn is working under the hood. However, I cannot get the parameters to match.
...
- 3
1
vote
0
answers
18
views
DNN quantile regression
I have a PyTorch model, the purpose of which is to predict quantiles over the output given an input. The output in this case is service time (minutes) for machine maintenance. The inputs detail ...
- 3,386
0
votes
0
answers
23
views
Detect paterns over time in multivariate dataset
I have a dataset representing the stock of a shop over several days. For each day, I have hourly inventories of the objects in the shop. Some products are sold, and others might temporarily disappear (...
0
votes
0
answers
10
views
Best Approach to Smooth ETA Transitions in Delivery Prediction Models
I'm working on displaying the ETA (Estimated Time of Arrival) for an item delivery using a predictive model that evolves over time. The issue I'm facing involves smoothing the transition between ...
- 23
0
votes
0
answers
17
views
Dynamic Factors with Statsmodels in Python
I am fitting a linear gaussian state space model in python using statsmodels.DynamicFactorMQ. It is giving me back only the model summary, but I want to extract the estimated AR(1) transition matrix ...
- 31
1
vote
0
answers
29
views
p values for a ARMA - GARCH model with exogenous variable
I am trying estimate the parameters for the following ARMA(1, 2) - GARCH(1, 1) model, with an exogenous variable as well. The model specification is as follows:
$ x_t = \mu + \beta_Y \cdot y_{t-1} + ...
- 21
0
votes
0
answers
19
views
What model or method to use for categorical data with repeated measures?
I am trying to analyze my data, it consists of 2 independent categorical variables (4 textures and 4 impact sounds - mixing and matching them gives me 16 different conditions that each person is shown)...
1
vote
0
answers
38
views
Bootstrap power analysis
Is power analysis through bootstrap sampling possible? Note, the desired output is MDE, not sample size. In context, assume sample size is fixed. We simply want to evaluate the smallest effect ...
- 3,386
1
vote
0
answers
58
views
Prediction of a time series AR(1) vs AR(1) with exogenous variables vs Random Forest, why is the performance so different?
Extension of the previous question that only compare AR(1) vs. AR(1) with exogenous variables:
I am currently working on forecasting a time series y using three models: an AR(1) model and an AR(1) ...
- 361
0
votes
0
answers
11
views
Q-learning model instable for a specific application
Assume we have this implementation of transport security Q-learning model
...
- 113
0
votes
0
answers
17
views
Model performing poorly after cross validation
After using cross-validation to see how a custom predictive function performs on unseen data, I applied to function to the original dataset, and the performance (based on coefficient of determination) ...
- 1
1
vote
1
answer
24
views
Designing complex mixed model analysis
I'm trying to analyze EEG data from patients using mixed models.. and I've been stuck on this for a while, I've read a bunch of papers and a primer textbook on it but still not sure what I'm doing.. I'...
- 11
1
vote
0
answers
27
views
Generating data for Gumbel's bivariate exponential distribution in python
For a bivariate random vector $(X,Y)$ with survival function $\bar{F}(x,y)=\mathrm{e}^{-ax-by-\theta xy}$, $\theta\in(0,1)$. I want to generate random data in python. Also I want to see mathematical ...
- 173
1
vote
0
answers
19
views
What can cause a negative R-Squared (between) with Fixed Effects estimation [closed]
I'm analyzing the S&P500 companies for my master thesis about employee happiness and entrepreneurial orientation. I have a panel dataset (2016-2020) with 208 complete entries. My problem is that ...
0
votes
0
answers
8
views
Positional Invariance missing after max pooling operation in custom CNN
I am an early career researcher in computational neuroscience, and I am currently trying to model a robust object recognition model.
My model takes a dataset (binary images; where each object is ...
- 1
3
votes
1
answer
181
views
Finding the right model for non-linear least squares fitting of data
I have a set of data to which I would like to fit a function. As a starting point, I tried fitting the data to a standard normal distribution with $2$ parameters; a Gaussian function with amplitude $A$...
- 177
1
vote
0
answers
58
views
One step ahead forecasts: Why is LSTM so much worse than XGBoost? [closed]
I am working on generating recursive one-step-ahead predictions for a time series y using a minimal set of regressors. I have found that linear models all perform similarly and fail to outperform ...
- 361
1
vote
1
answer
40
views
NER With Custom Tags, How to Approach
I am building a "field tagger" for documents. Basically, a document, in my case something like a proposal or sales quote, would have a bunch of entities scattered throughout it, and we want ...
0
votes
0
answers
14
views
Per-Item inter rater reliability with multiple values and raters
I am trying to find the best statistical calculation to measure the agreement between 72 different raters on one item. My goal is to convey in a statistic how spread the raters are on their rating and ...
- 101
0
votes
1
answer
27
views
Should I Use Regularization in Univariate Logistic Regression for Diagnostic Methods Comparison?
I am comparing two diagnostic methods, Method 1 and Method 2, where Method 2 is considered the gold standard. I am using Method 1 to predict the Method 2 using logistic regression. My dataset contains ...
1
vote
1
answer
46
views
Why the contribution of a categorical value in SHAP trained on Catboost differs from observation to observation
Context
Let's imagine I am interested in predicting sepal length in the iris dataset using catboost.
Objective
My main objective is understanding the effect of each categorical value for ...
- 138
0
votes
0
answers
9
views
How to Improve performance of deep learning timeseries forecasting model like LSTM? [duplicate]
I have historical data of 5 years (June 2019- June 2024). Data is in daily & csv file format. I have 4 features: Data, AQI, Raw Concentration, NowCast Concentration. I am trying to forecast only ...
0
votes
0
answers
23
views
How Random Forest handle missing value in sk-learn? [duplicate]
What is the technic used in Random Forest Regressor from scikit-learn to handle missing value ?
First I thought that a Random Forest regressor was able to natively handle missing value during training ...
- 111
2
votes
0
answers
117
views
Anomaly detection for Multivariate Time-Series data from multiple sensors
I work with tabular time-series data from multiple sensors and my goal is to detect abnormal behavior in battery discharge.
Here is an example of data (example contains records only for one device ...
- 71
0
votes
0
answers
29
views
How to measure the error on extrapolation from a double log fit? [duplicate]
I am writing residual gas analysis mass spectrometry software in Python. One of the functions of this software is to take the raw mass spec intensity data, $y$, and timestamps $t$, and fit them to the ...
3
votes
1
answer
46
views
Can I use simulated data only for testing a Random Forest regression already trained on real data?
I am working, using Python, on a Random Forest Regression for the prediction of a target variable. I have trained it and tested it on real data, obtaining satisfying results. Now, I would like to ...
-1
votes
1
answer
41
views
Calculating error on a double natural log fit
I am writing residual gas analysis mass spectrometry data reduction software in Python. The evolution of gas intensity $y$ over time $t$ in the mass spec is roughly a double natural logarithmic ...