Questions tagged [python]
Python is a programming language commonly used for machine learning. Use this tag for any *on-topic* question that (a) involves `Python` either as a critical part of the question or expected answer, & (b) is not *just* about how to use `Python`.
4,735
questions
0
votes
0
answers
25
views
Detecting and Forecasting Intermittent Time Series
I am building a model to forecast some metrics. Those metrics are quite seasonal giving me good forecasts as shown below:
However, some new requirements dictate that I target those forecasts per ...
- 21
0
votes
0
answers
8
views
Prediction when Target's lag values are part of Predictors
I'm using LGBM for regression, where the Target column's lagged values (7 columns for each lag day) are also used as predictors when training the model. Absence of the 7Day lag values severely ...
- 25
0
votes
0
answers
9
views
confidence interval for GEV distribution [closed]
I'm trying to port some code from MATLAB to Python. I'm stuck with the confidence intervals for the distribution estimated parameters.
This is the MATLAB code:
...
- 101
0
votes
0
answers
31
views
Preventing Data Leakage in Time Series Forecasting with Feature Engineering
In a previous question (linked here), I sought guidance on forecasting thousands of time series. Based on the suggestion to treat it as a regression problem, I used the LightGBM model with extensive ...
- 13
1
vote
1
answer
23
views
Bad performance of ARIMA model on online buzz data, Any suggestions?
I was wondering if the ARIMA model is constrained to predict online buzz data (time series data).
What I want to do: Use the past round 30 months data to predict next month; and I use Python
Here are ...
- 13
0
votes
0
answers
12
views
Why isnt StackingRegressor in sklearn reproducing the same output? [closed]
I am just getting familiar with StackingRegressor for a prediction setup I am building and unfortunately I can't seem to reconcile its predictions. Here is a simple example:
...
0
votes
1
answer
28
views
Which hyperparameters should I choose to tune for different ML models? [closed]
I'm applying hyperparameter tuning with techniques such as, randomized search and grid search in python. I could find important hyperparameters and tuned them for Random Forest Regressor algorithm, ...
- 101
-1
votes
0
answers
16
views
Logistic regression with industry×year fixed effects on Panel data in Python
I would like to use Python to run a logistic regression with industry×year fixed effects on Panel data (firmID, year). data contain the following variables :
firmID: firm ID (1000)
year : Year (2010-...
1
vote
0
answers
62
views
Is there any algorithmic-pseudocode/Python-implementation for the DAG-related API(s) in the R package dagitty? [closed]
The statistical software DAGgity offers a graphical web-based UI as well as an implementation in R that allows for finding conditional independences corresponding to d-separation, minimal adjustment ...
2
votes
1
answer
32
views
Statistical Method for Accurately Detecting Seasonality in Monthly Sales Data
I have a dataset containing monthly sales data for different product categories spanning five years (60 months of data). I am using a Python process to calculate the seasonality for each category, ...
0
votes
0
answers
18
views
Monotone constraints in decision tree regressor or random forest regression
after I've spent several weeks trying to fit a regression model to my flood damage data (x1=water height, x2=adaptation height, x3=(x1-x2), y=damage), it is now time for my very first question on ...
- 11
2
votes
1
answer
35
views
Forecasts fail for new period
I am working with prophet library in python where I do some forecasts. While I split to train and test to check, it shows very good performance, but when I forecast for the actual future period it ...
- 21
0
votes
0
answers
16
views
GLMM data analysis in R
I am analysing some data using the glmer analysis and I would like assistance as how I could solve the following error massages. I tried fixing the problem using some examples from the internet but I ...
- 1
0
votes
0
answers
30
views
VECM predict gives forecasting results that lag behind actual data
I am using Python's statsmodels.tsa.vector_ar.vecm.VECM to estimate VECM models and generate pseudo out-of-sample forecasts with the .predict() function to compare with actual data.
For example, I ...
0
votes
0
answers
18
views
Why am I getting negative components with my custom NIPALS algorithm
I've recently been learning about the Nonlinear Iterative Partial Least Squares (NIPALS) algorithm for computing the principal components of a dataset. I am trying to code a NIPALS class from scratch ...
- 41
1
vote
1
answer
20
views
Is there a way to automatically split large clusters that are greater than some maximum number of points?
I ran HDBSCAN on these coordinates and got some clusters but some are too large. HDBSCAN has a minimum cluster size parameter, but no maximum size. All I want is to intuitively divide larger clusters ...
- 11
0
votes
0
answers
11
views
Best way(s) to combine data on the same object
I have data of a solar cell's performance. Each cell consists of 8 pixels, each of which have their own data for various metrics of performance (current, etc.) throughout time. I would like to combine ...
1
vote
0
answers
34
views
Training Regression Models to Predict Continuous Probability Values in [0, 1] [closed]
I'm working on a machine learning project where my target variable represents continuous probability values that must fall within $[0, 1]$. While I understand regression models are suitable for this ...
0
votes
0
answers
10
views
Empirical Mode Decomposition(EMD) + CNN for time series forecasting
I'm currently working on a time series project, and I intend to employ the EMD+CNN technique for forecasting the output. Upon applying EMD to the training data, I obtained a total of 14 Intrinsic Mode ...
- 1
1
vote
0
answers
23
views
Can I fit a Poisson distribution to a continuous variable to apply event detection algorithm?
Preprocessing:
I have a time series of number of tweets per 10 minutes time interval that are all taken from a given discussion on a specific topic in a specific region. I preprocessed the data by ...
- 41
3
votes
1
answer
53
views
Overlapping circular bearing distributions on a plane
I have some directional hydrophones capable of recognizing transient signals/sound and estimating the circular probability density function of the bearing, or direction, that the sound came from. I ...
- 33
0
votes
0
answers
34
views
Hypothesis Testing (normality violated)
I have two sets of data, SetA and SetB - they are metrics of the same group from different time or paired. SetA score at 10 am and SetB score at 5 pm, same person (30 of them).
I want to check if the ...
- 1
0
votes
0
answers
12
views
Do all nodes have to be stored / indexed in a GNN?
Apologies in advance for what maybe a really silly question :(
I am trying to construct a Graph Neural Network, in which I would like to learn representations for certain nodes, but not others. To be ...
- 213
1
vote
2
answers
119
views
Python - need help with Linear Mixed Effects Model results interpretation
I analyse a set of physicochemical data from the river and two rows of wells - one located closer and the other further from the river (in the N-S direction).
The study's main aim is to investigate ...
- 103
0
votes
0
answers
9
views
How can I best correct / handle a slight right skew distribution in my residuals plot using stats.models mixed effects model?
I am using statsmodels mixedlm as follows:
...
- 1
1
vote
0
answers
24
views
Discrete P-values in Kolmogorov-Smirnov Test with Small Sample Sizes
I'm facing an issue while conducting the Kolmogorov-Smirnov (KS) test with two datasets of different sizes: 84 and 28 samples. I got these sets from normal distribution. I've noticed that the ...
0
votes
1
answer
40
views
KS test for Poisson distribution data
I have a dataset comprising 1000 integers. It represented as follows:
...
- 101
0
votes
0
answers
12
views
Significance Test Time Series Data Python
As a Business Analyst I may not be as experienced in performing complex statistical tests and this is the reason I am asking for your advice.
I have this sales data whose dates I have clustered into ...
0
votes
0
answers
19
views
Low CV-RMSE and negative $R^2$ (comparative)
I am trying to predict a numeric variable using XGBoost with optuna for hyperparameter optimization. I defined two objective functions for optuna, one optimized for very small datasets (5 to 17 ...
- 1
12
votes
1
answer
489
views
For Gamma distribution, use MLE or MoM?
For Gamma distribution, is it better to use MLE(maximum likelihood estimation) than MoM(method of moments) to estimate the shape and scale parameters?
Also, in python SciPy, does gamma.fit use MLE? I'...
- 221
0
votes
0
answers
24
views
What is the best way to compare means across groups if the data is not normally distributed, and sample sizes across groups can vary?
I am quite new to statistical analysis and have what is probably a newbie question. I have a dataset that contains year and state data for two different groups. I am trying to compare the group means ...
- 1
6
votes
2
answers
136
views
Which OLS Regression Results are worth putting into an article? Does my example look right?
Yesterday, I set up a topic outlining the problem I am currently working on.
After receiving many interesting responses, I added linear regression to my results, following this suggestion.
My research ...
- 103
0
votes
0
answers
19
views
Generating Predictive Models for Multiple Time Series
I have a dataset of historical film streaming data for the last few years. Each record is the Title, Genre, Rotten Tomatoes Score, Week of Streaming Data and then Raw Streaming. I am trying to find ...
0
votes
1
answer
44
views
Trouble Achieving Optimal Parameters for XGBoost Regressor with Sin(x) Time Series
I'm facing a challenge while attempting to optimize parameters for an XGBoost regressor (Python) using time series data. I've created a time series using the following code:
...
2
votes
1
answer
230
views
Confusion on Chi-Squared test results
Basically, I have the data where I am trying to assess whether there is any correlation between the gender of the manager and the gender of people within their team. I decided to do the chi-squared ...
- 21
1
vote
0
answers
18
views
Discrepancy in 2SLS Estimation Results
I've been exploring the 2SLS (Two-Stage Least Squares) estimation method to analyze a model involving endogeneity and instrumental variables. To better understand the process, I performed manual ...
- 11
0
votes
0
answers
11
views
Spliced Distributions Framework for python
There is an article Fat-Tailed Regression Modeling with Spliced
Distributions
that describes fat-tailed regression modeling by fitting the distribution consisting of N components (different ...
0
votes
1
answer
22
views
Clustering Algorithms for Complex Survey Studies
Which clustering algorithms can be used in complex survey designs? Can the kmeans algorithm be used?
I searched for literature using complex survey design data like NHANES, but they do not provide ...
- 43
1
vote
0
answers
15
views
When is it appropriate to use t-SNE and how can I trust its output to faithfully represent higher dimensional separation?
Given the discussion on this answer what appropriate assumptions and contingencies would allow tSNE to have an informative interpretation?
It's hard to come away from that answer feeling like I can ...
- 41
4
votes
2
answers
106
views
Scikitlearn: Why are hyperplane coefficients not available if kernel is not linear
I am interested in learning the math behind support vector machines. So far, I understand that SVMs attempt to find hyperplanes that maximize the margin distance between support vectors associated ...
- 41
0
votes
1
answer
22
views
Representing Nested Models as an SKLearn Pipeline [closed]
Goal: Represent nested models with SKLearn's Pipeline / ColumnTransformer / FeatureUnion setup.
Specific issue: I cannot figure out how to use the prediction from one model as a factor of a secondary ...
- 11
0
votes
0
answers
18
views
PCA with gram matrix produces different results from PCA done using covariance matrix?
I was trying PCA on a dataset (#samples=24, #dims=42) via eigendecomposition using numpy. I read that for matrices where the number of features exceeds the number of samples, we should use the gram ...
- 121
0
votes
0
answers
6
views
Trying to understand CGAN loss curves
I am trying to understand the loss curves of a CGAN model better. From what I've seen so far, it seems like there is no consistent answer as to what the loss curves should look like. Some tutorials ...
0
votes
0
answers
109
views
Ordinal regression using statsmodels OrderedModel - basic interpretation
I want to run an ordinal regression in Python. My dependent variable describes a medical condition in an ordered manner (e.g. 0 = healthy, 1 = affected, 2 = very affected, 3= severely affected). I was ...
- 13
0
votes
0
answers
9
views
How to parameterize SARIMAX in Python [duplicate]
I'm using SARIMAX as a statistical model to solve my problem of predicting a cost variable (Y) based on the past history of this dependent variable.
In particular, I use SARIMAX because I have ...
2
votes
2
answers
63
views
How do I approach this problem about the order of the events?
So let's imagine I have a dataset of children. For each of them a have a bunch of characteristics (generation, gender, race, class, urban/rural, religion, bmi, number of siblings etc..) and plus the ...
1
vote
1
answer
163
views
Forecasting a Time Series Model for 1000s of Time Series
I'm currently immersed in a challenging forecasting project centred around predicting the required work hours to complete various tasks within a team setting. My dataset comprises crucial attributes, ...
- 13
2
votes
0
answers
38
views
Confusion about the code for choosing "stumps" in Adaboost algorithm
(I actually asked the following question on Stack Overflow recently:
https://stackoverflow.com/questions/76842431/confusion-about-the-code-for-choosing-stumps-in-adaboost-algorithm
but then I ...
- 121
0
votes
1
answer
31
views
Do we only use cross-validation for hyperparameter tuning?
I am still unsure why we must use cross-validation here to validate the model, or it may be unnecessary.
Is it correct to use like indicated below? or do we have to combine it with hyperparameter ...
- 13
0
votes
0
answers
9
views
DNN: Does mini-batching of grouped data during training introduce information leakage?
I am trying to replicate a deep neural network from this notebook, which works with the French MTPL dataset. The NB is in R and mine is in Python.
The dataset contains policy entries with the same ...
- 1