Questions tagged [python]
Python is a programming language commonly used for machine learning. Use this tag for any *on-topic* question that (a) involves `Python` either as a critical part of the question or expected answer, & (b) is not *just* about how to use `Python`.
4,794
questions
0
votes
0
answers
34
views
How can I reduce fluctuations in my validation accuracy?
I'm training a CNN with pictures data for binary classification and while my training accuracy increases, my validation accuracy keeps fluctuating between small and high values of accuracy. I have a ...
0
votes
0
answers
13
views
DRF all features have positive contribution to SHAP values except one
I have trained a DRF model using h2o and the shap summary plot came out like this.
Why aren't feature contributions centered? It seems weird that one feature contributes negatively and all the others ...
1
vote
0
answers
102
views
Python statsmodels GLM - log likelihood of null model
I have an issue when calculating log-likelihood for null model to double-check GLMResults.llnull parameter:
https://www.statsmodels.org/devel/generated/statsmodels.genmod.generalized_linear_model....
0
votes
0
answers
27
views
Optimising a multivariate table of counts based on marginals
I've been stuck on a problem for a very long time now so I decided to post on this forum for the first time. Although I am using code to perform this task, I believe it uses some statistics and I ...
0
votes
0
answers
54
views
Forecasting in ARIMA (python)
I have a time-series data (Date and Total). This is actual data from the past 2 years.
I understand how to pick the (p, q, d) order for ARIMA. And I can divide my data into train and test, and the ...
1
vote
1
answer
40
views
Using a binomial distribution to mathematically quantify a thought experiment from my friend (python)
My friend asked me the following question:
...
0
votes
0
answers
19
views
Keras Implementation of a neural network found on a paper
I am trying to implement a neural network I found on an open access paper as I have a similar problem and I am struggling with it. I am using tensorflow.keras for the implementation.
Here is the paper:...
0
votes
1
answer
49
views
Creating a CNN model for multi-output prediction where one target variable is categorical, and others are numeric
I want to create a simple CNN model for multi-output prediction. The predicted values are four numeric values (all between 0-1) and one categorical value (4 classes). When I try to create a model ...
0
votes
0
answers
31
views
How to Change Architecture of DCGans?
https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html
I was refering this notebook but default size is 64*64
I want to change architecture to 256 or 512
Can anyone help me with training
...
0
votes
0
answers
72
views
Kendall's Tau Correlation for dataset with multiple types (binary, categorical, ordinal, continuous)
I have a dataset of many variables of varying data types (binary, categorical, ordinal, continuous), and I want to find the correlations between them. I encoded the binary as (0,1) and one-hot encoded ...
0
votes
0
answers
33
views
BorutaPy selects different features in different iterations
I came up to something strange. I know Boruta should select everything that is important.
I have a dataframe of 200 observations and 2000 features. if I shuffle the order of the features in the ...
1
vote
1
answer
289
views
Detecting and Forecasting Intermittent Time Series
I am building a model to forecast some metrics. Those metrics are quite seasonal giving me good forecasts as shown below:
However, some new requirements dictate that I target those forecasts per ...
0
votes
0
answers
65
views
Prediction when Target's lag values are part of Predictors
I'm using LGBM for regression, where the Target column's lagged values (7 columns for each lag day) are also used as predictors when training the model. Absence of the 7Day lag values severely ...
0
votes
1
answer
296
views
Preventing Data Leakage in Time Series Forecasting with Feature Engineering
In a previous question (linked here), I sought guidance on forecasting thousands of time series. Based on the suggestion to treat it as a regression problem, I used the LightGBM model with extensive ...
1
vote
1
answer
63
views
Bad performance of ARIMA model on online buzz data, Any suggestions?
I was wondering if the ARIMA model is constrained to predict online buzz data (time series data).
What I want to do: Use the past round 30 months data to predict next month; and I use Python
Here are ...
0
votes
1
answer
152
views
Which hyperparameters should I choose to tune for different ML models? [closed]
I'm applying hyperparameter tuning with techniques such as, randomized search and grid search in python. I could find important hyperparameters and tuned them for Random Forest Regressor algorithm, ...
1
vote
0
answers
130
views
Is there any algorithmic-pseudocode/Python-implementation for the DAG-related API(s) in the R package dagitty? [closed]
The statistical software DAGgity offers a graphical web-based UI as well as an implementation in R that allows for finding conditional independences corresponding to d-separation, minimal adjustment ...
2
votes
1
answer
69
views
Statistical Method for Accurately Detecting Seasonality in Monthly Sales Data
I have a dataset containing monthly sales data for different product categories spanning five years (60 months of data). I am using a Python process to calculate the seasonality for each category, ...
0
votes
0
answers
271
views
Monotone constraints in decision tree regressor or random forest regression
after I've spent several weeks trying to fit a regression model to my flood damage data (x1=water height, x2=adaptation height, x3=(x1-x2), y=damage), it is now time for my very first question on ...
2
votes
1
answer
41
views
Forecasts fail for new period
I am working with prophet library in python where I do some forecasts. While I split to train and test to check, it shows very good performance, but when I forecast for the actual future period it ...
0
votes
0
answers
20
views
GLMM data analysis in R
I am analysing some data using the glmer analysis and I would like assistance as how I could solve the following error massages. I tried fixing the problem using some examples from the internet but I ...
0
votes
0
answers
94
views
VECM predict gives forecasting results that lag behind actual data
I am using Python's statsmodels.tsa.vector_ar.vecm.VECM to estimate VECM models and generate pseudo out-of-sample forecasts with the .predict() function to compare with actual data.
For example, I ...
0
votes
0
answers
48
views
Why am I getting negative components with my custom NIPALS algorithm
I've recently been learning about the Nonlinear Iterative Partial Least Squares (NIPALS) algorithm for computing the principal components of a dataset. I am trying to code a NIPALS class from scratch ...
1
vote
1
answer
35
views
Is there a way to automatically split large clusters that are greater than some maximum number of points?
I ran HDBSCAN on these coordinates and got some clusters but some are too large. HDBSCAN has a minimum cluster size parameter, but no maximum size. All I want is to intuitively divide larger clusters ...
0
votes
0
answers
11
views
Best way(s) to combine data on the same object
I have data of a solar cell's performance. Each cell consists of 8 pixels, each of which have their own data for various metrics of performance (current, etc.) throughout time. I would like to combine ...
1
vote
0
answers
80
views
Training Regression Models to Predict Continuous Probability Values in [0, 1] [closed]
I'm working on a machine learning project where my target variable represents continuous probability values that must fall within $[0, 1]$. While I understand regression models are suitable for this ...
0
votes
0
answers
27
views
Empirical Mode Decomposition(EMD) + CNN for time series forecasting
I'm currently working on a time series project, and I intend to employ the EMD+CNN technique for forecasting the output. Upon applying EMD to the training data, I obtained a total of 14 Intrinsic Mode ...
1
vote
0
answers
56
views
Can I fit a Poisson distribution to a continuous variable to apply event detection algorithm?
Preprocessing:
I have a time series of number of tweets per 10 minutes time interval that are all taken from a given discussion on a specific topic in a specific region. I preprocessed the data by ...
4
votes
1
answer
97
views
Overlapping circular bearing distributions on a plane
I have some directional hydrophones capable of recognizing transient signals/sound and estimating the circular probability density function of the bearing, or direction, that the sound came from. I ...
0
votes
0
answers
38
views
Hypothesis Testing (normality violated)
I have two sets of data, SetA and SetB - they are metrics of the same group from different time or paired. SetA score at 10 am and SetB score at 5 pm, same person (30 of them).
I want to check if the ...
0
votes
0
answers
15
views
Do all nodes have to be stored / indexed in a GNN?
Apologies in advance for what maybe a really silly question :(
I am trying to construct a Graph Neural Network, in which I would like to learn representations for certain nodes, but not others. To be ...
1
vote
2
answers
244
views
Python - need help with Linear Mixed Effects Model results interpretation
I analyse a set of physicochemical data from the river and two rows of wells - one located closer and the other further from the river (in the N-S direction).
The study's main aim is to investigate ...
0
votes
0
answers
19
views
How can I best correct / handle a slight right skew distribution in my residuals plot using stats.models mixed effects model?
I am using statsmodels mixedlm as follows:
...
1
vote
0
answers
56
views
Discrete P-values in Kolmogorov-Smirnov Test with Small Sample Sizes
I'm facing an issue while conducting the Kolmogorov-Smirnov (KS) test with two datasets of different sizes: 84 and 28 samples. I got these sets from normal distribution. I've noticed that the ...
1
vote
1
answer
230
views
KS test for Poisson distribution data
I have a dataset comprising 1000 integers. It represented as follows:
...
0
votes
0
answers
20
views
Significance Test Time Series Data Python
As a Business Analyst I may not be as experienced in performing complex statistical tests and this is the reason I am asking for your advice.
I have this sales data whose dates I have clustered into ...
0
votes
0
answers
50
views
Low CV-RMSE and negative $R^2$ (comparative)
I am trying to predict a numeric variable using XGBoost with optuna for hyperparameter optimization. I defined two objective functions for optuna, one optimized for very small datasets (5 to 17 ...
12
votes
1
answer
774
views
For Gamma distribution, use MLE or MoM?
For Gamma distribution, is it better to use MLE(maximum likelihood estimation) than MoM(method of moments) to estimate the shape and scale parameters?
Also, in python SciPy, does gamma.fit use MLE? I'...
0
votes
0
answers
30
views
What is the best way to compare means across groups if the data is not normally distributed, and sample sizes across groups can vary?
I am quite new to statistical analysis and have what is probably a newbie question. I have a dataset that contains year and state data for two different groups. I am trying to compare the group means ...
6
votes
2
answers
159
views
Which OLS Regression Results are worth putting into an article? Does my example look right?
Yesterday, I set up a topic outlining the problem I am currently working on.
After receiving many interesting responses, I added linear regression to my results, following this suggestion.
My research ...
0
votes
0
answers
22
views
Generating Predictive Models for Multiple Time Series
I have a dataset of historical film streaming data for the last few years. Each record is the Title, Genre, Rotten Tomatoes Score, Week of Streaming Data and then Raw Streaming. I am trying to find ...
0
votes
1
answer
90
views
Trouble Achieving Optimal Parameters for XGBoost Regressor with Sin(x) Time Series
I'm facing a challenge while attempting to optimize parameters for an XGBoost regressor (Python) using time series data. I've created a time series using the following code:
...
2
votes
1
answer
246
views
Confusion on Chi-Squared test results
Basically, I have the data where I am trying to assess whether there is any correlation between the gender of the manager and the gender of people within their team. I decided to do the chi-squared ...
1
vote
0
answers
27
views
Discrepancy in 2SLS Estimation Results
I've been exploring the 2SLS (Two-Stage Least Squares) estimation method to analyze a model involving endogeneity and instrumental variables. To better understand the process, I performed manual ...
0
votes
0
answers
14
views
Spliced Distributions Framework for python
There is an article Fat-Tailed Regression Modeling with Spliced
Distributions
that describes fat-tailed regression modeling by fitting the distribution consisting of N components (different ...
0
votes
1
answer
76
views
Clustering Algorithms for Complex Survey Studies
Which clustering algorithms can be used in complex survey designs? Can the kmeans algorithm be used?
I searched for literature using complex survey design data like NHANES, but they do not provide ...
1
vote
0
answers
21
views
When is it appropriate to use t-SNE and how can I trust its output to faithfully represent higher dimensional separation?
Given the discussion on this answer what appropriate assumptions and contingencies would allow tSNE to have an informative interpretation?
It's hard to come away from that answer feeling like I can ...
4
votes
2
answers
135
views
Scikitlearn: Why are hyperplane coefficients not available if kernel is not linear
I am interested in learning the math behind support vector machines. So far, I understand that SVMs attempt to find hyperplanes that maximize the margin distance between support vectors associated ...
1
vote
1
answer
55
views
Representing Nested Models as an SKLearn Pipeline [closed]
Goal: Represent nested models with SKLearn's Pipeline / ColumnTransformer / FeatureUnion setup.
Specific issue: I cannot figure out how to use the prediction from one model as a factor of a secondary ...
0
votes
0
answers
60
views
PCA with gram matrix produces different results from PCA done using covariance matrix?
I was trying PCA on a dataset (#samples=24, #dims=42) via eigendecomposition using numpy. I read that for matrices where the number of features exceeds the number of samples, we should use the gram ...