Questions tagged [python]

Python is a programming language commonly used for machine learning. Use this tag for any *on-topic* question that (a) involves `Python` either as a critical part of the question or expected answer, & (b) is not *just* about how to use `Python`.

Filter by
Sorted by
Tagged with
1
vote
0answers
42 views

Is it possible that accuracy testing and the accuracy training are 1.00 in binary classification [closed]

I am working to improve classification results with more ML algorithm. I get 100 percent accuracy in both test and training set. I used GradientBoostingClassifier, XGboost , RandomForest and Xgboost ...
4
votes
1answer
43 views

Using n linear regression models for different subsets vs using one model for the entire dataset

I am working on the following regression problem: 1 dependent variable m continuous features 1 categorical feature c with n possible values (giving a total of m + n continuous features) My initial ...
6
votes
1answer
42 views

Difference between manifest and observed variables

I try to code SEM model using the semopy package (https://arxiv.org/pdf/1905.09376v1.pdf), but there are 3 types of variables there - except latent, there are ...
0
votes
0answers
30 views

get prediction intervals from Winters-Holt forecast method in python

I am trying to output predictions from a Winters-Holt forecasting model that I have made. I have found that libraries like Statsmodels have some tools to do that but impossible to find tutorials or ...
1
vote
0answers
9 views

Best practice/Ideas for clustering Event Sequence Embeddings?

My dataset consist of around 40 000 samples of event sequences. Sample of data [[Event 1, Event 2, Event 4, Event 5], [Event 1, Event 3, Event 4], [...]] I ...
2
votes
1answer
45 views

Linear function gradient descent

I am trying to implement a gradient descent algorithm for a simple linear function: y(x) = x Where initial hypothesis function is: ...
3
votes
1answer
38 views

Perform regression with lot of variables (alternative to stepwise)

I hope everyone out here is doing well. I am working towards a linear regression model. I am starting out with 470 variables , most of them are demographics variables by area (zip code). My target ...
1
vote
1answer
36 views

One-factor mean-reverting model parameter estimation with Kalman Filter - LinAlgError: 13-th leading minor of the array is not positive definite

My code so far is as follows, starting from the simplest case of estimating only k: ...
0
votes
0answers
22 views

Issue in using KNN in out-of-sample estimation

In search of the best possible features of forecasting, I have the idea of finding the KNN(5) of my Y variable to predict future values of Y. This is only possible in real time, using forecasts of X-...
1
vote
1answer
14 views

Testing for multi-collinearity after fitting a model with LassoCV from Sklearn in Python?

Is there a way to test for multi-collinearity, like VIF for example, after fitting a model with LassoCV from Sklearn in Python? https://scikit-learn.org/stable/modules/generated/sklearn.linear_model....
0
votes
1answer
84 views

The Hessian in XGBoost loss function doesn't look like a square matrix

I am following the tutorial for a custom loss function here. I can follow along with the math for the gradient and hessian, where you just take derivatives with respect to ...
4
votes
1answer
117 views

How do you uniformly sample spans from a bounded line?

Suppose you have a bounded and continuous line. For example, the line could include all real numbers between 0 and 3. How do you sample spans from the line such that... Any point on the line has an ...
2
votes
0answers
71 views

Does data point ordering matter in LASSO regression?

I am working with Lasso on Python with a dataset of size 85×2000. I find that the ordering of the rows influence the result of the Lasso run. I have other parameters fixed, including random states so ...
2
votes
2answers
109 views

Is Jaccard similarity/distance suitable for non-binary, quantitative data?

I have a dataset with each row a country and 10 columns with numerical features like GDP,Electrcity consumption, GNI etc. I am trying to use distance metrics to find similarity between the countries ...
1
vote
0answers
32 views

Why does my ARIMA model forecast drastic drop at the second timestamp in the prediction period?

I am trying to reproduce SPSS results with python. I'm using statsmodels SARIMAX package in python. My model produces forecasting results which shows sudden drop at '2020-08-01'. After '2020-08-01', ...
0
votes
1answer
30 views

Data leakage with clustered observations

I have (what I call) a clustered dataset, that is: for one client, I can have multiple observations that will have some variables in common and some variables will be specific to each observation. ...
1
vote
0answers
21 views

Modeling with Multiple Values per Variable, per Observation

I'm attempting to develop an autoencoder on top of medical claims data that have mutliple values per category because a claim often has multiple lines associated with it. For example, let's say (but ...
1
vote
1answer
27 views

Can anyone tell me more about this Prediction quality measure, Prediction Accuracy, used for regression evaluation?

I am trying to find more info about the attached Prediction Accuracy measure used for regression. It is quiet similar to R2 and Nash-sutcliffe Efficiency but not exactly. Googling leads to ...
2
votes
1answer
24 views

Artificial Neural Network for time series analysis - $R^2$ scores [duplicate]

I've looked at an introduction to simple ANNs for time series analysis https://mc.ai/an-introduction-on-time-series-forecasting-with-simple-neura-networks-lstm/. In the tutorial, the train dataset r2 ...
1
vote
1answer
52 views

Convergence Error Using lifelines.CoxTimeVaryingFitter Python

I want to evaluate my Cox model using the lifelines package for a time varying covariate problem. However, when I use the lifelines.CoxTimeVaryingFitter I get a ...
-1
votes
1answer
23 views

How to validate Anova assumption of gaussian residuals in python

I think I have some fundamental misunderstandings when it comes to the assumptions behind ANOVA. Lets suppose there is a function that maps y to x with the following equation, y=3x+2. Now suppose we ...
0
votes
0answers
9 views

TF object detection - The total number of detected objects is not increasing

I'm building a model to recognize fishes in the aquarium (150 different fishes). I'm using a faster_rcnn_inception_v2_coco_2018_01_28 model for transfer learning from TF object detection API. I have ...
1
vote
1answer
29 views

Computing the hastings ratio, g(x|x')/g(x'|x) for asymmetric proposal distributions in MH algorithm?

I understand the Metropolis algorithm. Where I get confused is the MH algorithm where asymmetric proposal distributions may be used. I understand that P(x) and P(x') represent the likelihood/...
0
votes
1answer
22 views

Outlier treatment in sales data comparisson

I'm working for a webshop and I'm trying to compare month for month this year with same month last year. In order to get a better picture of growth in revenue/loss in revenue I use z-scores to detect ...
0
votes
0answers
10 views

partial_fit API in minibatch K-Means Sci-kit Learn

The documentation on the partial_fit API from Sklearn is very sparse. I am trying to understand how it works with Sci-kit learns Minibatch K-means algorithm: https://scikit-learn.org/stable/modules/...
2
votes
1answer
41 views

Pointwise standard errors for a logistic regression fit with statsmodels

Source An Introduction to Statistical Learning with Applications in R, found here: https://faculty.marshall.usc.edu/gareth-james/ISL/ISLR%20Seventh%20Printing.pdf Task I'm trying to replicate the ...
0
votes
0answers
17 views

SIR Estimation Finding Cause of Error While Estimating Parameters With Python

Hello I had a question on stackoverflow and one user redirected me to here as this question is more related with statistics. I'm trying to fit SIR Epidemics Spread Model to the current new case data ...
0
votes
0answers
15 views

Monte Carlo replacement for T Test and Confidence Intervals using Percentiles

I am following the Hypothesis Testing Tutorial from ZedStatistics, and now I am trying to replicate the results using a Monte Carlo approach. However, when I try to reproduce the basic example (image ...
0
votes
1answer
34 views

Linear Regression : Why isn't my loss decreasing? [closed]

I have been trying to implement a linear regression model on my own with multiple variables. After implementing it, I see that my loss doesn't decrease. I think that I'm facing a dimension problem. ...
1
vote
0answers
21 views

How to define ARIMA parameters if some values of autocorrelation is too high?

I know, that parameters p, q, P and Q can be founded using autocorrelograms. As I understand q is the number of the last significant lag in the autocorrelogram, and p is the number of the last ...
0
votes
0answers
32 views

Granger causality and meaning of lag for time series $X_{t}$ and $X_{t+1}$

Consider 2 time series $X_{t}$ and $X_{t+1}$ created as follows ...
2
votes
1answer
73 views

Metropolis -> Metropolis-Hastings for asymmetric proposal distributions?

The below python code implements the Metropolis algorithm and samples from a single variable gaussian distribution. The initial value is sampled uniformly within 5 standard deviations of the mean. ...
1
vote
1answer
37 views

Statsmodel logit producing param nans, large std err, warnings, but model performance is fine [duplicate]

I keep getting warnings such as RuntimeWarning: invalid value encountered in greaterreturn (a < x) & (x < b) and my model summary is full of nans and ...
5
votes
1answer
83 views

How do I treat my Confounding variables in my multivariate Linear Mixed Model?

I'm trying to build a linear mixed model for 5 outcome variables ... Cholesterol 1,Cholesterol 2,Cholesterol 3,Cholesterol 4,Cholesterol 5 which will be melted into a single Cholesterol variable, ...
0
votes
0answers
13 views

How to know the total number of infected from the time-series of infected people?

From a SIR model, I have the time-series that provides the people infected at each instant t, where $t∈[0,T]$. How can I find out the total number of people who have been infected during the entire ...
0
votes
1answer
30 views

Algorithms for Anomaly Detection of Event Sequence Data [Python/R]

I am building an anomaly detection system of event sequence data (transactions). For each timestep, a transaction can be in any of 76 different stages. My dataset is therefore a 3D array of size(m,t,N)...
1
vote
1answer
36 views

Estimate the delay between two signals

I have two voltage time series from two sides of a battery with 20 seconds resolution. The signals are non-periodic and voltage2 is usually half the ...
0
votes
1answer
36 views

Higher Order Category Overlap Analysis

I am attempting to analyse the categorical overlap of a dataset to ultimately ascertain the optimal way of categorising the data to minimise the amount of used categories to describe the dataset. ...
0
votes
0answers
24 views

Multinomial logit returns nans in some folds of cross validation

I made this code that uses straified kfolds to split the dataset and fit a multinomial regression, and get the accuracy afterwards. My X is an array of has 19 ...
0
votes
0answers
31 views

Best way to analyse categorical data?

I'm fairly new to data science so forgive the newb question, but: what's the best way to work with categorical data when doing exploratory data analysis? Is there a dedicated field for that? In this ...
0
votes
1answer
66 views

How to do Data Augmentation and Cross validation at the same time

I have read somewhere that you should not use data augmentation on your validation set, and you should only use it on your training set. My problem is this: I have a dataset with few samples. I split ...
0
votes
1answer
34 views

How to find areas under the normal curve?

I was able to draw a normal curve using the data samples in Python with the help of import scipy.stats as stats library as: ...
1
vote
0answers
23 views

Polynomial regression multicollinearity assumption?

The difference between Linear regression and Polynomial regression is that in the last we manipulate our original explanatory ...
0
votes
0answers
31 views

Statistical significance with Newey–West standard errors with a one-lag correction used

I am trying to do statistical test to prove that the returns of a certain equity are statistically significant so that I can reject the null hypothesis that the mean (of returns) is equal to 0. I have ...
2
votes
1answer
106 views

Boostrap Confidence Interval Calculation

So, I have a sample dataset of size 500, and I've bootstrapped it 1000 times and took the mean of each bootstrap sample. So now, I have a list of means a.k.a ...
0
votes
0answers
9 views

Plot SVM boundary *after* training in Python

I want to train SVM on multiple features for high accuracy. Then I want to do the visualization: reduce dimensionality to 2D (with PCA, t-SNE or anything else) and plot the learned decision boundary. ...
0
votes
0answers
15 views

Does feature selection with mutual information require scaling?

I’m building a machine learning model that has continuous, discrete and one-hot encoded features. I would like to use mutual_info_classif for feature selection (through SelectKBest). Do I need to use ...
1
vote
1answer
60 views

Appropriate Feature Selection methods

I'm running a multinomial logistic regression and I'm torn regarding which variable selection method to apply... The ones I know are backwards elimination or forward selection, chi square feature ...
0
votes
0answers
22 views

How to randomly shuffle tiny spheres inside a big sphere? [duplicate]

I have a list of spheres with some known characteristics (ids, radii, masses, and positions) with ids, radii, and masses being 1D arrays with shape (511, ) and positions being 3D array with shape (511,...
1
vote
0answers
37 views

Is there any seasonal VAR or VARMA algorithm?

As I understand, VAR algorithms doesn't work with seasonal data. I mean, seasonal components are not in the autoregression equation. But, if I want find dependencies between several time series, that ...

1 2 3
4
5
70