Questions tagged [python]

Python is a programming language commonly used for machine learning. Use this tag for any *on-topic* question that (a) involves `Python` either as a critical part of the question or expected answer, & (b) is not *just* about how to use `Python`.

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

Understanding GARCH

I am new to finance and volatility forecasting and am trying to understand how garch model works. Even though there are many tutorials on how to use arch_model from Python, none of them gives ...
0
votes
0answers
14 views

Runnning a pearson's correlation for two variables that change from a third factor [closed]

I have a covid-19 dataset that I'm trying to analyze. This dataset is a time-series dataset and has many columns such as Date, number of new cases, and number of new vaccinations. To find out whether ...
0
votes
0answers
8 views

Improving robustness of XGBoost on large tabular dataset with small signal and lots of noise

I have been working with XGBoost on a large set of panel data. There are 20m+ rows with 200 features. The data includes weather related data points for 100s of cities, recorded every day, for several ...
0
votes
1answer
23 views

clustering using minimum spanning tree (MST)

I have an array (X), that contains scores values of 10 users with their score. I used Minimum Spanning Tree Clustering (MST) to cluster the users based on their values, but MST did not cluster the ...
0
votes
0answers
6 views

How to measure ML classification model stability?

It's well known that decision trees are unstable models, i.e. they have high variance. It can be easily shown by adding Gaussian noise to variable(s) and checking that completely different tree is ...
0
votes
0answers
6 views

What technique is best for dimensionality reduction OR clustering of mixed data?

All, I have data with mixed types: categorical e.g. colour and numerical. I am trying to reduce the number of dimensions. I thought about using FACTOR ANALYSIS of mixed data, however the python ...
0
votes
0answers
36 views

Panel Data Regression: Confusion about Random Effects Model

I have a panel data set covering 42 countries over a period of 14 years. My data set contains country sustainability scores (combined, economic, environmental, social) as well as the mean firm-level ...
0
votes
1answer
21 views

In Python, can I use mean, median, minimum, maximum, standard deviation, population size, and single sample to generate statistically identical data?

Example situation After each exam, the professor provides the following information. Minimum Score [Arithmetic] Mean Median Maximum Score Standard Deviation I also know what my score was as well as ...
1
vote
0answers
12 views

LSTM Time Series Forecasting

I am working with Bitcoin block data and attempting to forecast target hashrate ~one month in advance. I have condensed the raw block-by-block data into daily data and calculating metrics like ...
1
vote
0answers
21 views

How to model exponential time decay component with linear regression?

I was wondering if there was a way to run a regression on a formula that is not exactly a straightforward $Y=\beta X + \varepsilon$ Essentially what I have is a bunch of features $x_i$, with an ...
1
vote
0answers
11 views

What am I misunderstanding about the ACF and ACF plot?

I am using Python's statsmodels.tsa.stattools.acf on a series, specifying alpha: acf = acf(x, alpha=0.05). After this I'm using ...
0
votes
0answers
7 views

Encoding end of line while writing from pandas dataframe to csv [closed]

I created a dataframe that contain text with end of line "\n" as shown below. Then i writed it to csv as shown in the code, and newline character are interpreted. Is there an option to ...
0
votes
1answer
20 views

Chisquare statistic not working python

code is Python 3 I have a dataset of 5000 counts that I need to plot, compare to a Gaussian fit, and calculate the Chisquare statistic. The code I have used to plot my data is: ...
0
votes
0answers
13 views

Finding the (approximate) beginning of a novel based on the distribution of newline characters

I am working on a NLP project in the domain of literature. My dataset consists of a collection of books but most of the books are prepended with meta-information such as the copyright, the outline, ...
0
votes
0answers
10 views

Separate time-series based data into two trends with known temporal patterns

I am dealing with a problem that need to distangle different patterns of a time series-based data. Specifically, the concentrations of air pollutant for one spo always vary with time due to the ...
0
votes
0answers
10 views

HDBSCAN Soft clustering - Why everything becomes 1 class?

GOAL Using HDBSCAN Putting all data points in to a class and leaving behind 0 point as noise (in other words partitioning the data without leaving behind any unallocated data points) DONE I have a ...
0
votes
0answers
7 views

Adjusting plot legend so it no longer covers the plot (matplotlib and pandas)

I have a dataframe with 2 columns: ID and Heart Rate The dataset is in long format and there exactly 50 recorded time intervals per patient (n=10) corresponding to 50 consecutive hours. I've created a ...
0
votes
1answer
15 views

If data is square rooted to assume normality, should the data used to test with Levene's Test for Equality of Variance also be square rooted?

I am going to run a MANOVA, two of the assumptions are normally distributed data and equal variances. I had a data set that non-normally distributed. I square rooted it, and it is approximately ...
0
votes
1answer
31 views

KPSS test: p-value greater than 0.05 but test statistic is less than critical values

I am using Python Statsmodels to find out if my time series is stationary or not. My time series (after the first level shift) passed the ADF test and it suggested that the time series is stationary. ...
-1
votes
0answers
12 views

image with 4-channels [closed]

i have images with 4-channels that i created by stacking RGB and thermal data. as follow: ...
0
votes
0answers
10 views

Two variables Hermite polynomial fit [closed]

I have a question about how to generate 2 variables Hermite poynomial fit. We already have $$Y \approx \sum_{r=0}^{M} \hat{\beta}_{r} \psi_{r}(X)$$ We can get $\hat{\beta_{r}}$ by the code ...
0
votes
0answers
6 views

Orthogonal Standard Regression (OSR) and Orthogonal Distance regression (ODR) [closed]

As a newbie in this area, I just want to know, the difference between OSR and ODR? If there is any difference, Is there any package available to do the OSR in python?
0
votes
0answers
14 views

Code help about creating custom loss function in tensor flow

I am trying to create my custom loss function for solving the regression problem using the RNN layers. For my specific dataset, I need the following conditions for updating my loss function: ...
0
votes
1answer
23 views

PCA and eigendecomposition: Calculating loading vector in Python

For an assignment the Matlab code has to be rewritten into Python code. ...
0
votes
1answer
10 views

Univariate time-series data with 30 minutes interval - Which algorithm to use for forecasting?

I am working on a case study to predict # of calls for next 30 days. The data is at 30 minutes interval and start from 7 am to 23:30 PM (So 34 observations in a day). I have two questions: What would ...
0
votes
0answers
26 views

Implementing Hadamard Kernel [closed]

I want to use Hadamard Kernel for classification with SVM and trying to implement the following kernel function. Here is my implementation in Python using numpy: <...
1
vote
0answers
44 views

Resource of python statistics problems and their solutions

I am learning ML / DS, after learning Pandas, I found a lot of Pandas exercises to practice like this https://github.com/guipsamora/pandas_exercises Can anyone share such exercises with a solution for ...
0
votes
0answers
17 views

implementation of $\chi^2$ test in python

I'm trying to check if a set of data has a Poisson distribution and for that I'm trying to implement a chi square test. My input data is an array that counts the amount of events registered in 1 ...
1
vote
1answer
40 views

How does one define the sum of N random variables in Python? [closed]

Given $X_1 \cdots X_n \stackrel{iid}{\sim} exp(1)$ I want to show that $Y = 2\sum_{i=1}^{n}X_i \stackrel{}{\sim} \chi^2_{2n}$ I proved it by computing the MGF of Y as $M_{Y_1}(t) = M_{2\sum_{i=1}^{n}...
1
vote
0answers
18 views

Setup and analysis of deterministic simulation (Design of experiments, ANOVA)

My problem is as follows: I have a time series simulation (implemented in Python), which is roughly about an abstract modeled storage object digesting and creating energy flows. These (resulting) ...
0
votes
0answers
8 views

How to combine time series and factor data to build a model for forecasting in python?

I am working with electrical load data. I have a time-series data of load over time and some factor data (eg, day type, temperature, humidity, airspeed, etc). I am willing to build a machine learning ...
0
votes
0answers
24 views

Predict Company Sales - Design choices; how to store incoming sales/which model to choose? [closed]

I am trying to predict the bookings of a stand-up comedian cafe. There are a lot of features I can use which have an effect on the number of sales. (e.g. day of the year, weather, average sales last ...
0
votes
0answers
8 views

DBSCAN returns TypeError: invalid type promotion [closed]

Hi I have a dataframe that looks like this. When I try to run a dbscan to search for outliers or anomalies it return me this error "TypeError: invalid type promotion" Is there anyway I can ...
3
votes
1answer
40 views

Examples of a modern ARIMA algorithm from scratch?

Where can I find a modern/robust algorithm for the fitting of an ARIMA model? Or in other words, if one were to implement an ARIMA model from scratch in 2021 what would be the most correct/efficient ...
0
votes
0answers
23 views

How to group every data point with HDBSCAN to some group to have no noise?

TASK I am clustering products with about 70 dimensions ex.: price, rating 5/5, product tag(cleaning, toy, food, fruits) I use HDBSCAN to do it GOAL The goal is when users come on our site and I can ...
1
vote
1answer
17 views

Optimal reconciliation for store sales

I'm try out optimal reconciliation using hts package in python. I'm referring https://scikit-hts.readthedocs.io/en/latest/usage.html#reconcile-pre-computed-forecasts as an example. For my data I get ...
0
votes
0answers
11 views

How to find the optimal parameter for the likelihood with lasso penalty and kernel matrix?

Suppose we have the dataset $(X, Y)$, $X = (X_1, ..., X_n)$ is the covariates matrix and $Y = (Y_1, ... , Y_n)$ to be the response/observation. I'm dealing with such optimization problem in python: $ \...
0
votes
0answers
12 views

Interpreting the y values of a pdf [duplicate]

In trying to understand the y values of a normal distribution plot I use this code: ...
0
votes
0answers
18 views

generalized fisher score for feature selection

I m looking for an R or Python code for the Generalized Fisher Score. I have look at various packages dedicated to feature selection but I could only find information about the original fisher score. ...
0
votes
0answers
92 views

Value of Covariance term too high : multiple attempts to solve it

I have a first array 5x1 Bias_sp_fid and a second one 5x1 Bias_ph_fid (fid for fiducial). I ...
1
vote
1answer
40 views

Difference Between SciPy.Stats.powerlaw and powerlaw package

I am trying to simulate random variables that are power law distributed based on my understanding of the definition in this Wikipedia article and several other resources where the consensus is that a &...
0
votes
1answer
16 views

Significant difference/ANOVA test when some values occure only once

I have fake dataframe with neighbrhood codes and prices of houses in those neighbrhoods, with this amount of values for each negihborhood: ...
0
votes
0answers
13 views

CNN - Is L2 counted at every layer?

I build one CNN model with this code: ...
0
votes
0answers
29 views

Why does PanelOLS.from_formula from Python returns way more parameters than what the model used for it works with?

I tried to run a panel OLS on a panel data (comprised of 275 time entries for 12 individuals) in Python using: ...
0
votes
1answer
19 views

Question about using formula to compute PCA transform in Python

I try to understand how the Python function sklearn.decomposition.PCA working. I read the documentation of the code package in Github about transform function. And ...
0
votes
0answers
9 views

Comparing CDFs, Kolmogorov-Smirnov, Max/Average?

I'm in the process of building a python group of classes that will calculate many CDFs for a large number of samples. I then want to compare the CDFs of all the samples to see if any two samples could ...
5
votes
1answer
130 views

How to propagate measurement uncertainty in predictors *and* responses for multidimensional, non-parametric regression (and software to do it)?

Background Errors-in-variables models are defined as: regression models that account for measurement errors in the independent variables. In contrast, standard regression models assume that those ...
0
votes
1answer
58 views

Network analysis: density of communities/partitions (and other metrics)

After running a community detection algorhythm (e.g. edge betweenness, or greedy modularity), I like know the density of each seperate community, and potentially some other metrics, too. My desired ...
0
votes
0answers
14 views

Inter annotator agreement (or disagreement) for highly imbalance annotations?

I have a time-series dataset which is annotated by 4 individual annotators where I have following things in annotations. Its possible that one annotator has not annotated all samples (i.e. missing ...
0
votes
0answers
27 views

How can we introduce an anomaly class as a positive class to sklearn IsolationForest?

I inspired by this notebook, and I'm experimenting IsolationForest (IF) algorithm using scikit-learn==0.22.2.post1 for anomaly ...

1
2 3 4 5
76