Questions tagged [python]

Python is a programming language commonly used for machine learning. Use this tag for any *on-topic* question that (a) involves `Python` either as a critical part of the question or expected answer, & (b) is not *just* about how to use `Python`.

Filter by
Sorted by
Tagged with
1
vote
1answer
27 views

Joint optimization - Feature extraction and a classifier

I am dealing with a classification problem and high dimensional data. I am using a feature extraction method ( PCA - Principle Component Analysis) followed by a Support Vector Machine (SVM). I just ...
0
votes
0answers
21 views

What is the correct procedure for nested cross-validation?

I am trying to use scikit-learn to make a classifier and then predict the accuracy of the classifier. My dataset is relatively small and I am unsure of the best parameters. Hence I turned to nested ...
0
votes
0answers
11 views

calculating multinomial likelihood

I have the counts for four nucleotides: a,g,c,t at each position and the corresponding frequencies in a string. I want to compute multinomial likelihood of observing a given count e.g. 'a' given the ...
0
votes
0answers
9 views

How can PyMC3 handle uncertainty in the number of parameters in a Dirichlet Distribution?

I'm taking a look at the following to familiarize myself with Bayesian Inference in PyMC3: https://towardsdatascience.com/estimating-probabilities-with-bayesian-modeling-in-python-7144be007815 In this,...
0
votes
0answers
13 views

SVM predicts always the same class

I have a dataset with tf-idf values and their corresponding classes and I am trying to do predictions using SVM. The problem is that all the results that it produces have the same class. Most related ...
0
votes
0answers
11 views

ARIMA Forecasts (Python / Statsmodel)

Say I have an ARIMA model that I have fit and with with I want to make forecasts/predictions. Say the model is an ARIMA(2,0,2) with some seasonality (1,0,0,365) Since p and q are 2, we are relying on ...
0
votes
0answers
14 views

How can I “remove” variability in my data that is due to periodic signals, such as Temperature, RH and Solar radiation?

I have a measured signal that I know is affected by some periodic signals, such as Temperature, RH and Solar radiation. Is there a way that I can "remove" their influence from my measured ...
0
votes
0answers
18 views

Multi-level repeated-measures logistic regression in python/R

Accidentally posted this in stackoverflow, so reposting here. I have a repeated-measures design experiment. Each subject took part in some experimental conditions, each one associated with several ...
0
votes
0answers
11 views

Multi-level repeated-measures logistic regression in python/R [duplicate]

I have a repeated-measures design experiment. Each subject took part in some experimental conditions, each one associated with several events. Normally I would analyse this with an ANOVA by grouping ...
1
vote
0answers
14 views

Document AI : Using FUNSD dataset to train a GNN to classify 'Linked' entities

I have been using the FUNSD dataset to predict sequence labeling in unstructured documents per this paper: LayoutLM: Pre-training of Text and Layout for Document Image Understanding . The data after ...
0
votes
0answers
31 views

Forecasting Quarterly Time Series Data?

I've gotten very confused reading all the articles about forecasting time series data with seasonality on Medium and other sources. It seems that many provide useful background and importance of ...
0
votes
1answer
13 views

How can I extract the correct hyper-plane from sklearn.svm's LinearSVC

I'm not certain I understand how sklearn's Linear SVC works. I had assumed that it would find an optimal hyper-plane to divide one class from another. I tried to recover the separating hyper-plane ...
2
votes
2answers
38 views

LogisticRegression with GridSearchCV not converging

I'm trying to find the best parameters for a logistoic regression but I find that the "best estimator" doesn't converge. Is there a way to specify that the estimator needs to converge to ...
1
vote
1answer
19 views

What to do after knowing the model is overfitted?

So I was trying to run a model using scikit-learn. In order to tune the hyperparameters, I used RandomizedSearchCV, just like this: ...
0
votes
0answers
29 views

Is it possible to perform factor analysis over an unstandardized matrix?

I'm trying to do exploratory factor analysis (EFA) and it seems that all functions (in MATLAB, python or R) standardise the data matrix first. I was wondering if there is a way to perform factor ...
1
vote
1answer
18 views

Can logistic regression be used when the dataset has observations from the same users but are unique per day

I have a dataset that captures user information by day (the users are unique per day but often have observations on multiple days) and I want to analyze a binary outcome. Is there a more appropriate ...
0
votes
0answers
7 views

(python) dice roller, odds that groups or players win?

I am trying to simulate a dice game. All players roll a 1d6 and the highest number wins. (if multiple people win, they win.) I am getting proper odds for one player by redoing the game 10000x and ...
0
votes
0answers
18 views

How to properly measure accuracy with feature selection?

I applied a feature selector (with this great python package) in my dataset. This package uses the wrapper approach, where you define a classification model that runs on your data and find the best $k$...
0
votes
0answers
22 views

About xgboost tuning

I'm trying to tune an XGboost for a multiclass imbalanced problem, for classification. I've been reading about it and apparently, bayesian optimization is a good way to tune the hyperparameters, so I ...
1
vote
1answer
58 views

Deep Neural Network Not Learning Anything [closed]

I am training a simple Neural network with some Dense and Dropout Layers. But on running the fit function, there is no training taking place. My Model is: ...
0
votes
1answer
16 views

Trying to determine the failure rate of redundantly sending bits over a noisy transmission

We are sending a one bit message to someone. There is a 60% chance the message bit will be a 0. When transmitting the message there is a 3% chance a 1 will become a 0. There is a 5% chance a 0 will ...
0
votes
1answer
47 views

Computing the covariance of the truncated normal distribution

In the paper On Moments of Folded and Truncated Multivariate Normal Distributions on page 17, one can find the explicit expression for low order moments of the truncated multivariate normal ...
0
votes
0answers
7 views

Implementations for Conditional Average Treatment Effects that can be trained incrementally

I am currently working on a very large dataset (billions of rows) of A/B test data and want to implement some methods to estimate conditional average treatment effects. I basically need a forecast ...
0
votes
1answer
18 views

Training loss fluctuates but validation loss is nearly constant

I am training a network ESNet in Pytorch to predict vanishing point as per VPGNet ICCV 2017 paper. I am using SGD with 0.1 learning rate and ReducedLR scheduler with patience = 5. My loss curve is ...
0
votes
0answers
9 views

Balanced corpus in topic models

I am building a topic model for a corpus of webpages extracted from a random subset of domains, the topics seems to be ok, but often I see very similar topics, which makes me think I should reduce the ...
0
votes
0answers
18 views

How do I select number of bins to discretize the data?

So, I have been pondering on how I can select the number of bins in a dataset? I know we have different methods for selecting number of bins for histogram, but how do I select number of bins when ...
0
votes
0answers
13 views

Non Linear regression to obtain diminishing marginal effect / elasticity [duplicate]

I am working with some real estate data on housing units. For a given market, I have data on occupied units, rents, and control variables such as population, demographics, income levels etc. I'd like ...
1
vote
2answers
35 views

Violin plot of 2 numpy arrays with seaborn

I would like to compare the distribution of 2 numpy arrays using a violin plot made with seaborn. The maximal value in both arrays is 1. The plot suggests a higher maximum. Am I misunderstanding the ...
4
votes
1answer
46 views

Do 1PL IRT models measure both ability and difficulty, or just difficulty?

I'm trying to better understand Item response Theory (IRT) from a Bayesian perspective. Hypothetically, suppose I want to use a 1PL model and my data is a binary matrix ...
7
votes
1answer
94 views

Gaussian process - what am I doing wrong?

I have recently started to delve into Gaussian processes. During my review, I have found a book which states that one can interpret the mean of a Gaussian process as a combination of basis functions, ...
0
votes
1answer
21 views

Identify individual networks within dataset containing many (using SAS preferably)

I have a dataset of links/edges that looks something like the following (except significantly larger - approx. 50,000 links): ...
0
votes
0answers
26 views

Computing Karcher means in the space of all probability densities

Suppose I have samples $X_i \sim \mu_i$, and let us for simplicity assume all the higher moments exist and the supports of probability measures $m_i$ are on some compact subset of $\mathbb{R}^n$. Let ...
0
votes
0answers
7 views

Metrics for implicit data in the recommender system

Which metrics for analysis and evaluation for implicit data in a recommender system do you use? Which ones are there? I have found some metrics, but I can't see the wood for the trees. Looking forward ...
1
vote
0answers
9 views

Does TPOT support multi-label text classification?

How can I run TPOT to give me suggestions on what algorithm to use for a multi-label text classification? My data is already cleaned and divided into training and testing sets.
0
votes
0answers
9 views

Clustering DBSCAN's parameter Epsilon: How is eps related to scale of data being clustered?

How is scale of eps related to data to be clustered in DBSCAN? e.g. in image of 1024x1024, we have points as: ...
1
vote
0answers
33 views

Covariance matrix computation VECM, Lutkepohl (2005) p.287

In Lutkepohls book "New introduction to multiple time series analysis" (2005) on p.287 is outlined how to calculate your estimated parameters $[\hat \Pi,\hat \Gamma]$ sample covariance ...
0
votes
0answers
8 views

Looping over algorithms results in error (ensemble, cross-validation). The function changes the shape of the predictor value

I have a dataset that I want to test with several machine learning algorithms. The features I use have a shape of (100854, 94) and the predictor value has a shape ...
0
votes
1answer
74 views

How to estimate the optimal cutpoint for a binary outcome in python?

I have a dataset of diabetic patients which has been used to train an xgboost model in several outcomes such as stroke, amputation, and more. Originally we used the continuous numeric variables as-is, ...
0
votes
0answers
11 views

Modeling resulting actions as a parameter approaches a certain value

Python newbie here, so please bear with me. I'm trying to code a simple reinforcement learning program in which an agent repeatedly chooses from two different gambles of the form: win x1 with ...
0
votes
0answers
27 views

Transforming Heterogeneous Treatment Effect Models (in EconML) into Average Treatment Effect Model (from DoWhy)

This question relates to the steps one would need to take in order to reproduce an answer from the DoWhy tutorial, using the EconML library code for heterogeneous causal effects. In DoWhy, there is ...
1
vote
2answers
30 views

How to combine outputs from different non-machine learning prediction models?

I have a list of cancer mutations and I have run five different bioinformatics tools that predict whether a particular mutation will lead to a more aggressive form of cancer or not (0-neutral;1-...
0
votes
0answers
10 views

How to select 'n-splits' in ShuffleSplit

How many times can I do ShuffleSplit? What is the maximum/recomemend value of 'n-splits'? If, for example, I'm comparing two classifiers, can I perform ShuffleSplit as many times as needed on each ...
0
votes
0answers
16 views

Large dataset with many missing values (90% or more) - remove columns or still do one-hot-encoding?

I have a large dataset with 500.000 rows and 54 columns. I want to predict the sale price of machines, based on historical data. Many columns are nominal categorical (with no order) and have NaN; ...
1
vote
0answers
46 views

How to get coordinates when only pairwise distances are known? [duplicate]

I have $n$ points with pairwise distances known, $d_{i,j}: 0<i,j<n$. Their coordinates, $\vec{x}_i \in \mathbb{R}^k: 0 <i<n$, are unknown. I can set up $n^2 - n$ equations to solve for ...
3
votes
1answer
42 views

Repeated Nested Cross validation

I'm aware that nested cross-validation is used for hyperparameter tuning and model selection and that repeated k-fold cross-validation is used to improve the estimated performance of the model. My ...
0
votes
0answers
15 views

Testing for Correlation between Google NGram Series

I have two time series extracted from Google ngrams from two different corpora, over a 15 year period. I want to check if the two series are correlated or if the first series influences the second (i....
1
vote
1answer
29 views

How to set shape parameters for Johnson SU distribution in python scipy? [closed]

The Johnson SU distribution has 4 parameters ($\delta,\gamma,\lambda,\xi$), but scipy.stats.johnsonsu only has 2 parameters ($a,b$). Why the difference. how can I ...
1
vote
0answers
65 views

Unifying scale (1 to 10) represent user’s performance to standard across 100+ features

We are building a tool to assess the risk of language disorders in children. We have 600 participants (100 in each of the 6 age categories of 4 to 4.5, 4.5 to 5, and so on). This is the training data ...
0
votes
1answer
25 views

Understanding the Q-learning loss function?

Perhaps this can be explained a little more to me. I understand what's in literature but I'm struggling to understand why this is the preferred loss. If we have an agent that can move ↑↓→← and for ...
0
votes
0answers
16 views

Generate binary classification data in python?

Is there a simple way to generate binary classification data in python? I'd like to specify $X$ input parameter, $[x_1,...,x_n]$, and generate a dataset such that the (overall) McFadden's pseudo $R^2$ ...

1
2
3 4 5
70