Questions tagged [python]

Python is a general purpose programming language designed for ease of use. It is a commonly used platform for machine learning. Use this tag for any *on-topic* question that (a) involves `Python` either as a critical part of the question or expected answer, & (b) is not *just* about how to use `Python`.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

Keras predict probability too high

I've followed this tutorial to create a simple neural network that matches intents. The results are good and the process is relatively fast, however there's an issue with it. The probability is ...
-1
votes
1answer
23 views

Which is the correct method for outlier analysis on a dataset for modelling?

I'm trying to build a regression model but my data-set have many outliers points which I need to analyze and then remove them. There are two ways to do it, 1) First do all the analysis on every ...
0
votes
0answers
23 views

How to find the right optimization algorithm? [on hold]

I'm currently working out this optimization problem, but I'm kinda lost as to what optimization algorithm I should use. I'm using the optimization module of the SciPy library to provide the algorithm, ...
0
votes
0answers
5 views

Panel Data Models: Do they account for auto-correlation? [Python Package: Linearmodels]

I have a data set of measurements take over the last four years. I am interested in the effects of a treatment. I measure response variable Y in 50 entities with no treatment every month, and; I ...
2
votes
2answers
27 views

Anyone know Multivariate OLS on Statsmodels? [on hold]

I trying to run a mutivariate multiple regression using Statsmodels. Not sure if this is the best tool for this type of regression, so please do tell me if there's another package to which I should ...
0
votes
0answers
16 views

Derive and calculate winning probability of solitaire 1-2-3 [on hold]

There is a simple solitaire that involves a 40 card deck, 4 suits each with 10 cards from one to ten. It develops as following: Card Shuffling Start drawing from the deck one card at time and ...
1
vote
0answers
13 views

Avoid Retraining a Model When Executing a Program? [on hold]

I've started using OpenCV for some image processing projects and I'm wondering if there's a way to save time when it comes to processing test images against a database of faces. Issue: 10 pictures of ...
0
votes
0answers
10 views

Machine Learning Pipeline and remove weak correlated variables [on hold]

I am very new to machine learning and I am trying to understand the whole process of preparing the data for the machine learning part. I am making use of pipeline from sklearn. Lets say I have the ...
0
votes
0answers
24 views

When to use stratified split and when to use simple sklearn split or hashing based split [on hold]

When to use stratified split in sklearn for splitting the data into test and training set I have California housing dataset and i am splitting income feature. And i am getting more accurate results ...
0
votes
1answer
40 views

Clustering groups of data with Machine Learning

I want to cluster objects. There are two attributes (one categorial and one numerical). They should be clustered after the numerical attribut. But observations with the same categorical value should ...
0
votes
1answer
41 views

obtaining empirical CDF of a given data

I have a dataset of variable $x$ that has a value between 0 and 6. I would like to have a function that defines empirical CDF of variable $x$. Since $x$ does not have a specific distribution (such as ...
2
votes
3answers
33 views

How can I create a neural network that can recognize objects without having data for objects that aren't in the classification set?

I have a data set of 10,000 images of 5 different recycling items. The goal of my neural network is to tell me if an item is recyclable or not. The problem is that I only have data for the 5 different ...
0
votes
0answers
5 views

How to perform MANOVA with categorical independent variables

I have a dataset where the independent variable is sieve size (categorically either 25, 63, 100 or 150), and the dependent variables are length and width. ...
1
vote
0answers
10 views

How to implement 1D Convolutional Autoencoder with multiple channels?

I want to build a 1D convolution autoencoder with 4 channels in Keras. Instead of images with RGB channels, I am working with triaxial sensor data + magnitude which calls for 4 channels. I haven't ...
1
vote
1answer
17 views

SelectKBest score function with mixed categorical and continuous data

I am building a classification model where my label is categorical (0 or 1). I want to use scikit-learn’s SelectKBest to select my top 10 features, but I’m not sure which score function to use. I ...
0
votes
0answers
35 views

Why to use cross-validation instead of using all training data?

My question is specifically about kaggle competitions. Why would I need to use the cross-validation method if I can just use all training data and then see the accuracy on kaggle? The only reason of ...
0
votes
0answers
10 views

How to define a good threshold based on a numbered dataset

I have a set of ordered number list as follows. 25789, 20500, 18888, 15000, 13200, ...., 118, 8, 1, 0 In my dataset most of the values are in the range of (1000-...
1
vote
0answers
31 views

Is it possible to calculate LASSO by hand?

I'm using LASSO regression from Sklearn, and I wanted to see if it is possible to calculate the coefficients and predictions by hand? Currently I'm just fitting the model and then using the predict ...
0
votes
1answer
21 views

Measuring R-Squared by category

It makes sense to look at metrics like recall/precision by category when performing classification. sklearn has classification_report for this purpose. But what if I want to look at error by category ...
0
votes
0answers
26 views

Kernel Density Estimation - Comparison Between different sets of samples

Is there a way for compare the distribution of different set of samples? For example, I have three sets, for example: X1 = N(0, 1); X2 = N(0.5, 1); X3 = N(1, 1). Each set is drown with a specific (...
0
votes
1answer
14 views

how does class 0 scores in the classification report are calculated ( sklearn python )?

Here how these class-0 probability are calculated?? print(classification_report(y_true, y_pred, target_names=target_names)) precision recall f1-score support ...
0
votes
0answers
7 views

Keras how to implement custom loss function that requires matrix multiplication [closed]

I am trying to implement singular value decomposition using a DNN (I'm not super concerned with whether this is even possible, whether this makes sense, etc.) This ultimately might be a silly idea, ...
0
votes
0answers
19 views

Python code for correlated Poisson [closed]

Does anyone have a Python code or direct me to a site which has the following Yahav and Shmueli code (march 2008)? APPENDIX: Generating Multivariate Poisson Data in R ...
0
votes
1answer
19 views

Generating samples from target distribution after importance sampling

So I'm trying to understand importance sampling. So far I have the algorithm implemented like this: ...
-1
votes
0answers
14 views

Comparing two subsets of data for same observations

I have a data set with 50 test subject and 500 features. The features can be divided into two subsets: profile data of the test subject (pass/fail, age, education, IQ,...) and scoring on the test(...
0
votes
2answers
36 views

why we should use neural network in detecting objects like detecting license plates

Am learning some techniques to be able to detect license plate accurately But I saw some recent researches talk about DEEP learning or Neural network (NN) for reaching best results. do I should use it ...
2
votes
2answers
28 views

Getting very large coefficients from linear regression

I'm currently looking at rates for a study that vary between 0 and 100 with most of the rates falling between 0 and 1. I am running a linear regression on 70 dummy variables (coded 0-1) and nearly ...
0
votes
0answers
11 views
1
vote
1answer
14 views

How to identify a case of overfitting using stratified k fold cross validation?

I am working on a problem with a very small amount of data of 211 examples. The problem is a binary classification problem with 2 sets of classes. The data is highly imbalanced with 84% being the ...
0
votes
0answers
10 views

Passing Multiple Arguments to Generator in tf.data.Datasets.from_generator [closed]

I have a folder containing images of various sizes. I need to resize them all to (160,120) and create a tf.data.Dataset. For this I used the tf.data.Dataset.from_generator. But I can't seem to figure ...
1
vote
2answers
43 views

How should I go about doing regression with a series of exponents involved?

I have a table of data with input values and target values, and I was tasked to do something quite peculiar. I was tasked to run a sort of exponential regression and report back with the coefficients ...
2
votes
0answers
47 views

James Stein Estimator for more than one Sample

I have a hard time understanding the James-Stein Estimator. I show you how I tried to comprehend it by using a python example. I take a normally distributed random vector with mean $(0.1, 0.2, 0.3, 0....
0
votes
0answers
21 views

Equation to update slope vector using lasso regression

I am trying to figure out how sklearn lasso model works. I am using gradient descent process under which I am performing lasso Where I have defined cost function as ...
1
vote
0answers
20 views

ARMA-GARCH python implementation

When it comes to predicting timeseries with ARMA-GARCH, the conditonal mean is modeled using an ARMA process and the conditional variance with a GARCH process. I've seen tutorials predicting returns ...
0
votes
1answer
22 views

k-means clustering issue voice data

I'm getting an issue in my k-means I don't know if it my data-set or what anything else. Why i got thia flowing point in the right side of the image? ...
0
votes
0answers
10 views

Observations correction using HMM

I don't know if this is even a good idea, but I am trying to implement a HMM such that my hidden states are the same as my output states. So given a sequence of discrete, time-series observations: $$O ...
0
votes
0answers
17 views

How to build a propensity model when you have only one target class?

I have a use case in which I am trying to build a model which predicts most likely customer to invest in commodity (Share market data). My dataset consists of customers who invest in commodity ...
0
votes
0answers
12 views

Unsupervised binary classification with mixed data: Anomaly detection?

I apologize for the lengthy question, but this problem has been troubling me for quite some time now and I can't seem to find an answer to another question which directly applies to my situation. I ...
6
votes
2answers
500 views

How do I deal with large amout missing values in a data set without dropping them?

I am trying to build a binary classification model which predicts whether a patient would me infected with a certain disease at the the end of his hospital stay or not. The features that I have are ...
0
votes
1answer
30 views

Set up a SARIMA Model

I hope my question is not a dupplicate, I've searched on many forums and did not find the answer I'm looking for... I'm currently studying Data Analysis, and I'm working on a project with time ...
0
votes
1answer
34 views

Sarima : Fitting on train subset works but fitting on whole train does not [on hold]

I use SARIMAX from statsmodels.tsa.statespace.sarimax in Python. I have a simple one column energy consumption dataset with 27679 rows. The frequency is Hour. I do hyperparameters optimization ...
0
votes
1answer
9 views

Transition for scikit learn to xgboost: Where can I find a comprehensive documentation for xgboost? (Python)

As the internet seems to be conviced that xgboost is well worth a shot when working with decision trees anyways, I set out to try it. I deal with a binary classification problem. Up to now, I was ...
0
votes
1answer
56 views

Why does the mean for a beta distribution when input into the scipy beta cdf not return 0.5? [duplicate]

Very likely I am missing some fundamentals here, or my python coding is not up to scratch. When I use the standard formula to calculate the mean for a beta distribution ~ $B(\alpha,\beta)$ $$ \mu = \...
0
votes
1answer
24 views

Kernel PCA: Find most important variables for each PC

To find the most important variable for each Principal Component is easy with PCA: With data->X and variables->variable_names ...
0
votes
0answers
14 views

Why my Deming Regression line change so much when switching variables? If they seem to be a linear relationship betwen them?

I am trying to fit a line that best predicts the production of energy Y given the speed of wind X, a typical Y = xm + b , using deming regression. I am looking for the slope and the intercept of that ...
0
votes
0answers
35 views

Simulations of sampling distribution of variance

I'm trying to estimate the sampling distribution of a variance from a single sample to do a certain statistical test. To test the proof of concept, I take a normal distribution $N(200,5)$ with mean ...
0
votes
1answer
17 views

Saved PCA model produce different result

I'm using PCA to reduce my feature vector dimension. I'm saving its model and transformed output like this: ...
-2
votes
0answers
13 views

Normalizing data in classification [duplicate]

i am following a classification tutorial and i encountered this line of code and i dont understand it . it says that we are going to normalize our data. anyone with knowledge please explain ...
0
votes
1answer
29 views

Get individual features importance with XGBoost [closed]

I have trained an XGBoost binary classifier and I would like to extract features importance for each observation I give to the model (I already have global features importance). More specifically, I ...
1
vote
2answers
49 views

Forecasting daily data with zeros in Python

I'm currently testing some forecasts on daily sales quantities. However, out of ~2000 observations I have 16 zeros. How should I approach this? It's mainly Sundays and holidays that holds zero as ...