Questions tagged [python]

Python is a programming language commonly used for machine learning. Use this tag for any *on-topic* question that (a) involves `Python` either as a critical part of the question or expected answer, & (b) is not *just* about how to use `Python`.

Filter by
Sorted by
Tagged with
2
votes
1answer
13 views

Is there a way to determine if one picture is more random than another

I have two pictures. I only want to keep pictures that "display" randomness. Is there a way to distinguish more random from less? In these plots, time is the vertical axis. So, the second ...
1
vote
1answer
51 views

Differential model. Random Forest

I was having discussion with some colleagues and I would like to know some external opinion. Description: We have to decide, for a given person, whether that person would choose item EXPENSIVE or ...
0
votes
1answer
236 views

How to perform MANOVA with categorical independent variables

I have a dataset where the independent variable is sieve size (categorically either 25, 63, 100 or 150), and the dependent variables are length and width. ...
0
votes
1answer
16 views

Huber-M estimator for Robust Linear Regression R Vs Python [closed]

For Robust Linear regression in R using package Huber-M estimator, we use the below code: ...
15
votes
3answers
26k views

Using BIC to estimate the number of k in KMEANS

I am currently trying to compute the BIC for my toy data set (ofc iris (: ). I want to reproduce the results as shown here (Fig. 5). That paper is also my source for the BIC formulas. I have 2 ...
1
vote
0answers
9 views

How to decide whether to keep the duplicate rows or remove them. I have two duplicate records but they refer to two different persons

I am trying to build an NLP model on this data set where I have data from some accidents where I need to predict the Accident Level. There are a total of 13 duplicate rows. But on looking into them I ...
0
votes
0answers
14 views

Averaging the parameters of multiple weibull models

I have multiple different weibull models whose parameters have been estimated on different populations. I would like to "average" those different models into one model based on my beliefs of ...
0
votes
2answers
19 views

Large Negative r-Squared Scores using Cross-Validation

I am working working with the World Happiness Report dataset from Kaggle. When using either cross_val_score or GridSearchCV from ...
0
votes
1answer
497 views

Q-learning agent stucks in an infinite loop

I am simulating a mouse to find a cheese on an empty table. I randomly put a cheese on the table and let the mouse find the cheese without falling off the table. The problem is, in test part, agent ...
0
votes
1answer
74 views

p value in a testing [duplicate]

Why does a test return the p value? I am looking at the Diebold-Mariano-Test (DM Test): https://github.com/johntwk/Diebold-Mariano-Test Basically, this test gives me a DM statistics, and also the p ...
0
votes
0answers
23 views

Word2Vec input implementation to Machine/Deep Learning

I am running an experiment using NLP. I am using Word2vec in order to have a distributed vector representation of the input text and then feed these representations in different Machine Learning (ML) ...
0
votes
1answer
12 views

Does GridSearchCV actually fit the best model to the training data, or do you have to refit after hyperparameter optimisation?

I have this code, with the aim being to develop a neural network with cross validation and hyperparameter optimisation for a regression problem (continuous features, continuous label). ...
0
votes
0answers
15 views

Can I evaluate the performance of a model on the full dataset after hyperparameter tuning on the full dataset?

Below I'm using cross-validation to determine the best hyperparameters. To do that, I'm using my full dataset (X,y). I'm then using the full dataset again to determine the model accuracy with those ...
1
vote
0answers
14 views

Least median square in python

I couldn't find least median square technique on python, any one knows any library? Or how to adapt it from the common linear regression.
1
vote
1answer
16 views

Control group in ML model? [closed]

I have on my Data Frame column with name "group" with values: control_group / campaing_group. And my question is should I use observations from "control" in Machine Learning model ...
0
votes
0answers
27 views

How to measure the impact of a feature or set of features on the classification performance?

I have a dataset as follows: mydata: f1,f2,f3, ..., fn, target s1 34,56,32,...., 43, 0 s2 37,60,33,...., 54, 1 .... sm 89,86,56,...., 90, 0 I ...
2
votes
2answers
62 views

How to understand and interpret multicollinearity in regression models

I am using python to implement different regression models on a fantasy sports dataset. I am using a multivariable dataset which contains 5 independent variables to 2 regression models, which is Lasso ...
0
votes
0answers
5 views

Achieving a Target y efficiently given regression output

Let's say you have profit for a product as your y and it is a function of just marketing spend and price for a given week. You run a linear regression and everything looks good and you get reasonable ...
0
votes
0answers
10 views

Python : Number of features of the model [closed]

I'm getting an error when trying to predict my test data using a random forest model I built in sklearn : ValueError: Number of features of the model must match the input. Model n_features is 298 and ...
0
votes
0answers
30 views

How do I find a minimum sample size for the proportion test / T test?

What I want to figure out is how to find a minimum sample size for comparing a sample to the whole population when the ratio of the sample to the population is less than 5-10% (e.g. 30 to 1000). The ...
-1
votes
0answers
9 views

detecting threshold anomalies using time series and LSTM [closed]

I am using LSTM to detect anomalies of electrical machine input voltage. I have applied the algorithm and it gave me total mae=0.4 for both training and testing phase. Here my doubt is how much should ...
2
votes
1answer
19 views

Visualize tangent plane for mean squared error loss function

I would really appreciate your help on a rather simple issue that I just can't solve on my own. I'd like to visualize gradient descent for a simple linear regression problem with two parameters $\...
0
votes
1answer
133 views

is there a way to train the models in batches? i m not talking about deep learning

I have 800,000 records. The training with all the records at once is not possible. Is there a way i can train a model on 50k records then continue training on next 50k records and so on... basically i ...
0
votes
0answers
5 views

Compare variability of a panel of measures collected at five different timepoints

I have a panel of 60 measures that have been collected from 14 people on 5 consecutive days I have a second panel of 40 different measures that have been performed under the same conditions. Ideally, ...
0
votes
0answers
6 views

Fast searching for subsequences with highest correlation with pattern

Searching for highly correlated subsequence in large dataset is time consuming. I deal with 3Gb dataset which could be represented as single, one-dimensional time series. I've tried to use MASS (Mueen'...
2
votes
1answer
20 views

Measuring performance using cross-validation on the full dataset after training on a subset of that dataset?

In my Python code below, I'm measuring the performance of my model using cross-validation. I'm doing it with my full dataset (X,y). This is after I trained on a subset of that data (X_train, y_train). ...
1
vote
1answer
24 views

Is there any relationship between image input dimension, filter/kernel size and feature map?

Is there any relation between image input dimension (height, width), filter/kernel size and feature map? If for example I have this code: ...
1
vote
1answer
28 views

Why does Statsmodels' ARMAX function fit a linear regression model

I want to fit an AR(MA)X model to some data. The application is to describe the temperature of a single room. As disturbances/exogenous inputs to the room temperature is solar radiation, outdoor air ...
0
votes
1answer
133 views

Dealing with a dataset having target values on different scales?

I am currently working on a dataset having 10 features and one continuous target variable. One of the features is 'Country' , in which there are seven unique values [Argentina ,Denmark , France...etc]....
2
votes
1answer
39 views

Should the day of the week variable be splitted into 7 columns or is one enough? Time-series forecast

I have time-series data, in which I added a variable "dayofweek" with the day of the week varying from $0$ (Monday) to $6$ (Sunday) (Python's default). I'm using boosting models like GBM and ...
0
votes
0answers
9 views

StyleGAN as conditional GAN? [closed]

Can you modify the StyleGAN, so that you can generate images only from one specific class?
1
vote
0answers
13 views

Interrater reliability with multi-rater, multi-label dataset

I'm trying to calculate interrater reliability with 3 coders, and each question is multiple choice. An example tagging would be: ...
0
votes
0answers
7 views

How to predict near future value beyond the available data from my uploaded file in flask python [closed]

I m new to deployment for LSTM in flask python. I would like to predict the near future value using the current date from the uploaded file in flask python. This issues already stuck me for a while. ...
0
votes
1answer
16 views

what does the predict() method do in the lifeline package?

I see people using lifeline package and they use predict() method after fitting their data into kaplan_meier_fitter(). I'm not quite sure what predict method does? Does it predict the survival ...
1
vote
0answers
19 views

ARIMA variance of the residual values estimation in Python

I'm trying to estimate the variance of the error term in the best ARIMA(p,0,q) -- I'm making the hypotesis that my time series is stationary -- be with 'statsmodels.tsa.arima.model' command in Python. ...
2
votes
0answers
20 views

Random Forest Regressor Overfitting

I'm implementing a Random Forest Regressor using Random Forest algorithm in scikit-learn. Please consider the following performance in terms of RMSE: ...
0
votes
2answers
1k views

Fit Hawkes process to 1d data using python package TICK

How can one fit the 1-dimensional Hawkes process with exponential kernel to the experimental 1d dataset (t1,t2,t3...tn) and check the goodness-of-fit via tick python3 package? I found on official ...
0
votes
1answer
234 views

Statistical significance with Newey–West standard errors with a one-lag correction used

I am trying to do statistical test to prove that the returns of a certain equity are statistically significant so that I can reject the null hypothesis that the mean (of returns) is equal to 0. I have ...
3
votes
1answer
30 views

How to make Kalman filter results equivalent to linear regression?

Kalman filter appears to be a powerful estimator for linear problems. I understand one can tune the performance by adjusting parameters like process noise and measurement noise. Is it possible to ...
1
vote
0answers
22 views

Need help with questions about modeling and regression [closed]

I have two questions that I am struggling with. It's not coding questions but more like theory questions. Fill in the blanks ________(1) has an output that is dependent on initialization, whereas ...
1
vote
2answers
299 views

A hierarchical Bayesian model in pymc3

Suppose we have the following model: $X$ unobserved $Y$ such that $Y|X \sim \mathcal{N}(X,\sigma^2)$, observed $Z$ such that $Z|X \sim \mathcal{B}(1,X)$, observed and suppose, given observed data $...
1
vote
1answer
1k views

Determining samples sizes for an A/B test using z-test and chi-square test

Assumptions I consider an A/B test where there is a control group and a variant group. Each observation can either be true (converted) or false (not converted). I evenly and randomly split the ...
0
votes
0answers
9 views

Huge MAPE and MSE in ARIMA in Python, how to decrease these metrics? [closed]

I have enormous MAPE and MSE in my ARIMA model in Python, How can I decrease tehese errors ? MSE = 3328168.201388124 MAPE = 54533.6299854454
0
votes
0answers
11 views

ANN Train and Test set accuracy becomes 1.0 after second Epoch Keras Classifier

I have the following classification model built: ...
0
votes
1answer
23 views

Why macro F1 measure can't be calculated from macro precision and recall?

I'm interested in calculating macro f1-score by macro precision and recall manually. But the results aren't equal. What is the difference in the final formula between f1 and f1_new in code? Would you ...
1
vote
0answers
12 views

Correlation between two time-lagged factors

I have the following problem: I have a large dataset of a health facility where every (emergency) call was recorded.The calls have already been counted per day and transferred to a new table Besides ...
1
vote
0answers
9 views

What kind of regularization can I use for CNN aside from L1/L2/Dropout? [closed]

I am building a CNN to estimate a sequence of pitches existed in a song with this architecture: ...
2
votes
1answer
24 views

Should the data be balanced for Kaplan-Meier survival curve?

I have a dataset that contains some categorical info about my users (like gender and country), whether they have churned or not and what's the time from their subscription to churn. I want to use ...

1
2 3 4 5
75