Questions tagged [pandas]

Python library for data manipulation, implementing R-style data frames.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

When testing a trained model, what do you use for x_test and y_test when you started with a separate test set without a target variable? [closed]

In this html file, I'm trying to get a CSV file by doing the model testing on a test dataset - w/o a target variable - that I was given at the beginning of a project. I'd greatly appreciate it if ...
0 votes
0 answers
9 views

How to find whether two years of time series data are statistically significantly different from each other

I have several years of time series data with data points (# of cars) taken every 5 minutes (have filled in any missing data points). Wondering the best way to check whether there is a statistical ...
1 vote
0 answers
32 views

Time Series Seasonality and Frequency

I am working with a time series that indicates the number of times a user logs into an application per minute. I want to find an algorithmic way (in Python) that would detect if there is any pattern ...
0 votes
0 answers
15 views

Incremental value of a categorical variable

The question is to calculate the incremental value of running an advertisement for a client. I have provided a sample of my dataset. ...
0 votes
0 answers
22 views

How can reduce improve my ANN accuracy and reduce overfitting?

My ANN model produces classic overfitting characteristics, producing high R2 values (90-99%) but low accuracy scores (10-40%). I'm currently inputting 28655 data entries, using 8 input features to ...
's user avatar
2 votes
2 answers
21 views

Transformation of mix normal and skew features [duplicate]

I have a weather dataset containing four features that are continuous values. Temperature is almost normal, but precipitation is highly negatively skewed. In addition, wind speed and humidity are ...
  • 51
0 votes
0 answers
22 views

How does Pandas Interpolate Time work?

How does Pandas Interpolate method = time function work? From my understanding, it sounds very similar to an Exponential Moving Average, but it gives me different results, even when I play around with ...
  • 121
1 vote
1 answer
32 views

running multiple t-tests across large dataset

I'm trying to work out if a t-test is the most appropriate in this situation: I have a data frame which looks like the one below but my data frame has aprox 37,000 rows. I'd like to run a t-test on ...
  • 153
1 vote
1 answer
327 views

Implementing a Conditional Logit in Python StatsModels

I have a dataframe with some horseracing data, and each row contains a predicted speed rating for each of the runners. I am now trying to convert that information into a winning probability for each ...
  • 31
0 votes
0 answers
30 views

Why my default random forest works better then grid search cv random forest?

I am having problem understanding why the random forest method without the GridSearchCv works better then the one with it, can someone explain it please? (they both use the same X,y of course) the ...
0 votes
0 answers
27 views

Non linear knapsack optimization

Problem: Primarily problem to solve: Allocate budget for most revenue given ROIs at a given investment. Secondarily problem: Minimize budget to meet a certain revenue threshold. All while having some ...
  • 1,519
0 votes
1 answer
42 views

Understanding the idx column in h2o metrics [closed]

h2o model metrics results report generated like this. ...
  • 3
0 votes
0 answers
216 views

RMSE, MAE and MAPE reduced but RMSPE increased after removing outliers in linear regression

I am using a linear regression model with 465 datapoints to predict crop yield of wheat. The data points are spilt into training and testing sets in 80:20 ratio. Initially, I tried linear regression ...
1 vote
0 answers
18 views

How to deal with the miss# how to deal with the missing values in time series forecasting?

I have dataset in which there are records of stock market hour by hour. There are some missing values like each day should posses 24 values as it is hour by hour but I don't have the value of some ...
  • 121
0 votes
0 answers
59 views

How to measure relationship between two non-numerical variables in Python?

I am new to Data Science (and programming) and try to understand the data of the project I am currently working in. I want to understand the relationships between two categorical variables and also ...
1 vote
0 answers
112 views

What to do with missing days from timeseries when checking for seasonality?

I have a dataset which looks like this, with pm25 values and timestamp as index: ...
1 vote
0 answers
251 views

How to increase the accuracy of a multivariate LSTM network?

I am using a pollution dataset with values for pm2.5, pm10 and pm1 as features and I am predicting the values for the pm2.5. I built an LSTM network but the predicted values are quite from the real ...
0 votes
0 answers
164 views

Converting a code for 5-fold cross validation to stratified 5-fold cross validation for continuous target variable

Do you know how I can convert my code so it can do stratified 5-fold cross validation on a continuous target? df['score'] or y is a continuous variable. ...
1 vote
1 answer
32 views

How to predict a class between uneven datasets

I am trying to predict obesity using two datasets. The first (Diet) has 20k values in two columns: Diet [ID, Calories(float)] The second dataset also has two columns: Obesity [ID, Level(1,2,3)] ...
0 votes
0 answers
19 views

correlation of sample subset high but correlation of entire sample low?

This is more of a theoretical question but I'll use code to illustrate it. I have a dataset df with two numerical columns 'real' ...
  • 207
1 vote
1 answer
178 views

How to scale multidimensional time series data per group

I am dealing with panel data and want to scale it in order to use it for some ML models: id year A B C 1 2000 3,539,101 265.152 .0683649 1 2001 3,539.101 2,485.833 .0683649 1 2002 3,539.101 2,939....
4 votes
1 answer
3k views

Why am I getting different results on a prediction using the same Keras model and input? [closed]

posting here is my last resort cause I can't find anything like it online. I trained a model to classify embeddings into categories (a simple three layer Dense neural network). I'm using pandas for ...
  • 83
0 votes
1 answer
40 views

How do I roll data within groups in a dataframe? [closed]

The original dataframe is like df1. df1 = pd.DataFrame({'group': [1,1,1,1,1,2,2,2,2,2], 'in': list(range(1,11)), 'out': [0,0,0,1,1,0,0,0,1,1]}) df1 The output I ...
0 votes
1 answer
155 views

Dimensionality Reduction Techniques Ordinal Variables on different scales

I have a data set that is a mix between Ordinal variables and numerical variables. The problem is that the ordinal variables are on different scales, such as 0-2, 1-4. The data set has 35 variables. I ...
0 votes
2 answers
85 views

I'm performing a multi linear regression on the The Oxford Covid-19 Government Response Tracker. I have a couple of doubts about the process

I hope this is the right place to ask this. I'm currently working my way through a dataset and performing Multiple Linear Regression on it. The data is for Oxford Governement Response Tracker for the ...
1 vote
0 answers
66 views

Creating box plots from a count (or frequency) table

I have a census data where the number of population at each age is indicated and I am trying to create a box plot for the age of the entire population. At first I tried to repeat the 'age' by the ...
0 votes
0 answers
60 views

Using pandas to check stability in data

I have a conversions table & I am trying to identify Are the conversions in the 'conversions' table stable over time? to identify Any pattern? ...
  • 259
2 votes
2 answers
71 views

How do i interpret this Scatter Plot

I'm researching on the relationship between Scholarship and years. I have scatter plot below. The x axis is equal to the total amount of scholarship for a student. The y axis show the term a student ...
  • 21
0 votes
1 answer
362 views

How to do feature engineering with scikit learn pipelines? [closed]

When doing preprocessing I've always used pandas to impute, encode, or scale my data. In other words, I've done all of the steps "manually". However, this takes a long time, generates a lot ...
  • 641
0 votes
2 answers
136 views

How to name and treat such a distribution of extreme values / outliers?

In this sample data I have 4600 teenagers with their telephone calling frequency per week. Most of them are "normal" (max 5) but some are extreme with up to 75 calls per week. I learned to ...
  • 203
1 vote
0 answers
54 views

Imputation Methods for Multiple Variables

I am training Recurrent Neural Network models (including LSTMs) on a dataset that includes 6-10 variables. Each variable is a properly formatted numerical measurement (ie: length, pressure, ...
  • 608
0 votes
0 answers
22 views

Univariate multi-step prediction model for missing value imputation

I have a dataframe with columns of timestamp and energy usage. The timestamp is taken for every min of the day i.e., a total of 1440 readings for each day. I have few missing values in the data frame. ...
0 votes
0 answers
16 views

How to classify the features instead of rows

I have a dataset about some students' e-learning activities and their grades respectively. So in rows, I have the students and in columns (features) I have the activities. I want to classify the ...
1 vote
0 answers
123 views

Normalization PCA for dataframes?

I would like to know what is the correct way to normalize a dataframe before applying PCA. I have found two options and I got different results for each one: ...
1 vote
1 answer
366 views

What dummy variables to set for US state-and-year Differences-in-Differences model

As a sort of personal experiment, I'm trying to run a differences-in-differences (DiD) model on US state-level firearms restrictions and violent crime, to see if changes in the former impact the ...
0 votes
2 answers
78 views

Account for variable to add more variation to predictor variable scipy

Is there a test or method to account for the interaction between predictor variables to influence the outcome in a regression model. Specifically, if a variable increases in value, it may add more ...
  • 87
1 vote
1 answer
147 views

One hot coding in Train Validation and Test set (Production data) [closed]

For example I have below train set. name values 0 Tony 100 1 Smith 110 2 Sam 120 3 Shane 130 4 Sam 140 5 Ram 160 After ...
  • 111
1 vote
0 answers
65 views

What are the methods to find the maximum data size (in python and pandas) that can be handled smoothly by a system before even executing a dataset

I want to know about the various ways to calculate (or approximate) the maximum data size (in python and pandas) that can be handled smoothly by the current processor or the system as a whole before ...
0 votes
1 answer
226 views

Joining train and test dataset when preprocessing data with encoders

So I've been looking at several beginner Kaggle kernels for the 'Titanic - Machine Learning from Disaster' competition. In these kernels, I noticed that sometimes they combine train and test data, ...
0 votes
1 answer
250 views

Similarity/dissimilarity matrix over classes

I am trying to get something like a confusion matrix for different classes, but without training a model. The idea is to use some kind of distance between classes. The data set is like this, just for ...
2 votes
1 answer
41 views

how to know which features contribute more in the class labels

I have a data of million record, and each record has a label the sum of all labels in my data is 324521 label. I don't want to do classification; I want only to know which features contribute more in ...
  • 121
1 vote
0 answers
79 views

Mixed Linear Model Regression missing output

I'm running a mixed linear model regression and I get some missing output and for the one I get it doesn't make too much sense (or at least I think so as the p-value is always equal to 1). The columns ...
2 votes
1 answer
117 views

Difference between quantiles

As far as I understood from its definition, quantile borders should divide a dataset into equal parts (or at least into almost equal parts, if the dataset doesn't have enough entries or has an odd ...
0 votes
0 answers
136 views

How do I select number of bins to discretize the data?

So, I have been pondering on how I can select the number of bins in a dataset? I know we have different methods for selecting number of bins for histogram, but how do I select number of bins when ...
1 vote
1 answer
932 views

How can I develop a 0-100 composite or index score using multiple Z scores from multiple independent variables?

I'm really interested in index scores like the human development index or economic freedom index where they rank things on a 0-100 scale based off of a bunch of different variables (e.g. press freedom,...
1 vote
0 answers
207 views

How to programmatically differentiate between MCAR (Missing Completely at Random), MAR(Missing at Random), and MNAR(Missing Not at Random) in python

I found the following code in R. Im not sure how much does it serve this purpose. But I want to implement this in python. How does this mostly convert to?? I also want to differentiate between all ...
0 votes
1 answer
85 views

Outlier treatment in sales data comparisson

I'm working for a webshop and I'm trying to compare month for month this year with same month last year. In order to get a better picture of growth in revenue/loss in revenue I use z-scores to detect ...
2 votes
1 answer
60 views

Pearson correlation ratio

I try to build linear regression model. First of all I choose variables mostly correlated with target variable (price) using Pearson correlated. It is 9 variables. Then, I check correlation between ...
2 votes
1 answer
2k views

How to remove correlated features?

I have a small dataset (200 samples and 22 features) and I am trying to solve a binary classification problem. All my features are continuous and lie on a scale of 0-1. I computed the correlation ...
  • 123
1 vote
1 answer
72 views

Normality test of non "normally distributed" values prior to Anova test

I am trying to perform a normality test to multiple continuous values before doing an anova test. The p-value I am getting for the data does not make much sense and I want to make sure I am not ...