Questions tagged [pandas]

Python library for data manipulation, implementing R-style data frames.

Filter by
Sorted by
Tagged with
0 votes
0 answers
6 views

Group by in stata [closed]

Given an existing table that contains a list of observations: colA, colB, value 0, 0, 11 0, 0, 12 0, 1, 11 0, 1, 22 1, 1, 33 How do I ...
user avatar
0 votes
0 answers
14 views

Non linear knapsack optimization

Problem: Primarily problem to solve: Allocate budget for most revenue given ROIs at a given investment. Secondarily problem: Minimize budget to meet a certain revenue threshold. All while having some ...
user avatar
  • 1,489
-1 votes
0 answers
14 views

Solution for Session Crash in google Colab by applying KFold Cross Validation in Python [closed]

Is there any way to modify this below-mentioned code? As I am applying 5-Fold cross-validation by using the below code in Google Colab free version. My data frame consists of 5000 rows and 9000 ...
user avatar
0 votes
1 answer
18 views

Understanding the idx column in h2o metrics [closed]

h2o model metrics results report generated like this. ...
user avatar
  • 3
0 votes
0 answers
98 views

RMSE, MAE and MAPE reduced but RMSPE increased after removing outliers in linear regression

I am using a linear regression model with 465 datapoints to predict crop yield of wheat. The data points are spilt into training and testing sets in 80:20 ratio. Initially, I tried linear regression ...
user avatar
1 vote
0 answers
17 views

How to deal with the miss# how to deal with the missing values in time series forecasting?

I have dataset in which there are records of stock market hour by hour. There are some missing values like each day should posses 24 values as it is hour by hour but I don't have the value of some ...
user avatar
  • 121
0 votes
0 answers
50 views

How to measure relationship between two non-numerical variables in Python?

I am new to Data Science (and programming) and try to understand the data of the project I am currently working in. I want to understand the relationships between two categorical variables and also ...
user avatar
1 vote
0 answers
47 views

What to do with missing days from timeseries when checking for seasonality?

I have a dataset which looks like this, with pm25 values and timestamp as index: ...
user avatar
0 votes
0 answers
70 views

How to increase the accuracy of a multivariate LSTM network?

I am using a pollution dataset with values for pm2.5, pm10 and pm1 as features and I am predicting the values for the pm2.5. I built an LSTM network but the predicted values are quite from the real ...
user avatar
0 votes
0 answers
84 views

Converting a code for 5-fold cross validation to stratified 5-fold cross validation for continuous target variable

Do you know how I can convert my code so it can do stratified 5-fold cross validation on a continuous target? df['score'] or y is a continuous variable. ...
user avatar
1 vote
1 answer
31 views

How to predict a class between uneven datasets

I am trying to predict obesity using two datasets. The first (Diet) has 20k values in two columns: Diet [ID, Calories(float)] The second dataset also has two columns: Obesity [ID, Level(1,2,3)] ...
user avatar
0 votes
0 answers
13 views

correlation of sample subset high but correlation of entire sample low?

This is more of a theoretical question but I'll use code to illustrate it. I have a dataset df with two numerical columns 'real' ...
user avatar
  • 207
1 vote
1 answer
37 views

How to scale multidimensional time series data per group

I am dealing with panel data and want to scale it in order to use it for some ML models: id year A B C 1 2000 3,539,101 265.152 .0683649 1 2001 3,539.101 2,485.833 .0683649 1 2002 3,539.101 2,939....
user avatar
1 vote
1 answer
1k views

Why am I getting different results on a prediction using the same Keras model and input?

posting here is my last resort cause I can't find anything like it online. I trained a model to classify embeddings into categories (a simple three layer Dense neural network). I'm using pandas for ...
user avatar
0 votes
1 answer
27 views

How do I roll data within groups in a dataframe? [closed]

The original dataframe is like df1. df1 = pd.DataFrame({'group': [1,1,1,1,1,2,2,2,2,2], 'in': list(range(1,11)), 'out': [0,0,0,1,1,0,0,0,1,1]}) df1 The output I ...
user avatar
0 votes
1 answer
99 views

Dimensionality Reduction Techniques Ordinal Variables on different scales

I have a data set that is a mix between Ordinal variables and numerical variables. The problem is that the ordinal variables are on different scales, such as 0-2, 1-4. The data set has 35 variables. I ...
user avatar
0 votes
2 answers
72 views

I'm performing a multi linear regression on the The Oxford Covid-19 Government Response Tracker. I have a couple of doubts about the process

I hope this is the right place to ask this. I'm currently working my way through a dataset and performing Multiple Linear Regression on it. The data is for Oxford Governement Response Tracker for the ...
user avatar
0 votes
0 answers
21 views

How to make relationships between dataframes rows?

I have a such big json file with many nested dictionaries, lists etc. My approach is to divide them to standalone dataframes. There occurs a problem - how can I make relationships between them? Lets ...
user avatar
  • 1
0 votes
0 answers
11 views

Failing to converge when combining feature dataframe with word vectors

I am doing a basic text classification task (kaggle disaster tweets). My approach is to use both the vectorized tweets (via count vectorizer) as well as their sentiment as features for a model (e.g. ...
user avatar
0 votes
0 answers
24 views

Distribution of data in clusters

I am using python to perform Agglomerative Hierarchical Clustering(AHC) and K-mode to cluster my data in the Jupyter. My data has 8 features(columns) and most of features are skewed. For example, ...
user avatar
  • 31
1 vote
0 answers
45 views

Creating box plots from a count (or frequency) table

I have a census data where the number of population at each age is indicated and I am trying to create a box plot for the age of the entire population. At first I tried to repeat the 'age' by the ...
user avatar
0 votes
0 answers
43 views

Using pandas to check stability in data

I have a conversions table & I am trying to identify Are the conversions in the 'conversions' table stable over time? to identify Any pattern? ...
user avatar
  • 259
2 votes
2 answers
68 views

How do i interpret this Scatter Plot

I'm researching on the relationship between Scholarship and years. I have scatter plot below. The x axis is equal to the total amount of scholarship for a student. The y axis show the term a student ...
user avatar
  • 21
0 votes
1 answer
258 views

How to do feature engineering with scikit learn pipelines? [closed]

When doing preprocessing I've always used pandas to impute, encode, or scale my data. In other words, I've done all of the steps "manually". However, this takes a long time, generates a lot ...
user avatar
  • 641
0 votes
2 answers
80 views

How to name and treat such a distribution of extreme values / outliers?

In this sample data I have 4600 teenagers with their telephone calling frequency per week. Most of them are "normal" (max 5) but some are extreme with up to 75 calls per week. I learned to ...
user avatar
  • 63
1 vote
0 answers
35 views

Imputation Methods for Multiple Variables

I am training Recurrent Neural Network models (including LSTMs) on a dataset that includes 6-10 variables. Each variable is a properly formatted numerical measurement (ie: length, pressure, ...
user avatar
  • 608
0 votes
0 answers
19 views

Univariate multi-step prediction model for missing value imputation

I have a dataframe with columns of timestamp and energy usage. The timestamp is taken for every min of the day i.e., a total of 1440 readings for each day. I have few missing values in the data frame. ...
user avatar
0 votes
0 answers
16 views

How to classify the features instead of rows

I have a dataset about some students' e-learning activities and their grades respectively. So in rows, I have the students and in columns (features) I have the activities. I want to classify the ...
user avatar
1 vote
0 answers
95 views

Normalization PCA for dataframes?

I would like to know what is the correct way to normalize a dataframe before applying PCA. I have found two options and I got different results for each one: ...
user avatar
0 votes
0 answers
5 views

How do I set up my data to properly analyze skewness?

I'm trying to identify skew in my data set using pandas/scipy and I'd like to make sure that I'm setting up the data correctly. For example, if my data set is a list of words: ...
user avatar
  • 143
1 vote
1 answer
159 views

What dummy variables to set for US state-and-year Differences-in-Differences model

As a sort of personal experiment, I'm trying to run a differences-in-differences (DiD) model on US state-level firearms restrictions and violent crime, to see if changes in the former impact the ...
user avatar
0 votes
0 answers
15 views

How to check performance of the linear regression algorithm and improve?

Finally after countless tries I have successfully implemented Linear Regression using tensorflow on this dataset to predict Laptop prices after given specs. I plan to use this on a web app that can ...
user avatar
0 votes
2 answers
76 views

Account for variable to add more variation to predictor variable scipy

Is there a test or method to account for the interaction between predictor variables to influence the outcome in a regression model. Specifically, if a variable increases in value, it may add more ...
user avatar
  • 87
1 vote
1 answer
74 views

One hot coding in Train Validation and Test set (Production data) [closed]

For example I have below train set. name values 0 Tony 100 1 Smith 110 2 Sam 120 3 Shane 130 4 Sam 140 5 Ram 160 After ...
user avatar
  • 111
1 vote
0 answers
33 views

What are the methods to find the maximum data size (in python and pandas) that can be handled smoothly by a system before even executing a dataset

I want to know about the various ways to calculate (or approximate) the maximum data size (in python and pandas) that can be handled smoothly by the current processor or the system as a whole before ...
user avatar
0 votes
1 answer
138 views

Joining train and test dataset when preprocessing data with encoders

So I've been looking at several beginner Kaggle kernels for the 'Titanic - Machine Learning from Disaster' competition. In these kernels, I noticed that sometimes they combine train and test data, ...
user avatar
0 votes
1 answer
235 views

Similarity/dissimilarity matrix over classes

I am trying to get something like a confusion matrix for different classes, but without training a model. The idea is to use some kind of distance between classes. The data set is like this, just for ...
user avatar
2 votes
1 answer
24 views

how to know which features contribute more in the class labels

I have a data of million record, and each record has a label the sum of all labels in my data is 324521 label. I don't want to do classification; I want only to know which features contribute more in ...
user avatar
  • 121
1 vote
0 answers
63 views

Mixed Linear Model Regression missing output

I'm running a mixed linear model regression and I get some missing output and for the one I get it doesn't make too much sense (or at least I think so as the p-value is always equal to 1). The columns ...
user avatar
2 votes
1 answer
98 views

Difference between quantiles

As far as I understood from its definition, quantile borders should divide a dataset into equal parts (or at least into almost equal parts, if the dataset doesn't have enough entries or has an odd ...
user avatar
0 votes
0 answers
71 views

How do I select number of bins to discretize the data?

So, I have been pondering on how I can select the number of bins in a dataset? I know we have different methods for selecting number of bins for histogram, but how do I select number of bins when ...
user avatar
1 vote
1 answer
673 views

How can I develop a 0-100 composite or index score using multiple Z scores from multiple independent variables?

I'm really interested in index scores like the human development index or economic freedom index where they rank things on a 0-100 scale based off of a bunch of different variables (e.g. press freedom,...
user avatar
1 vote
0 answers
174 views

How to programmatically differentiate between MCAR (Missing Completely at Random), MAR(Missing at Random), and MNAR(Missing Not at Random) in python

I found the following code in R. Im not sure how much does it serve this purpose. But I want to implement this in python. How does this mostly convert to?? I also want to differentiate between all ...
user avatar
0 votes
1 answer
47 views

Outlier treatment in sales data comparisson

I'm working for a webshop and I'm trying to compare month for month this year with same month last year. In order to get a better picture of growth in revenue/loss in revenue I use z-scores to detect ...
user avatar
2 votes
1 answer
54 views

Pearson correlation ratio

I try to build linear regression model. First of all I choose variables mostly correlated with target variable (price) using Pearson correlated. It is 9 variables. Then, I check correlation between ...
user avatar
2 votes
1 answer
1k views

How to remove correlated features?

I have a small dataset (200 samples and 22 features) and I am trying to solve a binary classification problem. All my features are continuous and lie on a scale of 0-1. I computed the correlation ...
user avatar
  • 123
1 vote
1 answer
59 views

Normality test of non "normally distributed" values prior to Anova test

I am trying to perform a normality test to multiple continuous values before doing an anova test. The p-value I am getting for the data does not make much sense and I want to make sure I am not ...
user avatar
1 vote
0 answers
32 views

Crossing categorical features that are stored as integers

Newbie here. I'm experimenting with the following dataset: https://archive.ics.uci.edu/ml/datasets/Teaching+Assistant+Evaluation Data Set Information: The data consist of evaluations of teaching ...
user avatar
3 votes
1 answer
780 views

What are good methods to deal with outliers when calculating mean of data?

I have a dataframe with yearly energy uses of buildings over 5 years. In order to have a representative yearly energy use for data modelling, I'll have to take the mean of those data. As the data can ...
user avatar
1 vote
0 answers
375 views

PCA returns duplicated features for different components

I performed (sklearn) PCA on a (1416960,140) pandas DataFrame. The resulting components_ attribute is a matrix where each ...
user avatar
  • 11