Skip to main content

Questions tagged [pandas]

Python library for data manipulation, implementing R-style data frames.

Filter by
Sorted by
Tagged with
0 votes
1 answer
20 views

List of 100 teams, each team has 10 players, each player has 10 performance metrics. How can I structure this using pandas Dataframes? [closed]

Effectively I already have one dataframe with team names against player names and another with player names against performance metrics
Heiko's user avatar
  • 37
0 votes
0 answers
17 views

Showcasing Generalized data with data using matplotlib

Price of Bread in Afghanistan with Average We are currently looking into prices of food throughout the world. We want to show the general trend of a price in a certain country (In the example ...
user avatar
1 vote
1 answer
38 views

Reason for high MSE and negative R square value

I am getting really high MSE and negative R square value. Dataset: https://docs.google.com/spreadsheets/d/1moTZS_LgOn6d74NC44i9lVcWchj-abVx/edit?usp=sharing&ouid=100514649347129021200&rtpof=...
user avatar
0 votes
1 answer
51 views

Feature selection using backward feature selection in scikit-learn and PCA

I have calculated the scores of all the columns in my dataframe, which has 312 columns and 650 rows, using PCA. I used the following code: ...
Mostafa Bouzari's user avatar
3 votes
1 answer
229 views

How to rank data based on multiple variables

I need help in ranking data, says car models in this case, based on multiple variables. For some variables (eg. mpg), the higher the better. For some variables (eg. car age), the lower the better. For ...
zZzZ's user avatar
  • 79
0 votes
0 answers
11 views

Question about the effects of cardinality changing on the model's training

I'm analyzing a dataset that contains a feature 'street_name' with 5980 unique values. I used the LeaveOneOutEncoder class for encoding, but I noticed that the cardinality reduced a lot. There are now ...
Antonio Caipora's user avatar
3 votes
1 answer
38 views

How to find correlations of the features in a dataframe? The features are of mixed types (nominal, ordinal, discrete, and continuous)

I am working with python using pandas, and seaborn libraries. I have a dataframe, that I am using for some machine learning. My dataframe has a target variable, along with several other features. ...
letdatado's user avatar
  • 367
0 votes
1 answer
93 views

how can I evaluate the unbalanced data set

The output here shows the titles and descriptions of the comments written for the evaluation of the top 100 books in amazon with nltk vader, and the total reviewer rating for those analyzes, but there ...
Tunahan Deniz's user avatar
0 votes
0 answers
91 views

Kendall's Tau Correlation for dataset with multiple types (binary, categorical, ordinal, continuous)

I have a dataset of many variables of varying data types (binary, categorical, ordinal, continuous), and I want to find the correlations between them. I encoded the binary as (0,1) and one-hot encoded ...
cjwehr's user avatar
  • 1
1 vote
0 answers
27 views

Distributions in the context of geo-spatial data [closed]

I have geo spatial data with lat, lon of user activity. I have divided this data into geohashes with precision 6. For each geohash, how do I measure the distribution of points within them? How do we ...
kll's user avatar
  • 11
0 votes
0 answers
11 views

Formal terminology: metafeatures then groupby (GROUP BY) operation

In ML training and other analytics I often combine features to produce a 'metafeature' and then perform a 'groupby' (pandas) or 'GROUP BY' (SQL) query. What is the technical term for this operation? ...
M__'s user avatar
  • 113
0 votes
0 answers
170 views

How to get Pandas EWMA results to match online EWMA calculator results?

I wasn't getting much help in the programming forums and thought to try here. I am testing Pandas exponential weighted moving average (EWMA) function: ...
Ray Zhang's user avatar
  • 191
0 votes
0 answers
27 views

How to proceed when correlation plot doesn't show as much multicorrelation as is seen by statsmodels variance_inflation_factor in a regression task?

I am working with the kaggle Blueberry Yield prediction dataset. There are 17 columns including the target variable. Below is the correlation heat map: It can be seen that multiple features are ...
forest's user avatar
  • 101
1 vote
0 answers
54 views

How to structure combinations of dataframes for regression, without corruption/loss?

I have a data set, redacted sample below. My goal is linear regression. My question is: Have I created unintended results, due to how I structured the df, using concat and/or div? For example, ...
Megan Perry's user avatar
0 votes
0 answers
59 views

Handling Imbalance Categorical Value in A Dataframe Column

I have a dataframe contains id, gender, and class, for example: ...
user avatar
3 votes
1 answer
182 views

Determine correlation between two categorical columns with lots of data

I have a large dataset containing country names and names of musicians like this, with more than 50.000 rows: Country Musician australia Jimmy Barnes australia Grinspoon england Giles united ...
karkraeg's user avatar
  • 131
1 vote
1 answer
287 views

Encode multiple values of an attributes in Pandas

I have a dataset and one of the attributes of the dataset is Race. People have multiple races on the dataset. The values for the attribute Race are following ...
Encipher's user avatar
  • 175
0 votes
1 answer
37 views

Interpretation T-test

First of all, I guess I should start my question exposing what is the data about and how it looks right now, I have the following pd.Series object. They are basically the daily conversion of a few ...
INGl0R1AM0R1's user avatar
2 votes
1 answer
49 views

How do I normalize this feature, I've tried almost everything

I'm trying to normalize this skewed data as part of data preprocessing, but it doesn't normalize no matter which transformation I use to the point it's making me crazy :') . The methods which I've ...
Burhan C's user avatar
2 votes
2 answers
77 views

Can we change the estimated population mean in Hypothesis Testing?

My goal is to learn hypothesis testing. My understanding is that we do not have the population data. Therefore, we do not know the estimated population mean. So, we guess the population mean. If that ...
Jason Rich Darmawan's user avatar
0 votes
0 answers
58 views

What is hypothesis testing one sample one sided?

My goal is to learn hypothesis testing one sample one sided. Honestly, I don't get it. Hypothesis testing one sample RIGHT SIDED ...
Jason Rich Darmawan's user avatar
0 votes
0 answers
55 views

How to find optimum column values when they affect another column value in time series data?

Original data frame spans a million rows and has an year's data. Here's top ten rows: Code: ...
Mystic's user avatar
  • 1
0 votes
0 answers
37 views

Border values at which timeseries decrease and increase

I have a timeseries data of signals in stock market ([-1, 1]) and I want to find mean values at which I have down trend and upwards trend. I already used Moving ...
Michael's user avatar
  • 101
1 vote
0 answers
18 views

How to find whether two years of time series data are statistically significantly different from each other

I have several years of time series data with data points (# of cars) taken every 5 minutes (have filled in any missing data points). Wondering the best way to check whether there is a statistical ...
LizCat2222's user avatar
1 vote
0 answers
140 views

Time Series Seasonality and Frequency

I am working with a time series that indicates the number of times a user logs into an application per minute. I want to find an algorithmic way (in Python) that would detect if there is any pattern ...
Iamtrying's user avatar
0 votes
0 answers
90 views

How can reduce improve my ANN accuracy and reduce overfitting?

My ANN model produces classic overfitting characteristics, producing high R2 values (90-99%) but low accuracy scores (10-40%). I'm currently inputting 28655 data entries, using 8 input features to ...
user avatar
2 votes
2 answers
60 views

Transformation of mix normal and skew features [duplicate]

I have a weather dataset containing four features that are continuous values. Temperature is almost normal, but precipitation is highly negatively skewed. In addition, wind speed and humidity are ...
Asa Ya's user avatar
  • 73
0 votes
0 answers
531 views

How does Pandas Interpolate Time work?

How does Pandas Interpolate method = time function work? From my understanding, it sounds very similar to an Exponential Moving Average, but it gives me different results, even when I play around with ...
Kelsey's user avatar
  • 121
3 votes
1 answer
478 views

running multiple t-tests across large dataset

I'm trying to work out if a t-test is the most appropriate in this situation: I have a data frame which looks like the one below but my data frame has aprox 37,000 rows. I'd like to run a t-test on ...
Chip 's user avatar
  • 173
2 votes
1 answer
3k views

Implementing a Conditional Logit in Python StatsModels

I have a dataframe with some horseracing data, and each row contains a predicted speed rating for each of the runners. I am now trying to convert that information into a winning probability for each ...
AMJ's user avatar
  • 51
0 votes
1 answer
116 views

Understanding the idx column in h2o metrics [closed]

h2o model metrics results report generated like this. ...
John's user avatar
  • 3
0 votes
0 answers
569 views

RMSE, MAE and MAPE reduced but RMSPE increased after removing outliers in linear regression

I am using a linear regression model with 465 datapoints to predict crop yield of wheat. The data points are spilt into training and testing sets in 80:20 ratio. Initially, I tried linear regression ...
kakashiOfSharingan's user avatar
1 vote
0 answers
21 views

How to deal with the miss# how to deal with the missing values in time series forecasting?

I have dataset in which there are records of stock market hour by hour. There are some missing values like each day should posses 24 values as it is hour by hour but I don't have the value of some ...
Hamza's user avatar
  • 121
1 vote
0 answers
156 views

What to do with missing days from timeseries when checking for seasonality?

I have a dataset which looks like this, with pm25 values and timestamp as index: ...
user17856557's user avatar
1 vote
0 answers
515 views

How to increase the accuracy of a multivariate LSTM network?

I am using a pollution dataset with values for pm2.5, pm10 and pm1 as features and I am predicting the values for the pm2.5. I built an LSTM network but the predicted values are quite from the real ...
user17856557's user avatar
1 vote
1 answer
36 views

How to predict a class between uneven datasets

I am trying to predict obesity using two datasets. The first (Diet) has 20k values in two columns: Diet [ID, Calories(float)] The second dataset also has two columns: Obesity [ID, Level(1,2,3)] ...
Munir Yousef's user avatar
1 vote
1 answer
1k views

How to scale multidimensional time series data per group

I am dealing with panel data and want to scale it in order to use it for some ML models: id year A B C 1 2000 3,539,101 265.152 .0683649 1 2001 3,539.101 2,485.833 .0683649 1 2002 3,539.101 2,939....
Kristina Zhupunova's user avatar
4 votes
1 answer
8k views

Why am I getting different results on a prediction using the same Keras model and input? [closed]

posting here is my last resort cause I can't find anything like it online. I trained a model to classify embeddings into categories (a simple three layer Dense neural network). I'm using pandas for ...
Ramiro's user avatar
  • 93
0 votes
1 answer
107 views

How do I roll data within groups in a dataframe? [closed]

The original dataframe is like df1. df1 = pd.DataFrame({'group': [1,1,1,1,1,2,2,2,2,2], 'in': list(range(1,11)), 'out': [0,0,0,1,1,0,0,0,1,1]}) df1 The output I ...
Goldfish's user avatar
1 vote
3 answers
678 views

Dimensionality Reduction Techniques Ordinal Variables on different scales

I have a data set that is a mix between Ordinal variables and numerical variables. The problem is that the ordinal variables are on different scales, such as 0-2, 1-4. The data set has 35 variables. I ...
Shelby Labuschagne's user avatar
0 votes
2 answers
129 views

I'm performing a multi linear regression on the The Oxford Covid-19 Government Response Tracker. I have a couple of doubts about the process

I hope this is the right place to ask this. I'm currently working my way through a dataset and performing Multiple Linear Regression on it. The data is for Oxford Governement Response Tracker for the ...
Vipul Dass's user avatar
1 vote
0 answers
187 views

Creating box plots from a count (or frequency) table

I have a census data where the number of population at each age is indicated and I am trying to create a box plot for the age of the entire population. At first I tried to repeat the 'age' by the ...
Sophie Bae's user avatar
0 votes
0 answers
120 views

Using pandas to check stability in data

I have a conversions table & I am trying to identify Are the conversions in the 'conversions' table stable over time? to identify Any pattern? ...
Ranji Raj's user avatar
  • 259
0 votes
1 answer
514 views

How to do feature engineering with scikit learn pipelines? [closed]

When doing preprocessing I've always used pandas to impute, encode, or scale my data. In other words, I've done all of the steps "manually". However, this takes a long time, generates a lot ...
LeLuc's user avatar
  • 661
0 votes
2 answers
298 views

How to name and treat such a distribution of extreme values / outliers?

In this sample data I have 4600 teenagers with their telephone calling frequency per week. Most of them are "normal" (max 5) but some are extreme with up to 75 calls per week. I learned to ...
buhtz's user avatar
  • 282
1 vote
0 answers
105 views

Imputation Methods for Multiple Variables

I am training Recurrent Neural Network models (including LSTMs) on a dataset that includes 6-10 variables. Each variable is a properly formatted numerical measurement (ie: length, pressure, ...
jros's user avatar
  • 614
0 votes
0 answers
31 views

Univariate multi-step prediction model for missing value imputation

I have a dataframe with columns of timestamp and energy usage. The timestamp is taken for every min of the day i.e., a total of 1440 readings for each day. I have few missing values in the data frame. ...
Ashkan Lotfipoor's user avatar
0 votes
0 answers
19 views

How to classify the features instead of rows

I have a dataset about some students' e-learning activities and their grades respectively. So in rows, I have the students and in columns (features) I have the activities. I want to classify the ...
Wiss Dev's user avatar
1 vote
0 answers
201 views

Normalization PCA for dataframes?

I would like to know what is the correct way to normalize a dataframe before applying PCA. I have found two options and I got different results for each one: ...
Itzy Death's user avatar
3 votes
1 answer
2k views

What dummy variables to set for US state-and-year Differences-in-Differences model

As a sort of personal experiment, I'm trying to run a differences-in-differences (DiD) model on US state-level firearms restrictions and violent crime, to see if changes in the former impact the ...
Demosthenes's user avatar