Questions tagged [pandas]
Python library for data manipulation, implementing R-style data frames.
168
questions
0
votes
1
answer
68
views
how can I evaluate the unbalanced data set
The output here shows the titles and descriptions of the comments written for the evaluation of the top 100 books in amazon with nltk vader, and the total reviewer rating for those analyzes, but there ...
0
votes
0
answers
32
views
Kendall's Tau Correlation for dataset with multiple types (binary, categorical, ordinal, continuous)
I have a dataset of many variables of varying data types (binary, categorical, ordinal, continuous), and I want to find the correlations between them. I encoded the binary as (0,1) and one-hot encoded ...
- 1
1
vote
0
answers
26
views
Distributions in the context of geo-spatial data [closed]
I have geo spatial data with lat, lon of user activity. I have divided this data into geohashes with precision 6. For each geohash, how do I measure the distribution of points within them?
How do we ...
- 11
0
votes
0
answers
11
views
Formal terminology: metafeatures then groupby (GROUP BY) operation
In ML training and other analytics I often combine features to produce a 'metafeature' and then perform a 'groupby' (pandas) or 'GROUP BY' (SQL) query.
What is the technical term for this operation?
...
- 113
0
votes
0
answers
105
views
How to get Pandas EWMA results to match online EWMA calculator results?
I wasn't getting much help in the programming forums and thought to try here.
I am testing Pandas exponential weighted moving average (EWMA) function:
...
- 191
0
votes
0
answers
23
views
How to proceed when correlation plot doesn't show as much multicorrelation as is seen by statsmodels variance_inflation_factor in a regression task?
I am working with the kaggle Blueberry Yield prediction dataset. There are 17 columns including the target variable. Below is the correlation heat map:
It can be seen that multiple features are ...
- 101
0
votes
0
answers
13
views
Systematic/Repeatable Framework for Outlier Removal? [duplicate]
I am using both IQR and Z score > +/- 3SD for outlier detection.
It seems like Z score > +/- 3SD is more strict and yields fewer outliers than IQR, which is better for my purposes (Regression, ...
- 831
0
votes
0
answers
9
views
Efficient merging of multiple tables
Suppose you have 3 or more tables with different observations or hypotheses to be merged with a inner join. The tables have partially overlapping columns, and when completely merged yield a reduced ...
- 161
0
votes
0
answers
37
views
How to forecast sales for different product types and categories
I'm trying to predict sales for a tea export company with different tea types and weights. My goal is to predict sales for each product type category for the next 12 months.
Data set looked like this
<...
1
vote
0
answers
54
views
How to structure combinations of dataframes for regression, without corruption/loss?
I have a data set, redacted sample below. My goal is linear regression. My question is: Have I created unintended results, due to how I structured the df, using concat and/or div?
For example, ...
- 11
0
votes
0
answers
50
views
Handling Imbalance Categorical Value in A Dataframe Column
I have a dataframe contains id, gender, and class, for example:
...
Reganda
3
votes
1
answer
146
views
Determine correlation between two categorical columns with lots of data
I have a large dataset containing country names and names of musicians like this, with more than 50.000 rows:
Country
Musician
australia
Jimmy Barnes
australia
Grinspoon
england
Giles
united ...
- 131
0
votes
0
answers
19
views
Why is the mean in my model almost the same for all values?
I am currently in the process of creating a deep learning model to forecast stock prices. I am using AutoGluon since I am relatively new to deep learning/ML.
My pandas dataframe consists of the ...
1
vote
1
answer
209
views
Encode multiple values of an attributes in Pandas
I have a dataset and one of the attributes of the dataset is Race. People have multiple races on the dataset. The values for the attribute Race are following
...
- 175
0
votes
1
answer
37
views
Interpretation T-test
First of all, I guess I should start my question exposing what is the data about and how it looks right now, I have the following pd.Series object. They are basically the daily conversion of a few ...
- 105
2
votes
1
answer
46
views
How do I normalize this feature, I've tried almost everything
I'm trying to normalize this skewed data as part of data preprocessing, but it doesn't normalize no matter which transformation I use to the point it's making me crazy :') .
The methods which I've ...
- 21
2
votes
2
answers
72
views
Can we change the estimated population mean in Hypothesis Testing?
My goal is to learn hypothesis testing. My understanding is that we do not have the population data. Therefore, we do not know the estimated population mean. So, we guess the population mean. If that ...
0
votes
0
answers
54
views
What is hypothesis testing one sample one sided?
My goal is to learn hypothesis testing one sample one sided. Honestly, I don't get it.
Hypothesis testing one sample RIGHT SIDED
...
0
votes
0
answers
43
views
How to find optimum column values when they affect another column value in time series data?
Original data frame spans a million rows and has an year's data. Here's top ten rows:
Code:
...
- 1
0
votes
0
answers
36
views
Border values at which timeseries decrease and increase
I have a timeseries data of signals in stock market ([-1, 1]) and I want to find mean values at which I have down trend and upwards trend.
I already used Moving ...
- 101
1
vote
0
answers
15
views
How to find whether two years of time series data are statistically significantly different from each other
I have several years of time series data with data points (# of cars) taken every 5 minutes (have filled in any missing data points). Wondering the best way to check whether there is a statistical ...
- 11
1
vote
0
answers
126
views
Time Series Seasonality and Frequency
I am working with a time series that indicates the number of times a user logs into an application per minute. I want to find an algorithmic way (in Python) that would detect if there is any pattern ...
- 23
0
votes
0
answers
55
views
How can reduce improve my ANN accuracy and reduce overfitting?
My ANN model produces classic overfitting characteristics, producing high R2 values (90-99%) but low accuracy scores (10-40%). I'm currently inputting 28655 data entries, using 8 input features to ...
reubenturner
2
votes
2
answers
42
views
Transformation of mix normal and skew features [duplicate]
I have a weather dataset containing four features that are continuous values. Temperature is almost normal, but precipitation is highly negatively skewed. In addition, wind speed and humidity are ...
- 73
0
votes
0
answers
323
views
How does Pandas Interpolate Time work?
How does Pandas Interpolate method = time function work? From my understanding, it sounds very similar to an Exponential Moving Average, but it gives me different results, even when I play around with ...
- 121
3
votes
1
answer
373
views
running multiple t-tests across large dataset
I'm trying to work out if a t-test is the most appropriate in this situation:
I have a data frame which looks like the one below but my data frame has aprox 37,000 rows. I'd like to run a t-test on ...
- 173
2
votes
1
answer
2k
views
Implementing a Conditional Logit in Python StatsModels
I have a dataframe with some horseracing data, and each row contains a predicted speed rating for each of the runners. I am now trying to convert that information into a winning probability for each ...
- 51
0
votes
1
answer
105
views
Understanding the idx column in h2o metrics [closed]
h2o model metrics results report generated like this.
...
- 3
0
votes
0
answers
502
views
RMSE, MAE and MAPE reduced but RMSPE increased after removing outliers in linear regression
I am using a linear regression model with 465 datapoints to predict crop yield of wheat. The data points are spilt into training and testing sets in 80:20 ratio. Initially, I tried linear regression ...
1
vote
0
answers
19
views
How to deal with the miss# how to deal with the missing values in time series forecasting?
I have dataset in which there are records of stock market hour by hour. There are some missing values like each day should posses 24 values as it is hour
by hour but I don't have the value of some ...
- 121
1
vote
0
answers
152
views
What to do with missing days from timeseries when checking for seasonality?
I have a dataset which looks like this, with pm25 values and timestamp as index:
...
- 21
1
vote
0
answers
489
views
How to increase the accuracy of a multivariate LSTM network?
I am using a pollution dataset with values for pm2.5, pm10 and pm1 as features and I am predicting the values for the pm2.5. I built an LSTM network but the predicted values are quite from the real ...
- 21
1
vote
1
answer
33
views
How to predict a class between uneven datasets
I am trying to predict obesity using two datasets. The first (Diet) has 20k values in two columns:
Diet [ID, Calories(float)]
The second dataset also has two columns:
Obesity [ID, Level(1,2,3)]
...
- 11
1
vote
1
answer
1k
views
How to scale multidimensional time series data per group
I am dealing with panel data and want to scale it in order to use it for some ML models:
id
year
A
B
C
1
2000
3,539,101
265.152
.0683649
1
2001
3,539.101
2,485.833
.0683649
1
2002
3,539.101
2,939....
4
votes
1
answer
7k
views
Why am I getting different results on a prediction using the same Keras model and input? [closed]
posting here is my last resort cause I can't find anything like it online. I trained a model to classify embeddings into categories (a simple three layer Dense neural network).
I'm using pandas for ...
- 83
0
votes
1
answer
101
views
How do I roll data within groups in a dataframe? [closed]
The original dataframe is like df1.
df1 = pd.DataFrame({'group': [1,1,1,1,1,2,2,2,2,2], 'in': list(range(1,11)), 'out': [0,0,0,1,1,0,0,0,1,1]})
df1
The output I ...
- 23
1
vote
3
answers
548
views
Dimensionality Reduction Techniques Ordinal Variables on different scales
I have a data set that is a mix between Ordinal variables and numerical variables.
The problem is that the ordinal variables are on different scales, such as 0-2, 1-4. The data set has 35 variables. I ...
0
votes
2
answers
115
views
I'm performing a multi linear regression on the The Oxford Covid-19 Government Response Tracker. I have a couple of doubts about the process
I hope this is the right place to ask this.
I'm currently working my way through a dataset and performing Multiple Linear Regression on it. The data is for Oxford Governement Response Tracker for the ...
1
vote
0
answers
154
views
Creating box plots from a count (or frequency) table
I have a census data where the number of population at each age is indicated and I am trying to create a box plot for the age of the entire population. At first I tried to repeat the 'age' by the ...
- 11
0
votes
0
answers
106
views
Using pandas to check stability in data
I have a conversions table & I am trying to identify Are the conversions in the 'conversions' table stable over time? to identify Any pattern?
...
- 259
0
votes
1
answer
481
views
How to do feature engineering with scikit learn pipelines? [closed]
When doing preprocessing I've always used pandas to impute, encode, or scale my data. In other words, I've done all of the steps "manually".
However, this takes a long time, generates a lot ...
- 651
0
votes
2
answers
237
views
How to name and treat such a distribution of extreme values / outliers?
In this sample data I have 4600 teenagers with their telephone calling frequency per week. Most of them are "normal" (max 5) but some are extreme with up to 75 calls per week.
I learned to ...
- 232
1
vote
0
answers
95
views
Imputation Methods for Multiple Variables
I am training Recurrent Neural Network models (including LSTMs) on a dataset that includes 6-10 variables. Each variable is a properly formatted numerical measurement (ie: length, pressure, ...
- 604
0
votes
0
answers
29
views
Univariate multi-step prediction model for missing value imputation
I have a dataframe with columns of timestamp and energy usage. The timestamp is taken for every min of the day i.e., a total of 1440 readings for each day. I have few missing values in the data frame.
...
0
votes
0
answers
19
views
How to classify the features instead of rows
I have a dataset about some students' e-learning activities and their grades respectively. So in rows, I have the students and in columns (features) I have the activities.
I want to classify the ...
- 1
1
vote
0
answers
184
views
Normalization PCA for dataframes?
I would like to know what is the correct way to normalize a dataframe before applying PCA. I have found two options and I got different results for each one:
...
- 11
1
vote
1
answer
1k
views
What dummy variables to set for US state-and-year Differences-in-Differences model
As a sort of personal experiment, I'm trying to run a differences-in-differences (DiD) model on US state-level firearms restrictions and violent crime, to see if changes in the former impact the ...
- 13
0
votes
2
answers
84
views
Account for variable to add more variation to predictor variable scipy
Is there a test or method to account for the interaction between predictor variables to influence the outcome in a regression model. Specifically, if a variable increases in value, it may add more ...
- 87
2
votes
1
answer
378
views
One hot coding in Train Validation and Test set (Production data) [closed]
For example I have below train set.
name values
0 Tony 100
1 Smith 110
2 Sam 120
3 Shane 130
4 Sam 140
5 Ram 160
After ...
- 121
1
vote
0
answers
141
views
What are the methods to find the maximum data size (in python and pandas) that can be handled smoothly by a system before even executing a dataset
I want to know about the various ways to calculate (or approximate) the maximum data size (in python and pandas) that can be handled smoothly by the current processor or the system as a whole before ...
- 283