Questions tagged [pandas]

Python library for data manipulation, implementing R-style data frames.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

How to read complex number matrix from excel/csv using python? [closed]

I am trying to read complex numbers matrix saved in excel xlsx format. I used pd.read_excel('data.xlsx') to read using pandas. But after reading when I convert it to NumPy array using .to_numpy(), ...
0
votes
0answers
11 views

'DataFrame' object has no attribute 'map' [migrated]

I have the following command below: ...
2
votes
1answer
286 views

What are good methods to deal with outliers when calculating mean of data?

I have a dataframe with yearly energy uses of buildings over 5 years. In order to have a representative yearly energy use for data modelling, I'll have to take the mean of those data. As the data can ...
1
vote
0answers
35 views

PCA returns duplicated features for different components

I performed (sklearn) PCA on a (1416960,140) pandas DataFrame. The resulting components_ attribute is a matrix where each ...
0
votes
0answers
13 views

How to compare datasets of different scales in python

I have a dateset which consists of a timestamp and a column for some sensor data. Because the timestamp is of very high frequency(for example every millisecond), i downsampled the data ( to every 100 ...
0
votes
1answer
26 views

correlation of two features, which test? [closed]

I have the following table as pandas dataframe with features feat1 and feat2: ...
0
votes
0answers
13 views

How to handle time series data with multiple discrepancies?

I'm trying to classify time-series data where we have n hikers traveling between two locations. We're just trying to classify them without regards to seasonality The problem is that each hiker starts ...
0
votes
1answer
23 views

How to calculate nullity correlation matrix? [closed]

I am able to print nullity correlation matrix using Using pandas- df.isnull().corr() (this is how it is done is missingno). But what is the maths behind it ?How is nullity matrix calculated when ...
1
vote
1answer
17 views

Sklearn PCA Calculation Seems to Use Truncated Division as Opposed to Floating Point Division

I am working with the following dataset: Housing dataset From this dataset, I am only interested in the following columns: GrLivArea (independent variable), and SalePrice (dependent variable). What ...
1
vote
0answers
17 views

Calculating significant drop from grouped time series data

I have a time series dataset of number of visits to a particular website of a group of users so it looks something like this (Note the number of time pts per user will vary depending on when they ...
1
vote
3answers
53 views

Is it possible to detect outliers in two related columns [duplicate]

I have pandas data frame which has some related columns, like let's say I have a column that showing housing sale prices and other column shows total area of that house. If I look for outliers alone ...
3
votes
2answers
190 views

Creating a dataframe includes my cross validation scores

I'm kinda beginner and stuck at some basic part of my work. I want to create a pandas dataframe showing my CV scores and std's per model. I managed to get the results I want but I feel like I did it ...
0
votes
0answers
27 views

Feature Reduction Normalization

So I have a dataset with 13 columns and 10million rows, each is a different features like Power, Wind Speed etc. I wanted to apply extra tree classifier to find each feauture importance for data ...
2
votes
1answer
73 views

Why are p-values for python's Statsmodels so different from those reported by Stata?

I'm running a OLS regression in Stata and the same one in python's Statsmodels. Note that I adjust for clusters (for id and year). All the outcomes are very similar if not the same. But,the ...
0
votes
0answers
28 views

Right skewed distribution of a continuous variable with outliers: replace outliers with mode or median of that column?

When I replace my outliers with the median value of that column/feature, my mode for that column/feature also changes. Is that correct?
0
votes
0answers
41 views

Hypothesis Testing on German Credit Data

I am trying to perform hypothesis testing on German Credit Data ...
1
vote
0answers
22 views

Data Quality Checks---Taking the standard deviation on the number of rows in a dataset?

I am pulling in a handful of different datasets daily, performing a few simple data quality checks, and then shooting off emails if a dataset fails the checks. My checks are as plain as checking for ...
0
votes
0answers
19 views

What kind of method to use to predict multiple time-series with missing data points and of different lengths

I have to create a model to predict future sales of different restaurants to use in investment decisions. For about 1000 restaurants, I have the following data: Weekly sales (split out over product ...
2
votes
1answer
49 views

Dealing with large time series gaps

I have data with big missing gaps over the same months. Should I split this into separate dataframes? Impute over the gaps (seems like a bad idea). Any other ideas? Real lost and am having trouble ...
0
votes
1answer
24 views

Pandas concatenate function problem [closed]

I have a dataframe of shape (1388, 14) where two of the columns represent the years of education of each parent. Both these columns have missing values, which are ...
0
votes
0answers
20 views

Analysing arrays of image data with Machine Learning Models

I am trying to do Machine Learning on arrays or vectors describing images. The target variable is a category I am trying to predict. I have multiple features that each contain arrays describing the ...
0
votes
0answers
13 views

Reverse calculation of moving mean

i am currently trying to develop a time series prediction model with scikit-learn and after a very long time of trying, i have managed to get a very good r2-score. However, I had to adjust my target ...
2
votes
0answers
483 views

Hypothesis Testing and calculating p-value for Pandas dataset

I want to study the relationship between car accidents and weather temperature. So, I have a dataset for car accidents that have different attributes related to accidents and weather temperature for ...
1
vote
0answers
16 views

pandas_profiling - I am having trouble reading the correlation table

My data has five numeric columns: And this is what pandas_profiling generates as the correlation matrix. I can't understand it. I see that red means 1.0. But why does each pair of variables seem to ...
0
votes
0answers
12 views

Handling percent change variables with 0s

I'm building a model, and adding in some percent change features. I'm running into an issue when calculating percent change month over month. Let's say I have a product and I'm just calculating the ...
1
vote
0answers
27 views

Dataframe containing value 0 should be removed or replaced?

I have a question that mostly I get stuck at. I was looking at the data for diabetes patients and found that most of the rows have 0 values under most of their columns. Reference for the url https://...
0
votes
1answer
28 views

Using KStests to find my estimator is a good fit with my distribution

I am trying to build an estimator based on my distribution that has 1585 values in it. The distribution itself kinda looks normal so I created a KDE and MLE estimators to find the best fit for my ...
0
votes
0answers
9 views

Multi feature matching

this might be a really intro question, But if I have two dataframes like: ...
0
votes
1answer
120 views

correlation coefficient in pandas (pearson) [duplicate]

I have divided my data into training and testing, and I am outputting the error metrics on the testing data. This is what I get: ...
0
votes
0answers
13 views

How to find optimum matrix set based on determinant values using python

I am new at programming, so I want to find the optimum set of row values based on maximum determinant logic. 1) Set the 1st Column 'Serial_no' as index. 2) Take first 'N' row values as user input ...
1
vote
1answer
30 views

Which regression is useable for an ordinal dependent and multiple discrete/ordinal/binary independent variables?

For a paper of mine I am trying to figure out if there is a correlation between success as a musician and multiple other factors. These are my variables: Dependent variable: Success from 2016 ...
1
vote
1answer
39 views

Influence of trend on (supposedly) correlated time series

TL;DR: What is the impact of a linear trend on the correlation between time series that are (most likely) not spuriously correlated? I'm currently trying to reconstruct/cross-validate an analysis ...
0
votes
1answer
120 views

Generating synthetic data based on mean, skewness, standard deviation and autocorrelation

Given that I have the mean, standard deviation, skewness and autocorrelation, How do I generate 1000 years of random data based on the above parameters in python or Matlab? I know for example I can ...
4
votes
2answers
935 views

Getting very large coefficients from linear regression

I'm currently looking at rates for a study that vary between 0 and 100 with most of the rates falling between 0 and 1. I am running a linear regression on 70 dummy variables (coded 0-1) and nearly ...
1
vote
0answers
24 views

How to detect calendar effects for stock prices (day-of-week, month, etc.)

This is a python/pandas question just as much as it is a statistical one. How would I go about determining the typical delta for day-of-week and month effects of a given time series? Taking the day-...
0
votes
1answer
15 views

What analysis should I perform to determine the minimum food items necessary to satisfy daily nutrient values?

I have a table ("table A") where each row represents a single food, and each column represents the amount of a particular nutrient found in that food. ...
2
votes
3answers
998 views

predictions for AR(1) model

I don't understand how predictions can trace the actual data so closely (see the code below)? Does that make sense? The model is $Y_t = \theta Y_{t-1} + Z_t$ where $Z_t$ is random noise. Hence the ...
1
vote
1answer
96 views

Query with predicted output using Multiple linear regression

I'm hoping to better understand the predicted output of the dependent variable using a multiple linear regression. Specifically, I'm getting negative predicted outputs when altering specific ...
2
votes
1answer
80 views

Finding Outliers in Resource Allocation Forecast Data

I initially posted this under the DS stack exchange, but after much reading and browsing, I think this is the right place for this question. I'm a workforce analyst at a large retail company, I own ...
0
votes
0answers
36 views

Condensing values of categorical data

Beginner ML question here. I have a dataframe with a categorical column, a lot of the values are slightly different but essentially mean the same thing. Here's an example of such values: ...
1
vote
2answers
61 views

Finding Relationship between Categorical and Continuous data

A subset of my dataset looks as follows where cells in "cat1_ids" column contains list of "cat1" categories and cells in "person_id_list" column contains list of persons id. There are 2000 "cat1" ...
1
vote
2answers
49 views

Which Labeling should I use for my data

I am currently getting ready to preprocess my data for scikitlearn and was wondering if I should use one hot encoding or label encoding when working with values greater than 9. I may be wrong but when ...
3
votes
2answers
275 views

How to identify outliers in a time series with correlated variables

I am working with time series data of sensor measurements. I have nine sensors that are in the same ballpark location recording the same data every 10 minutes. The sensors are setup such that the ...
0
votes
1answer
37 views

AR(k)-GARCH(1,1) model. Why am I getting same Log-likelihoods and AICs?

I am trying to for loop an AR(k)-GARCH(1,1) model, however it seems that I am getting same log-likelihoods and AICs. I believe that my code is fine, since I manually checked the iterations. Is there ...
0
votes
0answers
28 views

How to proceed when testin for non-linear relationships between variables

I'm trying to predict the amount of money a customer spends on a product by using linear regression. I had the hypothesis that there was a non-linear connection between the Product_Category, the ...
1
vote
0answers
138 views

How to calculate volatility and trends in a time series analysis?

I have a table with data for different groups by months and totals, each groups are of different scales where the max and min ...
0
votes
0answers
50 views

Which machine learning technique suits this use case?

I have two different zones say for ex. IN and OUT and each zone with two features i.e., A & B. Now I have a target with the same two features and with the help of this, I need to identify, to ...
3
votes
1answer
629 views
2
votes
1answer
245 views

Pearson Correlation of same values

I got this problem while computing the pearson correlation of two datasets where one set consists of the same value. For example this pandas DataFrame: ...
1
vote
0answers
30 views

How to compute/plot the contribution of each original descriptor in a final PLA regression model?

New to scikit-learn. I am using v 20.2. I am developing PLS regression models.I would like to know how important each of the original predictors/descriptors are in predicting the response. The ...