Questions tagged [pandas]

Python library for data manipulation, implementing R-style data frames.

Filter by
Sorted by
Tagged with
0
votes
0answers
8 views

Creat Dataframe from Matrix Search Calculation Pandas [closed]

I am looking to to a double continiod Matrix Search and take the product. The second dataframe is the one I would like to have. I searches for the indenitiy "Haus1" and "Haus2" for the Kind "gas" and ...
0
votes
0answers
7 views

Trying to return more than just the top result from sklearn NearestNeighbors [closed]

I'm trying to compare a list of names (duplicated into a clean file and a messy file). I then compare the files against each other. My problem is that it returns only the top 1 result for each, ...
0
votes
0answers
24 views

How to run a jupyter notebook code on multiple cores? [closed]

I am trying to implement a K-Neighbours Classification model on a dataset with shape (60000,32,32) on my system (16 GB ram, I5 8th gen processor, 256 GB hard disk). Though I have normalized the data, ...
0
votes
0answers
4 views

accessing loaded data set within r code using rpy2 [migrated]

I am running some r code within python using the rpy2 module. I have data loaded into a pandas dataframe that I would like to access within r code that I pass as text to the embedded r process using ...
0
votes
1answer
17 views

Pandas concatenate function problem [closed]

I have a dataframe of shape (1388, 14) where two of the columns represent the years of education of each parent. Both these columns have missing values, which are ...
0
votes
0answers
19 views

Analysing arrays of image data with Machine Learning Models

I am trying to do Machine Learning on arrays or vectors describing images. The target variable is a category I am trying to predict. I have multiple features that each contain arrays describing the ...
0
votes
0answers
13 views

Reverse calculation of moving mean

i am currently trying to develop a time series prediction model with scikit-learn and after a very long time of trying, i have managed to get a very good r2-score. However, I had to adjust my target ...
2
votes
0answers
124 views

Hypothesis Testing and calculating p-value for Pandas dataset

I want to study the relationship between car accidents and weather temperature. So, I have a dataset for car accidents that have different attributes related to accidents and weather temperature for ...
0
votes
0answers
15 views

pandas_profiling - I am having trouble reading the correlation table

My data has five numeric columns: And this is what pandas_profiling generates as the correlation matrix. I can't understand it. I see that red means 1.0. But why does each pair of variables seem to ...
0
votes
0answers
11 views

Handling percent change variables with 0s

I'm building a model, and adding in some percent change features. I'm running into an issue when calculating percent change month over month. Let's say I have a product and I'm just calculating the ...
1
vote
0answers
27 views

Dataframe containing value 0 should be removed or replaced?

I have a question that mostly I get stuck at. I was looking at the data for diabetes patients and found that most of the rows have 0 values under most of their columns. Reference for the url https://...
0
votes
1answer
26 views

Using KStests to find my estimator is a good fit with my distribution

I am trying to build an estimator based on my distribution that has 1585 values in it. The distribution itself kinda looks normal so I created a KDE and MLE estimators to find the best fit for my ...
0
votes
0answers
8 views

Multi feature matching

this might be a really intro question, But if I have two dataframes like: ...
0
votes
1answer
60 views

correlation coefficient in pandas (pearson) [duplicate]

I have divided my data into training and testing, and I am outputting the error metrics on the testing data. This is what I get: ...
0
votes
0answers
10 views

How to find optimum matrix set based on determinant values using python

I am new at programming, so I want to find the optimum set of row values based on maximum determinant logic. 1) Set the 1st Column 'Serial_no' as index. 2) Take first 'N' row values as user input ...
1
vote
1answer
20 views

Which regression is useable for an ordinal dependent and multiple discrete/ordinal/binary independent variables?

For a paper of mine I am trying to figure out if there is a correlation between success as a musician and multiple other factors. These are my variables: Dependent variable: Success from 2016 ...
1
vote
1answer
28 views

Influence of trend on (supposedly) correlated time series

TL;DR: What is the impact of a linear trend on the correlation between time series that are (most likely) not spuriously correlated? I'm currently trying to reconstruct/cross-validate an analysis ...
0
votes
1answer
57 views

Generating synthetic data based on mean, skewness, standard deviation and autocorrelation

Given that I have the mean, standard deviation, skewness and autocorrelation, How do I generate 1000 years of random data based on the above parameters in python or Matlab? I know for example I can ...
3
votes
2answers
248 views

Getting very large coefficients from linear regression

I'm currently looking at rates for a study that vary between 0 and 100 with most of the rates falling between 0 and 1. I am running a linear regression on 70 dummy variables (coded 0-1) and nearly ...
1
vote
0answers
20 views

How to detect calendar effects for stock prices (day-of-week, month, etc.)

This is a python/pandas question just as much as it is a statistical one. How would I go about determining the typical delta for day-of-week and month effects of a given time series? Taking the day-...
0
votes
1answer
15 views

What analysis should I perform to determine the minimum food items necessary to satisfy daily nutrient values?

I have a table ("table A") where each row represents a single food, and each column represents the amount of a particular nutrient found in that food. ...
2
votes
3answers
505 views

predictions for AR(1) model

I don't understand how predictions can trace the actual data so closely (see the code below)? Does that make sense? The model is $Y_t = \theta Y_{t-1} + Z_t$ where $Z_t$ is random noise. Hence the ...
1
vote
1answer
68 views

Query with predicted output using Multiple linear regression

I'm hoping to better understand the predicted output of the dependent variable using a multiple linear regression. Specifically, I'm getting negative predicted outputs when altering specific ...
2
votes
1answer
79 views

Finding Outliers in Resource Allocation Forecast Data

I initially posted this under the DS stack exchange, but after much reading and browsing, I think this is the right place for this question. I'm a workforce analyst at a large retail company, I own ...
0
votes
0answers
13 views

Optimal pandas dataframe size for seasonal ARIMA predictions over multiple timeframes

For backstory, please see this related question: Q3: How many rows of data should I keep in my respective timeframe for accurate ARIMA modeling (using SARIMAX)? An example of the p,d,q's of a 1H ...
0
votes
0answers
34 views

Condensing values of categorical data

Beginner ML question here. I have a dataframe with a categorical column, a lot of the values are slightly different but essentially mean the same thing. Here's an example of such values: ...
1
vote
2answers
60 views

Finding Relationship between Categorical and Continuous data

A subset of my dataset looks as follows where cells in "cat1_ids" column contains list of "cat1" categories and cells in "person_id_list" column contains list of persons id. There are 2000 "cat1" ...
1
vote
2answers
49 views

Which Labeling should I use for my data

I am currently getting ready to preprocess my data for scikitlearn and was wondering if I should use one hot encoding or label encoding when working with values greater than 9. I may be wrong but when ...
3
votes
2answers
172 views

How to identify outliers in a time series with correlated variables

I am working with time series data of sensor measurements. I have nine sensors that are in the same ballpark location recording the same data every 10 minutes. The sensors are setup such that the ...
0
votes
1answer
36 views

AR(k)-GARCH(1,1) model. Why am I getting same Log-likelihoods and AICs?

I am trying to for loop an AR(k)-GARCH(1,1) model, however it seems that I am getting same log-likelihoods and AICs. I believe that my code is fine, since I manually checked the iterations. Is there ...
0
votes
0answers
26 views

How to proceed when testin for non-linear relationships between variables

I'm trying to predict the amount of money a customer spends on a product by using linear regression. I had the hypothesis that there was a non-linear connection between the Product_Category, the ...
0
votes
0answers
25 views

Classification of sample with only unseen words

I'm doing text classification (Product Name) where one example belongs to one class. "Some Product Name" -> MODEL -> {CLASS_1 | CLASS_2 | CLASS_3 | CLASS_4} ...
0
votes
0answers
85 views

Dropping One-hot-encoded columns in Pandas/Sklearn [duplicate]

When one-hot-encoding categorial features in python with pandas or sklearn, when should I drop one of the resulting columns? I recall something about having all columns present being a problem for ...
0
votes
0answers
80 views

How to calculate volatility and trends in a time series analysis?

I have a table with data for different groups by months and totals, each groups are of different scales where the max and min ...
0
votes
0answers
50 views

Which machine learning technique suits this use case?

I have two different zones say for ex. IN and OUT and each zone with two features i.e., A & B. Now I have a target with the same two features and with the help of this, I need to identify, to ...
3
votes
1answer
280 views
1
vote
1answer
146 views

Pearson Correlation of same values

I got this problem while computing the pearson correlation of two datasets where one set consists of the same value. For example this pandas DataFrame: ...
1
vote
0answers
22 views

How to compute/plot the contribution of each original descriptor in a final PLA regression model?

New to scikit-learn. I am using v 20.2. I am developing PLS regression models.I would like to know how important each of the original predictors/descriptors are in predicting the response. The ...
1
vote
1answer
549 views

Training data has more variables than test data

Given a train and test data that looks like the below: Im wondering if it is necessary to drip the id field in the training data if the id field is present in the test data. Also, if the test data ...
1
vote
1answer
381 views

Does it make sense to apply recursive feature elimination on one-hot encoded features?

Does it make sense to apply recursive feature elimination on a feature set pre-processed with One-Hot Encoding? This is my code for feature selection: ...
2
votes
1answer
57 views

I don't understand such a difference in the accuracy, please help

When I use a normalized values for the values of the target column in the following DL regression model I get a very good accuracy, and if I don't, the accuracy is a mess. However I've reading that ...
-2
votes
2answers
33 views

Choosing the right type of plot for my pandas dataframe

I have been confused because I don't know which type of plot I must choose. I have a data frame with two columns suppose the first one is the id of a person and the second one the number of houses ...
2
votes
1answer
539 views

Temperature time series forecasting predictions converging to a certain value

I am trying to forecast the value of the ambient temperature based on given data on Python. The data frequency is 15 minutes. In order to predict future values, I am using a simple autoregressive ...
1
vote
0answers
83 views

How to convert my Data set to Gaussian?

I want to convert an Attribute in My data set to Gaussian distribution, but i can't seem to find any formula that i can implement. I don't want to use any Builtin Functions. So is there a formula ? ...
1
vote
1answer
647 views

Deciding between get_dummies and LabelEncoder for categorical variables in a Linear Regression Model

I'm using the dataset http://www.stat.ufl.edu/~winner/data/airq402.dat whose description is here - http://www.stat.ufl.edu/~winner/data/airq402.txt. I'm planning to build a linear regression model ...
0
votes
1answer
33 views

Remove individual points and find slope

I am trying to delete one pair of x and y coordinates from a set of 10 data points and get the slope for the other 9 points. How do I go about this issue? Attached herewith is an image of what I am ...
1
vote
1answer
245 views

One-Hot Encoding and Feature Engineering While Avoiding Data Leakage

I have a Pandas dataframe for which I've performed some actions over categorical features: Feature Engineering One-Hot Encoding Let's say that in my dataset I have the features "person_income" and "...
0
votes
1answer
237 views

Do you have to create dummy variables for ordinal data?

Using Python and utilizing XGBoost for classification. I have three columns that is supposed to be categorical and are ordinal (ranging from 1-5). Since the columns do not contain strings, I was ...
0
votes
1answer
82 views

Dealing with a dataset having target values on different scales?

I am currently working on a dataset having 10 features and one continuous target variable. One of the features is 'Country' , in which there are seven unique values [Argentina ,Denmark , France...etc]....
0
votes
0answers
45 views

While performing label encoding or imputation, what should i do to the column with mostly 0's as values which is irrelevant to what column is about?

My DataFrame consists of 2919 rows. Now, for example I have this column "2ndFlrSF" 2ndFlrSF: Second floor's Area in square feet and these are the values in it after I run my Pandas command ...