Questions tagged [pandas]

Python library for data manipulation, implementing R-style data frames.

Filter by
Sorted by
Tagged with
-1
votes
0answers
16 views

Can we pass dataframe to fit() function in keras? [on hold]

According to the Keras documentation: fit(x=None, y=None, batch_size=None, epochs=1, ...) The arguments for fit function are numpy arrays: Arguments x: Numpy ...
-2
votes
0answers
17 views

How can I recreate an understandable path from a random forest specific prediction in python [closed]

I was asked to explain a specific prediction of a random forest model in production . I know there is the decision_path function from a random forest classifier object but I don't know how to ...
2
votes
2answers
31 views

Getting very large coefficients from linear regression

I'm currently looking at rates for a study that vary between 0 and 100 with most of the rates falling between 0 and 1. I am running a linear regression on 70 dummy variables (coded 0-1) and nearly ...
1
vote
0answers
16 views

How to detect calendar effects for stock prices (day-of-week, month, etc.)

This is a python/pandas question just as much as it is a statistical one. How would I go about determining the typical delta for day-of-week and month effects of a given time series? Taking the day-...
0
votes
1answer
15 views

What analysis should I perform to determine the minimum food items necessary to satisfy daily nutrient values?

I have a table ("table A") where each row represents a single food, and each column represents the amount of a particular nutrient found in that food. ...
0
votes
0answers
23 views

Pandas Predict Column with NaNs using sk-learn linear regression [closed]

I have a pandas DataFrame with a single column X, with some NaN values. The format of my df is something like this: ...
2
votes
3answers
159 views

predictions for AR(1) model

I don't understand how predictions can trace the actual data so closely (see the code below)? Does that make sense? The model is $Y_t = \theta Y_{t-1} + Z_t$ where $Z_t$ is random noise. Hence the ...
1
vote
1answer
56 views

Query with predicted output using Multiple linear regression

I'm hoping to better understand the predicted output of the dependent variable using a multiple linear regression. Specifically, I'm getting negative predicted outputs when altering specific ...
2
votes
1answer
73 views

Finding Outliers in Resource Allocation Forecast Data

I initially posted this under the DS stack exchange, but after much reading and browsing, I think this is the right place for this question. I'm a workforce analyst at a large retail company, I own ...
0
votes
0answers
11 views

Optimal pandas dataframe size for seasonal ARIMA predictions over multiple timeframes

For backstory, please see this related question: Q3: How many rows of data should I keep in my respective timeframe for accurate ARIMA modeling (using SARIMAX)? An example of the p,d,q's of a 1H ...
0
votes
0answers
30 views

Condensing values of categorical data

Beginner ML question here. I have a dataframe with a categorical column, a lot of the values are slightly different but essentially mean the same thing. Here's an example of such values: ...
1
vote
2answers
55 views

Finding Relationship between Categorical and Continuous data

A subset of my dataset looks as follows where cells in "cat1_ids" column contains list of "cat1" categories and cells in "person_id_list" column contains list of persons id. There are 2000 "cat1" ...
1
vote
2answers
47 views

Which Labeling should I use for my data

I am currently getting ready to preprocess my data for scikitlearn and was wondering if I should use one hot encoding or label encoding when working with values greater than 9. I may be wrong but when ...
3
votes
2answers
70 views

How to identify outliers in a time series with correlated variables

I am working with time series data of sensor measurements. I have nine sensors that are in the same ballpark location recording the same data every 10 minutes. The sensors are setup such that the ...
0
votes
1answer
31 views

AR(k)-GARCH(1,1) model. Why am I getting same Log-likelihoods and AICs?

I am trying to for loop an AR(k)-GARCH(1,1) model, however it seems that I am getting same log-likelihoods and AICs. I believe that my code is fine, since I manually checked the iterations. Is there ...
0
votes
0answers
25 views

How to proceed when testin for non-linear relationships between variables

I'm trying to predict the amount of money a customer spends on a product by using linear regression. I had the hypothesis that there was a non-linear connection between the Product_Category, the ...
0
votes
0answers
19 views

Classification of sample with only unseen words

I'm doing text classification (Product Name) where one example belongs to one class. "Some Product Name" -> MODEL -> {CLASS_1 | CLASS_2 | CLASS_3 | CLASS_4} ...
0
votes
0answers
85 views

Dropping One-hot-encoded columns in Pandas/Sklearn [duplicate]

When one-hot-encoding categorial features in python with pandas or sklearn, when should I drop one of the resulting columns? I recall something about having all columns present being a problem for ...
0
votes
0answers
40 views

How to calculate volatility and trends in a time series analysis?

I have a table with data for different groups by months and totals, each groups are of different scales where the max and min ...
0
votes
0answers
50 views

Which machine learning technique suits this use case?

I have two different zones say for ex. IN and OUT and each zone with two features i.e., A & B. Now I have a target with the same two features and with the help of this, I need to identify, to ...
3
votes
1answer
94 views
1
vote
1answer
63 views

Pearson Correlation of same values

I got this problem while computing the pearson correlation of two datasets where one set consists of the same value. For example this pandas DataFrame: ...
1
vote
0answers
17 views

How to compute/plot the contribution of each original descriptor in a final PLA regression model?

New to scikit-learn. I am using v 20.2. I am developing PLS regression models.I would like to know how important each of the original predictors/descriptors are in predicting the response. The ...
0
votes
1answer
81 views

Training data has more variables than test data

Given a train and test data that looks like the below: Im wondering if it is necessary to drip the id field in the training data if the id field is present in the test data. Also, if the test data ...
0
votes
1answer
228 views

Does it make sense to apply recursive feature elimination on one-hot encoded features?

Does it make sense to apply recursive feature elimination on a feature set pre-processed with One-Hot Encoding? This is my code for feature selection: ...
2
votes
1answer
49 views

I don't understand such a difference in the accuracy, please help

When I use a normalized values for the values of the target column in the following DL regression model I get a very good accuracy, and if I don't, the accuracy is a mess. However I've reading that ...
-2
votes
2answers
32 views

Choosing the right type of plot for my pandas dataframe

I have been confused because I don't know which type of plot I must choose. I have a data frame with two columns suppose the first one is the id of a person and the second one the number of houses ...
0
votes
0answers
64 views

Does this correlation make sense?

I'm trying to work on a ML project and I have a dataset and I'm trying to see if there is a correlation between some of the features in my dataset. The dataset contains inspection notes for car parts ...
0
votes
0answers
115 views

Inconsistent autocorrelation plots

I used pandas.tools.plotting.autocorrelation_plot in Python to plot autocorrelation functions of the same time series: for its first 100 and 1000 entries, respectively (code below). ...
0
votes
0answers
133 views

What are the effects of autocorrelation on logistic regression?

I need a simple way to estimate the probability of winning an auction as a function of bid amount. I modeled the auction using the LogisticRegression from pandas, ...
0
votes
0answers
194 views

Calculate corr/cov matrix for large Pandas dataframe/Numpy matrix

I have a large dataframe (gigs), and I would like to compute the corr matrix. What is the most efficient way of doing this in parallel. I have access to many machines and I'm open to using any ...
2
votes
1answer
367 views

Temperature time series forecasting predictions converging to a certain value

I am trying to forecast the value of the ambient temperature based on given data on Python. The data frequency is 15 minutes. In order to predict future values, I am using a simple autoregressive ...
1
vote
0answers
50 views

How to convert my Data set to Gaussian?

I want to convert an Attribute in My data set to Gaussian distribution, but i can't seem to find any formula that i can implement. I don't want to use any Builtin Functions. So is there a formula ? ...
1
vote
1answer
364 views

Deciding between get_dummies and LabelEncoder for categorical variables in a Linear Regression Model

I'm using the dataset http://www.stat.ufl.edu/~winner/data/airq402.dat whose description is here - http://www.stat.ufl.edu/~winner/data/airq402.txt. I'm planning to build a linear regression model ...
0
votes
1answer
30 views

Remove individual points and find slope

I am trying to delete one pair of x and y coordinates from a set of 10 data points and get the slope for the other 9 points. How do I go about this issue? Attached herewith is an image of what I am ...
1
vote
0answers
244 views

Update R2 and slope by removing outliers-python [closed]

I am using the following code to find out Linregress parameters: ...
1
vote
1answer
192 views

One-Hot Encoding and Feature Engineering While Avoiding Data Leakage

I have a Pandas dataframe for which I've performed some actions over categorical features: Feature Engineering One-Hot Encoding Let's say that in my dataset I have the features "person_income" and "...
0
votes
1answer
141 views

Do you have to create dummy variables for ordinal data?

Using Python and utilizing XGBoost for classification. I have three columns that is supposed to be categorical and are ordinal (ranging from 1-5). Since the columns do not contain strings, I was ...
0
votes
1answer
58 views

Dealing with a dataset having target values on different scales?

I am currently working on a dataset having 10 features and one continuous target variable. One of the features is 'Country' , in which there are seven unique values [Argentina ,Denmark , France...etc]....
0
votes
0answers
33 views

While performing label encoding or imputation, what should i do to the column with mostly 0's as values which is irrelevant to what column is about?

My DataFrame consists of 2919 rows. Now, for example I have this column "2ndFlrSF" 2ndFlrSF: Second floor's Area in square feet and these are the values in it after I run my Pandas command ...
10
votes
1answer
8k views

What does (pandas) autocorrelation graph show?

I am a beginner and I am trying to understand what an autocorrelation graph shows. I have read several explanations from different sources such as this page or the related Wikipedia page among ...
0
votes
1answer
42 views

How do you link multiple X input images to a single ground truth image in machine learning?

What I'm asking is basically manual data augmentation. I have some very specific data augmentation for my inputs so I have to create them first in another software instead of doing it on the fly. I ...
2
votes
1answer
46 views

Dealing with the order of features (sequences)?

Assume we have following sequence database that is subsequently converted with one-hot encoding: 1 2 3 4 0 A B C D 1 B A D NA 2 A D C NA One-hot encoded: <...
0
votes
0answers
21 views

How do I improve model accuracy predicting categorical outcome using categorial predictors?

I'm trying to predict Para using Cols. My data is in this format: ...
0
votes
0answers
180 views

Strange output while using Holt’s Linear Trend method

Here're the pictures of using Holt’s Linear Trend method: From tutorial (what it should be like): After running the code for my data Isn't it strange? Here's a code (method #5): ...
1
vote
0answers
33 views

Detrending sales based on historical

I have a dataset with month sales for 6 months per customer. I want to compare the sales average of the first 3 months versus the last 3, but I know this averages may be influenced by seasonality of ...
1
vote
0answers
445 views

Python ANOVA or MANOVA

I have a datasets with samples from various locations. At each of these locations I measured the concentrations of various genes. I saved the results in pandas dataframe and would like to compare the ...
1
vote
1answer
32 views

Approach to analysing big data [closed]

I’m trying to analyse quite a largish dataset. The sasdb7 is around 11GB and the csv is around 9GB. This is after merging the datasets, cleaning and removing unnecessary columns. I am comfortable ...
1
vote
1answer
589 views

Underfitting? Validation scores above training scores

I'm plotting a learning curve currently using the following code: ...
0
votes
0answers
944 views

How to find nearest neighbors using cosine similarity for all items from a large embeddings matrix?

I have an embeddings matrix of a large no:of items - of around 100k, with each embedding vector length of 100. So a matrix of size 100k x 100; From this, I am trying to get the nearest neighbors for ...