Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [pandas]

Python library for data manipulation, implementing R-style data frames.

2
votes
3answers
100 views

predictions for AR(1) model

I don't understand how predictions can trace the actual data so closely (see the code below)? Does that make sense? The model is $Y_t = \theta Y_{t-1} + Z_t$ where $Z_t$ is random noise. Hence the ...
0
votes
0answers
14 views

How to interpret if Mean square error is higher than 1 in Multiple Linear Regression model?

How to interpret if Mean square error is higher than 1 in Multiple Linear Regression model? Can I use get_dummies with Linear Regression from scikit-learn? I am building a multiple linear regression ...
0
votes
0answers
6 views

How can I convert rows to columns (with custom names) after grouping? [migrated]

I'm trying to get some row data as columns with pandas. My original dataframe is something like the following (with a lot more columns). Most data repeats for the same employee but some info changes, ...
1
vote
1answer
48 views
+50

Query with predicted output using Multiple linear regression

I'm hoping to better understand the predicted output of the dependent variable using a multiple linear regression. Specifically, I'm getting negative predicted outputs when altering specific ...
2
votes
1answer
72 views

Finding Outliers in Resource Allocation Forecast Data

I initially posted this under the DS stack exchange, but after much reading and browsing, I think this is the right place for this question. I'm a workforce analyst at a large retail company, I own ...
0
votes
0answers
9 views

Optimal pandas dataframe size for seasonal ARIMA predictions over multiple timeframes

For backstory, please see this related question: Q3: How many rows of data should I keep in my respective timeframe for accurate ARIMA modeling (using SARIMAX)? An example of the p,d,q's of a 1H ...
0
votes
0answers
15 views

Condensing values of categorical data

Beginner ML question here. I have a dataframe with a categorical column, a lot of the values are slightly different but essentially mean the same thing. Here's an example of such values: ...
1
vote
2answers
55 views

Finding Relationship between Categorical and Continuous data

A subset of my dataset looks as follows where cells in "cat1_ids" column contains list of "cat1" categories and cells in "person_id_list" column contains list of persons id. There are 2000 "cat1" ...
1
vote
2answers
46 views

Which Labeling should I use for my data

I am currently getting ready to preprocess my data for scikitlearn and was wondering if I should use one hot encoding or label encoding when working with values greater than 9. I may be wrong but when ...
3
votes
2answers
46 views

How to identify outliers in a time series with correlated variables

I am working with time series data of sensor measurements. I have nine sensors that are in the same ballpark location recording the same data every 10 minutes. The sensors are setup such that the ...
0
votes
1answer
28 views

AR(k)-GARCH(1,1) model. Why am I getting same Log-likelihoods and AICs?

I am trying to for loop an AR(k)-GARCH(1,1) model, however it seems that I am getting same log-likelihoods and AICs. I believe that my code is fine, since I manually checked the iterations. Is there ...
0
votes
0answers
24 views

How to proceed when testin for non-linear relationships between variables

I'm trying to predict the amount of money a customer spends on a product by using linear regression. I had the hypothesis that there was a non-linear connection between the Product_Category, the ...
0
votes
0answers
16 views

Classification of sample with only unseen words

I'm doing text classification (Product Name) where one example belongs to one class. "Some Product Name" -> MODEL -> {CLASS_1 | CLASS_2 | CLASS_3 | CLASS_4} ...
0
votes
0answers
75 views

Dropping One-hot-encoded columns in Pandas/Sklearn

When one-hot-encoding categorial features in python with pandas or sklearn, when should I drop one of the resulting columns? I recall something about having all columns present being a problem for ...
0
votes
0answers
25 views

How to calculate volatility and trends in a time series analysis?

I have a table with data for different groups by months and totals, each groups are of different scales where the max and min ...
0
votes
0answers
50 views

Which machine learning technique suits this use case?

I have two different zones say for ex. IN and OUT and each zone with two features i.e., A & B. Now I have a target with the same two features and with the help of this, I need to identify, to ...
1
vote
1answer
50 views

Pearson Correlation of same values

I got this problem while computing the pearson correlation of two datasets where one set consists of the same value. For example this pandas DataFrame: ...
1
vote
0answers
17 views

How to compute/plot the contribution of each original descriptor in a final PLA regression model?

New to scikit-learn. I am using v 20.2. I am developing PLS regression models.I would like to know how important each of the original predictors/descriptors are in predicting the response. The ...
0
votes
1answer
48 views

Training data has more variables than test data

Given a train and test data that looks like the below: Im wondering if it is necessary to drip the id field in the training data if the id field is present in the test data. Also, if the test data ...
0
votes
0answers
174 views

Does it make sense to apply recursive feature elimination on one-hot encoded features?

Does it make sense to apply recursive feature elimination on a feature set pre-processed with One-Hot Encoding? This is my code for feature selection: ...
2
votes
1answer
48 views

I don't understand such a difference in the accuracy, please help

When I use a normalized values for the values of the target column in the following DL regression model I get a very good accuracy, and if I don't, the accuracy is a mess. However I've reading that ...
-2
votes
2answers
32 views

Choosing the right type of plot for my pandas dataframe

I have been confused because I don't know which type of plot I must choose. I have a data frame with two columns suppose the first one is the id of a person and the second one the number of houses ...
0
votes
0answers
64 views

Does this correlation make sense?

I'm trying to work on a ML project and I have a dataset and I'm trying to see if there is a correlation between some of the features in my dataset. The dataset contains inspection notes for car parts ...
0
votes
0answers
101 views

Inconsistent autocorrelation plots

I used pandas.tools.plotting.autocorrelation_plot in Python to plot autocorrelation functions of the same time series: for its first 100 and 1000 entries, respectively (code below). ...
0
votes
0answers
101 views

What are the effects of autocorrelation on logistic regression?

I need a simple way to estimate the probability of winning an auction as a function of bid amount. I modeled the auction using the LogisticRegression from pandas, ...
0
votes
0answers
153 views

Calculate corr/cov matrix for large Pandas dataframe/Numpy matrix

I have a large dataframe (gigs), and I would like to compute the corr matrix. What is the most efficient way of doing this in parallel. I have access to many machines and I'm open to using any ...
2
votes
1answer
328 views

Temperature time series forecasting predictions converging to a certain value

I am trying to forecast the value of the ambient temperature based on given data on Python. The data frequency is 15 minutes. In order to predict future values, I am using a simple autoregressive ...
1
vote
0answers
33 views

How to convert my Data set to Gaussian?

I want to convert an Attribute in My data set to Gaussian distribution, but i can't seem to find any formula that i can implement. I don't want to use any Builtin Functions. So is there a formula ? ...
1
vote
1answer
280 views

Deciding between get_dummies and LabelEncoder for categorical variables in a Linear Regression Model

I'm using the dataset http://www.stat.ufl.edu/~winner/data/airq402.dat whose description is here - http://www.stat.ufl.edu/~winner/data/airq402.txt. I'm planning to build a linear regression model ...
0
votes
1answer
29 views

Remove individual points and find slope

I am trying to delete one pair of x and y coordinates from a set of 10 data points and get the slope for the other 9 points. How do I go about this issue? Attached herewith is an image of what I am ...
1
vote
0answers
229 views

Update R2 and slope by removing outliers-python [closed]

I am using the following code to find out Linregress parameters: ...
1
vote
1answer
166 views

One-Hot Encoding and Feature Engineering While Avoiding Data Leakage

I have a Pandas dataframe for which I've performed some actions over categorical features: Feature Engineering One-Hot Encoding Let's say that in my dataset I have the features "person_income" and "...
0
votes
1answer
115 views

Do you have to create dummy variables for ordinal data?

Using Python and utilizing XGBoost for classification. I have three columns that is supposed to be categorical and are ordinal (ranging from 1-5). Since the columns do not contain strings, I was ...
0
votes
1answer
43 views

Dealing with a dataset having target values on different scales?

I am currently working on a dataset having 10 features and one continuous target variable. One of the features is 'Country' , in which there are seven unique values [Argentina ,Denmark , France...etc]....
0
votes
0answers
25 views

While performing label encoding or imputation, what should i do to the column with mostly 0's as values which is irrelevant to what column is about?

My DataFrame consists of 2919 rows. Now ,For example I have this column "2ndFlrSF" 2ndFlrSF: Second floor's Area in square feet and these are the values in it after i run my Pandas command ...
7
votes
1answer
6k views

What does (pandas) autocorrelation graph show?

I am a beginner and I am trying to understand what an autocorrelation graph shows. I have read several explanations from different sources such as this page or the related Wikipedia page among ...
0
votes
1answer
40 views

How do you link multiple X input images to a single ground truth image in machine learning?

What I'm asking is basically manual data augmentation. I have some very specific data augmentation for my inputs so I have to create them first in another software instead of doing it on the fly. I ...
1
vote
0answers
115 views

How to calculate timesteps for a 2D-Tensor in LSTM-Keras [closed]

I'm feeding a 2-D tensor in my LSTM network in keras, but I saw that you need to feed 3-D tensor. I''m using embedding layer. Here's my code: ...
2
votes
1answer
43 views

Dealing with the order of features (sequences)?

Assume we have following sequence database that is subsequently converted with one-hot encoding: 1 2 3 4 0 A B C D 1 B A D NA 2 A D C NA One-hot encoded: <...
0
votes
0answers
19 views

How do I improve model accuracy predicting categorical outcome using categorial predictors?

I'm trying to predict Para using Cols. My data is in this format: ...
0
votes
0answers
167 views

Strange output while using Holt’s Linear Trend method

Here're the pictures of using Holt’s Linear Trend method: From tutorial (what it should be like): After running the code for my data Isn't it strange? Here's a code (method #5): ...
1
vote
0answers
33 views

Detrending sales based on historical

I have a dataset with month sales for 6 months per customer. I want to compare the sales average of the first 3 months versus the last 3, but I know this averages may be influenced by seasonality of ...
1
vote
0answers
405 views

Python ANOVA or MANOVA

I have a datasets with samples from various locations. At each of these locations I measured the concentrations of various genes. I saved the results in pandas dataframe and would like to compare the ...
1
vote
1answer
32 views

Approach to analysing big data [closed]

I’m trying to analyse quite a largish dataset. The sasdb7 is around 11GB and the csv is around 9GB. This is after merging the datasets, cleaning and removing unnecessary columns. I am comfortable ...
1
vote
1answer
536 views

Underfitting? Validation scores above training scores

I'm plotting a learning curve currently using the following code: ...
0
votes
0answers
899 views

How to find nearest neighbors using cosine similarity for all items from a large embeddings matrix?

I have an embeddings matrix of a large no:of items - of around 100k, with each embedding vector length of 100. So a matrix of size 100k x 100; From this, I am trying to get the nearest neighbors for ...
1
vote
0answers
94 views

MAPE yielding value of 1000+ for timeseries forecasting

I am comparing a model's outcome with the actual values for a span of 9 months. I am expecting a very large error, however my MAPE implementation yields 1404512.56 using python, pandas. Here is my ...
1
vote
0answers
1k views

python StatsModels use training parameters for test data summary

I ran a linear regression on a month of hourly Nox concentration using StatsModels ols function. how can I get summary OLS Regression Results on the test data? I ...
1
vote
0answers
35 views

Correlation of words and score

Say I have two fields: "Comments" (type text) and "Score" (type float, from 0 to 5) Sample data like: "I appreciate this ...