Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [pandas]

Python library for data manipulation, implementing R-style data frames.

0
votes
0answers
21 views

How to proceed when testin for non-linear relationships between variables

I'm trying to predict the amount of money a customer spends on a product by using linear regression. I had the hypothesis that there was a non-linear connection between the Product_Category, the ...
0
votes
0answers
13 views

Classification of sample with only unseen words

I'm doing text classification (Product Name) where one example belongs to one class. "Some Product Name" -> MODEL -> {CLASS_1 | CLASS_2 | CLASS_3 | CLASS_4} ...
0
votes
0answers
23 views

Dropping One-hot-encoded columns in Pandas/Sklearn

When one-hot-encoding categorial features in python with pandas or sklearn, when should I drop one of the resulting columns? I recall something about having all columns present being a problem for ...
0
votes
0answers
12 views

How to calculate volatility and trends in a time series analysis?

I have a table with data for different groups by months and totals, each groups are of different scales where the max and min ...
0
votes
0answers
49 views

Which machine learning technique suits this use case?

I have two different zones say for ex. IN and OUT and each zone with two features i.e., A & B. Now I have a target with the same two features and with the help of this, I need to identify, to ...
1
vote
1answer
42 views

Pearson Correlation of same values

I got this problem while computing the pearson correlation of two datasets where one set consists of the same value. For example this pandas DataFrame: ...
1
vote
0answers
15 views

How to compute/plot the contribution of each original descriptor in a final PLA regression model?

New to scikit-learn. I am using v 20.2. I am developing PLS regression models.I would like to know how important each of the original predictors/descriptors are in predicting the response. The ...
0
votes
1answer
35 views

Training data has more variables than test data

Given a train and test data that looks like the below: Im wondering if it is necessary to drip the id field in the training data if the id field is present in the test data. Also, if the test data ...
0
votes
0answers
100 views

Does it make sense to apply recursive feature elimination on one-hot encoded features?

Does it make sense to apply recursive feature elimination on a feature set pre-processed with One-Hot Encoding? This is my code for feature selection: ...
2
votes
1answer
43 views

I don't understand such a difference in the accuracy, please help

When I use a normalized values for the values of the target column in the following DL regression model I get a very good accuracy, and if I don't, the accuracy is a mess. However I've reading that ...
-2
votes
2answers
32 views

Choosing the right type of plot for my pandas dataframe

I have been confused because I don't know which type of plot I must choose. I have a data frame with two columns suppose the first one is the id of a person and the second one the number of houses ...
0
votes
0answers
63 views

Does this correlation make sense?

I'm trying to work on a ML project and I have a dataset and I'm trying to see if there is a correlation between some of the features in my dataset. The dataset contains inspection notes for car parts ...
0
votes
0answers
81 views

Inconsistent autocorrelation plots

I used pandas.tools.plotting.autocorrelation_plot in Python to plot autocorrelation functions of the same time series: for its first 100 and 1000 entries, respectively (code below). ...
0
votes
0answers
81 views

What are the effects of autocorrelation on logistic regression?

I need a simple way to estimate the probability of winning an auction as a function of bid amount. I modeled the auction using the LogisticRegression from pandas, ...
0
votes
0answers
98 views

Calculate corr/cov matrix for large Pandas dataframe/Numpy matrix

I have a large dataframe (gigs), and I would like to compute the corr matrix. What is the most efficient way of doing this in parallel. I have access to many machines and I'm open to using any ...
2
votes
1answer
267 views

Temperature time series forecasting predictions converging to a certain value

I am trying to forecast the value of the ambient temperature based on given data on Python. The data frequency is 15 minutes. In order to predict future values, I am using a simple autoregressive ...
1
vote
0answers
26 views

How to convert my Data set to Gaussian?

I want to convert an Attribute in My data set to Gaussian distribution, but i can't seem to find any formula that i can implement. I don't want to use any Builtin Functions. So is there a formula ? ...
1
vote
1answer
207 views

Deciding between get_dummies and LabelEncoder for categorical variables in a Linear Regression Model

I'm using the dataset http://www.stat.ufl.edu/~winner/data/airq402.dat whose description is here - http://www.stat.ufl.edu/~winner/data/airq402.txt. I'm planning to build a linear regression model ...
0
votes
1answer
29 views

Remove individual points and find slope

I am trying to delete one pair of x and y coordinates from a set of 10 data points and get the slope for the other 9 points. How do I go about this issue? Attached herewith is an image of what I am ...
1
vote
0answers
208 views

Update R2 and slope by removing outliers-python [closed]

I am using the following code to find out Linregress parameters: ...
-1
votes
0answers
31 views

Is it beneficial and practical to perform the same moving average, 10 times (in pandas, df.rolling operation) on an already rolled df?

I have a precipitation data which is very sharp in some places and in my case not predictable for the LSTM predictive model which I'm currently working on. Here is ...
1
vote
1answer
150 views

One-Hot Encoding and Feature Engineering While Avoiding Data Leakage

I have a Pandas dataframe for which I've performed some actions over categorical features: Feature Engineering One-Hot Encoding Let's say that in my dataset I have the features "person_income" and "...
0
votes
1answer
83 views

Do you have to create dummy variables for ordinal data?

Using Python and utilizing XGBoost for classification. I have three columns that is supposed to be categorical and are ordinal (ranging from 1-5). Since the columns do not contain strings, I was ...
0
votes
1answer
23 views

Dealing with a dataset having target values on different scales?

I am currently working on a dataset having 10 features and one continuous target variable. One of the features is 'Country' , in which there are seven unique values [Argentina ,Denmark , France...etc]....
0
votes
0answers
20 views

While performing label encoding or imputation, what should i do to the column with mostly 0's as values which is irrelevant to what column is about?

My DataFrame consists of 2919 rows. Now ,For example I have this column "2ndFlrSF" 2ndFlrSF: Second floor's Area in square feet and these are the values in it after i run my Pandas command ...
6
votes
1answer
5k views

What does (pandas) autocorrelation graph show?

I am a beginner and I am trying to understand what an autocorrelation graph shows. I have read several explanations from different sources such as this page or the related Wikipedia page among ...
0
votes
1answer
38 views

How do you link multiple X input images to a single ground truth image in machine learning?

What I'm asking is basically manual data augmentation. I have some very specific data augmentation for my inputs so I have to create them first in another software instead of doing it on the fly. I ...
1
vote
0answers
102 views

How to calculate timesteps for a 2D-Tensor in LSTM-Keras [closed]

I'm feeding a 2-D tensor in my LSTM network in keras, but I saw that you need to feed 3-D tensor. I''m using embedding layer. Here's my code: ...
2
votes
1answer
38 views

Dealing with the order of features (sequences)?

Assume we have following sequence database that is subsequently converted with one-hot encoding: 1 2 3 4 0 A B C D 1 B A D NA 2 A D C NA One-hot encoded: <...
0
votes
0answers
17 views

How do I improve model accuracy predicting categorical outcome using categorial predictors?

I'm trying to predict Para using Cols. My data is in this format: ...
0
votes
1answer
182 views

Automatic relevance determination Kernel for GPR in sklearn

I have checked the performance of my model with the different kernel and seems ARD kernels, where each estimator has different length scale performance better than normal kernels. ARD kernels are not ...
0
votes
0answers
143 views

Strange output while using Holt’s Linear Trend method

Here're the pictures of using Holt’s Linear Trend method: From tutorial (what it should be like): After running the code for my data Isn't it strange? Here's a code (method #5): ...
0
votes
0answers
34 views

is it better to train autoencoders on the whole XML file or just the textual fields using a Dataframe?

I have an XML file. I want to generate it again using the autoencoders network. I am confused which idea is better : Read the xml file to a pandas dataframe, target the textual fields, use the ...
0
votes
0answers
33 views

Dataset randomisation for more secure and robust analysis? A good idea?

I had an idea (below) for why (and how) it might be a good idea to randomise sensitive datasets before developing analysis for added security and reduction in bias. I realised (I think) it would have ...
0
votes
0answers
175 views

Transaction and Observation matrices in Kalman Filtering for univariate data

I am working on Kalman Filtering and I have questions about observation and transition matrices. So, I have a variable and I want to clean the noise from my variable by using Kalman Filtering. What ...
1
vote
0answers
29 views

Detrending sales based on historical

I have a dataset with month sales for 6 months per customer. I want to compare the sales average of the first 3 months versus the last 3, but I know this averages may be influenced by seasonality of ...
0
votes
0answers
12 views

Check whether two tables come from the same population

I have two data frames and I want to check whether they come from the same population. Essentially, I want to go through the data columns one by one and, depending on column type and how many ...
1
vote
0answers
363 views

Python ANOVA or MANOVA

I have a datasets with samples from various locations. At each of these locations I measured the concentrations of various genes. I saved the results in pandas dataframe and would like to compare the ...
1
vote
1answer
32 views

Approach to analysing big data [closed]

I’m trying to analyse quite a largish dataset. The sasdb7 is around 11GB and the csv is around 9GB. This is after merging the datasets, cleaning and removing unnecessary columns. I am comfortable ...
1
vote
1answer
453 views

Underfitting? Validation scores above training scores

I'm plotting a learning curve currently using the following code: ...
0
votes
0answers
840 views

How to find nearest neighbors using cosine similarity for all items from a large embeddings matrix?

I have an embeddings matrix of a large no:of items - of around 100k, with each embedding vector length of 100. So a matrix of size 100k x 100; From this, I am trying to get the nearest neighbors for ...
1
vote
0answers
79 views

MAPE yielding value of 1000+ for timeseries forecasting

I am comparing a model's outcome with the actual values for a span of 9 months. I am expecting a very large error, however my MAPE implementation yields 1404512.56 using python, pandas. Here is my ...
1
vote
0answers
1k views

python StatsModels use training parameters for test data summary

I ran a linear regression on a month of hourly Nox concentration using StatsModels ols function. how can I get summary OLS Regression Results on the test data? I ...
1
vote
0answers
32 views

Correlation of words and score

Say I have two fields: "Comments" (type text) and "Score" (type float, from 0 to 5) Sample data like: "I appreciate this ...
1
vote
1answer
26 views

Cache/Saved feature engineering result for reusing

I use python pandas for transforming a csv with panda dataframe for feature engineering (e.g., handle missing values, extract feature etc.) and pass it to machine learning algorithm for handling. ...
1
vote
1answer
51 views

Intuitive visualization of data

I am implementing a recommendation engine tool used by several users who are supplying feedback in the form of thumbs up (+1) or thumbs down (-1) for each recommendation. The recommendation engine ...
0
votes
1answer
74 views

Add value to DataFrame to achieve given Standard Deviation [duplicate]

Let's say I have a DataFrame that looks like this value 0 2 1 6 2 5 3 3 4 7 To get the ...
4
votes
1answer
3k views

Incremental learning with decision trees (scikit-learn)

I'm trying to train a regression tree with some very large data I have: approx 3Tb. I'm using scikit-learn and of course there is no way I can load that amount of data on memory. Doing some online ...
11
votes
5answers
3k views

How do I interpret this Scatter Plot?

I have a scatter plot which has sample size which is equal to the number of people on x axis and median salary on y axis, I am trying to find out if the sample size has any effect on the median salary....