Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-preprocessing]

A step of cleaning data in data mining for analysis purposes

0
votes
0answers
7 views

Features with missing entries are different in train data than in test data

I know there is a number of approaches to preprocess training data with missing entries: dropping features, imputing mean values, etc. I've compared few of such approaches and found that dropping ...
0
votes
0answers
9 views

Confused about z-score image normalization output

I am trying to normalize my input data for a convolutional network, I applied the z-score normalization technique to my image dataset as follows: Formula: (image - mean(image)) / std(image) ...
0
votes
0answers
18 views

Preprocessing and hyper-parameter tuning within cross-validation

My first approach was to split the data-set into training and test set. Thereafter, I preprocessed the training set (normalizing and imputing the missing values) and used cross-validation to tune the ...
1
vote
0answers
17 views

which normalization method is appropriate for regression problem?

I have a regression problem. I have 3 features: X1 is in the range (0.7 , 1.3) X2 is in the range (0.4 , 0.5) X3 is in the range (4.5 , 5.5) and Y is in the range (115 , 719) I started trying ...
0
votes
1answer
27 views

why does preprocessed test data change with change of calibration data in PLS-DA?

why does preprocessed test data change when calibration dataset (and model based on that data) changes? i have spectral, normalized datasets, the preprocessing was 1. derivative + autoscale. for ...
2
votes
1answer
44 views

In Convolutional networks, how to do input data normalization? is it necessary?

I'm wondering about data normalization in CNN, how can we do it for the input images?, what can it add to the model's performance? and what are the main pre-processing techniques before doing the ...
0
votes
0answers
18 views

Cross Validation and Feature Selection with Chronological Split and Feature Preprocessing

I have a task with daily entries for which I need to do binary classification. Suppose I have 18 months of data and the model is refit every month. In addition I've got about 150 one-hot encoded ...
0
votes
1answer
18 views

How to standardize data with low variance?

I have quarterly data of federal fund rate (test set), e.g.: ...
0
votes
0answers
13 views

WoE of test dataset

How to encode the categorical data to Weight of Evidence in the test dataset? I am interested to know about how to handle the categories which are not present in the training set.Or should we combine ...
0
votes
1answer
33 views

Categorical feature with one dominant value [closed]

in my dataset for machine learning I have many categorical features with one dominant category as you can see on the picture. How can I deal with this kind of categorical data in preprocessing?
1
vote
1answer
29 views

Preprocessing gene expression dataset

Given a gene expression dataset with 99 samples and 10000 features, it is required to find clusters of samples in the dataset. Now taking the features and finding their means and subtracting those ...
0
votes
0answers
11 views

DeepSets design when sets may have varying number of sample elements with some fixed value?

Trying to make use of a DeepSets machine learning design (as described here) where in order to model data that is given as per-item samples when sets of the samples should actually be considered ...
0
votes
1answer
41 views

Preprocessing - dealing with natural NaN values

I am wondering how to deal with a variable having what I call natural NaN values. For example, a measure of duration between 2 events. If one event did not occure the variable has no value. For that ...
0
votes
0answers
23 views

Audio files and their corresponding spectrograms for image classification process

Suppose I have a dataset of audio files that I have to use for whale sound classification. I am choosing the strategy of treating it as an image classification problem by using their corresponding ...
0
votes
1answer
50 views

Pre-Processing audio data for whale sound classification using CNN

Previous researchers have used techniques like Denoising using Spectral Subtraction method and calculating Short Time Fourier Transform (STFT) by dividing the audio data into fixed size chunks and ...
0
votes
1answer
26 views

prepare a data set to be fed into a machine learning model

so I am pretty new to the whole Machine Learning concept and I'm coming from a complete beginners standpoint. I am trying to detect whether there is steganography present within jpg files using a ...
0
votes
0answers
30 views

Multiple Entries for Same Participant

I have raw data that I need to transform and unsure as to how. Manually doing it is out of the question due to the thousands of entries. I have the data in excel and I’m looking to analyze in SPSS. ...
0
votes
1answer
49 views

How to deal with mixed data type in deep neural network

My dataset has 300 numeric features, each of them ranges from 1 to 500. In addition, I have 1000 categorical features (0 or 1), around 90% are 0's (kind of sparse). To run deep neural network, I ...
1
vote
0answers
52 views

Longitudinal Data with Equal Outcomes Within Individual Samples

I need to prepare some data for plugging into a predictive model. The data is in tidy format, but it comes from an audit table, i.e. every change made to a record is recorded and stored as a separate ...
0
votes
0answers
14 views

How to approach node/graph classification in an event?

I'm facing a new project and thought about maybe going in the direction of Graph-Neural-Networks. My data comes in the form of events (unrelated to each other), the data in each events contains a 2D/...
0
votes
1answer
24 views

Image Augmentation or incrementing dataset by flipping/mirroring?

My task is a regression task, where an input image results in another, transformed image. So far so good, works quite well. As my data set is fairly small, I want to take some actions. Here I wanted ...
0
votes
0answers
21 views

Types of preprocessing for Deep Learning NLP tasks

I am doing some research with Deep Learning NLP tasks. There are many ways of text preprocessing. Some are removing stop words. Others convert to lower case, do stemming, or lemmazation. Others do ...
0
votes
0answers
7 views

CART model: Performance differences depending on preprocessing

I am trying to fit a CART model on a spam dataset, trying to predict "spam" or "ham". Usually, I work with the "tidy" packages for preprocessing; however, I now learnt about another way in a tutorial,...
0
votes
0answers
12 views

how to calculate the difference between multiple distribution(or frequency list)

Here is the scenario: I have a dataset, which contains list of data points, each point has F features(i.e. float numbers) and a category(there are C categories). I want to compare the difference of ...
0
votes
1answer
53 views

text preprocessing using keras [closed]

I am getting started with NLP, in kaggle , and it dont get how this keras preprocessing works if anyone could explain the code would be much helpful,thanks ...
0
votes
1answer
28 views

neural network: How Standard Scaler can affect to node weights learning

I and learning about neural network, and i suspect about how effective of our feature scaling for deep model. In my opinion, Standard Scaler can lead to a bad results of neural network. Let look at ...
0
votes
1answer
64 views

Random Forest and preprocessing in Data Mining

When applying the Random Forest classification technique, do we have to do preprocessing or is it true that it is not needed for Random Forest ?
2
votes
1answer
58 views

Clustering / Grouping on image's pixels

I have an image, and im building a model to recognize a pattern in that image and classify it. There is however a lot of noise in the rest of the image, but the actual pattern to classify will always ...
1
vote
0answers
45 views

How do I measure information loss when converting categorical data to numerical?

Assume that a dataset has a mix of categorical and numerical attributes. The dataset has to undergo numeric processing which necessitates the conversion of the categorical attributes to numeric/...
0
votes
1answer
53 views

analysis of different time series

i have 26000 items of a shop (rows) and the sell quantity in every week of 2017( 52 columns). i'm doing forecasting: this is the goal. Now, however i'm in the data preprocessing step and i want ...
1
vote
1answer
57 views

Should I use other scaling methods for pre-processing the data rather than normalizing or MinMaxScaling?

I have a weather parameter (daily volume of inflow for a river in million cubic meters, MCM) time series data as follows: I want to scale this data and feed it to a ...
1
vote
1answer
131 views

Handling daily time series data for better accuracy

I have a daily observation of call volumes data starting from 28-01-2017 to 31-08-2018 a little over one and half year.On sundays calls volume are less and monday the highest showing weekly pattern. ...
0
votes
0answers
21 views

Improve Accuracy over Image dataset in Convolutional Neural Network

I a newbie to Convolutional Neural Networks and for some dataset I trained my ConvNet model and achieved some accuracy. Now, I increased some filters and one or two ConvNet layers and saw some ...
1
vote
2answers
232 views

Handle Categorical Variables in Machine Learning in Python [closed]

I have $4$ variables in the data-set, each has more than $50$ levels in them. I want to include all these variables in my predictive model. How should I handle these categorical variables? If I do ...
0
votes
0answers
36 views

How to pre-process features from different domains for Machine Learning models [duplicate]

Eventhough my question is applicable for all kind of models, I asked it in the scope of SVM for now. Assume I have 3 sentences in my training set and 2 sentence in my Test set. I would like to ...
0
votes
0answers
55 views

What is the best way to remove outliers in the task of forecasting demand?

I have dataset with over 2500 IDs of products. For each ID I need to remove outliers. I removed them by the condition, that everything what lies over 1.5 * IQR should be deleted, but it seems that ...
1
vote
1answer
27 views

How to approach preprocessing large number features for machine learning?

I used to apply supervised machine learning for maximum few dozen "normal", natural features like human interpretable ones in Boston House Prices table. I usually try to understand each of them, think ...
1
vote
1answer
92 views

Why we don't normalize the images?

I was watching the video from this stanford course on convolutional neural nets where the professor says (at 28:59) 'we do zero-mean the pixel values in image but we do not normalize the pixel values ...
0
votes
0answers
61 views

Data Scaling: is multi-dimension scaling equivalent to uni-dimension scaling for same range features?

I am given an $n \times m$ matrix $\mathbf R$ where $n$ is the number of $users$ and $m$ is the number of $items$ - this matrix is usually known as the rating matrix within recommender systems domain. ...
2
votes
1answer
72 views

“We have to apply feature scaling on test set using scaling parameters from train set.” Is this statement true? If yes, why?

I understand that in standard data cleaning and pre-processing pipelines, we have to make sure that the information from the test set (or what would be the test set after splitting) does not leak into ...
0
votes
1answer
30 views

List of machine learning classifiers that naturally assume data in normal distribution

I have searched about this question but no answer really made it comprehensive. To my knowledge, linear regression and most clustering algorithms naturally have the assumption that data need to be in ...
1
vote
1answer
222 views

Does it make sense to preprocess (normalise or standardise) this data for GAN?

I'm working on a project where I have a dataset for a dynamical system (pendulum) containing a trajectory, energy cost and corresponding control actions (See below). I'm using a generative adversarial ...
0
votes
1answer
30 views

How to preprocess the Categorical Data with large number of columns

I have large number of categorical columns in my dataset, I want to preprocess the data, I know that I have to do one hot encoding but in data set columns or not in specific order they are at random ...
1
vote
1answer
75 views

Some convenient way to transform time series into stationary one

In this Machine Learning Mastery post we read that in order to predict time series using LSTM network, it is good to make the data stationary first and then scale it to the interval $(-1, 1).$ In ...
0
votes
0answers
36 views

Should you reshuffle your dataset after you use five or ten crop data augmentation in general machine learning?

My data was shuffled randomly first then I applied a five crop data augmentation. Now my batch went from [8, 3, 256, 256] to ...
0
votes
0answers
52 views

Applicability of Queueing models for data pipelining?

I have to write a system that has several data processing steps, each varying in time complexity. I intend to make use of buffer queues and multithreading for each preprocessing step. I have limited ...
3
votes
1answer
44 views

Can a ConvNet see patterns that a human cannot?

I am training a ConvNet to detect different types of stripes in my images. As I am working on astronomical images, my pixel values are flux densities and therefore represent ground truth data. When I ...
2
votes
2answers
414 views

Should I normalize featurewise or samplewise

It might be a beginner question, but I'm not sure how to normalize my data. Let's suppose I have a NxM matrix with N samples of M dimensions each. If I want to normalize my data I can do it in two ...
1
vote
1answer
49 views

NN works well on preproccessed data, but results are poor after de-normalizing - predicting non-normalized prediction values using NN?

So I'm trying to build a neural network to fit an approximating function to a data set. The data set consists of (input,output) paris where the input features are discrete numbers [0,1,2,...,1027]. ...