The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [data-cleaning]

Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.

Filter by
Sorted by
Tagged with
0
votes
0answers
17 views

Creating a single scale using multiple variables

Hello everyone I am new to the forum. I have a question about how to operationalize variables in survey data. So, The survey asks respondents to give their opinion on a number of issues. There are 6 ...
2
votes
2answers
37 views

Is data that has been entered incorrectly treated the same as missing data?

I am doing an online study and have just started looking at the data. I noticed two of my participants have listed ages that they couldn't possibly be (e.g 450 and 220). I'm wondering what the ...
0
votes
1answer
33 views

Standardizing data produces negative values

I am working on a basic house price prediction problem with traditional ML algorithms, not NN since the size of data is small comparing to the number of features. The issue I am having is that many ...
2
votes
1answer
12 views

what should be done first, handling missing data or finding correlation between features and drop irrelevant features?

In data science, Which process should come first, handling missing data or finding correlation between features. I am asking this question because I have problem in following cases: 1) Handling ...
4
votes
2answers
149 views

what should be done first, handling missing data or dealing with data types?

In data science, Which process should come first, handling missing data or handling data types. I am asking this question because I have problem in following cases: 1) Handling Missing data first, ...
1
vote
1answer
26 views

Is there a point where you wouldn't use dummy variables? I.e., if getting dummy vars would lead to hundreds of variables? [duplicate]

I built a web scraper that drew in a bunch of data and I have more qualitative variables than I expected. Originally there were just a few quantitative variables that I had intended to consider but, ...
-1
votes
1answer
67 views

Which is the correct method for outlier analysis on a dataset for modelling?

I'm trying to build a regression model but my data-set have many outliers points which I need to analyze and then remove them. There are two ways to do it, 1) First do all the analysis on every ...
-1
votes
1answer
27 views

group_by() and summarize() is not resulting in one row in the output for each group [closed]

I am learning basics of {dplyr} package in R and working with summarize() function. When I create groups using group_by() ...
1
vote
1answer
58 views

Normalising data with only min and max values [closed]

I'm working with salary datasets and wants to normalize the data as much as possible. I have data in the form of: years of experience, salary range 0-1, 28-34 2-5, 32-44 ... Each dataset uses ...
0
votes
0answers
13 views

Time series classification - How much overlap?

I am trying to perform a sequence classification using LSTM. Let's say I have two time-series, series A and series B. Time series A has a length that is almost 100 times series B. I like to develop a ...
0
votes
0answers
31 views

How to build a machine learning model from sparse features with class imbalance?

I have around 10 numerical features and 1 class/target (e.g., visitors count of a website). All of them are sparse. Sparsity is around %70-80. The median of the class/target is zero. Is there a good ...
0
votes
0answers
11 views

Converting a grid (or correlation matrix) to a single-axis table which summarizes the relationships

Say I have a correlation matrix/ grid (in Excel), similar to as follows: I'd like to convert this into a table whereby each interaction is captured, such as: I'm an R novice, but would like to ...
-1
votes
3answers
95 views

How can missing values in the target variable be substituted using Python?

I have a dataset with some missing values in the target variable (label). Can I use clustering to find those missing label values? What other methods can be applied to resolve such an issue in Python? ...
0
votes
1answer
159 views

Pre-Processing audio data for whale sound classification using CNN

Previous researchers have used techniques like Denoising using Spectral Subtraction method and calculating Short Time Fourier Transform (STFT) by dividing the audio data into fixed size chunks and ...
0
votes
0answers
34 views

Multiple Entries for Same Participant

I have raw data that I need to transform and unsure as to how. Manually doing it is out of the question due to the thousands of entries. I have the data in excel and I’m looking to analyze in SPSS. ...
0
votes
0answers
20 views

Clearing out errors from a data set

Sorry for the vagueness of the title, I am having a hard time even coming up with sort of problem I am facing (if there is a specific name for it....) In a nutshell, I have a time series of points, ...
2
votes
2answers
403 views

Is it 40% or 0.4%? [closed]

A variable, which should contain percents, also contains some "ratio" values, for example: 0.61 41 54 .4 .39 20 52 0.7 12 70 82 The real distribution parameters ...
2
votes
0answers
66 views

How to fill NaN values that exist because there are no measures of certain features?

I'm currently doing a ML project (the goal is simply to clean the data set and apply some of the models we learned , like Random Forests, Ensemble learning, etc, and test the results) for a class and ...
0
votes
0answers
26 views

How to verify structural errors in a data set with huge number of variables

I have a data-set with 80 predictors and 1 outcome. As part of data cleaning i want to see the structure of each data, how they are distributed and whether there are any culprits(like NULL instead of ...
1
vote
0answers
23 views

Underperforming participants (<50% accuracy) in LMM

I have to perform a linear mixed model analysis on behavioural conflict paradigm data (ie analysis of congruency effects) and I'm struggling to find reliable sources on what to do with respect to ...
3
votes
2answers
157 views

Regression algorithm on [0,1] with lots of mislabeled data

I have a training set mapping some Likert-scale variables (integers between 1 and 7, rescaled to real numbers between 0 and 1) to predict a continuous variable between 0 and 1. The data set is ...
1
vote
1answer
269 views

How long does it take to clean data? [closed]

I am trying to plan out how long it will take me to clean my survey data. I have about 200 responses. The survey takes about 15 minutes, about 40-60 questions (depending on the logic). I have very few ...
1
vote
0answers
31 views

Two predictor variables are partially related. How do I handle them before performing logistic regression

I have a predictor variable which has three levels (success, failure, nonexistent). For all nonexistent values another predictor variable value is 0 but success and failure could fall within a range ...
0
votes
0answers
103 views

Distribution Comparison on Multiple Dimensions in Python

I work as an analyst at a company that is in the midst of transitioning to more cloud-friendly tools. That means we've gone from using SAS/TD SQL (from my actual analyst days) to working with Python ...
0
votes
2answers
32 views

What is the rule for deciding when to normalize Variables In pre-processing?

Some techniques, Like boosting For classification, Do not require The Variables to be normalized.For other techniques, Normalization seems very important How Do I know When I need to normalize My ...
0
votes
0answers
103 views

Simulate/generate more data points off small dataset and constraints

I have a small data set only 18 data points, is there a way i could simulate or fabricate more data points but in the confides of those 18 data points. Here is a screenshot of the data I hope my q ...
1
vote
0answers
20 views

is there any tools or fast-track techniques through which millions of review data can be clean automatically

My research domain pertains to Deep Learning, and with reviews. To make better feature selection and extraction by deep learning algorithm I have a collection of close to a million review data. But, ...
0
votes
1answer
35 views

How to remove cofounding effect on a variable?

I'm working in a team that is collecting data by bicycle : We have biometric t-shirts that measure our ventilation rate. The problem is that during the last data collection, participants used masks to ...
-1
votes
1answer
20 views

Concatenating distinct values contained within a string [closed]

I have a string variable (drugs) that contains a list of drugs prescribed at each line of treatment. I'm looking to create a new string that contains only the unique drugs from each line. In the ...
-1
votes
1answer
320 views

Conversion of Interval data to ordinal data

I recently learned about Level of measurements and I am really confused in this MCQ where I think the right answer is a and d both. Lets say the three participants finished in a race in 45 seconds,35 ...
2
votes
1answer
201 views

Is it possible that GridSearchCV make the test data leakage?

I have 1000+ records of dataset. After I clean all the training data then I perform the GridSearchCV library to the training data. So the cleaning step that I've done is filling some missing value and ...
7
votes
3answers
143 views

Using ML to assist human labelling in dataset with highly unbalanced classes

Are there scientific issues with using ML to assist human annotation? I've got a 3 class unlabelled dataset where only 1 in 500 elements belong to the 2 classes of interest. The labels arn't ...
1
vote
2answers
60 views

PCA variances pattern changed greatly after data cleaning

I have data, when I normalize it and then performed PCA, I calculated the variance of PC components, I found that, the first component is 72% and seconed component is 8% (total 72+8=80%) and so on. ...
0
votes
0answers
828 views

MCAR test for large number variables and small sample size

I have a dataset with 101 observations and 402 columns (those columns comprise several multiple-item questionnaires). Among those 402 columns, 10 of them are categorical and the remaining are ...
0
votes
1answer
28 views

How to handle a column having the number of children with some values being represented with some dummy value?

I have a dataset which has a column representing the number of children. The values vary from 0 to 6. But the issue is that there are many entries which have values like 84, 94, 97, 98 and 99, which ...
1
vote
1answer
620 views

Is Tomek Link undersampling the same as Edited Nearest Neighbours with 1 neighbour?

From what I've read I've understood that undersampling the majority class with Tomek Links or Edited Nearest Neighbours with 1 neighbour should yield the same result. However, I've tried it on this ...
1
vote
0answers
45 views

making n-grams, best practices

if I encode my data in 2-grams, thats (26+26+10)^2~3800 possible pairs if I use 3-grams that's ~200,000 possible triplets. We can reduce this number by using only lower case, but basic combinatorics ...
0
votes
1answer
44 views

Data cleaning: Derived variables

I am cleaning data that I will use with machine learning prediction algorithms. Several of my variables in my data set are sums of other variables. eg) given variables x1, x2, x3, x3=x2+x1 or even x4=...
0
votes
1answer
35 views

Should I be cleaning data that needs to be classified?

I have a Naive-Bayes model that does sentiment analysis. For the training of the model, the training data was cleaned, i.e.: stop words were removed, certain punctuation, etc... When I want to ...
0
votes
0answers
53 views

Handle data with high variation

I'm analyzing som product data that shows sells over the past day, week and month. In some cases I have a very high number but in most cases this number is quite low, and this is affecting quite a lot ...
1
vote
0answers
247 views

Multivariate imputation in R

I am working with a dataset from the world bank, it's a relatively simple dataset with 11 variables for 211 countries from 1990-2015: ...
2
votes
2answers
1k views

Outlier removal before Boosted Decision Tree Regression

I am using Boosted Decision Tree Regression to predict a value. I get a few values where the residual (True Value - Estimated Value) in my training and test sets are large. I have a reason to believe ...
0
votes
1answer
3k views

how to handle sparse data problem in unsupervised learning .i'm going to use k means on data set

how to handle sparse data problem in unsupervised learning .i'm going to use k-means on the dataset. I have 200 variables, nearly in each column have 70% zeros. how can I handle without discarding any ...
6
votes
4answers
7k views

Difference between preprocessing train and test set before and after splitting

Is there a difference between doing preprocessing for a dataset in sklearn before and after splitting data into train_test_split?...
1
vote
2answers
68 views

Which statistic test should I use?

For a task on data cleaning and analysis using SPSS I have two research questions for which I should pick an appropriate statistical test. It's been over a few years though since my last statistics ...
0
votes
0answers
1k views

Improving RMSE of my model

I'm trying to build a model based on some training set. The training set contains 1460 observations, with 79 variables each (features). I'm using linear regression to build a model and after that ...
1
vote
1answer
368 views

What criteria to use when having with 'unengaged' , ‘straightlined’ or ‘patterned’ responses in questionnaires?

Please consider the following situation, which happens a lot in real life applied research in social sciences. Some percentage of the respondents give ‘straightlined’ or ‘patterned’ responses to a ...
0
votes
1answer
41 views

Should I remove data with known computational error before doing linear regression?

I have a dataset having 252 observations. There are two variables having a defined and known linear relationship. By computation, I found 4 observations do not follow the linear equation, and I ...
6
votes
0answers
88 views

Does tidiness of data differ by application?

After reading a recent paper by Hadley (link), I got to thinking about whether what we'd refer to as tidy data changes by application. For example, consider a sample dataset: ...
2
votes
1answer
834 views

What is the best way to perform sentence segmentation for textual analysis?

I am working on textual dataset containing data from official documents like reports by companies, legal documents, speeches by directors to shareholders etc. The content of textual documents are like:...