# Questions tagged [dataset]

Requests for datasets are off-topic on this site. Use this tag for questions concerning creating, processing, or maintaining datasets.

1,812 questions
Filter by
Sorted by
Tagged with
1 vote
15 views

### Determine the effect on margins of a price increase

I hope you can help guide me in the right direction! Any advice is appreciated! Situation I'm currently analyzing the effect of a price increase from a retailer on a few 100 products. I'm interested ...
6 views

### Analysing Likert scale responses

I have a dataset consisting of Learner's response for Likert scale questions. The data consists of questions from different courses, therefore not all questions belonging to any particular group. ...
8 views

### How to convert multiple rows into single rows in Python for prediction for next t days?

I have time-series data. I have taken the dataset from Kaggle [https://www.kaggle.com/code/kp4920/s-p-500-stock-data-time-series-analysis/comments]. So, how can I bring multiple rows into single rows ...
10 views

### units of measure vs. unit of observation

I have collected panel data on country level, so I think it's correct to say that country is my unit of observation. However, the variables in my dataset have different units of measure. Some are in ...
32 views

### Skewed distributions: choosing the median or average

Goal: To determine whether to use the median or average (or a weighted combination of the two) from a data set based on whether ...
• 101
22 views

### Efficient storage of functional data

I have access to a sample (size $N$) of functional data. Each observation corresponds to $C$ functions. Each function $f_{n,c}$ is represented by $T_n$ points for $1\geq n \geq N, 1\geq c \geq C$. All ...
• 2,059
21 views

### Practical correlation metric for a large number of vectors

I am dealing with a timeseries consisting of input flow sampled every 5 minutes over 441 days. My aim is to find any possible correlation from data coming from: The same day of the week The same ...
• 11
23 views

### Balancing a dataset in object detection through data augmentation and then random oclusion of classes?

first of all some info about the model I'm using. I'm using YOLOR, which uses a YOLOV5 formatted dataset. I'm trying to detect components off of PCB's which are divided into some groups (visible on ...
11 views

### when we lacked of some colums in validation set, how to use validation set in regression problem?

I am doing the project related with regression problem like failure rate prediction. For training data set, I have columns like below I have 10 columns training data. And for validation set, it is ...
30 views

### Separating datasets vs one dataset with extra categorical feature

I have regression/classification problem. Dataset contains data from 4 sensors on 4 positions (1,2,3,4). Processes measured on all 4 positions are equivalent and same label and features describe all 4 ...
• 3
24 views

### How to choose the best recommender system? What evaluation metrics to use?

I want to build a recommender system to suggest similar songs to continue a playlist (similar to what Spotify does by recommending similar songs at the end of a playlist). I want to build two models: ...
• 1
125 views

### How to evaluate complementary datasets for ML models?

Evaluating ML models is a fundamental task and subfield of the Machine Learning practice. On the other hand, I was not able to find any existing materials, guides, protocols, papers on how to proceed ...
30 views

### Comparing impact of training data size - what testing data size?

I am training a classifier using BERT and want to check how the accuracy changes with increasing training data size. Up until now, I have 1k annotated training samples and tested the accuracy for ...
• 3
13 views

### Moving columns under other columns in R [migrated]

I have to work with an existing dataset and want to move the content of some columns (in this example the content of columns F to J) under the content of columns A to E so that the length of the ...
22 views

### Will a dataset with multiple labels perform better than with binary labels?

Suppose I have a dataset comprised of garbages. Will a model perform better if I only label the dataset with biodegradable or non-biodegradable? Or will it be better if I label them with plastics, ...
• 287
5 views

### Does combining train and test data introduces any potential bias?

Given an estimator $f$ and a dataset $D$ with $k>2$ classes where $S_{train}$ denotes the train set and equivalently $S_{test}$ the test set. Suppose we want to transform the problem and instead of ...
• 102
5 views

### How do I make a user analytics model using logged data

I have a productivity app that I'm currently working on which requires a user analytics panel that logs how much time they've studied, including how much time spent on app navigation, how much time ...
72 views

### Salary of a group of people is continuous or discrete

I have salary data of 3000 employees ranging from 3000 - 10000 dollars. Based on my understanding:(https://mathbitsnotebook.com/Algebra1/FunctionGraphs/FNGContinuousDiscrete.html) Continuous data is ...
• 123
30 views

### Finding an optimal distribution to fit to a highly skewed data vector (DV with missing values) in R

I hope that this question has not already been asked. I am analyzing data in R (and am a novice). I have a highly skewed data vector in a dataframe with missing values that I hope to set as the ...
21 views

### Rule based label - For attrition risk

I have 3 domains of supplier data (Jan 2017 to Jan 2022) and they are as follows a) Purchase data - Contains all the purchase (of product) data made by the suppliers with us. It contains columns such ...
• 1,804
1 vote
8 views

### For the given data find the clusters. Assume the relevant parameters needed [closed]

Below is the given data, how can I make clusters using symmetric matrix?
• 11
1 vote
28 views

### Paired t-test vs two-sample t-test - animal populations

From my understanding, a paired t-test is used when samples are dependent of each other. I'm having trouble deciding whether a paired t-test should be used when comparing the average population of a ...
• 113
14 views

### How do I evaluate if my data represent the target variable before training a machine learning algorithm?

I have a dataset of points cloud where each point in the point cloud has a variable. I am trying to relate the local geometry features to that point variable by using FPFH, This means I am generating ...
12 views

### Effect of duplicate/redundant labels on performance of model

I am training a CNN to predict age,mass and tone from images. The structure of my dateset is as follows ...
24 views

### What are the best-practices for validating phone records? [closed]

I have a bunch of telephone data including information on call start time, end time, and duration. I am trying to evaluate the quality of the dataset to determine if the phone call data are legitimate ...
• 494
15 views

### Hyperparameter tuning on training data vs validation data

If we divide the data into training data, validation data, and testing data, I remember the lesson from Andrew Ng saying we use the validation data for hyperparameter tuning purpose. (you can see this ...
• 957
22 views

### Text classification of posts with very small dataset

I hope this is the right place to ask this. I am a student working on a project about the prediction of the privacy policy applied to textual posts on facebook. The objective is to predict for a post ...
1 vote
8 views

### Finding a dataset for a computer vision project related to medical imaging (related to cancer/tumor) [closed]

I am trying to find a dataset of medical images related to tumor/cancer, there should be images different stages of the cancer and also preferably the details about the patient, their medical history, ...
43 views

### Cross-validation: error estimation and bias

When obtaining the error estimation of a model over a dataset using k-fold cross-validation, lower values of the error estimation necessarily imply a lower bias? Are both concepts, error estimation ...
• 55
64 views

### Synthasize data given mean, variance, skew, and kurtosis in python

I would like to generate synthetic data by specifying their mean, variance, skew, and kurtosis. However, I only know how to generate synthetic data with mean and var. Here is an example with mean and ...
• 121
39 views

### What is a good type of transformation to try on this data to get normal distribution?

I've tried a log-transformation but data becomes left-skewed. In general, when data is distributed like this, what is the next best transformation to try if you want to normalize the data?
18 views

### Question about proper Data Analysis methods for A Discrete Dataset

everyone. I am a fourth-year statistics student and I wonder if I could have some recommendations for proper analysis methods for this dataset. The dataset contains var1, var2, var3, var4, var5, var6 ...
12 views

### comparing data subset to whole

I am trying to write a story about my local high school and its gender diversity in AP programs. I have ethnic data for everyone enrolled, and for everyone in AP programs at the school. What/how is ...
129 views

### The percentile method

a research used the percentile method to find the prevalence in severe group as below: The outcome variable was categorized using percentiles to Mild, Moderate, and Severe in the Symptom Severity ...
41 views

### How to learn a filter on a dataset?

Let $X$ be a a tabular dataset with $N$ features indexed by row number and a categorical value "Cat". Let $A$ be an aggregation function, e.g. ...
• 229
25 views

### Grouping using percentiles

I did visual binning process in spss and made three cutpoints like in this image: I did check off included I want to know the percentage range or value for each group. If I describe it in this way is ...
• 57
16 views

### How do I decide the frequency of data capture for modeling?

I plan to capture data to predict energy consumption in a food processing plant. I want to capture production details such as how much each category of food is produced, what is the machine's output, ...
• 21
20 views

### Really poor result using transfer learning on medical images vs on bees/ants dataset

I am using an Inception V3 model pre-trained on ImageNet and when I train it on Bees and Ants dataset from PyTorch training, I get this result: Dataset statistics: ...
• 514
54 views

### "Ordinal" is to "level of measurement" as "dependent variable" is to __________ [closed]

The Question If I'm drawing up a table that describes some variables -- a data dictionary, say -- and I consider whether a variable is "ordinal", or "nominal", or "interval&...
• 455
66 views

### A/B test with a result of another deep learning analysis with 90% accuracy

I am planning to conduct a A/B test with data obtained through a deep learning algorithm. Say, I got a binary classification dataset through machine learning with about 100k rows classified into yes, ...
• 53
21 views

### About ideal data and its distribution

I was just thinking about what would be the properties of an ideal data set $X \in R^{n,d}$ where n is sample size, d represents features. I think (or at least I understood from reading text books) ...
1 vote
46 views

### Is it true that a larger, representative dataset is always better to use than a smaller, representative dataset?

By "representative" I mean that the data in the dataset faithfully reflects the "underlying signal" a model is trying to tap in to. Is it always true that, as long as increasing ...
• 121
89 views

### Is 6% of your dataset are outliers normal?

My dataset has 80,886 obs and 16 variables. I am using Mahalanobis Distance to detect outliers. And use P-value less than 0.001 as the cut-off. I am getting 5,423 obs as outlier which is 6% of total ...
1 vote
54 views

### In data setup for Cox regression, how to handle a subject's time before treatment of interest (i.e. before time-zero)?

Background I'm designing a study that models time-to-event for two groups of study subject: people who receive a treatment and those who do not. I'm fairly new to applied survival analysis, so I've ...
• 455
1 vote
121 views

### How to solve the problem of having sparse data that would become too small when aggregated?

I have a dataset that provides the count of cyber incidents since 2011 for different countries and different attack types, and I want to use this data in a machine learning model to predict future ...
32 views

### Estimating Minimum of Distribution Despite Measurement Error

I have a dataset $X$ distributed according to some unknown distribution $f$. I know that $f$ is bounded from below by some $M > 0$, but unfortunately, due to measurement error, there exist many ...
15 views

### Having the negative cases in the same batch vs. shuffling the dataset

I am working on a model for an NLP task. The model encodes the text and has a regression output layer. In this task, from each instance (positive), I create several negative cases using a specific ...
• 135
1 vote
24 views

### Measure how dataset is harmonious or organized

Suppose we have two set of numbers : A = [1,4,9,16,25,49...100] and B = [1,4,7,7,25,49,64...100]. As you seen the first one is consistently growing, elements of it is square of numbers. But although ...
12 views

### Comparing two uneven datasets, both of which were sampled in different ways

Dataset X is small (n=100). It was sampled based on a specific variable. Say, it took data based on 3 countries. Something like this: ...
• 101