Questions tagged [dataset]
Requests for datasets are off-topic on this site. Use this tag for questions concerning creating, processing, or maintaining datasets.
1,885
questions
2
votes
1
answer
170
views
Is batching needed for the test set?
I'm just starting to learn about CNN (convolutional neural networks). Does the test data also need to be divided into batches, similar to how it's done with the training data?
- 21
1
vote
1
answer
72
views
What is an “independent” effect?
According to one study by Wang and colleagues (2013), smoking cigars has “a positive and independent” effect on testosterone. What is an independent effect and what is the significance of one?
- 113
0
votes
1
answer
59
views
Generate marginally dependent (with predetermined covariance) but conditionally independent data from a Mixture of Gaussians
Suppose you have three variables $y\in\{0,1\}$ and $x_1\in\mathbb{R}$ and $x_2\in\mathbb{R}$. I want to produce data with the following generative process which corresponds to a Mixture of Gaussians (...
- 314
1
vote
0
answers
82
views
With more labelled samples in a grouped dataset, to check the correlation with another, is it better to duplicate samples or take the means? [closed]
Another issue is that dataset with more labels in groups, they have less deviation intra groups vs inter groups. So I could take the means of them (the 'second' index), or the whole label means (the '...
- 111
0
votes
1
answer
34
views
Data normalisation for a newbie
I am new to data analysis and I have been given a task to compare safety of trams compared to buses. Since there are far more buses than trams, I was introduced to the concept of normalisation by ...
0
votes
0
answers
11
views
Right Way to Sample a Validation Set
I am working on a project that uses training data selection techniques; it involves sampling the training set in some smart way rather than sampling randomly. The goal is to compare different data ...
- 217
0
votes
0
answers
29
views
Structuring of a large, multivariate, multi-site timeseries weather dataset for a machine learning project
I have access to a large timeseries weather dataset which has values for historical weather (temperature, pressure, humidity, wind, rainfall etc.) across the UK over a large timeframe (approx. 1853-...
- 1
2
votes
1
answer
64
views
Comparison of Data that looks at correlation and absolute values
I am looking for a way to compare two sets of data in order to find out how similar they are to each other.
My application: I try to compare multiple Measurement methods that both measure the sound ...
- 23
0
votes
0
answers
13
views
Random shuffle vs random access [duplicate]
It appears to me that in Machine Learning randomly shuffling data is more common than drawing a random minibatch in every step, e.g. for MNIST in julia
...
- 101
0
votes
0
answers
24
views
Which observations should I use for my panel data analysis?
I am currently working on my Master's thesis and for this I would like to examine the impact of the geographical location of Mutual Funds on their performance. For this I have panel data of mutual ...
- 1
0
votes
0
answers
17
views
Seeking Guidance on Predictive Maintenance with Censored Data
I am currently immersed in a project centered around predictive maintenance for hardware components. The primary objective is to anticipate the impending failure of hardware components or estimate ...
- 1
0
votes
0
answers
15
views
Questions on calculating probabilities from user choice data for GLM training when two objects are equivalent
I’ve designed a GLM to predict user preference when presented with a choice between two buttons. However, there is some disagreement about how I’m providing collected user data to the model, and I ...
- 1
1
vote
0
answers
8
views
Any image datasets with Inter-annotator agreement (IAA) values recorded? [closed]
Is anyone familiar with publicly accessible image datasets that include Inter-annotator agreement (IAA) scores? Ideally for object detection or classification tasks.
Thank you!
- 11
0
votes
1
answer
22
views
Representing Nested Models as an SKLearn Pipeline [closed]
Goal: Represent nested models with SKLearn's Pipeline / ColumnTransformer / FeatureUnion setup.
Specific issue: I cannot figure out how to use the prediction from one model as a factor of a secondary ...
- 11
1
vote
0
answers
20
views
Dataset for Swimming with Heartbeat and Motion Data [closed]
I am currently working on my final project CS degree that involves analyzing swimming data, specifically focusing on both heartbeat and motion data. I am looking for a dataset that includes these two ...
- 11
3
votes
1
answer
44
views
Comparing 2 Likert Scales for 1 Population
I have two 5-point Likert scales (Strongly Disagree to Strongly Agree) and I want to compare the results within one population sample. I expect participants to choose 'agree' or 'strongly agree' on ...
- 31
1
vote
2
answers
55
views
RStudio/Econometrics: transform yearly dataset into monthly dataset?
As explained in the title, I would like to transform a yearly dataset into a monthly one, but including a constraint.
My current dataset gives the yearly production of a commodity, and from year to ...
- 11
0
votes
1
answer
29
views
Why are some datasets denoted as D = {X,y} instead of D={x,y}?
In this question: On the importance of the i.i.d. assumption in statistical learning the dataset is denoted as D={X,y}. In statistics, capital letters are usually used to refer to a random variable, ...
1
vote
0
answers
40
views
How to generate random values based on mean, standard deviation, skew and kurtosis in Python?
Given these values, is it possible to generate random values that conform to this distribution (using Python, but preferably without the SciPy package)?
Statistic
Value
Mean
1.518
Std Dev
24.827
...
- 111
0
votes
0
answers
43
views
What is the best way to analyze differences in demographics data among groups of different sizes
I have groups of individuals in a population. These groups have different sizes--for example, group 1 has 50 individuals and group 2 has 1000 individuals. I want to show the differences in the groups' ...
- 11
1
vote
0
answers
16
views
Examples of interventions changing the Population distribution of a system
When learning about the Datasets of naturally occuring systems you encounter distributions, for example power law distributions in the domain of population sizes of cities.
In my understanding this ...
- 11
0
votes
0
answers
35
views
Interpretation of ACF plot of a Hidden Markov Model
I am trying to interpret the ACF plot of a 3 state HMM model but as it is my first time dealing with any kind of HMM it is causing a lot of trouble. I have also had a hard time finding a ressource on ...
0
votes
0
answers
23
views
Adding pre and post treatment dummies in DiD regression
I want to estimate the impact of economic sanctions on GDP and other macroeconomic indicators on Italy in 1935 with DiD. I have observations of GDP of Italy (treated) and of other countries (control) ...
3
votes
2
answers
92
views
Odds of 2 people sharing the same exact ticket set of a size up to 38
Say user 1 entered the raffle 18 times (18 tickets) and he had ticket ids [0, 2, 3, 4, 8, 9, 14, 16, 27, 28, 30, 31, 32, 33] (so they acquired the first ticket, missed the second (2nd hour ticket) ...
user390427
1
vote
0
answers
34
views
The covariance of a data matrix
Please let me know if the below statement is valid or not ;
Suppose that $X$ is an $n\times p$ data matrix with $p$ features and $n$ data samples.
Suppose further that each feature(column) is zero ...
- 11
2
votes
1
answer
145
views
Monte Carlo cross-validation on an imbalanced dataset
I have been doing a little research about the several cross validation methods but there is an issue that remains as a doubt for me in the Monte Carlo method. Let's suposse I have 2,000 data points. ...
- 79
1
vote
0
answers
16
views
I want to apply Jeffrey divergence / Jensenshannon divergence for two histogram. Where data of histogram is not a probability distribution [closed]
I found Jeffrey formula as
here hi and ki are bin data of two histograms.
https://www.cs.cmu.edu/~efros/courses/LBMV07/Papers/rubner-jcviu-00.pdf
I want to find Jeffrey divergence.
There is any ...
- 11
1
vote
0
answers
22
views
confirm which dataset is closer to benchmark
I have 3 datasets of different sizes and trying to select one that is the closest to the benchmark, in a sense of minimized pairwise differences. I checked for normality using QQ plots and Shapiro-...
- 111
0
votes
0
answers
8
views
What model should I use for this weird dataset? (Survey Response attributes)
Above is a snippet of a dataset I am working with. It is a pull from a research tool presenting a set of many different and distinct attributes that have been sorted into numerous "categories&...
- 11
0
votes
0
answers
12
views
Wrong predictions when using a trained model in deployment
Background: I have trained a deep neural network model with 3 hidden layers of size 32,32,16 for binary classification problem. I have 9 input features and a 0 or 1 as output. My aim is to use this ...
2
votes
0
answers
31
views
How to learn from a dataset of weighted polynomials
I have a dataset of weighted polynomials, i.e. each data point is a polynomial (of variable size/degree) together with a weight vector (of fixed size).
Each data point has an integer label that ranges ...
- 21
0
votes
0
answers
25
views
How to deal with a large dataset when implementing clustering?
I am working with what I believe is quite a big dataset, with $17$ columns (each corresponds to one variable) and $41714$ rows (each corresponds to one observations).
My goal is to implement ...
- 155
0
votes
1
answer
36
views
Two way Anova Hypothesis Testing
I am looking to perform Two way Anova on the following dataset :
...
- 1
0
votes
0
answers
6
views
What is the difference between these two methods of data collection?
I am trying to measure the effect of a campaign on the knowledge of a certain disease on population X
method 1: measure the same individual’s knowledge two times once before and once after passing ...
- 1
0
votes
0
answers
14
views
multiple datasets train-val-test split for time series
Suppose that I have data with dimension $(N,H,F)$, where $N$ represents the number of different datasets, $H$ is the history size and $F$ is the input size. how would you split it into a train-...
- 125
1
vote
0
answers
64
views
Best Datasets and Packages for Comparing LASSO, Elastic Net, and Ridge
I have been recently been working with the MASS, lars, and glmnet packages to study variable ...
1
vote
0
answers
37
views
How to transform percentage cover data into species abundance data in algae
I have collected percentage data for Rocky shore and summarised it per algae type like the example below:
...
- 11
1
vote
0
answers
10
views
Scaling datasets for multi-dataset time series
Suppose that I have training data with dimension $(N,H,F)$, where $N$ represents the number of different datasets, $H$ is the history size and $F$ is the input size. Normalizing each dataset over the ...
- 125
0
votes
0
answers
19
views
How can I use a fixed effects on age without having panel data?
I have a dataset consisting of a case group and a control group, each with 8,000 unique individuals. Among the various independent variables, age is also included for each individual. However, I am ...
- 1
2
votes
0
answers
34
views
How to calculate the area (or Time) spent in each "zone" on a stacked area graph?
I have a dataset of blood pressure recordings measured over 36 hours, which I plotted with BP (aka MAP) on the Y axis, and minutes on the X axis. I then plotted the blood pressure percentiles for age (...
- 21
0
votes
0
answers
25
views
What are the best metrics to use for data drift detection of a Binary classification model on a tabular imbalanced numerical data
I have a very unbalanced 2 data sets, the first is the one that the model was trained on (old training data), and the second is the newly collected data
I want to see if the model performance dropped ...
1
vote
0
answers
17
views
Measurements for calculating the difference of the Outliers has on a Data Set
Given a scenario where I compare a single column(independent variable) in a data set to the other columns(dependent variables). If I find the outliers from the independent variable. What could I test ...
- 11
2
votes
1
answer
33
views
What's the best way to visualize load testing results
I have a piece of code that I'm adding a feature onto, and I need to test performance before and after the feature addition. So I used a test instance with a large amount of data and triggered the ...
- 121
0
votes
0
answers
15
views
What is the appropriate way to input the data of working hours?
In my dataset, working hours is weekly working hours. If the person's employed, then working hours > 0. However, I'm not quite sure how to input the data for ...
- 131
1
vote
0
answers
42
views
Ranked data and hypothesis testing
I have a few questions regarding ranked data and hypothesis testing.
Hypothesis: Car drivers are more likely to consider the car as the most beneficial transportation method for mental health than ...
0
votes
0
answers
7
views
What is a way to estimate job match quality using new job wage, old job, wage, and how long a new job was had?
The data I have seeks to understand whether an individual's new job is of high quality. I have an individuals
old job wage: how much they were paid before,
...
- 3,402
0
votes
0
answers
32
views
Are these data points decaying exponentially or as a power law?
0
I have a set of data points. The first coordinate is time and the second coordinate is energy. I am trying to figure out how the energy is decaying over time. Particularly, I have to find if it is ...
1
vote
1
answer
101
views
How to deal with groups when splitting a data into train and test?
Say i have a dataset with groups that i want to use for a Regression problem that looks like the following where feature1 is the group column:
...
- 11
1
vote
0
answers
70
views
Efficient estimation of conditional probability density
The formulation of the conditional density is:
$$ f_{Y|X}(y|x) = \frac{f_{X,Y}(x,y)}{f_X(x)}. $$
I need to estimate this density from data and it's prohibitively time-consuming to calculate the joint ...
- 11
1
vote
0
answers
27
views
Transformation to compare many distributions with different means and variances
I have daily sales (total volume in dollars) from 200 stores of the same franchise, over two years. I would like to identify any store with anomalies or special patterns, which could be the sign of a ...
- 111