Questions tagged [synthetic-data]
The synthetic-data tag has no usage guidance.
87
questions
0
votes
0
answers
25
views
How can I model the multivariate probability distribution of a dataset with both continuous and discrete variables for sampling?
This might seem like a duplicate of the following link, but I think that one is asking how to create a completely new dataset with specific distributions, rather than how to model an existing dataset ...
- 252
1
vote
1
answer
31
views
Can I do a meta-analysis by Monte-Carlo synthetic data?
I'm trying to do a meta analysis of ~30 studies (total N = ~2000) on the correlation (X, Y). However, the heterogeneity is soooo high. My hypothesis (and what has been suggested in the literature) is ...
- 23
1
vote
0
answers
47
views
How to visualize time series using PCA?
I have two multivariate data sets comprised of 100s of time series, one is the actual recorded data set of time series and the other is a synthetically generated data set based on the recorded one.
...
- 31
5
votes
1
answer
69
views
What type of data should I generate to observe/amplify a crossing problem in quantile regression?
1. Background
Crossing problem in quantile regression can be observed when we want to estimate several conditional quantiles (e.g. τ = 0.1, 0.2, . . . , 0.9), as two or more estimated
conditional ...
- 135
0
votes
2
answers
171
views
Generating synthetic time series data with limited data
I would like some opinions on my current situation.
I have a set of time series data that I want to forecast. The data however is not very long (around 500 rows) so I was looking into generating many ...
1
vote
1
answer
40
views
Synthetic vs Data augmentation for low dimensionality data
I have problems understanding data augmentation. I currently have low-dimension features, each data point only has 3 features. A total of 20k non-linear data with only 3 features. I have generated ...
- 23
1
vote
0
answers
150
views
How to generate synthetic data that respects pairwise correlations of features of the real dataset?
Let us suppose that I have a dataset with 3 features and that I know the pairwise correlations among these features.
Let us suppose that I want to build a synthetic dataset that respects those ...
- 678
3
votes
1
answer
127
views
Why use a copula to generate synthetic data?
For class, I am tasked to generate synthetic stock data using the copula R package. The step-by-step process is picking 2 stocks (i.e., Amazon & Apple), fit their marginal distributions (I am ...
0
votes
1
answer
142
views
Creating synthetic data for time series, Hidden Markov Model
Suppose that I have a task of classifying a time series. I decide to use Hidden Markov Model $\lambda(A, B, \pi)$, where $A$ is a transition matrix, $B$ is an emission probability, $\pi$ is an initial ...
- 288
1
vote
2
answers
205
views
Is SMOTE any good at creating new points?
Cross Validated has a pretty thorough debunking of class imbalance being an inherent problem for SMOTE to solve.
However, SMOTE is explicitly a method for synthesizing new points.
Is SMOTE any good at ...
- 51.2k
1
vote
1
answer
70
views
Which tool is more suitable for visualizing the distribution of multiple real and synthetic image datasets, t-SNE or PCA?
I am doing a thesis on the generation of synthetic data for training a deep learning model and evaluating it on real data. I have a few different real datasets, and I generated multiple synthetic ...
- 177
0
votes
0
answers
37
views
Ordering when using scipys Jenson Shannon distance
I am currently the scipy implementation of the Jenson-Dhannon distance to compare 2 vectors sampled from 2 distributions.
I'd expect the distance to be zero if I get the same samples - regardless of ...
- 101
3
votes
1
answer
310
views
Why is the sum of individual Spearman's rho squared less than 1 as opposed to Pearson's r in a synthetic example?
A relatively low number of iid random vectors of a relatively high dimension (10,000) are added up together element wise:
$$\sum_{i=1}^{n}X_i=Y$$
where $dim(X_i)=dim(X_j)=dim(Y),\forall i,j$ and $dim(...
- 291
0
votes
0
answers
22
views
Where to start creating a -synthetic 3D object- based neural network in combination with sensor data?
I have a question on where to start with a project of mine. It includes a wide variety of expertise, so I am not sure if am at the right part of stack exchange. My project is as such:
I have a cube ...
- 1
1
vote
1
answer
67
views
Generate a syntetic log-normal two dimensional random field
I would like to test some functions that I wrote related to the kriging applied to rain data. In order to do that, I would like to generate a synthetic log-normal 2D random field.
The idea is to ...
- 111
1
vote
0
answers
116
views
FID as a metric to evaluate the quality of synthetic datasets (Non GAN generated) for training models for a given classification task
I am working on a problem of generating synthetic data (algorithmically by blender, not using GANs) to aid the training of some CNN for a classification ask. Ideally, I want to generate an algorithm ...
- 177
0
votes
1
answer
178
views
Tried so many different models but cant get good accuracy
I am very new to the field of Machine Learning. My college seniors provided me a dataset to analyze and predict.
The data is purely synthetic, with 14 feature columns and a target column having values ...
- 103
0
votes
1
answer
65
views
How to create synthetic data for this case?
Some weeks ago, I ran an experiment with 30 participants. For each participant two microphones were recording them while they were reading a phrase in 3 different simulated emotions (Happy, Neutral ...
- 45
0
votes
1
answer
162
views
Simulating Hierarchical Data
I want to simulate a dataset that has a "grand mean", and then group means (with some deviation from the grand mean). Nevertheless, I have a bit of a problem conceptualising the problem: if ...
- 137
1
vote
0
answers
36
views
Metric to assess similarity of distributions
I am working on clinical synthetic data and I would like to learn more about metrics to compare synthetic vs measurements distributions.
As there are methods to generate synthetic distributions with ...
0
votes
1
answer
98
views
Synthetic multivariate time series for anomaly detection
I built an anomaly detection classifier which worked perfectly with the anomaly detection task in my dataset (multivariate time series). Now I'm trying to understand what are its weakness and my idea ...
- 115
3
votes
1
answer
61
views
Synthetic data set generation for binary classification based on paper (interpretation problem)
I'm reading a research paper about fraud detection (unbalanced binary classification) where the authors go for synthetic data for evaluating their methods. I want to reproduce their synthetic data but ...
- 725
1
vote
1
answer
18
views
Why is the standard error in a diff-in-diff with 4 datapoints 'not well defined'?
I was looking at this online book https://matheusfacure.github.io/python-causality-handbook/15-Synthetic-Control.html in order to learn about synthetic control. The author gets two cities in a panel ...
- 73
1
vote
1
answer
39
views
Generating data for one-class dataset
In case I have a dataset that have only one class unlabeled (benign), could you please list some algorithms/papers that are used to generate complementary data (malignant) based on benign data only? I ...
- 249
0
votes
1
answer
24
views
Composite Indicator - Negative Correlation Issue
I'm trying to build a composite indicator that aims to measure poverty. I'm planning to aggregate a series of variables using PCA. However I have a doubt, all the variables I want to include are ...
- 13
1
vote
0
answers
63
views
Generate data that matches a frequency distribution while preserving the original spatial structure
I am dealing with a 3D array containing values representing the "importance" of each voxel. For my analysis, I would like to synthesize n new arrays from my original array to have a ...
- 161
0
votes
0
answers
23
views
Is creating artificial class imbalance in synthetic training data a good way to tackle hard cases in classification?
I have a problem where I need to classify something around 50 different classes. Some of the classes are very similar to one another and the algorithm tends to confuse them. However, I can create a ...
- 63
1
vote
0
answers
16
views
Learning Distribution of Data [duplicate]
Sometimes it's important to generate data due to data imbalance issues. I heard that data augmentation by leaning distribution of data is a hot topic now. Could you please give me some resources and ...
- 249
1
vote
3
answers
169
views
How to generate synthetic data from a balanced dataset?
Let say I have a balanced dataset that has a small training sample size (lack of data).
How do I increase the training sample size by generating synthetic data based on the original data?
I believe ...
- 29
0
votes
1
answer
63
views
What can be a good way to generate data(tabular) from statistical facts and probability of data?
For Example, If I have facts saying that:
50% of humans are male
30% of males are Indians
70% of Indians are brown
average age of Indians are 27
30% of indian females are working
80% of no-indian ...
0
votes
1
answer
29
views
Simulated data for statistical framework testing
I want to generate two sets of simulated data (numeric) for statistical testing. Is it possible to generate datasets with predefined RMSE or accuracy?
- 51
2
votes
1
answer
396
views
Generating samples on an exponential distribution
I am trying to generate a synthetic earthquake database where the number of events ($N$) with magnitude ($M$) in the range $[M, M+\delta_M]$ follows:
$\log_{10}(N) = a - bM$
where $a$ and $b$ are ...
- 21
3
votes
1
answer
517
views
Does it make sense to use the KL-divergence between joint distributions of synthetic and real data, as a evaluation metric?
The KL-divergence is defined as:
$D_{KL}(p(x_1)∥q(x_1))=\sum p(x_1)\, \log \Big( \dfrac{p(x_1)}{q(x_1)} \Big)$
I consider the Kullback-Leibler (KL) divergence as a performance metric for data ...
- 31
0
votes
0
answers
42
views
Constrains on the coefficients of SARIMA
I am trying to generate synthetic time-series through SARIMA random process by defining the model coefficients manually. Could any one help me how to generate coefficients? What are the constraints on ...
- 103
0
votes
0
answers
162
views
Generate more data for a small dataset
I have been working on a dataset which has 14 attributes and 303 rows(instances) along with the binary labels.
I want to generate more data so that I could train my neural networks so that I could ...
1
vote
0
answers
54
views
Regression on unevenly distributed high dimensional dataset
I have a very high dimensional (20K+ hand engineered features) biological dataset to predict a single continuous output variable (such as a mental state exam scores for a dementia patient). The output ...
- 11
0
votes
1
answer
56
views
how to synthesize a virtual control arm from multiple historical trials with different treatment in their control arms
We want to synthesize a virtual control arm for a single arm study, and we found two trials with data available that targeting the same population. One trial used Placebo+Standard of care in the ...
- 469
23
votes
10
answers
10k
views
Best term for made-up data?
I'm writing an example and have made up some data. I want it to be clear to the reader this is not real data, but I also don't want to give the impression of malice, since it just serves as an ...
- 10.6k
0
votes
1
answer
115
views
Generation of synthetic data for Hierarchical clustering
I wanted to test various hierarchical clustering algorithms to check which algorithm performs best. For this, I was considering simulating some ground truth. Is the possible to generate a correlation ...
- 205
1
vote
1
answer
761
views
Combining treatment units for synthetic control method
Cross-posting this from the Stata forum to increase visibility - but let me know if that's an issue!
I am thinking about using the synthetic control method to evaluate an intervention that took place ...
- 193
0
votes
0
answers
69
views
Using Synthetic Data Based on Real Data for Classification
The goal is to classify three different cell types based on certain features (e.g. area, shape tensor etc.). However the amount of labelled training data I have is very small.
Therefore, it was ...
- 1
0
votes
1
answer
107
views
Synthetic Control Method with Correlated Variables
I am interested in using the Synthetic Control Method to measure the effects of a certain policy. I am afraid that the synthetic control will not be accurate because the underlying variables that make ...
- 103
1
vote
0
answers
113
views
Creating synthetic data from existing data
I have data with 8 features and 900 rows each. Those features are dependent on each other in some way. I want to conduct machine learning, but the data size is small.
I was suggested to generate ...
- 11
2
votes
0
answers
283
views
Generating Correlated Random Variables following Weibull Distribution
I am working on analyzing power generation from geographically distributed wind farms. I have wind speed data that have been collected at multiple locations, but the measurement periods for different ...
- 21
0
votes
1
answer
54
views
Sample whole number from distribution with average less than 1
I am trying to create some simulated data, where for each participant, the average number of events that occur in a given week can be any positive number, but usually in the range of 0 to 5. Trouble ...
- 161
0
votes
0
answers
142
views
Handling imbalanced data for classification [duplicate]
What are the best ways to deal with imbalanced datasets for classifying whether or not individuals pay their tuition? The data is 75% positive class (paid) and 25% negative (unpaid). Some approaches I ...
- 980
0
votes
0
answers
40
views
Data Synthesis based on Deep Learning
Is there any open source tool available to synthetically generate a new dataset with the same statistics than the original one?
The objective is to create a new tabular dataset that is private but ...
- 341
1
vote
0
answers
5k
views
"Add White Gaussian Noise with SNR" vs. "Add 5% Gaussian Noise"
I have a noise-free dataset which is a vector of numbers, $\mathbf{d}$, with length $N$.
I want to "add noise" to this data. My understanding is that there are two ways to do this.
1) Add some ...
- 885
12
votes
1
answer
633
views
Creating an Imbalanced Dataset
I would like to have my trained model tested on an imbalanced dataset. Is there any algorithms available to generate synthetic data from a balanced labelled dataset (spam/non-spam)?
- 381
1
vote
0
answers
149
views
Methods for generating regression data whose target follows an arbitrary distribution
I'd like to generate a regression dataset (independent vars that combine somehow into a target or dependent variable). Generating such a dataset is, in general, easy: we can e.g. pick some $x_i$ at ...
- 11