Skip to main content

Questions tagged [synthetic-data]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
11 views

Is this a reasonable way to check the quality of simulated data in MCMC inference?

I have a hierarchical Bayesian model that looks like this: $\alpha_i \sim \mathcal{N}\left(\mu_\alpha, \sigma_\alpha\right) \tag{1}$ $\beta_i \sim \mathcal{N}\left(\mu_\beta, \sigma_\beta\right) \tag{...
chesslad's user avatar
  • 211
4 votes
1 answer
41 views

Generating synthetic data with multiple records per ID

I would like to generate a synthetic dataset where there are multiple records per ID, and self-consistency is maintained among records of each ID. For example, imagine a dataset where the ID is a ...
user12138762's user avatar
3 votes
0 answers
45 views

References for Generation of Synthetic Data

What are some of the introductory textbooks/references specifically on the task of generating synthetic data (from real data)? If possible, such a text is expected to cover a range of methods, be it ...
0 votes
0 answers
31 views

Can I generate a time series with same features as a given dataset, but add a known linear trend coefficient (not just trend strength)?

I want to generate data that matches features of environmental data (that is often analysed with nonparametric tests due to nonrmality, skewness etc). I want to know how to best capture any linear ...
Justin Murphy's user avatar
1 vote
0 answers
12 views

Creating a dataset which provides specific regression output

I need to create a synthetic dataset with 1000 rows for two variables X and Y. I need the relationship between X and Y to be set up such that when I run a regression model, it provides specific output,...
mp9828's user avatar
  • 11
1 vote
1 answer
52 views

Are synthetic data produced by Gretel, YData, MostlyAI, etc. of higher quality than sdv-dev CTGAN?

There are some online services that we can use to generate synthetic data. On the other hand, we can also use sdv-dev CTGAN from GitHub. Are synthetic data produced by Gretel, YData, MostlyAI, etc. of ...
user366312's user avatar
  • 2,195
0 votes
0 answers
15 views

sampling correlated random variables using copula

i have a very small set of data, which is a collection of vectors with 4 element (for the sake of simplicity). the 4 marginal distributions are quite diversify (they are gaussian-like or sine-like). ...
Physics Student's user avatar
0 votes
0 answers
40 views

Imputing missing observations of zip code level data

I am looking for a sufficient imputation method for missing observations in my zip code level data, using R. I have a random sample consisting of households which live in different zip codes within ...
Ottibanane123's user avatar
0 votes
0 answers
32 views

How can I create realistic noisy data from distributions?

I want to create synthetic data from stitched distributions in order to test some models on them (for example Gaussian stitched with a GPD at quantile q). I'm currently simply sampling N*q points from ...
Philippe Ear's user avatar
0 votes
0 answers
68 views

asking for synthetic control method with multiples outcomes and multiple treated units stats command please

I am trying to do synthetic control method with 2 treated provinces, 10 non-treated provinces, and 7 outcomes. Can anyone let me know which stata command I can use?
Kimdung's user avatar
0 votes
0 answers
17 views

Generate a Synthetic Bivariate Data for Testing Simple Linear Regression Using Excel

I come up with a solution as below. What do you think? Do you have any other way? Linear regression modeled as; y(i) = a + b*x(i) + e(i) a and b is constant, thus ...
Maaruf Bussri's user avatar
1 vote
0 answers
194 views

How to generate random values based on mean, standard deviation, skew and kurtosis in Python?

Given these values, is it possible to generate random values that conform to this distribution (using Python, but preferably without the SciPy package)? Statistic Value Mean 1.518 Std Dev 24.827 ...
m01010011's user avatar
  • 111
2 votes
0 answers
28 views

How can I introduce dependence (to varying degrees) into a synthetic dataset to measure the effect on my method?

I'm using a synthetic dataset in which I sample from three independent Bernoulli random variables x1, x2 and x3 with p=p1, p=p2 and p=p3 respectively. I wish to "introduce dependence," or ...
Vance's user avatar
  • 21
1 vote
0 answers
61 views

Synthetic data for PCA [closed]

I was trying to evaluate different algorithms for PCA like eigenvalue decomposition, SVD, Lanczos Algorithm, Power iteration. I also want to do some other analysis I could find any papers concerning ...
pppp_prs's user avatar
0 votes
0 answers
123 views

How can I model the multivariate probability distribution of a dataset with both continuous and discrete variables for sampling?

This might seem like a duplicate of the following link, but I think that one is asking how to create a completely new dataset with specific distributions, rather than how to model an existing dataset ...
quanty's user avatar
  • 252
1 vote
1 answer
38 views

Can I do a meta-analysis by Monte-Carlo synthetic data?

I'm trying to do a meta analysis of ~30 studies (total N = ~2000) on the correlation (X, Y). However, the heterogeneity is soooo high. My hypothesis (and what has been suggested in the literature) is ...
Ken Chan's user avatar
1 vote
0 answers
82 views

How to visualize time series using PCA?

I have two multivariate data sets comprised of 100s of time series, one is the actual recorded data set of time series and the other is a synthetically generated data set based on the recorded one. ...
Unistack's user avatar
5 votes
1 answer
112 views

What type of data should I generate to observe/amplify a crossing problem in quantile regression?

1. Background Crossing problem in quantile regression can be observed when we want to estimate several conditional quantiles (e.g. τ = 0.1, 0.2, . . . , 0.9), as two or more estimated conditional ...
Recology's user avatar
  • 135
1 vote
2 answers
539 views

Generating synthetic time series data with limited data

I would like some opinions on my current situation. I have a set of time series data that I want to forecast. The data however is not very long (around 500 rows) so I was looking into generating many ...
codinator's user avatar
  • 123
1 vote
1 answer
50 views

Synthetic vs Data augmentation for low dimensionality data

I have problems understanding data augmentation. I currently have low-dimension features, each data point only has 3 features. A total of 20k non-linear data with only 3 features. I have generated ...
kasyful's user avatar
  • 23
1 vote
0 answers
396 views

How to generate synthetic data that respects pairwise correlations of features of the real dataset?

Let us suppose that I have a dataset with 3 features and that I know the pairwise correlations among these features. Let us suppose that I want to build a synthetic dataset that respects those ...
Zaratruta's user avatar
  • 1,018
3 votes
1 answer
308 views

Why use a copula to generate synthetic data?

For class, I am tasked to generate synthetic stock data using the copula R package. The step-by-step process is picking 2 stocks (i.e., Amazon & Apple), fit their marginal distributions (I am ...
W. Hunter Giles's user avatar
1 vote
1 answer
423 views

Creating synthetic data for time series, Hidden Markov Model

Suppose that I have a task of classifying a time series. I decide to use Hidden Markov Model $\lambda(A, B, \pi)$, where $A$ is a transition matrix, $B$ is an emission probability, $\pi$ is an initial ...
thesecond's user avatar
  • 390
2 votes
2 answers
723 views

Is SMOTE any good at creating new points?

Cross Validated has a pretty thorough debunking of class imbalance being an inherent problem for SMOTE to solve. However, SMOTE is explicitly a method for synthesizing new points. Is SMOTE any good at ...
Dave's user avatar
  • 64.7k
1 vote
1 answer
145 views

Which tool is more suitable for visualizing the distribution of multiple real and synthetic image datasets, t-SNE or PCA?

I am doing a thesis on the generation of synthetic data for training a deep learning model and evaluating it on real data. I have a few different real datasets, and I generated multiple synthetic ...
Manveru's user avatar
  • 177
0 votes
0 answers
46 views

Ordering when using scipys Jenson Shannon distance

I am currently the scipy implementation of the Jenson-Dhannon distance to compare 2 vectors sampled from 2 distributions. I'd expect the distance to be zero if I get the same samples - regardless of ...
T A's user avatar
  • 101
3 votes
1 answer
342 views

Why is the sum of individual Spearman's rho squared less than 1 as opposed to Pearson's r in a synthetic example?

A relatively low number of iid random vectors of a relatively high dimension (10,000) are added up together element wise: $$\sum_{i=1}^{n}X_i=Y$$ where $dim(X_i)=dim(X_j)=dim(Y),\forall i,j$ and $dim(...
ayorgo's user avatar
  • 301
0 votes
0 answers
28 views

Where to start creating a -synthetic 3D object- based neural network in combination with sensor data?

I have a question on where to start with a project of mine. It includes a wide variety of expertise, so I am not sure if am at the right part of stack exchange. My project is as such: I have a cube ...
Bende's user avatar
  • 1
1 vote
1 answer
85 views

Generate a syntetic log-normal two dimensional random field

I would like to test some functions that I wrote related to the kriging applied to rain data. In order to do that, I would like to generate a synthetic log-normal 2D random field. The idea is to ...
diedro's user avatar
  • 111
1 vote
0 answers
191 views

FID as a metric to evaluate the quality of synthetic datasets (Non GAN generated) for training models for a given classification task

I am working on a problem of generating synthetic data (algorithmically by blender, not using GANs) to aid the training of some CNN for a classification ask. Ideally, I want to generate an algorithm ...
Manveru's user avatar
  • 177
0 votes
1 answer
586 views

Tried so many different models but cant get good accuracy

I am very new to the field of Machine Learning. My college seniors provided me a dataset to analyze and predict. The data is purely synthetic, with 14 feature columns and a target column having values ...
pjmathematician's user avatar
0 votes
1 answer
86 views

How to create synthetic data for this case?

Some weeks ago, I ran an experiment with 30 participants. For each participant two microphones were recording them while they were reading a phrase in 3 different simulated emotions (Happy, Neutral ...
Fran's user avatar
  • 55
0 votes
1 answer
232 views

Simulating Hierarchical Data

I want to simulate a dataset that has a "grand mean", and then group means (with some deviation from the grand mean). Nevertheless, I have a bit of a problem conceptualising the problem: if ...
Zlo's user avatar
  • 137
1 vote
0 answers
48 views

Metric to assess similarity of distributions

I am working on clinical synthetic data and I would like to learn more about metrics to compare synthetic vs measurements distributions. As there are methods to generate synthetic distributions with ...
Andrea Zaliani's user avatar
0 votes
1 answer
136 views

Synthetic multivariate time series for anomaly detection

I built an anomaly detection classifier which worked perfectly with the anomaly detection task in my dataset (multivariate time series). Now I'm trying to understand what are its weakness and my idea ...
Fabio's user avatar
  • 115
3 votes
1 answer
74 views

Synthetic data set generation for binary classification based on paper (interpretation problem)

I'm reading a research paper about fraud detection (unbalanced binary classification) where the authors go for synthetic data for evaluating their methods. I want to reproduce their synthetic data but ...
Fredrik's user avatar
  • 725
1 vote
1 answer
20 views

Why is the standard error in a diff-in-diff with 4 datapoints 'not well defined'?

I was looking at this online book https://matheusfacure.github.io/python-causality-handbook/15-Synthetic-Control.html in order to learn about synthetic control. The author gets two cities in a panel ...
new_to_ds's user avatar
1 vote
1 answer
40 views

Generating data for one-class dataset

In case I have a dataset that have only one class unlabeled (benign), could you please list some algorithms/papers that are used to generate complementary data (malignant) based on benign data only? I ...
Avv's user avatar
  • 249
0 votes
1 answer
44 views

Composite Indicator - Negative Correlation Issue

I'm trying to build a composite indicator that aims to measure poverty. I'm planning to aggregate a series of variables using PCA. However I have a doubt, all the variables I want to include are ...
Gigi39's user avatar
  • 13
1 vote
0 answers
81 views

Generate data that matches a frequency distribution while preserving the original spatial structure

I am dealing with a 3D array containing values representing the "importance" of each voxel. For my analysis, I would like to synthesize n new arrays from my original array to have a ...
Johannes Wiesner's user avatar
0 votes
0 answers
24 views

Is creating artificial class imbalance in synthetic training data a good way to tackle hard cases in classification?

I have a problem where I need to classify something around 50 different classes. Some of the classes are very similar to one another and the algorithm tends to confuse them. However, I can create a ...
vcucu's user avatar
  • 73
1 vote
0 answers
16 views

Learning Distribution of Data [duplicate]

Sometimes it's important to generate data due to data imbalance issues. I heard that data augmentation by leaning distribution of data is a hot topic now. Could you please give me some resources and ...
Avv's user avatar
  • 249
1 vote
3 answers
195 views

How to generate synthetic data from a balanced dataset?

Let say I have a balanced dataset that has a small training sample size (lack of data). How do I increase the training sample size by generating synthetic data based on the original data? I believe ...
Aqee's user avatar
  • 39
0 votes
1 answer
63 views

What can be a good way to generate data(tabular) from statistical facts and probability of data?

For Example, If I have facts saying that: 50% of humans are male 30% of males are Indians 70% of Indians are brown average age of Indians are 27 30% of indian females are working 80% of no-indian ...
Mahesh Mistry's user avatar
0 votes
1 answer
30 views

Simulated data for statistical framework testing

I want to generate two sets of simulated data (numeric) for statistical testing. Is it possible to generate datasets with predefined RMSE or accuracy?
Msilvy's user avatar
  • 51
2 votes
1 answer
518 views

Generating samples on an exponential distribution

I am trying to generate a synthetic earthquake database where the number of events ($N$) with magnitude ($M$) in the range $[M, M+\delta_M]$ follows: $\log_{10}(N) = a - bM$ where $a$ and $b$ are ...
RustyC's user avatar
  • 21
4 votes
1 answer
707 views

Does it make sense to use the KL-divergence between joint distributions of synthetic and real data, as a evaluation metric?

The KL-divergence is defined as: $D_{KL}(p(x_1)∥q(x_1))=\sum p(x_1)\, \log \Big( \dfrac{p(x_1)}{q(x_1)} \Big)$ I consider the Kullback-Leibler (KL) divergence as a performance metric for data ...
Eui-Jin Kim's user avatar
0 votes
0 answers
43 views

Constrains on the coefficients of SARIMA

I am trying to generate synthetic time-series through SARIMA random process by defining the model coefficients manually. Could any one help me how to generate coefficients? What are the constraints on ...
AidinZadeh's user avatar
0 votes
0 answers
184 views

Generate more data for a small dataset

I have been working on a dataset which has 14 attributes and 303 rows(instances) along with the binary labels. I want to generate more data so that I could train my neural networks so that I could ...
VIPUL VAIBHAV's user avatar
1 vote
0 answers
66 views

Regression on unevenly distributed high dimensional dataset

I have a very high dimensional (20K+ hand engineered features) biological dataset to predict a single continuous output variable (such as a mental state exam scores for a dementia patient). The output ...
user254813's user avatar