Questions tagged [overlapping-data]
The overlapping-data tag has no usage guidance.
84 questions
1
vote
0
answers
31
views
How do you specify a regression model when both the response and explanatory variables are autocorrelated?
I have two sets of data points:
Each data point (y_t) in the first set represents the annualized return for the next ten years of monthly return data, i.e., this is forward-looking data, so my last ...
- 11
1
vote
1
answer
60
views
Comparing correlations for two variables in different overlapping groups
I have a sample where I have applied five (wildly differing) diagnostic definitions for the same disease. For example, 50% of people who are disease cases under diagnostic definition 1 are not disease ...
- 326
2
votes
1
answer
69
views
Time series regression on mixed frequency overlapping data
I have an hourly univariate time series. I am trying to see if the next hour, day, week etc changes are forecastable from the past changes. The ACF and PACF of the data both look similar and show some ...
- 633
0
votes
0
answers
7
views
Characterise distribution overlap
I'm applying VAEs to sections genomic data (haplotypic vcf format, so binary variables), with one model being trained on each section. Depending on which section I'm applying these methods to, my ...
- 93
1
vote
1
answer
83
views
Comparing forecasts under overlapping rolling evaluation windows
Two models with differing assumptions each purport to provide a forecast for a vector of values 1 year in the future. Each model has been run at the start of each month for three years:
Run Date
...
- 525
0
votes
1
answer
18
views
Comparing distribution of a biomarker among patients with overlapping clinical features
I have a question how to compare the distribution of one tested biomarker among patients with different overlapping clinical syndromes.
In my case, I am investigating patients with multiple sclerosis ...
- 1
1
vote
1
answer
224
views
rolling mean series - strong autocorrelation
I need to works with time-series of rolling means of a certain variable that comes from the data. I have this rolling means for N different individuals. Call them Y. I would like to test some ...
- 190
0
votes
1
answer
29
views
How to organize an overlapping ordinal variables?
I am analyzing some data with attributes about how long it takes to commute, and it is an overlapping ordinal variable like the following:
Time
Number of People
1-5 minutes
10
6-10 minutes
24
less ...
0
votes
0
answers
66
views
Measuring similarity between two datasets
This is quite straightforward:
I have a dataset with 8 features containing cell measurements from a patient. I want to compare two patients to a reference one. To do this I perform UMAP to reduce the ...
- 13
1
vote
1
answer
110
views
Best way to rank list of values having overlapping uncertainties
I have a list of individuals and their success rates on a task. The count of tasks for each varies widely, so the uncertainty in the estimates of their proficiency varies. Furthermore, typical ...
- 111
0
votes
1
answer
263
views
Overlapping vs non-overapping windows in random walk
Consider a simple random walk. I am trying to compute the variance of differences over a window of certain size dw which could, for example, model returns of a ...
- 217
2
votes
1
answer
367
views
When two distributions overlap, how to separate one distribution from the mixture distribution if I the other distribution is known?
I have a data which can be classified into two groups. As you see, Figure(a) shows that they are easily classified into group A and B. However, sometimes they are overlapped and it is impossible set a ...
1
vote
1
answer
198
views
If a class overlay is too high for one to another, is it possible to do classification?
This is a figure of PCA Map of several class data that i am trying to classify:
By looking at it, I wanted to be sure if this is not really applicable to be classified? (since it has a high overlap ...
- 159
1
vote
2
answers
155
views
Logistic regression for observations on a monthly grid - impact of overlapping information of a given customer
My question is based on the usual setting in credit scoring. Assume we have historical monthly observations of customers, their risk factors $R_{i,t}$ and the flag whether they are currently in ...
- 3,506
0
votes
0
answers
64
views
How many "effective" observations from sliding window sampling?
I have data on the trajectory of a particle and am interested in the displacement of the particle in a given time period. In particular, I have information on how much a particle has moved in each ...
- 1
0
votes
1
answer
209
views
Timeseries analysis with partially overlapping date ranges
I am working with a dataset of car/vehicle measurements at a Target location. In this dataset, there are multiple providers that give measurements about the number of vehicles at that Target (based on ...
1
vote
0
answers
262
views
Comparing proportions from overlapping samples with a complex survey design
I am trying to test whether there has been a statistically significant increase in the prevalence of a condition between two years of a complex survey using R. The samples are partly overlapping ...
0
votes
1
answer
179
views
What is correct way to calculate the expected overlap of protein sets in two related biological system
I have a question regarding the statistics of protein set differences in two biological samples.
I have performed quantitative proteomics analyses on both WT and KO cells (derived from the WT cells by ...
- 11
0
votes
0
answers
93
views
Can overlap impact the EM algorithm?
I am using EM algorithm for clustering. I see that if there is an overlap, the accuracy is reduced. However, it is improvesd if no overlap exists.
My question is, can overlap affect the performance of ...
- 670
2
votes
0
answers
137
views
How to quantify temporal overlap of species abundance?
I want to compare the extent of overlap of species abundances in the following two secanrios. In the first scenario, species barely overlap so the index or overlap value should be zero. in the second ...
- 633
4
votes
1
answer
355
views
Probability of outcomes from overlapping samples of a random variable
I have a feeling there is a straight forward answer to my question, but I'm not sure what the appropriate term to search for is. I'm trying to get a sense of what the probability is of finding some ...
- 41
1
vote
0
answers
220
views
What are pros and cons of AR and other time series models with overlapping datapoints
Anyone have any links or resources on pros/cons of building a timeseries model with overlapping data points? Generally, a lot of text build models on daily returns, but let's say the daily variable ...
- 3,263
1
vote
0
answers
1k
views
Overlapping data in time series model?
I have some time series data that I am trying to model - basically historical elasticity reads with daily sales data. I really am just interested in finding the 'mean' value - not necessarily predict ...
- 3,263
0
votes
1
answer
98
views
Population overlap of random distributions of binary choices
I’m trying to figure out how to calculate the overlap (concurrence) of certain building features, for example: high-efficiency windows, thick insulation, efficient lights, etc. I know the probability ...
- 35
1
vote
2
answers
1k
views
Average count of overlap between random samples
If I randomly sample n numbers from 1..m, x times, what would be the average count of ...
- 13
1
vote
1
answer
232
views
Properties of OLS estimator for overlapping observations' regression
I am running regressions of long-horizon gross financial returns $$R_{t+k} = \prod_{i=1}^{k} R_{t+i}$$ where $R_t=1+r_t$, on current dividend-price ratios $DP_t=\frac{D}{P}_t$:
$$R_{t+k}=\alpha +\beta ...
- 349
1
vote
1
answer
3k
views
How to deal with visually overlapping clusters
I have done hierarchal clustering (average) and silhouette method shows 6 as the best k. However visualising I see that most of the clusters are overlapping. Any way to deal with the overlapping ...
- 13
2
votes
2
answers
311
views
How to get the estimated means and standard deviations of two overlapping circular normal distributions?
I have the following plot:
Overlapping circular normal distributions
I would like to estimate the means and standard deviations of the apparent overlapping normal distributions. This is slightly ...
0
votes
0
answers
46
views
How to quantify the overlap of 2 distribution
I have three events in time that follows a Lognormal distribution and that, in theory, occur in order T0 then T1 then ...
- 171
2
votes
0
answers
65
views
Averaging methods to deal with overlapping time series
Consider the scenario where one is trying to fit a distribution to historical yearly excess returns of an equity index (say S&P 500 excess return over T bills). Monthly data are available but we ...
2
votes
1
answer
48
views
Why do two confidence intervals only overlap when their difference $(\theta_1 − \theta_2) \pm z(SE_1 + SE_2)$ includes 0?
I am able to check if the interval contains 0 which I can find out by just plugging numbers into the above formula but I am curious as to why this interval containing zero means that the two CI's of $\...
- 23
2
votes
1
answer
1k
views
Quantifying overlap between two categorical distributions with some non-identical categories
I'm looking for a way to measure the overlap (or general similarity) between two categorical distributions in which some of the categories are shared between each and some are not. For example, if the ...
- 405
6
votes
0
answers
584
views
Robust standard errors under overlapping observations: confusing simulation results from alternative methods
I have a time series of $h$-step-ahead forecasts $\hat{y}_{t+h|t}$ for $t=1,\dots,T-h$ with $h>1$. I also have the corresponding realized values of their targets $y_{t+h}$ and the corresponding ...
- 69.5k
0
votes
0
answers
75
views
Stationarity in quarterly overlapping data
If I have a time series of quarterly default rates (e.g. of mortgages), by definition they will be 75% correlated, thus leading in non stationarity.
What should I do in this case? Can I ignore lags up ...
- 483
2
votes
1
answer
176
views
Bayesian framework - Prior and Likelihood independence
I've a question on how, if ever needed, one can deal with the possible overlapping dependence of prior information and current data.
Basically, I was given the output parameters of a regression that ...
- 337
3
votes
0
answers
238
views
Correction for sample overlap in a meta-analysis of prevalence (metafor package)
I want to conduct a meta-analysis of around 30 studies (from a systematic review).
Some background of the studies: The quantity of interest is the prevalence of RSV infection. Different studies ...
- 71
2
votes
0
answers
156
views
Standard Error Bias from Monte Carlo Simulations
I am currently working through this paper on overlapping observations by Britten-Jones, Neuberger and Nolte 1, which presents an estimation method for correcting standard errors when performing ...
2
votes
0
answers
406
views
How to adjust R^2 and the Coefficient of Determination when using Overlapping Observations?
I am currently working through the paper Improved Inference and Estimation in Regression With Overlapping Observations [1] which presents an elegant way to do inference on $\beta$ in a linear ...
2
votes
1
answer
267
views
Estimating moments of annual return distribution
I am interested in estimating moments of annual return distributions of indices with limited historical data. I.e. equity/fixed income indices which have only 10-20 years of daily data available. The ...
2
votes
0
answers
487
views
Effective sample size of a time series of overlapping observations
Question: How do I calculate the effective sample size of a time series under overlapping observations?
By overlapping observations of $x_t$ where the overlap is of length $k$ I mean a case where $...
- 69.5k
4
votes
1
answer
343
views
Bias of overlap sampling
Say I have a series $x_1, ...., x_N$ independently drawn from $Normal(0,1)$
Say I create sub-samples
$y_1 = x_1+..x_n$,
$y_2 = x_2+..x_{n+1}$,
$y_m = x_{N-n}+..x_N$,
I was expecting $std(\{y\}) =...
- 143
1
vote
0
answers
49
views
Is it allowed to reduce a dataset of moving averages to run an AR(1) model properly?
I run a simple AR(1) and AR(2) model with the following code:
ar.ols(df$y, order.max = 1)
ar.ols(df$y, order.max =2)
My dataset is as follows: I do have yearly ...
- 39
0
votes
1
answer
179
views
How to change the observation for the first lag in an AR(1) model?
I run a simple AR(1) model in my analysis using ols:
ar.ols(df$y, order.max = 1))
However, I work with generations as my unit of analysis. Therefore, the first ...
- 39
6
votes
1
answer
793
views
$R^2$ and adjusted $R^2$ in presence of overlapping observations
Given a linear model
$$
y=X\beta+\varepsilon,
$$
the population value of $R^2$ is
$$
R^2=1-\frac{\text{Var}(\varepsilon)}{\text{Var}(y)}.
$$
The vanilla estimator of $R^2$ is
$$
\hat R^2=1-\frac{\...
- 69.5k
1
vote
0
answers
23
views
Multiple Comparisons Problem for non-independant subsets
I'm investigating whether a program was effective or not using BACI analysis and noticed that overall it had no effect (The 2-way ANOVA and non-parametric approach gave a p-value of about 0.5). I want ...
- 11
2
votes
1
answer
491
views
Training ARIMA based on overlapping hourly weather forecasts
I am working with hourly water level data and I plan to forecast each day the next two weeks (on an hourly base meaning 14*24 = 336 forecasts each day).
My regressor is an hourly weather forecast.
...
- 3,661
0
votes
1
answer
39
views
Cross checking AI Selections vs Random Chance
Fifty Three Images
AI Selects 20 images based on various criteria.
I Select 20 images based on personal criteria.
In a two group selection of all 53 items what is the likely overlap due to random ...
0
votes
1
answer
289
views
Survival Analysis on recurrent behavior time series predictor
We are trying to build a credit model to predict the default time (or finally closed the loans as censored).
The predictor is a high-dimension time series of current observed previous payment behavior....
- 106
4
votes
0
answers
212
views
Spatial regression with overlapping areas
Is it statistically correct to calculate a regression with overlapping areas? I have market areas as the spatial unit with different sociodemographic and (macro)economic variables and I´m examining ...
- 141
1
vote
0
answers
31
views
(Another) What Statistical Test Should I Use?
If I ask 600 different people to rate 3 different images from 0-10 (i.e 3 samples of 200 people per image) and I calculate the mean scores per image as well as the overall mean for all 3 images, can I ...
