Skip to main content

Questions tagged [overlapping-data]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
2 votes
1 answer
43 views

Time series regression on mixed frequency overlapping data

I have an hourly univariate time series. I am trying to see if the next hour, day, week etc changes are forecastable from the past changes. The ACF and PACF of the data both look similar and show some ...
dayum's user avatar
  • 643
0 votes
0 answers
6 views

Characterise distribution overlap

I'm applying VAEs to sections genomic data (haplotypic vcf format, so binary variables), with one model being trained on each section. Depending on which section I'm applying these methods to, my ...
Whitehot's user avatar
1 vote
1 answer
65 views

Comparing forecasts under overlapping rolling evaluation windows

Two models with differing assumptions each purport to provide a forecast for a vector of values 1 year in the future. Each model has been run at the start of each month for three years: Run Date ...
David's user avatar
  • 515
0 votes
1 answer
18 views

Comparing distribution of a biomarker among patients with overlapping clinical features

I have a question how to compare the distribution of one tested biomarker among patients with different overlapping clinical syndromes. In my case, I am investigating patients with multiple sclerosis ...
Ieva M's user avatar
  • 1
1 vote
1 answer
131 views

rolling mean series - strong autocorrelation

I need to works with time-series of rolling means of a certain variable that comes from the data. I have this rolling means for N different individuals. Call them Y. I would like to test some ...
user9875321__'s user avatar
0 votes
1 answer
25 views

How to organize an overlapping ordinal variables?

I am analyzing some data with attributes about how long it takes to commute, and it is an overlapping ordinal variable like the following: Time Number of People 1-5 minutes 10 6-10 minutes 24 less ...
Po Sang Yu's user avatar
0 votes
0 answers
43 views

Measuring similarity between two datasets

This is quite straightforward: I have a dataset with 8 features containing cell measurements from a patient. I want to compare two patients to a reference one. To do this I perform UMAP to reduce the ...
amr95's user avatar
  • 13
1 vote
1 answer
93 views

Best way to rank list of values having overlapping uncertainties

I have a list of individuals and their success rates on a task. The count of tasks for each varies widely, so the uncertainty in the estimates of their proficiency varies. Furthermore, typical ...
user3897315's user avatar
0 votes
1 answer
216 views

Overlapping vs non-overapping windows in random walk

Consider a simple random walk. I am trying to compute the variance of differences over a window of certain size dw which could, for example, model returns of a ...
Botond's user avatar
  • 217
2 votes
1 answer
296 views

When two distributions overlap, how to separate one distribution from the mixture distribution if I the other distribution is known?

I have a data which can be classified into two groups. As you see, Figure(a) shows that they are easily classified into group A and B. However, sometimes they are overlapped and it is impossible set a ...
Yong Hwan Kim's user avatar
1 vote
1 answer
164 views

If a class overlay is too high for one to another, is it possible to do classification?

This is a figure of PCA Map of several class data that i am trying to classify: By looking at it, I wanted to be sure if this is not really applicable to be classified? (since it has a high overlap ...
Jovan's user avatar
  • 159
1 vote
2 answers
138 views

Logistic regression for observations on a monthly grid - impact of overlapping information of a given customer

My question is based on the usual setting in credit scoring. Assume we have historical monthly observations of customers, their risk factors $R_{i,t}$ and the flag whether they are currently in ...
Richi W's user avatar
  • 3,446
0 votes
0 answers
62 views

How many "effective" observations from sliding window sampling?

I have data on the trajectory of a particle and am interested in the displacement of the particle in a given time period. In particular, I have information on how much a particle has moved in each ...
arm61's user avatar
  • 1
0 votes
1 answer
142 views

Timeseries analysis with partially overlapping date ranges

I am working with a dataset of car/vehicle measurements at a Target location. In this dataset, there are multiple providers that give measurements about the number of vehicles at that Target (based on ...
Augustine's user avatar
1 vote
0 answers
244 views

Comparing proportions from overlapping samples with a complex survey design

I am trying to test whether there has been a statistically significant increase in the prevalence of a condition between two years of a complex survey using R. The samples are partly overlapping ...
Mark O'Donovan's user avatar
0 votes
1 answer
167 views

What is correct way to calculate the expected overlap of protein sets in two related biological system

I have a question regarding the statistics of protein set differences in two biological samples. I have performed quantitative proteomics analyses on both WT and KO cells (derived from the WT cells by ...
Suola's user avatar
  • 11
0 votes
0 answers
87 views

Can overlap impact the EM algorithm?

I am using EM algorithm for clustering. I see that if there is an overlap, the accuracy is reduced. However, it is improvesd if no overlap exists. My question is, can overlap affect the performance of ...
Alice's user avatar
  • 650
2 votes
0 answers
127 views

How to quantify temporal overlap of species abundance?

I want to compare the extent of overlap of species abundances in the following two secanrios. In the first scenario, species barely overlap so the index or overlap value should be zero. in the second ...
slicer's user avatar
  • 613
3 votes
1 answer
298 views

Probability of outcomes from overlapping samples of a random variable

I have a feeling there is a straight forward answer to my question, but I'm not sure what the appropriate term to search for is. I'm trying to get a sense of what the probability is of finding some ...
Max Miller's user avatar
1 vote
0 answers
209 views

What are pros and cons of AR and other time series models with overlapping datapoints

Anyone have any links or resources on pros/cons of building a timeseries model with overlapping data points? Generally, a lot of text build models on daily returns, but let's say the daily variable ...
confused's user avatar
  • 3,273
1 vote
0 answers
1k views

Overlapping data in time series model?

I have some time series data that I am trying to model - basically historical elasticity reads with daily sales data. I really am just interested in finding the 'mean' value - not necessarily predict ...
confused's user avatar
  • 3,273
0 votes
1 answer
89 views

Population overlap of random distributions of binary choices

I’m trying to figure out how to calculate the overlap (concurrence) of certain building features, for example: high-efficiency windows, thick insulation, efficient lights, etc. I know the probability ...
Determinant's user avatar
1 vote
2 answers
926 views

Average count of overlap between random samples

If I randomly sample n numbers from 1..m, x times, what would be the average count of ...
rolledback's user avatar
1 vote
1 answer
191 views

Properties of OLS estimator for overlapping observations' regression

I am running regressions of long-horizon gross financial returns $$R_{t+k} = \prod_{i=1}^{k} R_{t+i}$$ where $R_t=1+r_t$, on current dividend-price ratios $DP_t=\frac{D}{P}_t$: $$R_{t+k}=\alpha +\beta ...
Mr Frog's user avatar
  • 339
1 vote
1 answer
2k views

How to deal with visually overlapping clusters

I have done hierarchal clustering (average) and silhouette method shows 6 as the best k. However visualising I see that most of the clusters are overlapping. Any way to deal with the overlapping ...
user340744's user avatar
2 votes
2 answers
286 views

How to get the estimated means and standard deviations of two overlapping circular normal distributions?

I have the following plot: Overlapping circular normal distributions I would like to estimate the means and standard deviations of the apparent overlapping normal distributions. This is slightly ...
Mark Solinski's user avatar
0 votes
0 answers
41 views

How to quantify the overlap of 2 distribution

I have three events in time that follows a Lognormal distribution and that, in theory, occur in order T0 then T1 then ...
Hattori's user avatar
  • 151
2 votes
0 answers
60 views

Averaging methods to deal with overlapping time series

Consider the scenario where one is trying to fit a distribution to historical yearly excess returns of an equity index (say S&P 500 excess return over T bills). Monthly data are available but we ...
Trevor Hansen's user avatar
2 votes
1 answer
45 views

Why do two confidence intervals only overlap when their difference $(\theta_1 − \theta_2) \pm z(SE_1 + SE_2)$ includes 0?

I am able to check if the interval contains 0 which I can find out by just plugging numbers into the above formula but I am curious as to why this interval containing zero means that the two CI's of $\...
Anthony's user avatar
  • 23
2 votes
1 answer
1k views

Quantifying overlap between two categorical distributions with some non-identical categories

I'm looking for a way to measure the overlap (or general similarity) between two categorical distributions in which some of the categories are shared between each and some are not. For example, if the ...
joshisanonymous's user avatar
6 votes
0 answers
544 views

Robust standard errors under overlapping observations: confusing simulation results from alternative methods

I have a time series of $h$-step-ahead forecasts $\hat{y}_{t+h|t}$ for $t=1,\dots,T-h$ with $h>1$. I also have the corresponding realized values of their targets $y_{t+h}$ and the corresponding ...
Richard Hardy's user avatar
0 votes
0 answers
70 views

Stationarity in quarterly overlapping data

If I have a time series of quarterly default rates (e.g. of mortgages), by definition they will be 75% correlated, thus leading in non stationarity. What should I do in this case? Can I ignore lags up ...
adrCoder's user avatar
  • 483
2 votes
1 answer
172 views

Bayesian framework - Prior and Likelihood independence

I've a question on how, if ever needed, one can deal with the possible overlapping dependence of prior information and current data. Basically, I was given the output parameters of a regression that ...
r_user's user avatar
  • 337
3 votes
0 answers
225 views

Correction for sample overlap in a meta-analysis of prevalence (metafor package)

I want to conduct a meta-analysis of around 30 studies (from a systematic review). Some background of the studies: The quantity of interest is the prevalence of RSV infection. Different studies ...
bienco88's user avatar
2 votes
0 answers
148 views

Standard Error Bias from Monte Carlo Simulations

I am currently working through this paper on overlapping observations by Britten-Jones, Neuberger and Nolte 1, which presents an estimation method for correcting standard errors when performing ...
Hans-Peter Schrei's user avatar
2 votes
0 answers
378 views

How to adjust R^2 and the Coefficient of Determination when using Overlapping Observations?

I am currently working through the paper Improved Inference and Estimation in Regression With Overlapping Observations [1] which presents an elegant way to do inference on $\beta$ in a linear ...
Hans-Peter Schrei's user avatar
2 votes
1 answer
240 views

Estimating moments of annual return distribution

I am interested in estimating moments of annual return distributions of indices with limited historical data. I.e. equity/fixed income indices which have only 10-20 years of daily data available. The ...
p.vitzliputzli's user avatar
2 votes
0 answers
456 views

Effective sample size of a time series of overlapping observations

Question: How do I calculate the effective sample size of a time series under overlapping observations? By overlapping observations of $x_t$ where the overlap is of length $k$ I mean a case where $...
Richard Hardy's user avatar
4 votes
1 answer
319 views

Bias of overlap sampling

Say I have a series $x_1, ...., x_N$ independently drawn from $Normal(0,1)$ Say I create sub-samples $y_1 = x_1+..x_n$, $y_2 = x_2+..x_{n+1}$, $y_m = x_{N-n}+..x_N$, I was expecting $std(\{y\}) =...
user1612986's user avatar
1 vote
0 answers
49 views

Is it allowed to reduce a dataset of moving averages to run an AR(1) model properly?

I run a simple AR(1) and AR(2) model with the following code: ar.ols(df$y, order.max = 1) ar.ols(df$y, order.max =2) My dataset is as follows: I do have yearly ...
R-User's user avatar
  • 39
0 votes
1 answer
167 views

How to change the observation for the first lag in an AR(1) model?

I run a simple AR(1) model in my analysis using ols: ar.ols(df$y, order.max = 1)) However, I work with generations as my unit of analysis. Therefore, the first ...
R-User's user avatar
  • 39
6 votes
1 answer
731 views

$R^2$ and adjusted $R^2$ in presence of overlapping observations

Given a linear model $$ y=X\beta+\varepsilon, $$ the population value of $R^2$ is $$ R^2=1-\frac{\text{Var}(\varepsilon)}{\text{Var}(y)}. $$ The vanilla estimator of $R^2$ is $$ \hat R^2=1-\frac{\...
Richard Hardy's user avatar
1 vote
0 answers
22 views

Multiple Comparisons Problem for non-independant subsets

I'm investigating whether a program was effective or not using BACI analysis and noticed that overall it had no effect (The 2-way ANOVA and non-parametric approach gave a p-value of about 0.5). I want ...
user258110's user avatar
2 votes
1 answer
480 views

Training ARIMA based on overlapping hourly weather forecasts

I am working with hourly water level data and I plan to forecast each day the next two weeks (on an hourly base meaning 14*24 = 336 forecasts each day). My regressor is an hourly weather forecast. ...
Arne Jonas Warnke's user avatar
0 votes
1 answer
39 views

Cross checking AI Selections vs Random Chance

Fifty Three Images AI Selects 20 images based on various criteria. I Select 20 images based on personal criteria. In a two group selection of all 53 items what is the likely overlap due to random ...
Rikk Flohr's user avatar
0 votes
1 answer
283 views

Survival Analysis on recurrent behavior time series predictor

We are trying to build a credit model to predict the default time (or finally closed the loans as censored). The predictor is a high-dimension time series of current observed previous payment behavior....
Shusheng's user avatar
  • 106
4 votes
0 answers
210 views

Spatial regression with overlapping areas

Is it statistically correct to calculate a regression with overlapping areas? I have market areas as the spatial unit with different sociodemographic and (macro)economic variables and I´m examining ...
Mapos's user avatar
  • 141
1 vote
0 answers
31 views

(Another) What Statistical Test Should I Use?

If I ask 600 different people to rate 3 different images from 0-10 (i.e 3 samples of 200 people per image) and I calculate the mean scores per image as well as the overall mean for all 3 images, can I ...
Justin Fisher's user avatar
1 vote
0 answers
164 views

Probability of Finding Two Matching Subsequences in a Sequence

I'm currently studying DNA sequencing and am trying to find a formula which gives the probability that a subsequence of length $k$ appears twice in a sequence of $L$ bases (characters); this is pretty ...
BodneyC's user avatar
  • 11
0 votes
0 answers
66 views

How to analyze combined data from studies with overlap in subjects?

I have data from multiple placebo-controlled cross-over trials (each with $n \simeq 20$) with unique phenotypes measured during the placebo phase. I'd like to combine the data from these measurements ...
Frank Brube's user avatar