0
$\begingroup$

I have a dataset of points in space, sampled at a specified time. Now i want to obtain a 'spatial correlation plot, only between points close in time'. What i did is creating a dataframe of pairs of observations that are within a specified time-gap (for example 6 months). The columns of my dataframe are the following

-    Source_x    Source_y    Source_val    Target_x    Target_y    Target_val    spaceDistance

Where

  • Source indicates the starting point and Target the other point of each pair
  • x and y are spatial coordinates
  • val is the value associated with each point
  • spaceDistance is the spatial distance between points

I want to obtain some sort of correlation between the distance and how much values differ for each pair.
I thought i could do this simply by calculating the Manhattan distance of values and correlating those to the distance but this seems biased and difficult to interpret.

What do you think is the smartest way to analyze this type of data?

Note:

  • I don't want to transform the dataset in a long format and calculate all possible correlations between points because if i would do that i lose the information of 'time-adjacent' measurements

EDIT Here's more information:

  • The biggest dataset I have is around 3 millions pairs of measurements (within 6 months). Not all datasets are this big and i think i might need to subsample in those cases.
  • I have no clue of what pattern there might be between points, my hypothesis is that there should be inhomogeneity. That is: there is an underlying clustering of points that affects the actual difference in values.
  • The original data is always available.
  • Source_val and Target_val are floats (numbers). The theoretical range is infinite but the measured values range from 0 to 5 millions (maybe is should log the values?).
Improve this question
$\endgroup$
5
  • $\begingroup$ Cool problem - there could be some approaches within the theory / statistics of marked space-time point processes that fit :-). Could you give a bit more information about how big, roughly, your dataset is (hundreds or thousands of points within 6 months)? Are the point patterns more or less homogeneous in space, or is there inhomogeneity, such as a gradient? Is information about the original time still accessible? (some methods could use it instead of a fixed time frame) More down to earth question: what kind of data is Source_val and Target_val? Numeric? Fixed or infinite range? $\endgroup$
    – Ute
    Commented Jun 6, 2023 at 10:30
  • $\begingroup$ I added in the edit section the info you were looking for $\endgroup$
    – Mirk
    Commented Jun 6, 2023 at 10:49
  • $\begingroup$ The points can also be considered random, right? Cell phone data? $\endgroup$
    – Ute
    Commented Jun 6, 2023 at 10:54
  • $\begingroup$ For simplicity i would say to consider the sample points random, but i think in reality they might have been sampled with an underlying unknown pattern. But looking at the map even if there was a pattern is not clearly visible $\endgroup$
    – Mirk
    Commented Jun 6, 2023 at 11:20
  • $\begingroup$ Hi Mirk (hope you are still there once in a while :-)) Is it correct that you start out with a data set just with points in space-time, and determine source-and-target pairs from that set? or do you know a priory what belongs together? If you set up the pairs yourself, depending on closeness in time, do you allow one source point to have several targets / one target to have several sources, or do you have a decision rule to select which points belong together? $\endgroup$
    – Ute
    Commented Jul 22, 2023 at 15:47

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.