How to test if these two samples are from the same distribution? (These are tweets located near some tourist attractions.
If yes, is there a simple implementation in Python?
$\begingroup$
$\endgroup$
13
-
3$\begingroup$ You should provide more information about the process underlying the tweets to determine if there is a test that would work. Do you view the tweets as coming from a fixed location with some sort of measurement (boy vs. girl, type of tweet, time of day, etc.)? Or is there a stochastic aspect to the spatial distribution of tweets such that they could be viewed as a realization of a point process (or marked point process)? $\endgroup$– coreydevinandersonCommented Dec 3, 2017 at 15:21
-
$\begingroup$ The data I have are scraped from a specific hashtag, containing user id, caption, location, time. It's tourists' data so it's very random, concentrated around tourist attractions. $\endgroup$– monotonicCommented Dec 4, 2017 at 9:07
-
$\begingroup$ Can you give a example of both? $\endgroup$– monotonicCommented Dec 4, 2017 at 9:15
-
$\begingroup$ I think you have point pattern data. You want to understand whether the pattern of tweets with a certain hash tag represent a realization from the same underlying distribution. There is not a simple answer to this question, but I think I can point you in the right direction, but you will need to update your question with the information in your comment so that the admin can reopen it. Provide as much detail as possible (and perhaps even a map, if feasible). $\endgroup$– coreydevinandersonCommented Dec 5, 2017 at 4:04
-
2$\begingroup$ 1. The diagrams should be explained more clearly. What are the x and y axis, what are the colour intensity of the dots? (shushwap lake?) 2. The situation should be sketched with more context. There seems to be clustering along two lines but can we explain this somehow? Such explanation can help to devise a model that allows a more powerful test. $\endgroup$– Sextus EmpiricusCommented Oct 25, 2018 at 8:47
|
Show 8 more comments
3 Answers
$\begingroup$
$\endgroup$
This is a seemingly simple...problem, but it is made more complicated by the fact that you have a replicated point pattern on a linear network (roads). I don't think you will get very far with this in Python. Your best bet is the spatstat package in R.
There has been a good deal of work on replicated point patterns (including some relatively simple non-parametric tests to compare the patterns), but not all of it may be applicable here since most of it is intended for planar (2D) data...and the situation gets even trickier if you are dealing with an inhomogeneous pattern...or more "complicated" point process (such as some sort of cluster model or Gibbs model)...on a linear network. The good news is that there are new tools emerging almost on a monthly basis; what is not possible today could be possible tomorrow.
I would start by do some exploratory analyses on each pattern separately. We usually start by testing a simple null model (such as a homogeneous Poisson process, a.k.a. CSR) using a summary function (here...network Ripley's K or, better yet, L). If you can reject CSR, then you need to explore point process models that might be more suitable. For example, if the data are more clustered than expected under CSR, then you either have an inhomogeneous pattern or you are dealing with a cluster process...or both). There are also pooled versions of these summary functions that would allow you to simultaneously compare the variability. The nature of the process itself might also be a good guide to the sort of point process model that is most suitable...and just the pattern of the summary functions over distance should help you to gauge the similarity and/or dissimilarity of the patterns.
Because I am not certain if there exists functionality (at the present time) for fitting a linear point process model to replicated patterns simultaneously, if you find that a Poisson model suffices, you could try fitting a generalized linear model (log-link) with group as a covariate (and perhaps network distance as an offset), followed by a likelihood ratio test (to compare the intensity of each pattern). An alternative approach would be to build a good point process model for one of the patterns and then test its fit for the other pattern (using a goodness-of-fit test). In theory, a mixed effects point process model might be most suitable here (since random effects might be applicable in your case), but, again, I am not sure if the functionality exits at the present time.
Chapter 16 (Replicated Point Patterns and Designed Experiments) of "Spatial Point Patterns Methodology and Applications with R" (and references therein) would be a good place to start.
answered Dec 8, 2017 at 20:01
$\begingroup$
$\endgroup$
2
If you assume they have a parametric distribution, then try to find an MLE estimate of the parameters, one method would be the RANSAC which can minimally get at the parameters. A non parametric distribution free approach would be the kolmogorov smirnov test which has a python implementation https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.ks_2samp.html
-
$\begingroup$ What distribution can be assumer in this situation? For the non-parametric case, Kolmogorov smirnov test only support 1-dimenison samples? $\endgroup$ Commented Dec 6, 2017 at 7:01
-
$\begingroup$ There are many possible multivariate distributions you could consider, normal, dirichlet, von mises. I dont know of any hypothesis test to compare multivariate distributions $\endgroup$– knkCommented Dec 6, 2017 at 7:45
$\begingroup$
$\endgroup$
1
You can use Kolmogorov-Smirnov test for that purpose. Assuming your samples are stored in arrays a and b,
from scipy import stats
print(stats.ks_2samp(a, b))
will give you KS statistic and P value. If P value is low enough, then you can reject null hypotheses that a and b come from the same distribution, like for instance here:
Ks_2sampResult(statistic=0.036719903980796165, pvalue=0.002286054946379626)
And vice versa.
-
1$\begingroup$ It should be noted that the Kolmogorov-Smirnov test is nonparametric, and gives no information about what underlying distribution either sample comes from - which is fine if OP has no assumptions about an underlying distribution. More detail about the applicability of the test would clarify whether or not it is appropriate here. $\endgroup$ Commented Sep 22, 2018 at 14:52