2
$\begingroup$

Suppose I have a dataset (see the explanation of the background later) as shown in the plot below, where each dot is a sample. To human eyes, there is an obvious boundary which separates samples above it (normal data points) from those underneath it (outliers). My question is what statistical technique can be used to find this imaginary boundary?

enter image description here

I had actually tried using Guassian Process prior with skewed normal distribution as likelihood, and implemented the model with PyMC3, but problem with this approach is the training process (i.e. MCMC sampling) is extremely slow. Even after I downsampled the dataset to only use 200 datapoints, the training took about 1 hour. More importantly, the downsampling process will inevitably drive the cure found by the model away from that imaginary boundary as most samples after downsample will be away from the boundary. So I was trying to see whether there is other better techinique for this task.

In case you are wondering what the dataset is, here is the background. Each datapoint corresponds to a listing on a real rental listing site. Y axis is the first review date of the listing and the X-axis is the listint id. The task is to infer the creation date of each listing. Given that the first review data is the upper bound of the listing creation date, I think that imaginary boundary should be a quite good estimation of the relation between listing id and listing creation date.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Care to post a generator for those data?

"Know your data" is an important part of any modeling process, in which case guessing a simple "sqrt(x)" from visual inspection seems like a starting place for a relatively simple dataset like this.

But to the question: It sounds like you'd like to perhaps find a set of parameters alpha, beta, and a function f so that p(x, y) ~ sigmoid(alpha * max(0, y - f(x)) + beta * min(0, y - f(x))). Allow alpha and beta to separately model the "sharpness" of the positive and negative sides of a sigmoid curve, and then you can fit f to the function. Still needs some kind of prior over functions f, though, perhaps a * x ** b + c.

Improve this answer
$\endgroup$
5
  • $\begingroup$ The first advice is very good. But the specific suggestions in the last paragraph are not flexible enough. There are more than enough data here to perform the detection in a nonparametric way, too. $\endgroup$
    – whuber ♦
    Oct 9, 2020 at 20:58
  • $\begingroup$ @Brian, the data is not a synthesized one, so I do not really have a generator. I updted my post to explain what the dataset is. Check it out if you are interested. $\endgroup$
    – victorx
    Oct 10, 2020 at 23:00
  • $\begingroup$ @whuber, do you mind elaborating a bit more about your idea of 'nonparametric way? $\endgroup$
    – victorx
    Oct 10, 2020 at 23:01
  • $\begingroup$ @whuber, P.S. I modified the post to add my previous attempt of a GP-based model and the assocated issue. Keen to see what nonparametetric modeling tool is in your mind. $\endgroup$
    – victorx
    Oct 10, 2020 at 23:20
  • $\begingroup$ @Victorx Two natural nonparametric approaches are (1) quantile regression with splines of the explanatory variable and (2) 2D kernel density estimation followed by edge detection. $\endgroup$
    – whuber ♦
    Oct 11, 2020 at 14:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.