2
$\begingroup$

I have a function $y=sin(x)$

I sample the points $ x_i=\{0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1\}$ and $$ y_i=sin(x_i) + \epsilon \ \ \epsilon \sim iid\ Normal$$

Now given only the set $\{x_i,y_i\}$ I want to uncover the underlying functional form $f(x)$ given as

$$y=f(x)+\epsilon \ \ \ \ \epsilon \sim iid\ Normal $$

My question is : ARE $y_i$'s independent ?

My intuition is that $y_1$ and $y_2$ are not independent. This is because given a new ${x}$ say ${x_\star}$ we can use $y_i$'s to learn some information about ${y_\star}$.

Does the fact that the input points come from the same underlying function make them somehow correlated or posses commonalities ? and if not why do most methods (ex: Non-parametric splines, Gaussian process) use the $y_i$'s to get information and learn ${y_\star}$ ? Isn't the fact that if samples are independent then they can't learn new information from each other ?

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Statistical independence in this case depends on how you sample your points $x_i$ (with replacement? Without? ...?) Trivially, $y_i$ will be independent of $y_j$ conditional on all $x_k$'s because your normals are iid normal. But from your description, not much more can be said about independence. $\endgroup$
    – Jeremias K
    Jan 24, 2017 at 14:36
  • $\begingroup$ @JeremiasK if $x_k$'s are sampled with replacement, the responses belong to the same underlying function. Therefore, even if $x_k$'s are independent there should be some connection between the $y$'s. I am always thinking this is not a random walk, there is a function beneath the points. Is my intuition correct, is it called Independence in this case or something else ? $\endgroup$
    – Wis
    Jan 24, 2017 at 18:24
  • $\begingroup$ Your question is puzzling, because (a) you explicitly assert the $y_i$ are independent (that's what one of the i's in "iid" means) but (b) then you ask whether the "$y^\prime$" are independent or not. Obviously, then, the $y^\prime$ don't refer to the $y_i$--but what do they refer to? $\endgroup$
    – whuber ♦
    Jan 24, 2017 at 19:03
  • $\begingroup$ @whuber When I said $y_i's$ i meant $y_i$'s. The error is $iid$ however the response vector $y_i$'s come from the same function. Doesn't that mean they have serial dependence (auto correlation) or they are linked in some manner ? $\endgroup$
    – Wis
    Jan 25, 2017 at 2:07
  • $\begingroup$ @raw5 I hope I understood your question from the comments now. In my answer, I assume that you sample iid from the set $S$ and that the $\epsilon_i$ and $x_i$ are sampled independently from each other $\endgroup$
    – Jeremias K
    Jan 25, 2017 at 12:00

1 Answer 1

Reset to default
0
$\begingroup$

If I interpret the comments correctly, you sample the $x_i$ independently of each other. Denote the random sequence that you sample by $X_1, X_2, \dots$ with realizations in the set $S = \{0, 0.1, \dots 1 \}$. The fact that you use an iid-sample from $S$ amounts to saying that for all $x_i, x_j \in S$, $\mathbb{P}(X_i = x_i | X_j = x_j) = \mathbb{P}(X_i = x_i) = 1/11$. (This is the definition of statistical independence that holds for the sequence $X_1, X_2, \dots$ by assumption of an iid sample from $S$.)

Now consider any arbitrary function $f:S \to \mathbb{R}$. As long as the function $f$ has a unique inverse $f^{-1}$ on $S$, it holds that each point in $S$ has a unique point in $\mathbb{R}$ it is associated with. Thus, it holds that the exact same probablity mass is on the event $\{f(X_i) = f(x_i) \}$ as on the event $\{X_i = x_i \}$. In other words, it holds that $\mathbb{P}(f(X_i) = f(x_i)) = \mathbb{P}(X_i = x_i)$! Another way to look upon this is to say that if $f$ has a unique inverse on $S$, sampling at random with replacement from $S$ and then plugging it into $f$ is identical to sampling at random with replacement from $f(S) = \{f(0), f(0.1), \dots f(1) \}$, as there is a one-to-one correspondence between the two procedures.

Clearly, the function $f = \sin(x)$ satisfies the condition outlined in the paragraph on $S$. We thus have that the above paragraph can be applied to $f$. Hence, $y_i$ is a sum of two independent random variables, namely $f(X_i)$ and $\epsilon_i$. It follows that $y_i$ and $y_j$ are independent.

Improve this answer
$\endgroup$
5
  • $\begingroup$ thank you for your comment. I do now understand that the $y_i$ and $y_j$ are independent. Sometimes when we have many responses $y_i$'s from an underlying function we can model these responses to infer the underlying function (spline, Gaussian process etc..). However, when responses are independent we cannot use them to infer any knowledge to another random variable. On what basis are these methods working ? is there some kind of serial dependence (auto correlation) as in dependence with time increments so that I can combine the responses and infer the underlying function ? $\endgroup$
    – Wis
    Jan 26, 2017 at 1:57
  • $\begingroup$ Well, you can infer the function $f$ given that you have the tuples $(y_i, x_i)$! You will then try to find a function (spline, polynomial, linear regression, ... ) that gives you a prediction of $y_i$ given $x_i$. In fact, this is precisely what kernel regressions do, for example. You are trying to build a function $\hat{y} = f(x)$ that gives you a prediction if you feed it with $x$. In the process, you either estimate $f$ or you pretend that you know it. $\endgroup$
    – Jeremias K
    Jan 26, 2017 at 11:43
  • $\begingroup$ mathematically, if we consider the tuples $\{y_i,x_i\}$ why can we learn the model. Do the tuples become dependent and in what mathematical sense ? can we still say that $P(y_i | x_i)$ independent of $P(y_j \given\ x_j)$ $\endgroup$
    – Wis
    Jan 26, 2017 at 16:22
  • $\begingroup$ Well, yes they are dependent on each other by virtue of the specified function $y = f(x) + \epsilon$. So you have that $\mathbb{E}(Y|X=x) = sin(x)$. Since in your case, $f$ is invertible on $X$, you could in principle also construct $y - \epsilon = f(x) \Longleftarrow x = f^{-1}(y - \epsilon)$, but then your dependence relationship in $\mathbb{E}(X|Y=y) = \mathbb{E}(f^{-1}(y - \epsilon))$ is simply more complicated and cannot be computed easily in closed form (because you do not observe $\epsilon$). $\endgroup$
    – Jeremias K
    Jan 26, 2017 at 16:30
  • $\begingroup$ I am starting to get the point, just one last question. How can we write mathematically the dependence between $P(y_i|x_i)$ and $P(y_j|x_j)$ assuming we the tuples $\{y,x\}$ are from the same function $f(x)$. $\endgroup$
    – Wis
    Jan 26, 2017 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.