# Sampling: If one sample produces multiple outputs, how many samples do I actually have to work with?

A strangely worded question so let me explain. Consider the following image

I have a distribution $P$ at the top from which I can generate a number $x$. I have two sets of vectors, $S_1$ and $S_2$. Finally, I have a decider $D$ or classifier. Let's assume that $S_1$ has 10 vectors and $S_2$ has 5 vectors. A generated number $x$ from $P$ modifies vectors in both $S_1$ and $S_2$ (say multiplies them or whatever). Each modified vector then produces a 0 or 1 after it goes through the decider. My goal is to fit a distribution to the decider, and it is important for me to know how many samples I have so I can say what my confidence intervals are.

Here's the question: If I generate one $x$ from $P$ and use it to modify all the vectors in $S_1$ and $S_2$ and go down the chain, I get 15 outputs from the decider. So do I have 1 sample or do I have 15 samples? My guess is 1 but I can't explain why.

Now let's assume I generate $x_{1}$ from $P$ and use it to modify the vectors in $S_{1}$, and generate another $x_{2}$ from $P$ and use it to modify the vectors in $S_{2}$. Again, I have 15 outputs from the decider. So do I have 2 samples or 15?

• Back up. What do the samples and vectors represent in the first place? What are you trying to do? Etc. – Kodiologist Aug 22 '17 at 17:12
• I'm making the problem more abstract so we don't have to discuss application specific details. The vectors are just arrays of numbers that get crunched by the decider. The goal is to obtain samples to characterize the decider (fit a distribution or whatever you want to do). The question as I mentioned, is to find out what a "sample" is in this case. – Mr. Fegur Aug 22 '17 at 17:41
• @MrFegur But the application matters. You might have an XY problem. – Kodiologist Aug 22 '17 at 17:53
• What is not specific about: My goal is to characterize D. – Mr. Fegur Aug 22 '17 at 20:34
• The lack of context. – Kodiologist Aug 22 '17 at 22:12

You have one sample of $x$ and 15 sample vectors.
I think you might be worried that your 15 sample vectors can be thought of as a $15 k$-dimensional representation of a one-dimensional manifold (where $k$ is the dimension of each vector). But if your goal is to best approximate the mapping from the space of vectors to the output of $D$ then you have a discriminative problem, which means you don't care about the distribution from which your input was obtained. You only care that your input samples decently span the domain of $D$. Whether your points decently span the domain if they're only transformed from a single sample $x$ depends on the transformation process.
Consider the following example. Suppose $f(x) = \mathcal{N}(x; 0,1)$ is a mean-zero unit Gaussian distribution. You are given a data set of $N$ input-output pairs $\{x_i, y_i\}$, where the $x$ values are drawn from a different distribution, namely $g(x) = \mathcal{N}(x; 3, 25)$, and $y$ is evaluated as $y = f(x) + \epsilon$ where $\epsilon$ is a zero-mean Gaussian noise term (for example, a measurement error). Then, if you are trying to do a regression problem of finding $f(x)$ by evaluating $y$ then this would be a perfectly fine data set to do so, because outputs from $g(x)$ span the domain of $f(x)$ very well, even if $g(x)$ is nothing like $f(x).$ On the other hand, if $g(x)$ was simply a dirac delta function, then you would have a problem.
You just have to ask yourself if the resulting vectors span the domain of $D$ well enough to let your estimation of $D$ properly generalize. Given what you told us in your question, we don't know if that's true. For example, it's possible that there are only $15$ possible realizations of a vector in the vector space with which you're concerned, and they're all obtained by generating any sample $x$ (the only thing $x$ determines is which vector gets represented by which transform). In that case, you only need one sample $x$ to perfectly describe $D.$ On the other hand, it could be that the value of every vector is just the value $x$ repeated $k$ times, in which case you get an extremely narrow piece of information from one sample (i.e. only one vector value). Obviously the true scenario will lie somewhere between those extreme cases.
• Thinking about it in terms of spaces really cleared it up for me. I had to draw some 3 dimensional plots but ultimately the answer is that one $x$ gives me one sample. Cheers. – Mr. Fegur Aug 22 '17 at 23:52