In their text on machine learning, Ben-David and Shalev-Shawrtz (see https://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/understanding-machine-learning-theory-algorithms.pdf) develop a setting in which learning takes place in chapter 3. Essentially you have a hypothesis class $\mathcal{H}$, an input space $\mathcal{X}$, an output space $\mathcal{Y}$, a probability distribution $\mathcal{D}$ on $\mathcal{Z=X\times Y}$ and a loss function $l:\mathcal{H\times Z}\rightarrow\mathbb{R_{\geq 0}}$.

They note on page 49 that for each $h\in\mathcal{H}$, $l(h,.)$ is a random variable. Now assume that you have a sequence of $z_1,\ldots,z_n$ iid sample of points in $\mathcal{Z}$ sampled according to $\mathcal{D}$. On page 56 they have the following: "Getting back to our problem, let $\theta_i$ be the random variable $l(h,z_i)$".

I find this confusing.

1) According to the data it seems that $l(h,z_i)$ is a single real value, say $c_i$. Is it that you think of this real value as the random variable $f_i$ that is constant on $\mathcal{Z}$ with $f_i(z)=c_i$ for all $z\in\mathcal{Z}$?

2) Since the sample is drawn in an iid fashion there are iid random variable $\theta_i$ such that the $z_i$ are realizations of the $\theta_i$. Is it that they referring to these $\theta_i$. Also, if this is the case, it seems to me that $\theta_i (z)=l(h,z)$. But this seems to contradict the iid assumption, unless you assume that $\theta_i$ talks about the $i^\text{th}$ pick from the data sample. Even then the calculations then should be far simpler as the $\theta_i$ can be replaced by $l(h, .)$ which leads me to think that I'm making a mistake somewhere.

So what is the correct viewpoint here?

  • $\begingroup$ Welcome to Cross Validated. I recommend you choose a more descriptive title for your question, this way it is more likely to be found on the site. $\endgroup$ – deemel Apr 24 at 14:22
  • $\begingroup$ @Rickyfox: Thanks for pointing that out. I was going to put a more descriptive title but I forgot and ended up submitting the question. $\endgroup$ – user233343 Apr 24 at 14:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy