Given that "The y values are probabilities of being of a certain class, obtained from averaging classifications done manually by people," I strongly recommend doing a logistic regression on your base data. Here is an example:
Assume you are looking at the probability of someone agreeing to a proposal ($y=1$ agree, $y=0$ disagree) given an incentive $x$ between 0 and 10 (could be log transformed, e.g. remuneration). There are two people proposing the offer to candidates ("Jill and Jack"). The real model is that candidates have a base acceptance rate and that increases as the incentive increases. But it also depends on who is proposing the offer (in this case we say Jill has a better chance than Jack). Assume that combined they ask 1000 candidates and collect their accept (1) or reject (0) data.