I am solving a machine learning task in which I need to predict a label $\tau$ from input $\vec x$. The input $\vec x$ can be considered as two parts $\vec u$ and $\vec v$ ($\vec x$ can be thought of as a concatenation of $\vec u$ and $\vec v$).
The property of the problem is that $\vec v$ sufficiently and uniquely determines (possibly with very small noise) the label $\tau$ but the decision boundary may be very complex. Thus, with few data, inferring $\tau$ from $\vec v$ is hard. As number of data increases, a classifier with very low bias (e.g. SVM) can do a really good job.
However, the goal is to produce relatively accurate prediction with few data. $\tau$ depends on $\vec u$ in a way that has more noise than $\vec v$ but a much nicer boundary. Thus, something like a multi-class logistic classifier can do reasonably well with few data. However, it does not uniquely determine $\vec x$.
Thus, here is what I intend to do: use a (relatively simple) probabilistic classifier to produce $\Pr(\tau|\vec u)$. This is the approximate probability of each label given the "obvious but not accurate" part of the independent variable $\vec u$. Then use another (relatively complicated) probabilistic classifier to produce $\Pr(\tau|\vec v)$. As number of data increases, the final prediction should favor more the opinion of $\Pr(\tau|\vec v)$ but when there are not many data, we should not put too much weight on $\Pr(\tau|\vec v)$ (because it may just be overfitting).
So I think it is reasonably to consider $\Pr(\tau|\vec u)$ as a "prior" on labels and $\Pr(\tau|\vec v)$ the "likelihood" (using Bayesian terms). Thus, initially prior should have a larger weight but as we see more data, we should shift and rely on what is suggested by the likelihood.
So can anyone suggest a more concrete procedure to do this? For example, what formulas should I use to combine the two probabilities, $\Pr(\tau|\vec u)$, $\Pr(\tau|\vec v)$?
A very simple way would be just take some sort of weighted average of the two predictions. Then the weight should shift gradually from $\Pr(\tau|\vec u)$ to $\Pr(\tau|\vec v)$ as more data come in. However, this approach completely lacks any theoretical support or guarantee that it is not convincing. For example, you cannot mathematically answer questions such as how to determine the initial weight, and how the relative weight should change as a function of time concretely. On the other hand, it seems to me like Baysian approach (described above using "prior" and "likelihood") is a natural way to consistently model this kind of behavior.