How to improve classification performance based on multiple known classification results I am working on a classification problem, which may contains a unknown number of data classes, typically 5-50 classes in each sample. I had several classification algorithms, each gives me a classification output based on a given sample. However, these classification outputs are almost unlikely to agree with each other completely. 
When I looked into these classification results, I noticed that there always exists a better classification output in a subset of a given sample. This means that I shall be able to get a better classification output if I can use these existing classification outputs properly. However, the problem is I do not know how to do this job.
So far, what I have is 500 manually annotated sample data, and corresponding classification results for these training data. Is there any simple way that I can train a system to automatically form a new classification output for me?
Another thing that bothered me is that I am completely lost in formulating my task in mathematics. I know my following description is problematic, please help me to correct it.
Suppose we have a training sample composed of $m$ 2D data points $\Omega =\{d_1,d_2,\cdots,d_m\}$ with a corresponding known label set $C_{\rm gt} = \left\{l(d_i|{\rm gt})\big|i\in\Omega\right\}$, where $l(d_i|{\rm{gt}})=k$ means the label of data point $d_i$ is of the $k$th class in ground truth (gt). Assume I have $n$ classification algorithms $A_1(\cdot),\cdots,A_n(\cdot)$ and each takes the entire dataset as input and generates a classification result $C_j = \left\{l(d_i|A_j)\big|i\in\Omega\right\}$, which is the label collection for all data points using algorithm $A_j$. And my objective is to find a function $A_{new}=f(A_1,\cdots,A_j)$ such that 
$$A_{new} = \arg\min_{f(\cdot)}\sum_{i\in\Omega}\left\|l(d_i|A_{new})-l(d_i|\rm{gt})\right\|.$$
This is some formulation that I learnt in signal estimation, but in this problem a label is quite different from a random variable:
1) it is discrete rather than continuous
2) even if one algorithm assigns a label $k'$ to a data point which is marked as class $k$ in the groundtruth, it does not necessarily mean it is wrong, because what we really care about is whether the $k$th class in the groundtruth has been marked as one single class in our output, and we do not care about which label it used in the output. 
Finally, I have no idea how to construct a function $f(\cdot)$. 
 A: If you have some labeled data, you can probably perform what is known as "semi-supervised clustering". That is, you find the "optimal" clustering based on the complete data using both labeled and unlabeled data. The EM algorithm is often used for this type of problem depending on what type of model you use. One flexible approach may be to fit a Normal mixture model. You can try this for an varying number of classes (as you note that you do not know the true number of classes) and use some information criterion or cross-validation approach to select the correct number of classes. 
A: Since you have a sample with known labels, you want supervised learning algorithms here (perhaps that's what you're talking about, but "clustering" usually refers to unsupervised learning).
Supervised learning involves learning a classification rule from training data that can be generalized to test data of the same format. Some examples are multinomial logistic regression, support vector machines, and random forest classifiers. Without knowing more specifics about your data I can't make a specific recommendation.
The next part of your question involves pooling results across various classifiers: this would fall under the domain of ensemble classifiers. Aggregated results from multiple classification rules often outperform the individual classification schemes comprising them. There are many different methods for aggregation across multiple classification schemes. Your question is very broad, so hopefully this is enough to get you started.
