I am looking for a way to build and train an end-to-end CNN that contains two steps: 1) a CNN for finding a face and hands in the image and 2) CNN that works on the crops of the face and hands. To accomplish the first step I was thinking of: 1) finetuning a YOLO network for detection of faces & hands, but YOLO has tons of parameters for detecting many classes of objects; 2) building a custom YOLO-style network and training both CNNs at once. Any other ideas? Any suggestions or links are highly welcome!
I'm not sure what you mean by "2) CNN that works on the crops of the face and hands". Are you attempting to simply detect hands and faces?
If you're looking to just detect faces and hands in an image, then option 1: finetuning an existing pretrained YOLO network is your best bet. Just modify the last layer to output 2 classes (hands and faces) and train the model on your dataset. The large number of parameters are not due to the number of classes. The number of classes only influence the output of the final fully connected layer.
Assuming you don't just want to do detection but instead do something like:
Given an image: find faces and hands and then determine the orientation of the hand i.e. number of fingers the person is holding up, is it a clenched fist or in the case of faces, perform emotion recognition, etc.
In this case you may need to use a secondary CNN that is trained for your particular task.
In the event I've misunderstood the question or for future reference, I think you should provide a few more details regarding your objective.