How to combine multiple signal data in my ML model?

Question

I'm doing a task where I need to work with healthcare data from a few different sources. For example, one is an audio signal recording while another is biometric signal reading such as ECG.

Both of these will be available for each patient, and we are trying to do some machine learning classification to do some predictive diagnosis. In fact, we have created a multiclass classifier model of the audio signal, as well as the ECG data.

Now, we are trying to combine the two signals modalities into one single, better model. What would be the best approach for this kind of problem? We are researching multimodal ML algorithms but would like to get a feel of what is the best way to go about creating this model.

Answer 1

I have not worked with multimodal ML, but from my experience I would try two different approaches to your problem:

1. Feed a machine learning algorithm with all the data, prior to data normalize the inputs, your algorithm, e.g. a SVM, should be able to learn the relations between different sources.
2. Make one ML algorithm for each signal source, and combine the outputs of these algorithms in a new one, however this a bit more tedious a error prone.