I am training a neural network to do voice emotion recognition with:

100 input layer size.

25 hidden layer size.

6 labels (output layer).

I have divided the data set into train set and test set, then extracted features from the voice using MLFCC (Mel Frequency Cepstral Coefficients) which returns a matrix with different sizes. So, I used 100 features of them for each time.

The accuracy on the training set is 100%, but when it comes to the test set it is about 30-40%.

I still don't know a lot about this domain, but it is obviously overfitting the problem (maybe not, but that is what I have learned). I have done some adjustments to avoid this problem:

Increasing the lambda, decreasing the number of features, adding an extra hidden layer. The accuracy got better, but never bigger than 40%.

What would be the problem?

Here is the implementation of MLFCC:

function [cepstra,aspectrum,pspectrum] = melfcc(samples, sr, varargin)

if nargin < 2;   sr = 16000;    end

% Parse out the optional arguments
[wintime, hoptime, numcep, lifterexp, sumpower, preemph, dither, ...
 minfreq, maxfreq, nbands, bwidth, dcttype, fbtype, usecmp, modelorder, ...
 broaden, useenergy] = ...
    process_options(varargin, 'wintime', 0.025, 'hoptime', 0.010, ...
          'numcep', 13, 'lifterexp', 0.6, 'sumpower', 1, 'preemph', 0.97, ...
      'dither', 0, 'minfreq', 0, 'maxfreq', 4000, ...
      'nbands', 40, 'bwidth', 1.0, 'dcttype', 2, ...
      'fbtype', 'mel', 'usecmp', 0, 'modelorder', 0, ...
          'broaden', 0, 'useenergy', 0);

if preemph ~= 0
  samples = filter([1 -preemph], 1, samples);

% Compute FFT power spectrum
[pspectrum,logE] = powspec(samples, sr, wintime, hoptime, dither);

aspectrum = audspec(pspectrum, sr, nbands, fbtype, minfreq, maxfreq, sumpower, bwidth);

if (usecmp)
  % PLP-like weighting/compression
  aspectrum = postaud(aspectrum, maxfreq, fbtype, broaden);

if modelorder > 0

  if (dcttype ~= 1) 
    disp(['warning: plp cepstra are implicitly dcttype 1 (not ', num2str(dcttype), ')']);

  % LPC analysis 
  lpcas = dolpc(aspectrum, modelorder);

  % convert lpc to cepstra
  cepstra = lpc2cep(lpcas, numcep);

  % Return the auditory spectrum corresponding to the cepstra?
%  aspectrum = lpc2spec(lpcas, nbands);
  % else return the aspectrum that the cepstra are based on, prior to PLP


  % Convert to cepstra via DCT
  cepstra = spec2cep(aspectrum, numcep, dcttype);


cepstra = lifter(cepstra, lifterexp);

if useenergy
  cepstra(1,:) = logE;

And here is my implementation:

clear ; close all; clc


input_layer_size  = 70; 
hidden_layer_size = 100; 
hidden2_layer_size = 25;                         
num_labels = 6;          

fu = [input output];size(fu)
input = fu(:,1:70);
output = fu (:,71:76);

crossIn = input(201:240,:);
trainIn = input(1:200,:);

Theta1 = randInitializeWeights(input_layer_size, hidden_layer_size);
Theta2 = randInitializeWeights(hidden_layer_size,hidden2_layer_size);
Theta3  =randInitializeWeights(hidden2_layer_size,num_labels);

initial_nn_params = [Theta1(:) ; Theta2(:);Theta3(:)];
options = optimset('MaxIter',1000);

%  You should also try different values of lambda


costFunction = @(p) nnCostFunction(p, ...
                                   input_layer_size, ...
                                   hidden_layer_size, ...
                                   hidden2_layer_size,num_labels, trainIn, trainOut, lambda);    

[nn_params, cost] = fmincg(costFunction, initial_nn_params, options);

               num_labels, (hidden_layer_size + 1));
Theta1 = reshape(nn_params(1:hidden_layer_size * (input_layer_size + 1)), ...
                 hidden_layer_size, (input_layer_size + 1));

Theta2 = reshape(nn_params((1 + (hidden_layer_size * (input_layer_size + 1))):( (hidden_layer_size * (input_layer_size + 1)))+(hidden2_layer_size*(hidden_layer_size+1))), ...
                 hidden2_layer_size, (hidden_layer_size + 1));
Theta3 = reshape(nn_params(((1 + (hidden_layer_size * (input_layer_size + 1)))+(hidden2_layer_size*(hidden_layer_size+1))):end), ...
                num_labels, (hidden2_layer_size + 1));
 %[error_train, error_val] =  learningCurve(trainIn, trainOut, crossIn, crossOut, lambda,input_layer_size,hidden_layer_size,num_labels);    

pred = predict(Theta1, Theta2,Theta3,trainIn);
[dummy, p] = max(trainOut, [], 2);
[pred trainOut]
fprintf('\nTraining Set Accuracy: %f\n', mean(double(pred == p)) * 100);    

pred = predict(Theta1, Theta2,Theta3,crossIn);

[pred crossOut]
[dummy, p] = max(crossOut, [], 2);
fprintf('\nTraining Set Accuracy: %f\n', mean(double(pred == p)) * 100);

And here is the code for getting the patterns:

function [ input,output ] = gettingPatterns()
myFolder='C:\Users\ahmed\Documents\MATLAB\New Folder (3)\homeWork\speech'; 
for k = 1:length(wavFiles)

baseFileName = wavFiles(k).name;
if baseFileName(3:5)=='ang',sampleOutput(1)=1;,end;
if baseFileName(3:5)=='fea',sampleOutput(2)=1;,end;
if baseFileName(3:5)=='bor',sampleOutput(3)=1;,end;
if baseFileName(3:5)=='sad',sampleOutput(4)=1;,end;
if baseFileName(3:5)=='joy',sampleOutput(5)=1;,end;
if baseFileName(3:5)=='neu',sampleOutput(6)=1;,end;

fullFileName = fullfile(myFolder, baseFileName);
wavArray = wavread(fullFileName);

%Normalized Features
for i=1:length(smalledArray)

Please note the following:

I have done this test on the nn toolbox, I got the same result. The only reason for implementing it myself is to be able to add an extra hidden layer.

The implementation of the cost function, forward propagation and backward propagation are 100% correct, so I did not include them in this question.

  • $\begingroup$ Actually, 40%, in a 6 class problem is not that bad. If you were assigning class randomly you would have an accuracy of 16%. Besides, how did you tune your neural network ? They are pretty hard to tune and another set of parameters could achieve better results. $\endgroup$
    – RUser4512
    Commented Mar 4, 2016 at 9:41
  • $\begingroup$ I dont think I understood what you mean ? what should I change ? $\endgroup$
    – user111
    Commented Mar 4, 2016 at 21:14

1 Answer 1


Emotion recognition is a challenging problem although it is 7-class classification problem. Your task is even harder with respect to using visual clues to predict emotion. There is a challenge at kaggle on emotion recognition by using images. The best submitted model is around 71%. More interestingly human level also low with respect to other challenges goodfellow et al.:

On this dataset, human accuracy was 68$ \pm$5%. FER-2013 could theoretical suffer from label errors due to the way it was collected, but Ian Goodfellow found that human accuracy on FER-2013 was 65$ \pm$5%

I think the main problem is the classes are not exactly distinct (or mutually exclusive). If you consider visual clues, some inputs can be assigned to different classes even by humans. Labeling error is significant with respect to other challenges.

  • $\begingroup$ actually it is ENN based on voice I have seen some researches on internet with higher accuracy $\endgroup$
    – user111
    Commented Mar 5, 2016 at 7:54
  • 1
    $\begingroup$ I didn't mean it is not possible to reach high level accuracy. On toronto face dataset, www-etud.iro.umontreal.ca/~rifaisal/material/… reached to 85%. As I don't know your dataset, I'm not sure if your results are close to state-of-art or not. And some datasets are collected in lab environment with controlled environment, so they can be easier to reach higher accuracies. Could you give more information about dataset and the results from the other papers with links? $\endgroup$ Commented Mar 5, 2016 at 8:47
  • $\begingroup$ wav files of polish people Fs = 44,1 kH , 16 bits in each sample number of samples 240 $\endgroup$
    – user111
    Commented Mar 5, 2016 at 20:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.