1
$\begingroup$

I have a model implemented in Python with 2 free parameters. I would like to find the parameter values that provide the best fit to empirical data comprising of response times and accuracy of human participants in various tasks and conditions.

The approach I intend to take is to perform a gridsearch over the range of parameter values using scipy.optimize.brute and computing some kind of fitness metric. From what I understand, the Kullback-Leibler (KL) Divergence measures the "distance" between two distributions although it cannot be considered as a distance metric because it is not symmetric.

$$D_{KL}\big(p(x)||q(x)\big) = \sum_{x_i} p(x_i) log\bigg(\frac{p(x_i)}{q(x_i)}\bigg) $$

  1. Can I interpret KL divergence as a distance measure if I make sure I always use model data as $p$ and empirical data as $q$?

  2. Can I combine the KL divergences relating to different metrics (response times and accuracy), different conditions (i.e. different stimuli) and different tasks by just summing them? Below is my implementation of the error function. Elements in to_compare are numpy arrays containing data for each participant (or random generator seed in the case of my model).

class Data():
    @staticmethod
    def kl_div(a,b):
        q, bin_edges = np.histogram(b)
        p, bin_edges = np.histogram(a, bins=bin_edges)

        # normalize histograms
        p = p / p.sum()
        q = q / q.sum()

        return np.sum(np.where(p != 0, p * np.log(p / q), 0))

    @property
    def fitness_error(self, empirical):

        error = 0

        
        to_compare = [
            [
                self.task1_accuracies, 
                self.task1_RTs, 
                self.task2_accuracies, 
                self.task2_RTs, 
                self.task3_accuracies, 
                self.task3_RTs
            ],

            [
                empirical.task1_accuracies, 
                empirical.task1_RTs, 
                empirical.task2_accuracies, 
                empirical.task2_RTs, 
                empirical.task3_accuracies, 
                empirical.task3_RTs
            ]
        ]

        for i in range(len(to_compare[0])):
            model_data = to_compare[0][i]
            empirical_data = to_compare[1][i]
            for stimulus in range(4):
                error += Data.kl_div(model_data[stimulus], empirical_data[stimulus])

        return error
  1. I feel like I might be reinventing the wheel here, but I cannot find any paper about using KL divergence for model selection with respect to both accuracy and response times in various conditions. I guess it is more common to use mean squared error with error rates and average response times. I read a bit about AIC and BIC model selection methods but as I have a fixed number of parameters, I think they are not relevant in that context. Can you please link me some related references, preferably in the field of cognitive modelling?

  2. Does this approach make sense overall? What precautions should I take?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.