How to choose best value of 2 discrete parameters simultaneously? I have devised an algorithm that takes as input a dataset $s$ and two parameters, $k$ and $d$. $k$ is discrete while $d$ is categorical. My algorithm outputs a vector $v$ of length 20 which contains real numbers between 0 an 1.
I run my algorithm on three different datasets $s_1$, $s_2$, $s_3$. For each dataset, I vary $k$ in the range {5, 10, 20, 30, 40, 50, 60}, and I vary $d$ from {1, 2, 3}. I run each combination and save the resulting $v$.
My goal is to find out which values of $k$ and $d$ show the best average performance (in terms of $v$, larger values are better) across all three datasets. I'm ideally looking for a test or method that will tell me exactly which value of $k$ and $d$ are the best overall values for these three datasets.
How can I do this?
 A: Your algorithm can be conceptualised as a map $\mathcal{A}: S \times \mathbb{N}\times \mathbb{N} \mapsto \mathbb{R}^{20}$ which we can write in familiar notation as $\vec{v}(S; d,k) = \mathcal{A}(S; d,k)$.
Your problem can be formally stated in at least of a couple of ways: single-objective optimization or multi-objective optimization.
Single-Objective Optimization
In this formulation of the problem you must pick some scalar function to assess how 'good' a vector is, which is to say your loss function $\mathcal{L}:\mathbb{R}^{20}\mapsto\mathbb{R}$. One simple example is to sum the components of the vector (achieved below by taking the dot product of $\vec{V}$ with a row vector of ones $\vec{1} = [\underbrace{1, \cdots, 1}_{20}]$)
$$(d^*, k^*) = \arg\max_{d,k} \mathbb{E}_{S} \left[ \vec{1} \cdot \vec{v}(S; d,k) \right]$$
which you wish to estimate with
$$(d^*, k^*) \approx \arg\max_{d,k} \frac{1}{3} \sum_{i=1}^{3} \vec{1} \cdot \vec{v}(s_i; d,k)$$
where $d \in \{1,2,3\}$ rather than $d \in \mathbb{N}$ and $k \in \{5, 10, 20, 30, 40, 50, 60\}$ rather than $k \in \mathbb{N}$.
Multi-Objective Optimization
Alternatively to the single-objective case where you would aggregate the vectors by some function, such as summation of the components. But perhaps you think that each of the 20 components of $\vec{v}(S; d,k)$ should be considered an objective in their own right. This opens up a wonderful, but complicated, area of optimization called "multiobjective optimization" (MO).
For this I (almost unilaterally) recommend Emmerich & Deutz 2018 on this subject because:

*

*They give the required mathematical background (e.g. Pareto orders and Pareto fronts) to MO at an undergraduate level (i.e. it is relatively accessible)

*Surveys a few long-standing techniques (e.g. Chebyshev scalarization)

*Introduce the use of evolutionary algorithms for MO (which for your case can include discrete parameters)


Footnote: I agree with @whuber's comment that explaining the algorithm would likely yield answers more useful to your specific use case. After looking at your history I've surmised that your question was answered in a follow-up post.

