Finding the best cookie recipe. Hyper-parameter optimization using noisy local comparison I was watching applied science explore the state space while making cookies in this video https://youtu.be/8YEdHjGMeho. He was setting everything manually and it looks like he was searching a 10 dimensional recipe space.
I was thinking that it's pretty hard to give a consistent score to each cookie you taste but you could easily rank pairs of cookies tasted. 
How could someone optimize hyper-parameters in a framework where there is no global score function. Only a noisy rank between two points. I think its fair to assume choice is transitive and that your taste doesn't change too much throughout this experiment.
It looks like his function can run in batches of 20-30 points at a time but there is a pretty high latency for each batch. 
 A: If you're interested in use (more than in development), you should give a try to rankade, our ranking system. Rankade is free and easy to use, it can manage small to large playing groups (composed by players or 'cookies', as per your needs, or whatever), and it features rankings, stats, and more. It doesn't cover all of your tasks, maybe, but it should be useful to rank $n$ cookies from best to worst, both with global and partial rankings.
Ghosts' feature allows you to create a group without any account but your. Rankade's algorithm can manage, within different scopes, any kind of match. Due to faction structure, you can record outputs for 1-on-1 comparison, or for 2+ items as well (even mixing both kinds of 'matches'), while Bradley-Terry model or other ranking system (like most known Elo or Glicko - here's a comparison) don't. It's surely true that it's pretty hard to give a consistent score to each cookie you taste, but a multiple comparison (e.g. 3-6 cookies per 'match') should be suitable and useful, in your work. If you have more than one tester with different reliability, you can use weight feature for their 'matches'. 
