Automatic experimental design with expert in the loop
Date:
Manchester Workshop on Bayesian Experimental Design 2026
AIchemy and Leverhulme Research Centre Conference 2026
I gave two variants of the same talk, one in Bayesian Experimental Design workshop at Manchester Centre for AI Fundamentals, June 24, and AIchemy and Leverhulme Research Centre Conference, June 29. I spoke about a set of our recent papers and was asked about the papers - so here a brief summary with links to those main papers.
I spoke of two main topics: What to do with unknowns in the model, and what do do with unknowns in the goal.
Unknowns in the model
Do Bayesian inference on the unknowns, and measure new data if you can. Which data to measure: Use Bayesian decision theory to choose measurement actions, or other decisions, to maximize expected utility. Bayesian optimization is an instance of this.
We have known for some time already how to automate all these operations, but only in principle. In practice, the computations have been much too heavy for interactive use in actual practical experimental design tasks.
We have published a series of papers that transform this by amortization, which is an “embarrassingly simple” idea of transferring the on-line computation to off-line pre-computation. Essentially, learn form a massive number of pre-computed simulated designs a function that maps a context (experimental design history) to optimal design. Details are not embarrassingly simple, though… This can be generalized to multi-objective Bayesian optimization, even dimension-agnostically, to preferential (pairwise) inputs, and even to noisy or perturbed inputs from biased experts with partial knowledge. Here links to the papers:
ALINE: Joint Amortization for Bayesian Inference and Active Data Acquisition, NeurIPS 2024
Amortized Probabilistic Conditioning for Optimization, Simulation and Inference, AISTATS 2025
PABBO: Preferential Amortized Black-Box Optimization, ICLR 2025
In-Context Multi-Objective Optimization, ICLR 2026
In-Context Black-Box Optimization with Unreliable Feedback, arXiv
Unknowns in the goal
Assume an expert in the loop, who is in charge of the experiments, and has tacit knowledge about the goal. That is, even if they cannot write down their precise utility function, they will be able to give feedback on which candidate solutions are more promising. Now the AI can choose to ask the expert for preferencial feedback, interleaved with simulatied experiments and measuring new data.
An early example in drug design
Interpreting the input from the expert requires a model of the expert, a “user model”. It does not need to generate all human behaviour, but needs to give a likelihood for responses given the expert’s tacit preferences on the goal. This we can get from the principle of computational rationality. Again the computations would be too heavy in practice - but again we can amortize.
Preference Learning of Latent Decision Utilities with a Human-like Model of Preferential Choice, NeurIPS 2025
Modeling needs user modeling. Position piece.
Toward AI assistants that let designers design. Position piece.
Acknowledgements
This is joint work with: Daolang Huang, Luigi Acerbi, Xinyi Wen, Ayush Bharti, Paul Chang, Nasrulloh Loka, Ulpu Remes, Xinyu Zhang, Nicolas Samuel Blumer, Conor Hassan, Julien Martinelli, Iiris Sundin, Ola Engkvist, Mustafa Mert Celikok, Pierre-Alexandre Murena, Sebastiaan De Peuter, Antti Oulasvirta, Shibei Zhu, Yujia Guo, Andrew Howes. I hope I did not miss anyone…
As the main take-home message of the talks I also introduced ELLIS - European Laboratory for Learning and Intelligent Systems, and its second institute among the total of 44 sites. ELLIS Institute Finland is a startup institute launched last year and now in a scale-up mode, with 50 research groups by the end of this year, 20 of them new PI recruitments since the launch. In total ~500 bleeding-edge machine learners very soon, assuming average group size is 10.