Reinforcement Learning for
Experimental Sciences

Wave energy is a substantial renewable resource, but commercial devices must extract energy competitively under changing waves. For the coupled D-Spar converter, this creates two linked problems: predicting nonlinear device motion accurately and choosing power-take-off (PTO) damping quickly enough for control. High-fidelity numerical models retain coupled motion, radiation memory and nonlinear forces, but are expensive to evaluate repeatedly. Rather than further reducing the device dynamics, we learn the expensive nonlinear-force correction inside the full nine-degree-of-freedom model. The remaining excitation, radiation, mechanics, PTO and RK4 calculations stay explicit. On development data, a free-recursive 10-s forecast obtains 0.382% macro NRMSE over all displacement and velocity channels. A rank-5 proper orthogonal decomposition (POD5) decoder then reconstructs a 15-action damping–energy curve from only three rollouts, with 0.202% RMS-relative curve error, 100/110 exact sampled optima, exact-or-adjacent selection in every case and 0.0596% maximum regret. The complete measured K3+POD5 controller is not yet 10-Hz real time. Residual RL benefits from POD context but remains 0.606% below greedy. By contrast, exact two-wave sequence MPC gains 4.21% over matched wave-synchronous greedy and 3.57% over canonical 10-Hz greedy across three development seeds, while losing on one seed. These perfect-preview results motivate learning a compact wave-level policy from the MPC value landscape; they do not yet establish RL superiority, unseen-seed performance or real-time deployment.

Low-dimensional optimization problems in complex experimental environments are common and often take the form of black-box optimization problems. We present a protocol for automated alignment of a capillary optic that focuses extreme-ultraviolet light for angle-resolved photoemission spectroscopy (ARPES). This alignment is a demanding procedure typically performed by expert beamline scientists and is generally inaccessible to non-expert users. In spatially resolved ARPES, a small beam spot is essential for measurements of challenging materials and functional devices. We compare several optimization methods in simulations based on experimental data and demonstrate successful automated focusing in experiments at the SGM4 microARPES beamline of the ASTRID2 synchrotron radiation source in Aarhus.

Whole-slide images (WSIs) are gigapixel digital pathology slides where sparse diagnostic evidence makes exhaustive high-magnification inspection computationally expensive. Pathologists instead examine selected regions across magnifications, retain global context and examination history, update their assessment as evidence accumulates, and decide where to look next and when to stop. This naturally frames WSI examination as a partially observable sequential decision problem, for which reinforcement learning (RL) provides a principled framework to learn observation, navigation, and stopping policies. Existing RL and adaptive-sampling methods largely provide task-specific solutions, while existing environments expose limited examination states that do not reflect the richer information available to a pathologist during sequential slide examination and can be costly to execute at RL scale. We introduce WSI-Lab, a reusable, Gymnasium-compatible, pathologist-inspired environment combining the current viewport, global slide context, location and examination history, and an evolving machine diagnostic-belief estimate from acquired evidence. Agents navigate across magnifications and terminate with an explicit diagnosis. Tissue-aware preprocessing, precomputed navigation structures, caching, and efficient state updates reduce repeated online work. RL ablations show that WSI-Lab's complete examination state, tissue-aware navigation, and explicit diagnosis actions support substantially stronger training than reduced variants. Under matched environment workloads, WSI-Lab is additionally up to 2.24x faster while exposing a logical observation payload up to 10.7x larger than existing environments. Code is available at https://anonymous.4open.science/r/wsi-lab-0EEC/.

Antibodies recognise their targets through hypervariable complementarity-determining regions (CDRs), which are interleaved with conserved frameworks in sequence space, making *de novo* CDR design an infilling problem. Autoregressive models generate residues left-to-right, which precludes full framework context during CDR generation and conflates framework and CDR likelihoods, leaving no natural prompt–response interface for feedback to steer generation. We present GenCDR, a family of LLaMa-based autoregressive language models that read all frameworks as a conditioning prompt and generate all CDRs jointly as a variable-length response, making CDR likelihoods a clean, separable target for reward attribution. The family comprises *IgGenCDR*, *p-IgGenCDR*, and *NanoGenCDR*, trained on unpaired, paired, and nanobody chains, respectively. GenCDR achieves the highest CDR recovery among autoregressive models and produces natural, diverse, human-like CDRs whose likelihoods correlate with fitness and developability assays. The prompt–response boundary also enables principled alignment: reward signals for binding affinity, expression, or developability can be composed to steer CDR generation. Over four rounds of alignment against antibody–antigen co-folding and developability objectives, we find that NanoGenCDR can reach *in silico* structural interface metrics competitive with those of a structure-conditioned diffusion pipeline at roughly half the sampling budget, with more natural, developable designs. The same interface can be extended to integrate experimental feedback, opening a path to closed-loop antibody *de novo* design.

Terrestrial-based systems for early wildfire detection often use cameras deployed in remote environments, where images from these cameras are routed through the network and processed by ML models. However, determining an optimal policy for how to route and process collected images is challenging due to highly variable energy availability and network capacity at the edge. Existing methods typically adopt static approaches for determining such policies. They also rely on centralized execution, introducing a potential single point of failure. Multi-agent reinforcement learning (MARL) provides an attractive alternative. However, learned policy performance for MARL-based solutions is dependent on the quality of reward modeling, environment diversity, re-start strategies, and reward credit assignment, all of which can be difficult to define and if done poorly can lead to inconsistent and sub-optimal performance in live deployments. To address these challenges, we propose MARL-based System Optimization for Wildfire Sensing Networks (MARL-OptWSN), a framework for optimizing highly variable image-based wildfire sensing networks. MARL-OptWSN uses a custom training regime built around a Gurobi-based oracle, while leveraging decentralized execution to improve resilience and adapt to changing environmental conditions. Our training approach combines progressive training, oracle guidance, and an agent-specific blame system that identifies decisions responsible for poor outcomes. Empirical experiments show that our approach achieves near-oracle performance with substantially lower decision time, while retaining resilience under highly variable and constrained conditions that negatively affect centralized approaches.

To date, Bayesian experimental design (BED) has largely studied how a single adaptive policy should choose experiments. Yet many real-world data-acquisition systems distribute experimental capacity across multiple actors, such as laboratories, field teams, or decentralised robots. We introduce multi-actor Bayesian experimental design (MA-BED) as a general paradigm for BED in which experimental decisions are distributed across multiple actors, and within this paradigm focus on a fully cooperative setting with decentralised execution. In this regime, actors share an inferential objective but choose experiments using only their local histories, creating a coordination problem: individually informative policies may collectively acquire redundant information. We formalise this failure through an information-theoretic coordination gap and show that, under conditionally independent acquisition, it is exactly characterised by the total correlation among actor histories. To address this problem, we adapt centralised training with decentralised execution (CTDE) to BED, jointly optimising locally executable policies against the information contained in their pooled experimental histories. We instantiate this framework with DAD-based algorithms trained using a contrastive information objective. Empirically, joint training reduces coordination failure, induces greater actor specialisation under heterogeneous experimental costs, and outperforms independently trained policies without requiring communication at execution. These results establish coordination as an important challenge in Bayesian experimental design when data acquisition is distributed across multiple actors.

The Low Energy Ion Ring (LEIR) at CERN turns long lead-ion pulses from Linac3 into the dense bunches required by the Large Hadron Collider's (LHC) high intensity ion program. It accumulates successive multi-turn injections through six-dimensional phase space painting under electron cooling. The accumulated intensity depends sensitively on nine parameters, and the optimum drifts on every relevant timescale: continuously, as the stripper foil that produces the injected charge state ages, and discontinuously, as the machine is recommissioned every yearly campaign or after critical hardware faults. In this paper, we describe an operational Reinforcement Learning (RL) pipeline deployed on LEIR that treats this drift as the central design constraint and outperforms routine operational performance by more than 10\%. Using four years of archived operational data and system identification between 2023 and 2026, we first quantify the drift: generative topographic maps of the nine-dimensional setting space show that both the sampled region and the high-efficiency region move from year to year, and surrogate models fitted on one campaign transfer to the next with near zero rank correlation. We then show how to choose a low-dimensional observation robust to drift: a joint denoising autoencoder with physics-informed auxiliary heads, benchmarked against variational, quantized, and joint embedding alternatives under measured acquisition noise, compresses $\sim$136\,000 bin longitudinal Schottky spectrograms into nine latents that remain informative across campaigns. Policies are trained on per-year heteroscedastic ensemble digital twins and deployed by warm finetuning on the machine: relocking to a new campaign takes a few hundred machine cycles (1 to 2 dedicated hours of training), $\approx$33$\times$ fewer than training from scratch. Finally, exploration is framed as a constrained Markov Decision Process (MDP) with a learned, per-campaign vacuum risk cost: with a fixed Lagrange multiplier the constrained agent explores an order of magnitude more safely than its unconstrained ancestor.

In process experimentation, chemical reactors, and bioprocesses, reinforcement learning agents may only have access to a few hundred trajectories. Each is a real experiment and therefore, the employed algorithm must use them efficiently. Furthermore, any tuning of algorithm hyperparameters can further waste experimental budget. Standard neural Q-learning offers no closed-form uncertainty estimates, and requires additional tuning to achieve satisfactory performance. Non-parametric alternatives can improve sample-efficiency through improved function approximation, however existing methods either fix a constant step or target exploration regret. We propose Heteroscedastic Gaussian Process Q-learning (HGPQ): a closed-form Bayesian gain, obtained by projecting both moments of the value posterior under the distributional Bellman operator, replaces the scalar learning rate of prior Gaussian Process Q-learning, with a heteroscedastic variance update making the gain itself state-action-dependent. Under a fixed, small interaction budget across chemical process-control and biological-growth case studies, we compare HGPQ against specialized low-data RL algorithms and standard vanilla RL algorithms. HGPQ matches or improves on every low-data comparator, and deep actor-critic algorithm.

State-of-the-art deep reinforcement learning (RL) methods have achieved remarkable performance in continuous control tasks, yet their computational complexity is often incompatible with the constraints of resource-limited hardware, due to their reliance on replay buffers, batch updates, and target networks. The emerging paradigm of streaming deep RL addresses this limitation through purely online updates. Existing streaming methods, however, are studied in isolation, trained from scratch until convergence, whereas on-device learning is most useful as a continuation of a pre-trained policy. In this work, we propose two novel streaming deep RL algorithms, Streaming Soft Actor-Critic (S2AC) and Streaming Deterministic Actor-Critic (SDAC), designed as streaming counterparts of SAC and TD3, with performance comparable to state-of-the-art streaming baselines on standard benchmarks. Building on them, we study the batch-to-streaming transition as a problem in its own right: we show that a naive transition can cause a severe and lasting loss of the pre-trained policy, trace the failure to a mismatch between the optimizers used on either side of the switch and show that aligning the optimizers enables streaming deep RL in settings such as Sim2Real.

Plasma shape control in tokamaks requires a real-time controller that tracks dynamically changing shape targets while tolerating diagnostic failures. Classical approaches decompose the problem into equilibrium reconstruction followed by a linear controller, and assume a fixed, fully operational sensor set. We present a reinforcement learning agent that addresses both limitations simultaneously. The agent is trained in NSFsim, a high-fidelity tokamak simulator configured for DIII-D, on a curated dataset of 120 experimental plasma shapes. The shape targets are resampled as random step changes every 0.25\,s, exposing the agent to diverse transitions across the full shape envelope. At test time the agent zero-shot tracks dynamic shape sequences; on a held-out static configuration in simulation it achieves a mean shape error of 2.01\,cm, and dynamic trajectory following is demonstrated qualitatively in simulation and on the physical device. Diagnostic dropout randomly masks 30\% of magnetic sensors per episode, yielding a single policy robust to a distribution of sensor subsets used for tokamak operation, without backup controllers or mode-switching logic. An asymmetric actor-critic architecture with privileged equilibrium information improves value estimation under partial observability; an auxiliary shape reconstruction head on the actor enables end-to-end shape reconstruction from raw diagnostics and serves as an interpretability tool for policy analysis. The policy transfers to experimental DIII-D shots, where it directly commands the coil actuators on two dynamic shape maneuvers, and to the independent GSevolve simulator.

In active sequential testing, also termed pure exploration, a learner is tasked with the goal to adaptively acquire information so as to identify an unknown ground-truth hypothesis with as few queries as possible. This problem has several motivating applications, including Best-Arm Identification (BAI) in bandits, where actions index hypotheses, and generalized search problems, where strategically chosen queries reveal partial information about a hidden label. In many modern settings, however, the hypothesis, or recommendation space, is continuous: for example, identifying a near optimal action in a continuous-armed bandit, localizing an $\epsilon$-ball contained in a target region, or estimating the minimizer of a function from noisy observations. Existing methods are predominantly frequentist and model-specific, while learned approaches have been limited to finite recommendation spaces. We introduce C-ICPE, a theory-guided learned model for Bayesian fixed-confidence pure exploration with continuous recommendations. C-ICPE meta-trains sequential architectures over a task prior to jointly learn exploration, stopping and recommendations strategies. At inference time, it actively gathers evidence on tasks and identifies an -optimal recommendation without parameter updates.

Autonomous and adaptive experimentation systems are increasingly proposed for the physical sciences, often on the intuition that choosing which measurement to take next, rather than measuring at random or on a fixed schedule, should make better use of a limited observation budget. This project tests this intuition directly on a physical-parameter calibration problem: a stochastic wildfire simulator with three unknown physical parameters, calibrated from field observations under a fixed budget. This project compares an active, uncertainty-seeking sensing policy against random sampling and a fixed monitoring-grid sweep, evaluating all three on held-out calibration loss, burned-area agreement with the true fire, and parameter-recovery error. An initial run with 5 independent fire draws suggested active sensing had a real edge. A properly-powered rerun with 12 independent fire draws did not reproduce it: none of the three strategies are statistically distinguishable on any of the three metrics (paired Wilcoxon signed-rank test, $p \geq 0.456$ pooled over the full budget sweep). A second scenario with a different wind direction, ignition point, and terrain map does show a significant active-sensing advantage on calibration loss ($p=0.0009$), so the effect is not universally absent, it is scenario-dependent and easy to mistake for a general property of the method from a single underpowered pilot. This project argues that claims about adaptive experimental design improving physical-parameter calibration should be treated as empirical, checkable questions specific to a deployment scenario, not assumed a prior, and it reports the initial false positive as a worked example of why.

Multi-arm bandit algorithms are increasingly used in online platforms, clinical trials, and social science experiments, but valid statistical inference on their performance remains an open challenge. After deploying bandits, a natural question is whether one can construct a confidence interval for its mean reward and assess whether it reliably outperforms a baseline policy. More broadly, one may wish to assess a new candidate algorithm's expected reward before committing to deployment. Standard inference methods break down because bandit algorithms introduce complex dependencies in the collected data, leading to inflated Type-I error. Moreover, existing inference methods for bandit data only apply to estimands such as the mean reward under a fixed action, which do not depend on the data-collection algorithm. We propose Bandit Simulation for Inference (BSI), a framework that fits a simulator of the bandit environment from on- or off-policy data and uses it to estimate the mean reward under any evaluation policy, including adaptive blackbox algorithms. BSI formally propagates simulator estimation error into the confidence interval construction, requires only weak exploration assumptions on the behavior policy, and avoids importance weighting. We prove that BSI yields asymptotically valid confidence intervals, and demonstrate empirically that it maintains nominal coverage in settings where standard off-policy evaluation methods fail.

Food factories rely on fixed operating procedures because they are stable andeasy to audit. These procedures work well in normal production, but they canreact slowly to rare contamination events. We study whether reinforcementlearning can find when a factory should clean earlier. Our simulator follows areported factory practice of 18 hours of production followed by six hours ofcleaning. It also includes hidden failures, noisy sensor readings, and delayedtests. The current procedure earns a steady profit in normal operation, butloses an average of 3,276 dollars in a simulated crisis. We use simulated futureoutcomes to learn the value of cleaning after an alarm. The learned policyreduces crisis loss by 182 dollars compared with a strong two-alarm rule whilepreserving routine profit. The learned values also point to a simplestrategy: clean after the first strong alarm. On new simulations, this rulereduces crisis loss by 586 dollars and gains 5.83 dollars per day. The main lesson is notthat learning should replace the factory procedure. It is that learning canhelp find a small and useful change to a mature process.

Generative molecular design is often framed as a reinforcement learning (RL) problem in which a policy proposes candidate molecules and is rewarded according to desired properties. Because these properties cannot be measured during generative optimization, they are instead estimated using predictive models, where the reward is typically constructed from multiple in silico molecular property objectives formulated as a multi-parameter optimization (MPO) scalarization. The choice of weights is an important design decision, yet assigning them is difficult: equal weighting implicitly assumes objectives are independent and equally important, and yet drug-discovery objectives are frequently correlated or antagonistic, while manually specified weights rely on expert assumptions that may not reflect statistical dependencies among objectives. We integrate correlation-aware MPO (caMPO), a data-driven scalarization that down-weights redundant objectives, into a molecular scaffold-based RL generator (LibINVENT), recomputing weights online at every optimization step from the molecules generated so far, allowing the reward to adapt to the evolving correlation structure among property objectives. Across 17 scaffolds derived from a public PDE10A dataset, and comparing to a standard equal-weight approach, correlation-aware reward shaping improved property-space quality and alignment with target property profiles, with no significant impact on chemical novelty for both aggregations, and broadened Pareto coverage under arithmetic mean aggregation. These results suggest correlation-aware scalarization may serve as a simple, generator-agnostic reward-design mechanism for RL in experimental molecular science, enabling objective weights to be inferred from data rather than specified manually.

Training animals to perform complex behavioral tasks is a major bottleneck in systems neuroscience. A common training strategy is shaping: animals begin with easier versions of a task and progress toward the target behavior as their performance improves. This process resembles automated curriculum learning (ACL), where task difficulty is adapted to the learner's performance. Standard ACL formulations, however, typically assume that the learner remains available for continued training. In animal shaping, participation is not guaranteed. Animals may disengage after repeated failures or effortful trials. Reward can help sustain participation, but its motivational value declines with satiety and each session has a finite reward budget. Here, we formulate shaping as a sequential control problem over task difficulty and reward magnitude. Difficulty determines what the animal practices, whereas reward allocation influences whether future training opportunities remain available. We characterize how these decisions should be coordinated using a tractable three-step simulation, then develop a scalable strategy that adapts difficulty from recent performance and reward from the animal's within-session state. Adaptive reward allocation approaches a full-state dynamic-programming benchmark and provides the greatest benefit when declining participation would otherwise shorten training sessions. The same principle reduces sessions to mastery in longer sequential and perceptual-learning tasks. These results establish limited trial availability as a key feature of animal shaping and suggest that efficient training requires managing not only what an animal practices, but also how limited reward is allocated to sustain future learning opportunities.

Structured prediction tasks such as tubular segmentation require topology preservation over long-range geometric dependencies. Pixel-wise classification methods are limited in preserving fine-scale connectivity and recovering vascular structure. Connectivity is therefore the property that matters: a segmentation broken into disconnected fragments misrepresents the network even when its pixel overlap is high. Reinforcement learning (RL) offers an alternative by explicitly tracing image geometry, but policies trained without domain adaptation may become overly sensitive to low-level image statistics and generalize poorly under distribution shift. We propose a memory-augmented actor--critic framework that traces tubular structures sequentially, using accumulated trajectory context to resolve local ambiguities. The policy jointly models exploration and branching to reconnect fragmented structures. To enable zero-shot generalization, we train the policy through an adaptive curriculum on synthesized data covering an expansive distribution of noise and geometry.When applied without adaptation to the outputs of a DRIVE-trained segmenter, the policy reduces the number of connected components ($\beta_0$) from 29.1 to 4.5 while recovering vessels missed by the segmenter, increasing centerline recall from 85.8\% to 87.7\% at 94.6\% on-vessel precision. Without retraining or modification, the same policy transfers to ISBI-12 electron-microscopy membranes, despite their distinct appearance and topology, reducing $\beta_0$ from 10.8 to 4.2 while maintaining 94.1\% recall. These results demonstrate that a sequential tracing policy learned entirely from synthetic data can improve topological coherence across substantially different imaging domains.

We propose stochastic decision horizons (SDH), a theoretically grounded framework for survival-based constrained reinforcement learning from per-step violation signals. Rather than treating violations as expenditure of a cumulative budget, SDH shortens the effective decision horizon when violations occur, reducing current reward credit and future bootstrapping. We show that this construction represents a geometric-horizon survival constraint and, for a fixed Lagrange multiplier, remains the same SDH problem up to a constant reward offset that acts as an implicit incentive for survival. Using control as inference, we derive two off-policy and replay-compatible, regularized algorithms with post-violation semantics. Absorbing-state ends the decision process, so only surviving decisions pay policy KL, yielding AS-SAC. Virtual termination stops reward credit while the process continues, retaining ordinary discounted KL and yielding VT-MPO. To connect SDH with cumulative-cost CMDPs, we introduce violation-depth profiles. Under exponential-tail single-scale profiles, the SDH statistic determines cumulative cost; departures from this form mark where the connection breaks. Experiments validate both the method and this predicted scope. On the 90-muscle H2190 humanoid in Hyfydy, VT-MPO matches the EWA reference in peak gait realism, with its highest observed gait-match checkpoint at 21M environment steps versus 90M for EWA and markedly more stable training. On Safety Gymnasium, violation-depth profiles predict the regimes in which SDH achieves strong reward-violation trade-offs.

Closed-loop design systems are now common in the experimental sciences, but few of them report which part of the loop produced the improvement.We report such a system to increase the activity of large serine recombinases (LSRs), which catalyze unidirectional, site-specific integration of multi-kilobase DNA cargoes, making them attractive for therapeutic gene insertion.Each round aligns a 3B-parameter masked protein language model to activity data with a learn-to-rank objective,draws multi-mutant variants by Gibbs MCMC under a soft trust region on mutational distance,assays them by in vitro transcription-translation, and returns the measurements.Across two campaigns, five rounds and 392 assayed designs, the best variant carries fivesubstitutions and reads 165 times parent activity on the same plate.We then ask which component was responsible.Of the components we tested, only the trust region has an effect we can measure:distance to the previously measured set is inversely correlated with activity, absorbing the whole difference between policy-update schedules.We report these findings as a template for future closed-loop campaigns.

Facilities that consider reinforcement learning for control almost always possess a trusted classical controller, and a recurring proposal is to keep that controller in the training loop as a *teacher*. Many teacher designs exist, but no controlled experiment compares them: published evidence comes from different learners, tasks, and tuning budgets. We provide that experiment. On a nuclear-microreactor load-following benchmark, one on-policy learner (PPO) is trained under six guidance mechanisms and an unguided control, with a one-step nonlinear model-predictive controller (NMPC) as teacher and classical baseline. Every configuration receives its own hyperparameter search, then twenty evaluation seeds disjoint from the tuning seed, with best-checkpoint selection. Peak accuracy does not separate the methods; reliability separates them sharply, in an order given by one design property: how much of the teacher's behavior enters the learner's training data. Mechanisms that inject teacher actions into the on-policy gradient succeed on 0/20 (action blending) and 6/20 (warmstart) seeds, at or below the unguided control's 8/20, while mechanisms that confine the teacher to shaping states, rewards, or a base action succeed on 19/20 to 20/20 ($p \le 4.5\times10^{-4}$, Fisher exact, Holm-corrected). Guidance reduces the training-time constraint-violation rate by up to $17\times$ and eliminates overpower events; a residual configuration trains to teacher-level accuracy with *zero* violations in 200,600 episodes. Deployment-robustness ablations complete the accounting: under plant–model mismatch the learned policies degrade gracefully while the nominal-model teacher incurs errors one to two orders of magnitude larger; under sensor noise the relationship reverses; under process noise the two are comparable. Resampling our own seeds shows a 3-seed version of this study could have reported any unguided success rate from 0% to 100%, so small-seed comparisons of guidance methods are uninformative.

Analytic Moment Matching enables multistep uncertainty propagation in Gaussian-Process Model Predictive Control, but requires specialised derivations. We study an uncertainty-aware RBF kernel over Gaussian-distributed inputs as a practical alternative. For deterministic training inputs, we show that the resulting GP predictor has exactly the same posterior mean and marginal-likelihood objective as analytic Moment Matching. On Pendulum-v1 our method shows comparable performance with an $8\times$ wall-clock speedup and an $11\times$ lower memory footprint.

Multi-gene intervention planning is critical to achieve a desired final cell state from either an embryonic stem or an intermediary cell state, while simultaneously modeling epistatic gene interactions. Current perturbation foundation models are not reliably trained to predict multi-gene perturbations and are conditional-mean estimators rather than intervention operators, so composing perturbations under a reinforcement learning (RL) policy queries the model off-distribution. Here, we introduce a generative structural causal world model in which transcription factor (TF) perturbations are modeled as latent interventions with support for multi-perturbations where any k-factor combination is solved in closed form rather than rolled out. On held-out TF overexpressions, the model matches or outperforms baselines on both single and combination perturbations, with cocktails of up to eight TFs, and predicts held-out gene-set epistasis. Epistatic interaction is explicitly learned through a low-rank interaction head trained on multi-TF perturbations by injecting set-dependent latent shifts through the causal graph, allowing the model to predict non-additive effects in held-out, experimentally validated TF combinations. Over this pre-trained world model, we built a policy to design perturbation cocktails that steer human embryonic stem cells (hESCs) to downstream differentiated fates including hematopoietic stem and progenitor cells (HSPC). We implement a goal-conditioned GFlowNet policy that proposes TF cocktails scored by the frozen world model's do-operator under an ensemble-pessimistic reward, penalizing cocktails whose value depends on simulator error. The policy generalizes to held-out fate signatures and recovers the driver TFs of a human embryonic stem cell to HSPC screen alongside other human fetal atlas annotated fates, matching model-predictive-control (MPC) search on broad recovery and exceeding it on top-rank metrics, while searching at roughly 36-fold lower inference cost and proposing far more diverse cocktails. Furthermore, since both the gene-sets that differentiate to the HSPC fate and the fate itself were held out in the world model and the policy, respectively, its zero-shot proposals for HSPC surface more validated answer-key drivers than the search and heuristic baselines at matched budget, nominating canonical blood-lineage regulators including LYL1, KLF1, and KLF2. Together, our work provides a framework for an exact, compositional causal world model allowing the implementation of combinatorial cell-fate design as an amortized, highly efficient policy.

Chain-of-thought reasoning has enabled language models to solve complex problems through explicit intermediate steps. We ask whether this paradigm can be transferred from language into chemical space. We introduce \emph{Chain-of-Molecules} (CoM), a molecular-generation paradigm in which the intermediate reasoning states are molecules and the transitions between them are explicit chemical edits. In Agentic CoM, a symbolic environment executes valid edits, rejects infeasible ones, and returns the resulting molecule, allowing the model to adapt its subsequent actions, explore alternatives, backtrack, and commit. We evaluate CoM on constrained molecular optimization across five property objectives, requiring generated molecules to attain a requested property value while remaining within a specified graph-edit-distance interval. A controlled comparison of four generation formats isolates the effects of representing intermediate molecular states and interleaving generation with environment execution. Each format follows the same three-stage training pipeline: pretraining teaches molecular syntax and search, supervised finetuning grounds the true property objectives, and GRPO directly optimizes property and structural success. GRPO increases in-distribution joint success by a factor of $4.2$--$4.4$ across all formats. Agentic CoM combines the property success enabled by intermediate molecular states with the structural control provided by environment execution, reaching joint success rates of $0.547$ in distribution and $0.573$ on held-out interpolation targets while producing the most structurally diverse successful solutions across all three target regimes.

Artificial intelligence systems are increasingly proposed for treatment decisions in the intensive care unit (ICU), raising the question of how such systems should be evaluated.Treatment decisions by clinicians are assessed in randomized trials and large prospective studies. Decisions produced by learned policies are instead evaluated retrospectively, either by concordance with human experts or by off-policy evaluation (OPE). We therefore test whether \OPE{} supports a stable, credible, and clinically sensible choice among policies for red-blood-cell transfusion in the ICU.We treat clinical decision support as a reinforcement learning policy learned from offline data, and we score 21 learned, guideline-based, and planted control policies with six value estimators on two large databases under a fixed evaluation protocol.Our results show that the estimators do not agree. Every setting produces between three and six different winners, the median policy moves 8 to 12 rank positions across estimators, and a planted control takes the first rank 14 times out of 144. The strongest agreement occurs among importance-weighted estimators, but it is not independent evidence, since all four reuse the same likelihood ratios. Among the policies repeatedly ranked among the best, the database-specific median effective sample sizes are in the single digits despite thousands of held-out stays. Changing only the behaviour model changes the selected policy in 26 of 32 estimator--setting comparisons. Highly ranked policies can still recommend transfusion at normal hemoglobin levels. Retrospective evaluation therefore does not substantiate a claim that an offline policy improves on recorded clinician care.

Scientific machine learning reports predictive performance. It does not report whether the same prediction would survive a different draw of training data. Across 9 chemistry benchmarks, two classifiers trained on independent bootstraps of the same training set agree on aggregate accuracy to within 1.3–4.2 percentage points but disagree on the class label of 8.0–21.8% of test molecules. We call this gap cross-sample prediction churn. The standard parameter-side techniques (deep ensembles, MC dropout, stochastic weight averaging) do not reduce this gap; two data-side methods do. The first is K-bootstrap bagging, which cuts the rate 40–54% on every dataset at no accuracy cost (K×-ERM compute). The second is twin bootstrap, our proposal: two networks trained jointly on independent bootstraps with a sym-KL consistency loss between their predictions, which at matched 2×-ERM compute reduces churn a further median 45% beyond bagging-K=2. Cross-sample prediction churn deserves a column alongside predictive performance in scientific-ML benchmark reports, because without it the parameter-side and data-side methods are indistinguishable on the metric they actually differ on.

We introduce Preference-Graph Policy Improvement (PGPI), a method forlearning biological sequence edit policies directly from noisy localcomparisons. PGPI uses per-variant measurement uncertainties to constructa confidence-gated graph of local improvements, converts its edges intoweighted state--action transitions, and trains an iterative edit policy byconfidence-weighted behaviour cloning, without fitting an absolute scalarreward. The construction is motivated by two properties: localdifferencing is insensitive to smoothly varying absolute bias, and acalibrated confidence gate bounds the expected fraction of misdirectededges, subject to a graph-coverage trade-off.Across seven measured fitness landscapes and four noise models, PGPImatches or improves upon direct surrogate ascent most clearly in long,sparse, and noisy regimes, but direct surrogate ascent remains stronger on short, densely sampled landscapes.A controlled corruption study shows that robustness comes from learningover graph-supported transitions rather than from reward-freeness alone;component ablations identify the hard confidence gate and transitionrepresentation as the main source of the effect.

In some multi-agent systems, the quantity to be optimized is not an externally specified reward but the information acquired about unknown properties of the environment as done in active sequential hypothesis testing (ASHT) problems. However, the ASHT literature tends to focus on finite single-agent problems with well-specified models, while there is currently a gap for practical multi-agent methods that can perform active sequential testing. We fill this gap with ICMAPE, a Bayesian learning-based framework for decentralized multi-agent pure-exploration driven by inference objectives. ICMAPE converts the fixed-confidence identification objective into a reward derived from inference confidence, so that standard reinforcement learning machinery can be applied to decentralized pure exploration. It jointly learns a centralized neural inference network that estimates a posterior distribution over hypotheses from global trajectory data, and decentralized policies that select actions from local observation histories and learn when to stop collecting data once the target confidence is reached.On two synthetic benchmarks and a Maryland nitrate concentration monitoring task based on real-world data, ICMAPE-TD3 achieves target accuracy with fewer exploration steps.

Personalizing drug dosing is a problem of sample-efficient optimization: candidate regimens are tried one at a time on a patient, and evaluations are expensive and potentially risky. Here, we use a pharmacokinetic/pharmacodynamic (PK/PD) model of thedrug, a mechanistic model calibrated on population data with quantified parameter uncertainty, to reparameterize thissearch. A dosing network is trained offline against the model, amortizing the map from patient PK/PD parameters to the regimen predicted to be optimal, and online Bayesian optimization then searches over this learned, low-dimensional representation rather than over raw dosages. Because the network's image is a subspace of pharmacologically coherent regimens, the search remains sensible by construction, improving safety and convergence speed. Unlike model-informed precision dosing, ourmethod admits heterogeneous feedback (symptom scores, preferences, wearables), and applies toany drugs for chronic disease with a PK/PD model where no real-time biomarkers can be measured that provide a comprehensive measure of patients’ state, and disease management implies taking into account patient reported outcomes. On an in-silico population of Parkinson's patients under Levodopa treatment, itconverges to better regimens with fewer trials than baselines optimization methods.

In reinforcement learning (RL) for healthcare, reward functions often encode clinical endpoints like survival and death. This results in a sparse reward structure with non-zero rewards only at terminal transitions. However, the exact reward magnitude and combination of survival rewards and death penalties vary across studies, with the implicit assumption that these choices are interchangeable. In this work, we theoretically and empirically examine three common sparse reward designs: survival-only, death-only, and mixed. We prove that, under the assumptions of terminal-only rewards, guaranteed absorption, and no discounting, the corresponding value functions of the three designs have an equivalence relationship and lead to the same optimal policy; while breaking any of these conditions can break the equivalence in consequential ways. On simulated environments, we verify the theoretical results and demonstrate how relaxing these assumptions affect the equivalence relationship. Finally, we consider a healthcare domain of sepsis treatment based on real-world clinical data, in which all theoretical assumptions are violated. Overall, we find that the death-only reward design consistently underperforms and leads to indecisive policies, while survival-only and mixed rewards perform similarly with differences arising from how they handle trap states and interact with intermediate rewards and discounting. Our work provides the first systematic characterization of these failure modes in healthcare RL, offering theoretically grounded guidance for practitioners on reward design choices.