Accelerating Small-Molecule Discovery with Reinforcement Learning and Evolutionary Search

Developing a new drug is notoriously difficult. On average, it takes 12 years and billions of dollars to bring a single compound from the lab to the clinic. Most candidates fail. With success rates below 0.01%, the odds are discouraging—not just for pharmaceutical companies but for patients waiting on new treatments.

In response, researchers have increasingly turned to generative AI models to accelerate early-stage discovery. These models can propose entirely new molecular structures based on learned chemical patterns, helping scientists move beyond traditional libraries and heuristic design. But despite their promise, many generative approaches either struggle with invalid outputs or fail to meaningfully optimize key pharmacological properties—especially when those properties depend on complex, non-differentiable simulations like molecular docking.

That’s where Quantiphi’s EPOSMol framework comes in, developed by our Philabs research team.

Why Drug Discovery Needs a Smarter Search Engine

Finding a viable drug starts with identifying molecules that bind to a target protein. But the universe of possible small molecules is staggering—estimates range from 10²³ to over 10⁶⁰. Traditional computational tools can’t search this space efficiently. Even state-of-the-art generative models, like variational autoencoders (VAEs) or graph neural networks (GNNs), have limitations.

Most fall into one of two traps:

They generate molecules that are invalid or redundant
They fail to optimize effectively for key properties like binding affinity

Gradient-based optimization works well when a property (e.g., lipophilicity) is easy to predict. But it struggles when that property—like docking score—is determined by a black-box simulation. Gradient-free methods like genetic algorithms are better suited for these cases but are often slow or get stuck in narrow parts of the search space.

EPOSMol was designed to solve these problems by combining the best of both worlds.

What Is EPOSMol?

EPOSMol (Evolutionary Policy Optimization for Small Molecules) is an evolutionary policy gradient approach that searches latent space—the compressed, learnable space of molecular representations—in a more intelligent, adaptive way.

Here’s how it works:

Learn from billions of molecules
We train a VAE to encode molecules into a latent space that captures their core chemical features. The decoder allows us to sample and reconstruct novel molecules that are likely to be chemically valid.
Score with a compound reward
EPOSMol scores each molecule using a flexible reward function that can combine any set of properties—such as binding affinity, drug-likeness, scaffold diversity, or others depending on the research goal. Unlike simpler models that rely on a single metric and often “reward hack” by generating the same molecules repeatedly, EPOSMol balances multiple objectives to keep the search adaptive and diverse. This ensures optimized candidates are not only valid and effective but also meaningfully novel.
Search with policy gradients
Rather than guessing randomly, EPOSMol uses policy gradient reinforcement learning. It samples a population of latent vectors, evaluates each one’s reward, and adjusts the policy toward better-performing regions. It’s like a GPS that learns the fastest route based on feedback from past trips.
Adapt the search over time
To avoid local optima, EPOSMol dynamically adjusts its strategy—using large, noisy populations early for exploration, then narrowing in on promising regions later. It also restarts when progress stalls and runs multiple populations in parallel to ensure broad coverage of chemical space.

What Did We Achieve?

Metric	EPOSMol	REINVENT 4.0 (Baseline)
Valid Hit Rate (BA ≤ -8.0 & QED ≥ 0.7)	18.59%	1.85%
Docking Score (Best)	-12.1 kcal/mol	-10.1 kcal/mol
Unique Hits (Extended Run)	4,137	119
Unique Scaffolds	570+	Lower, not specified

10× Higher Hit Rate – Compared to leading models like REINVENT, EPOSMol delivered up to 10× more hits—molecules with docking scores ≤ -8.0 and QED ≥ 0.7.

Stronger Binding Affinity – Top candidates showed binding affinities as low as -12.1 kcal/mol, significantly outperforming baseline methods.

Greater Chemical Diversity – Over 570 unique scaffolds were identified in optimized runs, meaning EPOSMol doesn’t just overfit to one type of molecule—it explores widely and creatively.

Broad and Fringe Exploration – In extended tests, EPOSMol generated more than 36,000 novel compounds, with over 11,800 high-affinity hits—even when drug-likeness constraints were relaxed. This allows for frontier exploration beyond typical design heuristics.

How to Understand the Magic (Without a PhD)

If some of this sounds technical, here are a few analogies:

Latent space is like a 3D map of molecule ideas. We explore this map instead of starting from scratch.
Policy gradients are like adjusting your aim after each shot at a target, learning where to aim better next time.
Evolutionary search mimics natural selection—testing, scoring, and refining the best molecules over generations.
Dynamic scheduling means casting a wide net early and focusing narrowly once the fish start biting.

Why It Matters

While many generative methods promise novelty, EPOSMol delivers optimized novelty—compounds that are not just different, but better aligned with target-specific objectives.

Its modular design means:

It can integrate with different docking engines or generative backbones
It balances exploration and exploitation with minimal hyperparameter tuning
It supports dynamic, scalable optimization for a wide range of drug targets

In short, EPOSMol isn’t just pushing molecules out of a black box. It’s enabling targeted, intelligent exploration that empowers scientists—not replacing them.

Let’s Redefine What’s Possible in R&D

At Quantiphi, we specialize in applying AI to real-world life sciences challenges—from drug discovery and preclinical testing to pharmacovigilance and manufacturing.

With 2,500+ AI projects delivered and 300+ HCLS experts, we’re ready to help your team turn complex research into actionable breakthroughs.

Read the full report or Contact us to explore how our EPOSMol framework for intelligent small-molecule optimization—and other AI-first innovations—can accelerate your R&D.