Enhancing Medical Disease Diagnosis Agents with Advanced Reasoning and Reinforcement Learning
Generative AI advancements are revolutionizing medical diagnostics, presenting exciting opportunities to enhance clinical decision-making and patient care. Intelligent LLM-based agents show promise in medical diagnosis assistance by improving accuracy, consistency, and speed in clinical assessments, and the integration of advanced reasoning and reward modeling techniques into the backbone LLM has the potential to further elevate their effectiveness.
In this blog, we show how to improve the backbone LLM of a medical disease diagnosis agent using outcome and process-based reward tuning—particularly optimizing model behavior by comparing relative rewards among generated responses—and how it significantly augments the traditional LoRA-based Supervised Fine-Tuning (SFT). While outcome-based reward focuses on the final result, process-level reward encourages improvements in the reasoning steps leading to that outcome, ensuring both accuracy and the reasoning quality behind decisions. We also infuse diverse reasoning strategies inspired from human-like diagnostic thinking into the agent and evaluate their impact. For our study, we employ Qwen-2.5-7b-Instruct on the MIMIC III disease diagnostic dataset. We share key insights from our experiments showcasing how different reasoning approaches, when paired with the two fine-tuning methods, can improve the diagnostic accuracy of the models behind the medical agents.
Fine-Tuning for Incorporating Medical Diagnostic Expertise
Fine-tuning is crucial for adapting pre-trained language models to the specific demands of medical disease diagnosis tasks. In our experiments, we explored two key fine-tuning approaches for our diagnostic agents:
LoRA-Based Supervised Fine-Tuning (SFT): LoRA (Low-Rank Adaptation) offers a computationally efficient method for fine-tuning large language models by introducing low-rank matrices to the model’s layers. By freezing the base parameters and adding only a delta for the new task, it plays a critical role in injecting task-specific knowledge into pre-trained models, enabling them to adapt to new domains and objectives without full model retraining. However, SFT often encourages pattern memorization over flexible reasoning, limiting its ability to generalize. To address this, Reinforcement Learning (RL) techniques have emerged as a complementary approach—enabling models to optimize for downstream outcomes and encouraging more adaptable, reasoning-driven behavior.
GRPO-Based Reward Tuning: Group Relative Policy Optimization (GRPO) enhances reasoning by optimizing the model’s policy through group-relative advantage, comparing sampled responses for a given context. This method eliminates the need for a separate critic model, reducing computational costs and improving reasoning efficiency. We use two reward mechanisms for tuning our model.
Outcome-Based Reward Tuning: GRPO leverages RL principles by applying outcome-based rewards to fine-tune the diagnostic agent. This method directly aligns the agent’s reasoning process with the desired diagnostic outcomes, leading to more accurate and robust decision-making.
Process-Based Reward Tuning: In addition to outcome-based rewards, we incorporate process-level rewards at intermediate steps of the diagnostic workflow—such as test recommendations or clinical investigations—to better guide the agent’s reasoning. These intermediate rewards are derived from ‘silver label’ clinical-test recommendations from the expert-validated suggestions generated by high-performing reference agents, serving as proxies for clinically sound decision-making. By aligning the model’s behavior with these reference pathways, the agent is encouraged to follow structured diagnostic reasoning, prioritize relevant investigations, and build more effective diagnostic trajectories that ultimately lead to improved final predictions.
Why SFT Remains Essential Before Reinforcement Learning: To better understand the role of the two fine-tuning paradigms, we consider the effects of SFT alone, RL alone, and their combination. While outcome- and process-based reward tuning can significantly improve diagnostic reasoning, relying solely on RL remains computationally demanding and less practical for most real-world use cases. Although DeepSeek R1-Zero is an example that reasoning in models can be learned from scratch (starting with a base non-reasoning model like DeepSeek-v3) using RL alone, the associated resource requirements and complexity make this approach inefficient as a standalone strategy. Additionally, we also echo similar thoughts as the recent finding (April 2025) that the reasoning paths preferred by RL-trained models are already present in the base model’s sampling distribution. This implies that in the shorter term, RL primarily explores and ‘reweights’ responses toward reward-aligned reasoning, rather than instilling fundamentally new capabilities. Therefore, SFT remains a foundational step for encoding task-specific knowledge, with RL serving as a targeted refinement mechanism to steer the model toward more reliable and effective diagnostic reasoning.
Leveraging Reasoning Strategies for Robust Medical Disease Diagnosis
Now that we’ve discussed the role of fine-tuning—both supervised and reward-based—the next key aspect in enhancing medical diagnosis agents is shaping how they reason. In this section, we explore reasoning strategies that help guide models toward more accurate and clinically meaningful diagnostic outcomes.
Standard Reasoning with Self-Correction
Modern reasoning models, such as Qwen QwQ-32B, Deepseek-R1, Gemini 2.5, Claude 3.7 Sonnet and OpenAI’s o3, have increasingly adopted a standard reasoning paradigm that includes self-critiquing capabilities, which is highly relevant for medical disease diagnosis agents. In this paradigm, the diagnostic process for an agent unfolds in a structured manner:
- Generates an Initial Diagnostic Hypothesis: Initially, the agent processes the patient’s clinical information and produces a preliminary diagnosis along with a chain of thought outlining its reasoning.
- Self-Evaluates the Diagnostic Reasoning: The agent then reviews its intermediate steps, identifying any inconsistencies, errors in logic, or areas where further justification is needed.
- Refines the Final Diagnosis: Based on the feedback from its self-assessment, the agent revises its initial hypothesis, leading to a more accurate and well-supported final diagnosis.
This approach of refining solutions and hypotheses, now a key aspect of advanced reasoning in AI, enhances both the transparency and performance of medical disease diagnosis agents by enabling error correction and ensuring that the final diagnosis is well-justified by the evidence.
Incorporating Diverse Reasoning Strategies
Effective medical practice hinges on sound clinical reasoning—the cognitive processes clinicians employ to evaluate patient information, formulate hypotheses, and arrive at a diagnosis. To emulate these crucial processes, medical disease diagnosis agents are increasingly incorporating diverse reasoning strategies, including:
- Deductive Reasoning: Deriving specific diagnostic conclusions from established medical principles.
- Inductive Reasoning: Forming broader diagnostic generalizations based on specific patient observations.
- Decompositional Reasoning: Breaking down complex diagnostic problems into simpler, more manageable sub-problems to facilitate analysis.
- Abductive Reasoning: Inferring the most likely disease explanation for a given set of patient symptoms and findings.
The goal of integrating these reasoning strategies into AI agents is to mirror human-like diagnostic thinking, thereby enhancing the reliability and accuracy of AI-assisted medical decisions.
The Power of Composite Reasoning
One of our active research works is on implementing a composite reasoning strategy specifically designed for medical disease diagnosis agents. This strategy extends beyond simply choosing one reasoning mode; it dynamically switches among deductive, inductive, decompositional, and abductive reasoning as required throughout the diagnostic process. Integrating multiple reasoning strategies is crucial in medical diagnosis, as it enables the model to adapt to the complexity and variability of medical cases, enhancing diagnostic accuracy and robustness. Crucially, it also incorporates a self-critique loop. This self-critique mechanism allows the agent to reflect on its intermediate reasoning steps, identify potential errors or inconsistencies, and iteratively refine its diagnostic conclusions. By enabling the agent to select the most appropriate reasoning mode for a given clinical context—and then critically evaluate its own reasoning—the composite strategy aims to achieve more robust and adaptable diagnostic performance.
Example Thought of Each Reasoning Strategy:
Example: Chain of Thought
Example: Standard Reasoning
Example: Composite Reasoning
Results and Insights
Dataset for Training and Evaluation
Our experiments were conducted using the MIMIC III disease diagnostic dataset—a widely recognized benchmark in medical AI research. This dataset provides disease diagnoses for patients but lacks explicit recommendations for diagnostic tools or tests. To enrich the training data for our medical disease diagnosis agent, we employed the MIMIC agent with GPT-4o as the LLM. This agent generated potential tool and test recommendations for each patient. Then, we filtered these recommendations, retaining only those that aligned with the correct disease predictions. These verified recommendations were treated as ‘silver labels’ providing valuable additional guidance for fine-tuning the diagnostic model and enhancing its ability to reason about the diagnostic process. This approach allows us to incorporate both the provided disease labels and the additional expert-validated recommendations, thereby strengthening the training process and ultimately improving the diagnostic performance of the agents. Utilizing this process, in total 1500 unique patient samples were created. Of these, 1000 were used for training — 750 samples were utilized for LoRA finetuning, while 250 samples were used for GRPO based reward modelling. The remaining 500 samples were used as the hold-out test set.
Experimental Findings
We conducted a series of experiments using the Qwen-2.5-7b-Instruct model with different reasoning strategies and fine-tuning techniques, specifically evaluating their impact on medical disease diagnosis. The experiments assessed performance across two key metrics:
- Tests & Diagnosis Accuracy: (written as Tests shorthand in the table) Evaluating the model’s ability to interpret clinical tests (simulated through the silver labels) and arrive at a correct diagnosis.
- Final Diagnosis Accuracy: (written as Final Diagnosis shorthand in the table) Focusing solely on the correctness of the final disease diagnosis provided by the agent.
How Effective Are The Fine-Tuning Approaches?
Table 1 compares the inference scores on the base model versus a fine-tuned variant with the Chain of Thought (CoT) Reasoning. The delta change with GRPO is calculated as a percentage of (LoRA+GRPO – LoRA)/LoRA accuracy.
| Disease Diagnosis Accuracy | ||||||
| Tests | Tests (after GRPO reward tuning) | Delta change in Tests (with GRPO) | Final Diagnosis | Final Diagnosis (after GRPO reward tuning) | Delta change in Final Diagnosis (with GRPO) | |
| Base Model | 22.5% | 30.5% | 35.55% | 19.7% | 44% | 123.35% |
| Fine-tuned Model | 30.5% | 39% | 27.86% | 39.5% | 52% | 31.64% |
| Delta Improvement from fine-tuning | 35.55% | 27.86% | — | 100.5% | 18.18% | — |
Table 1: Comparison of Base vs Fine-tuned Qwen2.5-7B-Instruct model on Chain of Thought (CoT) Reasoning
We observe that in all cases, as expected, fine-tuning outperforms the base model. Interestingly, direct reward modelling through RL is beneficial for the base model, with a (35.55%, 123.25%) improvement over the baseline for tests and diagnosis, but we believe the reason for the spike in improvement is due to the base model already having prior knowledge of the CoT pattern on generic domains. In terms of absolute value, fine-tuning with LoRA first followed by reward modelling gives better accuracy (52%) for the final diagnosis, and there is a 2x (~100%) improvement already observed on the fine-tuned model after LoRA.
How Effectively Does Each Reasoning Strategy Perform?
Next, we evaluate our medical disease diagnosis agent on different reasoning strategies after being (i) finetuned on LoRA on the specific reasoning strategy, and (ii) after GRPO-based reward tuning, observations in Table 2. The delta change with GRPO is calculated as a percentage of (LoRA+GRPO – LoRA)/LoRA accuracy. A detailed analysis is provided under the Discussion section.
| Reasoning Types | Disease Diagnosis Accuracy | |||||
|---|---|---|---|---|---|---|
| Tests (LoRA) | Tests (LoRA + GRPO) | Delta change in Tests (with GRPO) | Final Diagnosis (LoRA) | Final Diagnosis (LoRA + GRPO) | Delta change in Final Diagnosis (with GRPO) | |
| Generating Direct Output | 18% | 25.5% | 41.67% | 22% | 29% | 31.81% |
| Chain of Thought (CoT)Reasoning | 30.5% | 39% | 27.86% | 39.5% | 52% | 31.64% |
| Standard Reasoning | 26% | 33% | 26.92% | 39.5% | 39.5% | ~0% |
| Composite Reasoning | 26% | 27% | 0.38% | 26.7% | 37.2% | 39.32% |
Table 2: Evaluation of Different Reasoning Types on Fine-tuned Models
Where Do We Stand Compared to Other Medical Reasoning Models?
Finally, in Table 3, we assess how our model—fine-tuned specifically for disease diagnosis, without explicitly adapting to the general medical domain—performs in comparison to other models explicitly trained for medical reasoning using large volumes of diverse domain-specific data.
| Model | Disease Diagnosis Accuracy | |
|---|---|---|
| Tests | Final Diagnosis | |
| HuatuoGPT-o1-7B | 31% | 18.39% |
| FineMedLM-o1-8B | 6.5% | 34.48% |
| JSL-MedLlama-3-8B-v2.0 | 7.5% | 39.1% |
| Qwen2.5-7B Instruct (Base) | 22.5% | 19.7% |
| Direct Output (LoRA + GRPO) | 25.5% | 29% |
| CoT Reasoning (LoRA + GRPO) | 39% | 52% |
| Standard Reasoning (LoRA + GRPO) | 33% | 39.5% |
| Composite Reasoning (LoRA + GRPO) | 27% | 37.2% |
Table 3: Comparison against other publicly available medical-reasoning models
We observe that our models fine-tuned specifically on just 1,000 task-specific samples—for test recommendation and disease diagnosis—outperform domain-adapted medical models applied out-of-the-box, and this is consistent across different reasoning strategies. This highlights the continued importance of task-specific data, even when extensive domain adaptation is present. Notably, HuatuoGPT-o1 demonstrated a strong preference for recommending tests over providing final diagnoses. This behavior persisted even though the prompt was explicitly structured to choose between recommending an action or providing a conclusive diagnosis. The model’s inclination suggests an over-reliance on reasoning, potentially at the expense of definitive diagnostic conclusions. On the contrary, FineMedLM-o1 and JSL-MedLLama-3 focused on providing diagnosis directly in most of the cases bypassing the clinical tests. This tendency may stem from the nature of the underlying datasets, which are primarily centered around medical question answering—where direct diagnoses are common, but references to clinical tests are relatively rare. While these models are applied as-is for this study, fine-tuning them on our disease diagnosis dataset could yield improved performance—an avenue reserved for future work.
Discussion
-
Reasoning Style Influences the Fine-Tuning Efficacy
Our experiments with LoRA and GRPO reward tuning across various reasoning styles for disease diagnosis yield the following insights into how these methods effectively integrate reasoning strategies into a base non-reasoning model, enabling it to serve as a strong backbone LLM for a disease diagnosis agent:
-
Direct Output: Fast Answers, Missed Longer Term Benefits?: Direct output strategies, which lack an explicit reasoning trace, show moderate improvements with GRPO. However, they may miss out on the more substantial benefits of reward tuning that are observed with reasoning strategies that provide detailed intermediate steps.
-
Chain-of-Thought Shines with Outcome-Aware Tuning: Detailed, step-by-step reasoning in diagnostic agents benefits substantially from outcome-based tuning, suggesting that the richness of the reasoning trace provides more granular reward signals for refining the diagnostic process and the final diagnosis.
-
Standard Reasoning with Self-Critique – Stable, but Hits a Ceiling: The relatively stable final diagnosis accuracy suggests that the inherent self-critiquing mechanisms in standard reasoning models already provide a robust framework for accurate diagnosis. GRPO tuning further enhances the interpretation of intermediate diagnostic information but seems to reach a performance ceiling for the final diagnostic outcome.
-
Composite Reasoning with Self-Critique – Untapped Potential with Dynamic Thought Paths: Composite reasoning holds promise for medical disease diagnosis through dynamic switching, adaptive thinking, and self-assessment. However, its complexity may require further refinement and specialized optimization beyond standard GRPO to fully realize its benefits. While its current performance lags behind more familiar styles like CoT, this can be attributed to the novelty of the reasoning paradigm itself. Unlike CoT, which aligns with patterns already present in the model’s pretraining, composite reasoning introduces a fundamentally new mode of thought. As such, it likely requires additional finetuning on targeted data to internalize this reasoning approach. Once learned, reward-based tuning can help the model explore diagnostic trajectories more dynamically than CoT. Even with this undertrained version of the model, reward tuning on the composite reasoning style is able to improve upon the LoRA-tuned performance much more ((37.2-26.7)*100/26.7 = 39.32%) than other LoRA-tuned models (CoT: (52-39.5)*100/39.5 = 31.64%; SR: 0% ), highlighting the potential of Composite Reasoning when coupled with reward tuning.
-
-
GRPO Unlocks Major Gains Across Reasoning Styles
Outcome-based reward tuning (GRPO) consistently yields substantial improvements in the performance of medical disease diagnosis across various reasoning strategies. This is particularly evident with the Chain-of-Thought approach, where final diagnosis accuracy nearly doubles.
-
Process Matters: Silver-Labeled Rewards Drive Diagnostic Depth
Incorporating process-level rewards based on silver-label test recommendations provides an additional layer of guidance, encouraging the model to learn effective diagnostic processes and consider relevant investigations, which ultimately contributes to improved accuracy.
-
Explicit Reasoning Supercharges Reward-Based Learning
Reasoning strategies that produce explicit intermediate steps (like Chain-of-Thought) appear to benefit the most from both outcome-based and process-level reward tuning. This suggests that providing the model with a structured thought process allows for more targeted and effective learning signals.
-
Implications for Developing Advanced Diagnostic Agents
Our findings underscore the importance of leveraging both outcome-based and process-level rewards in conjunction with structured reasoning strategies like Chain-of-Thought for developing advanced medical disease diagnosis agents. This combined approach can lead to more accurate, reliable, and clinically relevant diagnostic support.
Conclusion
Our study showcases the significant potential of integrating advanced reasoning strategies with reinforcement learning techniques, specifically outcome-based and process-level reward tuning, for enhancing medical disease diagnosis agents. The results demonstrate that reinforcement learning-based reward tuning, especially when combined with detailed, step-by-step reasoning, consistently outperforms traditional supervised fine-tuning. The incorporation of process-level rewards, derived from expert-validated test recommendations, provides an additional layer of guidance that encourages the development of more effective diagnostic pathways.
By learning to dynamically adapt and switch between reasoning strategies while incorporating self-critique, composite reasoning offers a powerful framework for tackling complex diagnostic challenges. Due to the complex nature of the composite reasoning it is observed that the thought often makes abrupt jumps between reasoning styles, which can even mislead the thought trajectory. While this definitely encourages more exploration, careful curation of rewards tailored for composite reasoning could be needed to encourage better exploitation along with exploration. Similarly, the inherent strengths of standard reasoning models can be amplified through targeted reward mechanisms like length-based rewards explored by multiple works.
We believe these findings contribute meaningful progress and pave the way toward the next generation of AI-driven medical disease diagnosis agents, supporting more accurate and reliable clinical decision support, ultimately leading to improved patient outcomes.
