In the fast-paced world of software development, the need for seamless translation across programming languages is more critical than ever. Code translation is central to ensuring interoperability between different languages and enhancing software adaptability. This process has long been a challenge, with traditional methods relying heavily on handcrafted transformation rules. While these manual rules have served their purpose, they are often rigid, time-consuming, and lack scalability, which makes them less effective in today’s dynamic development environments.

Recent advancements in natural language processing have paved the way for sophisticated language models that can understand and generate human-readable code. While open-source alternatives show promise, their translation performance frequently falls short compared to proprietary solutions. In contrast, proprietary closed-source API-based language models deliver superior performance but raise substantial concerns regarding data privacy, control, and dependency on third-party providers. These issues can be particularly concerning for businesses working with proprietary or sensitive codebases.

To address these challenges, we introduce Auto-Train for Code Translation (ACT), a novel framework designed to enhance code translation capabilities by enabling in-house fine-tuning of open-source LLMs.

Let’s dive deep into the methodology of the solution to understand how it works under the hood.

Methodology

The Auto-Train for Code Translation (ACT) framework consists of four primary components that work in tandem to automate and enhance the code translation process. These components include: (i) Data Generation, (ii) Fine-tuning, (iii) Evaluation, and (iv) Controller. Each component plays a crucial role in automating the process of generating synthetic code data, fine-tuning machine learning models, evaluating their performance, and managing iterative workflows. Below, we provide a detailed breakdown of how each component contributes to the effectiveness of the ACT framework.

Figure 1: Overview of the ACT framework

Data Generation

Typically, only a limited number of source-target code pairs are available as input, which is insufficient for effectively fine-tuning a model. To address this, a data generation pipeline is employed to create high-quality source-target code pairs. For example, starting with just 10 user-provided input seed samples, the pipeline expands them into 100–200 samples, providing a robust dataset to fine-tune the model effectively. This section outlines source dataset expansion, translating source to target code, generation of unit test cases and data validation.

Figure 2: Workflow of the Data Generation Process

Source Dataset Expansion

ACT starts by taking a “seed code sample” in the source language, along with a specified “diverse factor” and “use case requirements” that control the extent of data expansion. The framework captures a wide variety of coding styles and structures to enhance the robustness of the model. This diversity helps in building a robust training dataset, with the “diverse factor” (ranging from 1 to 10) controlling how broadly or deeply the data is expanded. A higher diverse factor indicates a stronger emphasis on breadth-based expansion over depth-based expansion and vice versa. Below are two key strategies employed here:

Figure 3: Breadth-based & Depth-based data expansion

Depth-Based Data Expansion: This adds complexity to the code by introducing or changing a (requirement/time-space complexity/logical flow), enabling the model to learn more intricate code patterns.

Constraints Prompt: E.g. “Add new constraints or requirements to the original code, adding approximately 10 additional words.”

Figure 4: Generated code sample using Constraints Prompt

Deepen Prompt: E.g. “Propose lower or higher time or space complexity requirements, but please refrain from doing so frequently.

Figure 5: Generated code sample using Deepen Prompt

Concretizing Prompt: E.g. “Replace a commonly used requirement in the programming task with a less common and more specific one.”

Figure 6: Generated code sample using Concretizing Prompt

Reasoning Prompt: E.g. “If the original code can be solved with only a few logical steps, please increase or decrease the reasoning steps.”

Figure 7: Generated code sample using Reasoning Prompt

Breadth-Based Data Expansion: Focuses on diversifying the generated code by changing the objectives of the initial sample (e.g., transforming a function that checks if a number is even to a function that checks for palindromes).

Figure 8: Generated code sample for Breadth-based Data Expansion

Translating to Target Language

Once the source code samples are generated, ACT proceeds with translating them into the target programming language. To handle large code files, the translation is done in smaller segments—such as classes, functions, or transformation steps.

Figure 9: Translating Code

Unit Test Case Generation

For every translated code sample, ACT generates unit test cases to ensure the functional correctness of the code. These tests are derived from the source code, focusing on function declarations to avoid biases from translation errors. Additionally, mutation testing is employed to create “mutants”—buggy versions of the code—to identify any gaps in the initial test cases. By running the unit tests against these mutants, ACT ensures that the test suite is robust and comprehensive.

Figure 10: Unit Test Case Generation

Figure 11: Mutation Testing

Figure 12: Updating Unit Test Cases

Validation of Synthetic Data

To ensure that the generated code works as expected, the synthetic target code undergoes validation in a controlled environment using Docker containers. This automated setup installs the necessary dependencies, runs the unit tests, and verifies that only fully functional code proceeds to the fine-tuning stage. If any test fails, the code is returned to the Data Generation phase for further expansion, ensuring high-quality, executable data for training.

Fine-tuning

Once high-quality synthetic data is generated, the next phase is fine-tuning. The goal is to adapt pre-trained models for the specific task of code translation. This process is driven by an “automatic parameter configuration” module that selects optimal fine-tuning strategies and training configurations based on the characteristics of the data and the specific model being fine-tuned. The distributed training pipeline leverages Deepspeed and Pytorch Lightning for faster and memory-efficient fine-tuning, incorporating techniques like Mixed-Precision Training, Quantized Training, Gradient Checkpointing and Accumulation, Flash Attention, and Deepspeed Zero strategies along with CPU offloading.

Automatic Parameter Configuration

ACT’s Automatic Parameter Configuration module autonomously selects the necessary training hyperparameters and configurations required for fine-tuning the model using an LLM. This automatic configuration occurs through two key steps:

Training Strategy Selection: Depending on the evaluation scores of the base model on the test data and the size of the training dataset, the framework determines which fine-tuning strategy to employ – Full Supervised Fine-tuning (SFT) or Low-Rank Adaptation (LoRA).
Training Configuration Selection: This step automatically adjusts various training parameters, such as the number of epochs, training and validation batch sizes per GPU, gradient accumulation size, Deepspeed strategy with/without CPU offloading, and learning rate. These decisions are based on factors such as dataset size, model size, and available GPU configuration.

The fine-tuning process follows an iterative approach, where evaluation occurs after each training stage to assess model performance. Based on these evaluation scores and the patterns in training and validation loss curves, decisions are made regarding whether to continue training, adjust parameters, generate additional data, or conclude the experiment with the best-performing model as the output. This ensures that users can fine-tune LLMs without needing any deep technical expertise.

Evaluation

ACT’s Evaluation phase ensures that the fine-tuned models are performing at their best. Evaluation is done after each iteration of fine-tuning, with performance metrics computed based on how well the translated code passes predefined unit test cases.

Execution-Level Evaluation

To quantify the model’s translation accuracy, ACT uses an execution-level evaluation method. The generated target code is executed within a secure Docker container against the generated unit test cases. This process computes pass@k values, providing a quantitative measure of model performance. Given a source code, the model generates the top k most probable outputs (i.e., translated code samples). The pass@k metric is defined as the probability that at least one of the k-generated outputs passes predefined test cases. This approach provides a rigorous evaluation of the code’s functional correctness and helps guide further fine-tuning decisions.

Controller – Orchestrating the Workflow

The Controller orchestrates the ACT framework, managing the interplay between Data Generation, Fine-tuning, and Evaluation across multiple stages. At each stage, it dynamically decides whether to: i) Continue finetuning the model if it consistently improves. ii) Generate additional synthetic data if the evaluation suggests that data diversity or quality is a limiting factor in model improvement. iii) Terminate the fine-tuning process if further iterations lead to diminishing returns. This iterative optimization process, powered by an LLM-based decision mechanism, optimizes each stage based on prior results, maximizing model performance while minimizing unnecessary training cycles, time, compute, and cost, leading to a more efficient fine-tuning pipeline.

A key innovation of the Controller is its ability to adaptively determine the optimal data size at each finetuning stage. Instead of generating large datasets upfront, which can lead to unnecessary computation, the Controller ensures that only the necessary volume of high-quality synthetic data is generated. At each stage, the Controller evaluates model performance based on the training and validation loss values and evaluation scores of the base model, finetuned model of the current stage, and previous stages to guide decisions.

Targeted Data Generation: Synthetic examples are iteratively crafted to address failed cases identified in prior stages, ensuring context-aware data augmentation.

Adaptive Finetuning: Hyperparameters — epochs, learning rate, batch size, and gradient accumulation steps are dynamically adjusted at every stage based on loss and evaluation trends. This adaptive approach prevents overfitting while accelerating convergence.

By leveraging data-driven decisions, LLM optimization, and historical tracking, the Controller enhances training efficiency, ensuring Auto-Train remains scalable and effective for large-scale code translation and generative AI tasks.

Solution Preview

Here’s a preview of the solution in action. The following demo video demonstrates key features, outputs, and performance to help you visualize its capabilities.

Demo Video: Preview of the Code Translation Solution

Experimental Setup

The study evaluates two open-source models, namely Qwen2.5-Coder-7B-Instruct (Hui et al., 2024) and DeepSeek-Coder-V2-Instruct (Zhu et al. 2024) on code translation tasks for Java→Go and C++→Rust to benchmark their performance before and after fine-tuning.

A seed dataset of 150 code pairs from HumanEval (Chen et al., 2021) was expanded using GPT-4o and Claude-3.5-Sonnet through breadth and depth augmentation strategies. From each seed, one breadth-expanded and four depth-expanded variants were created, yielding diverse training samples.

For quality control, 30 samples per task were manually reviewed by developers, with equal parts marked correct and incorrect by the data generation pipeline during the execution-based evaluation. Samples were scored from 0 to 2 based on translated code correctness and unit test coverage, where 0 indicates completely incorrect and unaligned, 1 represents partial alignment, and 2 signifies complete correctness and alignment.

Hyperparameters critical to finetuning, such as LoRA-specific parameters (r=16/32, α=64), epochs (ranging from 2 to 5), learning rates from (3e-5 to 1e-6), and batch sizes (1–4) were optimized by the automatic parameter configuration module with the model loaded in 8-bit by default. Training was conducted on a single 80GB A100 GPU, starting with 450 samples initially and adding 100 per stage, along with 50 samples for validation, over up to four controller-managed stages.

Task	Unit Test Case		Translated Code
Task	Correct	Incorrect	Correct	Incorrect
C++ → Rust	1.93	1.47	2.00	1.60
Java → Go	1.80	1.47	2.00	1.53

Table 1: Human evaluation scores reflect the evaluation of both translated code and unit test cases, with ratings (on a scale of 0-2) based on correctness and coverage.

Observations & Results

The human evaluation results, as shown in Table 1, validate the effectiveness of the data generation pipeline in producing high-quality training data. Correctly filtered samples received near-perfect scores for both translated code and unit test cases, indicating accurate translations and minimal loss in test coverage. In contrast, the incorrectly filtered samples scored lower due to syntax errors and logic flaws. This confirms that the data generation pipeline effectively filters, validates, and produces high-quality training data.

Models	Java → Go		C++→Rust
Models	pass@1	pass@5	pass@1	pass@5
Qwen2.5-Coder-7B-Instruct	0.4461	0.5904	0.4184	0.5646
ACT-Qwen2.5-Coder-7B-Instruct	0.4865	0.6642	0.3950	0.6192
DeepSeek-Coder-V2-Instruct	0.5348	0.6048	0.4500	0.5612
ACT-DeepSeek-Coder-V2-Instruct	0.6248	0.7019	0.5653	0.6328

Table 2: Execution-based metric comparing accuracy between base model & ACT fine-tuned model

The experimental results (Table 2) show that both ACT-Qwen2.5-Coder-7B-Instruct and ACT-DeepSeek-Coder-V2-Instruct outperform their base models in the Java → Go translation task, with DeepSeek demonstrating notable adaptability to fine-tuning. In the C++→Rust translation task, ACT-DeepSeek-Coder-V2-Instruct achieves substantial improvements, while ACT-Qwen2.5-Coder-7B-Instruct shows a slight decline (pass@1), hinting at a balance between precision and generalization challenges for Qwen-based models. However, the fine-tuned model still outperforms the base by a big margin when evaluated for pass@5, demonstrating the effectiveness of the iterative fine-tuning approach. A significant improvement in execution-based evaluation scores was observed at each stage of fine-tuning.

Figure 13: Execution-based evaluation scores (pass@1 and pass@5) of DeepSeek-Coder-V2-Instruct for Java→Go translation across finetuning stages, showcasing iterative improvements

The experiments highlight the effectiveness of the iterative finetuning approach, with significant improvements in execution-based evaluation scores observed at each finetuning stage. As depicted in Figure 20, Stage 1 saw DeepSeek-Coder-V2-Instruct generate 500 Java→Go samples from 150 seed samples and undergo fine-tuning for three epochs. In Stage 2, the controller determined that two additional epochs would yield further improvements, resulting in a modest increase in evaluation scores. In Stage 3, the controller generated 100 targeted samples from previous failures and fine-tuned the model for two more epochs, leading to a substantial performance boost. Improvement was consistently observed across all experiments. In another scenario, the controller terminated the process after two stages when no further gains were anticipated, underscoring its adaptive, performance-aware decision-making for optimal finetuning and model performance.

In conclusion, the controller-driven framework effectively enhances code translation tasks, by dynamically adjusting fine-tuning and generating targeted data, ensuring continuous model improvement.

Developer Acceleration in Real-Time Migration Projects (as observed within our Organization)

One of the key benefits of our framework is its ability to accelerate developer workflows by automating code translation between different programming languages. As shown in Table 3, our solution significantly reduces the effort required for real-time migration projects by handling complex transformations efficiently.

How Does It Help?

Automated Translation: Converts legacy code into modern frameworks, reducing manual rewriting.
Syntax & Logic Adaptation: Ensures that the translated code adheres to the best practices of the target language.
Faster Deployment: It speeds the transition from development to production by generating production-ready code.

Impact on Developer Productivity

Our framework has significantly expedited real-world migration projects, as demonstrated below. We conducted a few controlled experiments within our organization where two developer groups worked on the same migration tasks—one using manual translation and the other leveraging our ACT framework. The results showed that developers using ACT completed translations 35% faster on average, reducing migration time across various codebases.

Migration Use Case	Developer Acceleration with ACT
‘Pentaho kettle jobs’ → ‘PySpark’	35% faster
‘Talend job-scripts’ → ‘Airflow’	25% faster
‘Javascript’ → ‘Typescript’	50% faster

Table 3

These findings highlight how our framework streamlines migration, enabling faster deployment of production-ready code across diverse environments.

Conclusion

In summary, ACT is a game-changer in the field of code translation. By enabling the fine-tuning of open-source LLMs and offering an automated data generation and evaluation pipeline, ACT empowers organizations to achieve high-quality code translations, ensuring smooth migrations and software adaptability. It strikes a perfect balance between performance, data security, and user control, allowing for more efficient and reliable translation across a wide range of programming languages, all without the need for extensive datasets or complex hyperparameter tuning.