Optimizing AI with Fine-Tuned Open Source LLMs & SLMs: Boost Efficiency in Telecom using NVIDIA NeMo Microservices
The telecom industry is rapidly adopting Generative AI to enhance operations, from network management to customer service analytics. As organizations ramp up their AI investments, they’re also seeking ways to optimize high operational costs. Relying on costly API calls to large models for high-volume tasks like analyzing millions of customer queries can quickly become a significant expense. This economic need is driving a shift towards Open-source Large Language Models (LLMs) & Small Language Models (SLMs). Fine-tuning these more efficient models for specific telecom tasks offers a compelling path to retaining AI capabilities while drastically reducing inference costs and improving speed. This blog explores how telecom companies can leverage fine-tuned models to create cost-effective, efficient, and customized AI solutions that address their unique operational challenges without compromising on quality.
One of the leading U.S. telecom companies handles hundreds of thousands of customer interactions daily. To enhance their customer service operations, they leverage advanced proprietary models, including GPT-4 for retrieval-augmented generation (RAG)-based search, helping optimize critical contact center KPIs like Average Handling Time (AHT), Wait Time, and First Contact Resolution (FCR).
Despite the initial success, the client encountered operational limitations:
- High Latency: Prolonged response times adversely affected agent productivity and downstream customer experience.
- Escalating Costs: API usage associated with proprietary models significantly increased operational expenses.
In response, the client developed a platform to replace these production LLMs with more cost-effective, scalable, and low-latency models by fine-tuning open-source small language models. The goal was to reduce dependency on expensive LLM API calls while preserving the accuracy and relevance of responses.
Key Considerations
During a comprehensive technical engagement with Quantiphi, the following needs were identified:
- Standardized pipeline for fine-tuning SLMs
- Availability of high-quality datasets for training
- Industry-compliant evaluation framework
- Need for a performant and scalable model deployment platform
Lack of the above requirements created bottlenecks across the model development lifecycle—slowing experimentation, hindering model reliability, and ultimately extending time-to-market for production-grade AI applications.
The Solution: Fine-Tuning-as-a-Service Platform
To address these challenges, Quantiphi, in collaboration with the client, developed an end-to-end modular fine-tuning and deployment platform that automates the entire customization lifecycle using NVIDIA technologies:
NVIDIA NeMo Curator: Equipped functionalities to facilitate synthetic data generation and pre-processing data tailored to use cases.
- NV-Embed-v2: Provided robust embedding generation for downstream tasks.
- RAPIDS cuML: Accelerated dimensionality reduction and clustering for topic modeling.
- Llama 3.1 Nemotron 70B Instruct: Served as the core LLM-as-a-Judge model in data quality assessments.
NVIDIA NeMo Customizer: Enabled fine-tuning using LoRA and QLoRA methods via an API-driven interface.
NVIDIA NeMo Datastore: Served as a centralized repository for datasets and model artifacts.
NVIDIA NIM: Supported high-performance inference for real-time deployment.
NVIDIA NeMo Evaluator: Automated evaluation on custom datasets using both traditional and LLM-as-a-judge-based metrics.
- Llama 3.1 Nemotron 70B Instruct: Served as the core LLM-as-a-Judge model in data quality assessment.
This architecture enabled rapid onboarding of new use cases while significantly reducing both cost and latency.
Key Components of the Platform
-
NeMo Curator
The data curator serves as the foundational step in the fine-tuning process. In building Q&A RAG data generation pipeline, we utilized several core features found in the NeMo Curator framework:
Topic-Aware Question Generation
- Using NeMo Curator, we extracted key topics from document chunks to ensure domain coverage and semantic diversity.
- Further, we leveraged NV-Embed-V2 to generate contextual embeddings and used RAPIDS cuML for dimensionality reduction and clustering—ensuring high-quality topic grouping.
LLM-Based QA Creation
- Through LLM-driven prompt pipelines, we were able to generate high-quality, topic-informed questions and answers, supporting both open-ended and closed formats.
- This ensures relevance, naturalness, and diversity in the generated QA pairs.
Quality Control and Filtration
- Utilized NeMo Curator for advanced deduplication, including fuzzy and exact match filtering with GPU-accelerated performance.
- It supports semantic quality control using LLM-based evaluations to ensure responses meet high standards for relevance, groundedness, and fluency.
-
Fine-Tuning with NeMo Customizer
To fine-tune a variety of language models on our curated datasets, we utilized NeMo Customizer, which enabled streamlined model fine-tuning using parameter-efficient techniques such as LoRA and P-Tuning. The easy-to-use API-driven interface facilitated seamless integration with the training pipeline, enabling efficient training on NVIDIA GPU clusters.
Core Configuration Components:
The platform offered the following configuration components, building on NeMo Customizer’s capabilities:-
Base Model Selection: Selecting and configuring pre-trained models suited to our domain.
-
Dataset Integration using NeMo Datastore: Seamless integration of curated datasets using NeMo Datastore as a centralized repository for storing, organizing, and retrieving datasets for the development workflows.
-
Training Hyperparameters, including:
- Fine-Tuning Method
- Number of Epochs
- Batch Size
- Learning Rate
- Weight Decay
- Adapter Dropout
- Adapter Dimension
-
-
Evaluation
In the evaluation pipeline, we utilized several key functionalities provided by the NVIDIA NeMo Evaluator to assess the performance of the fine-tuned language models. Here’s how we integrated these features:
-
ROUGE-L for Answer Quality
- We used ROUGE-L to evaluate the lexical overlap between generated answers and ground truth responses. This was particularly useful in determining how closely the generated answers matched the phrasing and structure of reference answers—an important indicator of fluency and relevance in QA tasks.
-
BERTScore for Semantic Similarity
- For a deeper semantic evaluation, we applied BERTScore, which compares embeddings of generated and reference answers. This helped us quantify how well the model understood and responded to questions—even when the surface wording varied—ensuring that semantically accurate answers were recognized.
-
LLM-as-a-Judge for Holistic QA Evaluation
- We also employed the LLM-as-a-Judge approach, using Llama 3.1 Nemotron 70B Instruct to rate responses across multiple critical dimensions: relevance, correctness, groundedness, and fluency. This method was especially effective in QA scenarios, where the quality of the answer often depends on subtle aspects of phrasing, domain grounding, and factual accuracy.
-
By integrating these evaluation methods from the NeMo Evaluator, we were able to conduct a comprehensive assessment of the models.
Customer Case Study: Enhancing RAG System for a Customer Care Use Case using FTaaS
As one of the first real-world applications of the Fine-Tuning-as-a-Service (FTaaS) platform, we started with a use case including approximately 20,000 document chunks, and we set aside 233 human-annotated Q&A pairs for evaluation. Leveraging NeMo Curator, we generated 500 synthetic Q&A samples, ensuring topical diversity and grounding for fine-tuning.
We used the NeMo Customizer within the FTaaS platform to fine-tune the Mistral-7B-Instruct-v0.3 model with LoRA adapters (R=16, α=32, dropout=0.05), training it for 2 epochs with a learning rate of 2e-4. All training outputs—weights, logs, checkpoints, and metrics—were tracked using MLflow.
Once deployed, the fine-tuned model—together with improvements to the RAG pipeline—we were able to achieve up to 40% improvement in ROUGE F1 over the base model and more than 8% boost in BERT F1 scores compared to the model before fine-tuning. Moreover, its performance aligned closely with a production GPT-4 setup, while significantly reducing operational costs—demonstrating the efficiency and impact of the FTaaS offering.
Additionally, several client’s business units have adopted the FTaaS platform for production use cases , achieving significant results. For example, a fine-tuned Mistral model for Q&A RAG use case reached 97%+ accuracy, outperforming the existing production model by 17%, and improving relevance by 19% and faithfulness by 18%. Similarly, a fine-tuned Llama model for knowledge management Q&A RAG use case showed a 10% increase in relevance and 7% in faithfulness, leading to its adoption and substantial cost savings.
Conclusion
Quantiphi’s collaboration with NVIDIA marks a pivotal shift for telecom companies—demonstrating that they can move beyond high-cost proprietary models without compromising on performance or innovation. By strategically using NVIDIA NeMo microservices, we successfully fine-tuned open-source SLMs, delivering a generative AI platform optimized for performance and cost-efficiency. The result? A robust, future-ready solution that balances performance, cost-efficiency, and adaptability—empowering telcos to accelerate transformation while maintaining control over their AI infrastructure.
