LLMs at your Service: Intent-Guided Fine-Tuning for Large Language Models for Customer Support Applications
In our study we set up a two-stage pipeline. Stage 1 focussed on labeling customer support conversations with intents (on the message level) using a closed-weight LLM (GPT-4o), and stage 2 focussed on modeling, where open-weight LLMs were fine-tuned using the labels created in stage 1.
Here we share the code to replicate the most important parts of our work. The stage 1 folder contains the GIFA algorithm, and the different GPFA backbone algorithms: random search, greedy search, SEE and Hyperband-SEE. Furthermore the stage1.ipynb file runs all the algorithms. In our work, we ran all experiments using the closed-weight GPT-4o LLM and the ADA-003 embedding model. Since not everyone has access to these models, we share a simulated run of the whole pipeline on dummy data.
The dummy data created are 25 raw customer support conversations (generated by GPT-4o). All steps used to go from raw conversations to intent-labeled conversations are saved to rerun.
The generative intent finding algorithm (GIFA). GIFA iteratively calls an LLM (GPT-4o) to label each message with intents per conversation, and keeps an expanding known intent database. We provided a simulated GIFA run (toy_intents.csv) that used GPT-4o inside GIFA run on the dummy dataset. GIFA also uses a intent search setup, selecting the most relevant intents for a new conversation to label. This search is done using embeddings: with a new message, we want to find the most similar historical messages and extract their intents and add them to the prompt. To find the most similar messages embeddings are needed. In our simulation these embeddings can be found in GIFA->retrieval->embeddings.csv and are create with the ADA-003 model.
GIFA's performance is highly dependent on the prompt used. To optimize the prompt without human interference, we introduced the generative intent finding algorithm (GIFA). GIFA iteratively optimizes a prompt through feedback based on the reported clustering metrics and examples. GIFA can use multiple background search algorithms, which guide the process of which prompts to select during optimization. Four of them are included: greedy search; random search; SEE and Hyperband-SEE.
Each optimization has been run using GPT-4o and the results were saved in the following format:
- Random search simulation can be found in the rpo_toy folder: each prompt found is evaluated using GIFA, and the results for those are in the GIFA folder; the prompts folder shows the different prompts found through the process, and history provided more meta-data on the process.
- Greedy search works the same.
- SEE works the same, but also some embeddings (ADA-003) are stored, these embeddings are used when using a semantic-wise crossover round.
- Hyperband-SEE (hyperband_see folder) is more nested. Each round inside Hyperband-SEE runs SEE itself. So for each round the SEE run is saved with inside of it the GIFA runs.
Finally we provide a notebook for stage 2 (Step2_finetuning.ipynb) to imitate the CoT or response-only fine-tuning, using the intent-labeled conversations.
The combination of stage 1 and stage 2 can be found in LLMs_at_your_service.ipynb
The easiest way to try our pipeline is via the Colab notebook (Examples for every function/class and additional information can be found here):
Simply open the notebook, run all cells, and you will see the simulated pipeline running on dummy data.
- Stage 1: GIFA + backbone algorithms on toy conversations
- Stage 2: Fine-tuning on labeled conversations
No API keys are required for the simulated run.
git clone https://github.com/ramirosi/Intent-Guided-Fine-Tuning.git
cd Intent-Guided-Fine-Tuning
jupyter notebook LLMs_at_your_service.ipynb Update the API calls in
- GIFA/gpt_api.py (for the LLM)
- GIFA/retrieval/ada_fast.py (for embeddings) Provide your OpenAI API key or a custom model endpoint.
Run LLMs_at_your_service.ipynb with your own dataset (look at dummy data for required format), and turn dummy_mode=False