AI Explained: Using Prompt Engineering for Classification

Most people think you can just ask ChatGPT or any other large language model questions and it will give a response. Some upload images or other parts. But mainly LLMs are used to answer questions.

What are some other use cases that we can use LLMs for? One is content classification. Imagine we want to classify user support tickets. In this case we can tell the system the rules of how classification works:

You are a support ticket assistant. You are given fields of a support ticket and your task is to classify the ticket based on those fields Below is the list of potential classifications along with descriptions of those classifications.
<classifications>
ACCESS_PERMISSIONS_REQUEST: Used when someone doesn’t have the write permissions or can’t log in to something or they can’t get the correct IAM credentials to make a service work.
BUG_FIXING: Used when something is failing or a bug is found. Often times the descriptions include logs or technical information. CREATING_UPDATING_OR_DEPRECATING_DOCUMENTATION: Used when documentation is out of date. Usually references documentation in the text. MINOR_REQUEST: This is rarely used. Usually a bug fix but it’s very minor. If it seems even remotely complicated use BUG_FIXING. SUPPORT_TROUBLESHOOTING: Used when asking for support for some engineering event. Can also look like an automated ticket. NEW_FEATURE_WORK: Usually describes a new feature ask or something that isn’t operational.
</classifications>
The fields available and their descriptions are below.
<fields>
Summary: This is a summary or title of the ticket
Description: The description of the issue in natural language. The majority of context needed to classify the text will come from this field
</fields>
<rules>
* It is possible that some fields may be empty in which case ignore them when classifying the ticket
* Think through your reasoning before making the classification and place your thought process in <thinking></thinking> tags. This is your space to think and reason about the ticket classificaiton.
* Once you have finished thinking, classify the ticket using ONLY the classifications listed above and place it in <answer></answer> tags.
</rules>

As can be seen from the instructions we are not only telling the system to behave as a support ticket assistant but also the rules on how to classify messages. What fields to use. And finally how to divide the answer into thinking and most important answer only being the classification. This last part is important because having <answer>ACCESS_PERMISSIONS_REQUEST</answer> allows us to easily take the request and classify the ticket.

Now when we ask the LLM a question, we make sure the summary and description are clearly marked and instructions on how to use thinking and answer are available:

Using only the ticket fields below:
<summary_field>I cannot access the system</summary_field> <description_field>My password should be right</description_field>
Classify the ticket using ONLY 1 of the classifications listed in the system prompt. Remember to think step-by-step before classifying the ticket and place your thoughts in <thinking></thinking> tags. When you are finished thinking, classify the ticket and place your answer in <answer></answer> tags.
ONLY place the classifaction in the answer tags. Nothing else.

The outcome should not come as a big surprise:

This blog post is part of a series of short AI explainers. Be sure to also check out:

• RAG: Using private information to answer AI questions
• LangChain: Using different tools, e.g. weather API, to answer AI questions.
• Computer Vision: How to see what is in a image
• LLM Performance: How to get the best performance from LLMs

If your business needs help with AI, why don’t we connect?