{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code-graded eval: classification task\n",
    "\n",
    "In this lesson, we'll implement a slightly more complex code-graded evaluation from scratch to test a customer complaint classification prompt.  Our goal is to write a prompt that can reliably classify customer complaints into the following categories: \n",
    "\n",
    "* Software Bug\n",
    "* Hardware Malfunction\n",
    "* User Error\n",
    "* Feature Request\n",
    "* Service Outage\n",
    "\n",
    "For example, the following complaint text: \n",
    "\n",
    "> The website is completely down, I can't access any pages\n",
    "\n",
    "Should be classified as `Service Outage`\n",
    "\n",
    "In some cases, we may want allow up to two applicable classification categories, as in this example: \n",
    "\n",
    "> I think I installed something incorrectly, and now my computer won't start at all\n",
    "\n",
    "which should be classified as both `User Error` and `Hardware Malfunction`\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Evaluation data set\n",
    "\n",
    "We'll start by defining our evaluation data set of inputs and golden answers.  Remember that generally we want an evaluation data set of around 100 inputs, but to keep these lessons simple (and quick and affordable to run), we're using a slimmed down set.\n",
    "\n",
    "This test set consists of a list of dictionaries where each dictionary contains a `complaint` and `golden_answer` key:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_data = [\n",
    "    {\n",
    "        \"complaint\": \"The app crashes every time I try to upload a photo\",\n",
    "        \"golden_answer\": [\"Software Bug\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"My printer isn't recognized by my computer\",\n",
    "        \"golden_answer\": [\"Hardware Malfunction\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"I can't figure out how to change my password\",\n",
    "        \"golden_answer\": [\"User Error\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"The website is completely down, I can't access any pages\",\n",
    "        \"golden_answer\": [\"Service Outage\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"It would be great if the app had a dark mode option\",\n",
    "        \"golden_answer\": [\"Feature Request\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"The software keeps freezing when I try to save large files\",\n",
    "        \"golden_answer\": [\"Software Bug\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"My wireless mouse isn't working, even with new batteries\",\n",
    "        \"golden_answer\": [\"Hardware Malfunction\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"I accidentally deleted some important files, can you help me recover them?\",\n",
    "        \"golden_answer\": [\"User Error\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"None of your servers are responding, is there an outage?\",\n",
    "        \"golden_answer\": [\"Service Outage\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"Could you add a feature to export data in CSV format?\",\n",
    "        \"golden_answer\": [\"Feature Request\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"The app is crashing and my phone is overheating\",\n",
    "        \"golden_answer\": [\"Software Bug\", \"Hardware Malfunction\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"I can't remember my password!\",\n",
    "        \"golden_answer\": [\"User Error\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"The new update broke something and the app no longer works for me\",\n",
    "        \"golden_answer\": [\"Software Bug\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"I think I installed something incorrectly, now my computer won't start at all\",\n",
    "        \"golden_answer\": [\"User Error\", \"Hardware Malfunction\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"Your service is down, and I urgently need a feature to batch process files\",\n",
    "        \"golden_answer\": [\"Service Outage\", \"Feature Request\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"The graphics card is making weird noises\",\n",
    "        \"golden_answer\": [\"Hardware Malfunction\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"My keyboard just totally stopped working out of nowhere\",\n",
    "        \"golden_answer\": [\"Hardware Malfunction\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"Whenever I open your app, my phone gets really slow\",\n",
    "        \"golden_answer\": [\"Software Bug\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"Can you make the interface more user-friendly? I always get lost in the menus\",\n",
    "        \"golden_answer\": [\"Feature Request\", \"User Error\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"The cloud storage isn't syncing and I can't access my files from other devices\",\n",
    "        \"golden_answer\": [\"Software Bug\", \"Service Outage\"]\n",
    "    }\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--- \n",
    "\n",
    "## An initial prompt\n",
    "\n",
    "We'll start with a basic prompt and measure how it performs.  The prompt-generating function below takes a `complaint` as an argument and returns a prompt string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def basic_prompt(complaint):\n",
    "    return f\"\"\"\n",
    "    Classify the following customer complaint into one or more of these categories: \n",
    "    Software Bug, Hardware Malfunction, User Error, Feature Request, or Service Outage.\n",
    "    Only respond with the matching category or categories and nothing else.\n",
    "\n",
    "    Complaint: {complaint}\n",
    "\n",
    "    Classification:\n",
    "    \"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Collecting outputs\n",
    "\n",
    "Next, we'll write the logic to evaluate the prompt.  This logic is a bit more complex than our \"leg-counting\" example from the previous lesson:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "\n",
    "def get_model_response(prompt):\n",
    "\n",
    "    bedrock_client = boto3.client(service_name='bedrock-runtime', region_name=\"us-west-2\")\n",
    "    model_id = \"anthropic.claude-3-haiku-20240307-v1:0\"\n",
    "\n",
    "    # Send the message.\n",
    "    response = bedrock_client.converse(\n",
    "        modelId=model_id,\n",
    "        messages=[{\"role\": \"user\", \"content\": [{\"text\":prompt}]}]\n",
    "    )\n",
    "\n",
    "    return response[\"output\"][\"message\"][\"content\"][0][\"text\"]\n",
    "\n",
    "def calculate_accuracy(eval_data, model_responses):\n",
    "    correct_predictions = 0\n",
    "    total_predictions = len(eval_data)\n",
    "    \n",
    "    for item, response in zip(eval_data, model_responses):\n",
    "        golden_set = set(category.lower() for category in item[\"golden_answer\"])\n",
    "        prediction_set = set(category.strip().lower() for category in response.split(','))\n",
    "        \n",
    "        if golden_set == prediction_set:\n",
    "            correct_predictions += 1\n",
    "    \n",
    "    return correct_predictions / total_predictions\n",
    "\n",
    "def evaluate_prompt(prompt_func, eval_data):\n",
    "    model_responses = [get_model_response(prompt_func(item['complaint'])) for item in eval_data]\n",
    "    accuracy = calculate_accuracy(eval_data, model_responses)\n",
    "    \n",
    "    print(f\"Accuracy: {accuracy:.2%}\")\n",
    "    \n",
    "    for item, response in zip(eval_data, model_responses):\n",
    "        print(f\"\\nComplaint: {item['complaint']}\")\n",
    "        print(f\"Golden Answer: {item['golden_answer']}\")\n",
    "        print(f\"Model Response: {response}\")\n",
    "    return accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `evaluate_prompt` function does the following:\n",
    "\n",
    "1. It passes each input into our prompt-generating function and runs the resulting prompt through the model using the `get_model_response` function, collecting the responses as they're generated.\n",
    "2. It calculates the accuracy by comparing the model output answers to the golden answers in our data set. To do this it calls the `calculate_accuracy` function.\n",
    "3. The `calculate_accuracy` function checks to see if the appropriate classification categories are present in each of the model's outputs, using a `set`.  Remember, this is not an exact-match eval like our previous \"leg-counting\" eval.\n",
    "4. `calculate_accuracy` returns an accuracy score\n",
    "5. `evaluate_prompt` prints the final results\n",
    "\n",
    "**Note that instead of grading via exact string match, as we did in the previous lesson, our grading logic uses a `set` to check for the presence of values in the model output.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test it out with our initial `basic_prompt`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 85.00%\n",
      "\n",
      "Complaint: The app crashes every time I try to upload a photo\n",
      "Golden Answer: ['Software Bug']\n",
      "Model Response: Software Bug\n",
      "\n",
      "Complaint: My printer isn't recognized by my computer\n",
      "Golden Answer: ['Hardware Malfunction']\n",
      "Model Response: Hardware Malfunction\n",
      "\n",
      "Complaint: I can't figure out how to change my password\n",
      "Golden Answer: ['User Error']\n",
      "Model Response: User Error\n",
      "\n",
      "Complaint: The website is completely down, I can't access any pages\n",
      "Golden Answer: ['Service Outage']\n",
      "Model Response: Service Outage\n",
      "\n",
      "Complaint: It would be great if the app had a dark mode option\n",
      "Golden Answer: ['Feature Request']\n",
      "Model Response: Feature Request\n",
      "\n",
      "Complaint: The software keeps freezing when I try to save large files\n",
      "Golden Answer: ['Software Bug']\n",
      "Model Response: Software Bug\n",
      "\n",
      "Complaint: My wireless mouse isn't working, even with new batteries\n",
      "Golden Answer: ['Hardware Malfunction']\n",
      "Model Response: Hardware Malfunction\n",
      "\n",
      "Complaint: I accidentally deleted some important files, can you help me recover them?\n",
      "Golden Answer: ['User Error']\n",
      "Model Response: User Error\n",
      "\n",
      "Complaint: None of your servers are responding, is there an outage?\n",
      "Golden Answer: ['Service Outage']\n",
      "Model Response: Service Outage\n",
      "\n",
      "Complaint: Could you add a feature to export data in CSV format?\n",
      "Golden Answer: ['Feature Request']\n",
      "Model Response: Feature Request\n",
      "\n",
      "Complaint: The app is crashing and my phone is overheating\n",
      "Golden Answer: ['Software Bug', 'Hardware Malfunction']\n",
      "Model Response: Software Bug, Hardware Malfunction\n",
      "\n",
      "Complaint: I can't remember my password!\n",
      "Golden Answer: ['User Error']\n",
      "Model Response: User Error\n",
      "\n",
      "Complaint: The new update broke something and the app no longer works for me\n",
      "Golden Answer: ['Software Bug']\n",
      "Model Response: Software Bug\n",
      "\n",
      "Complaint: I think I installed something incorrectly, now my computer won't start at all\n",
      "Golden Answer: ['User Error', 'Hardware Malfunction']\n",
      "Model Response: User Error\n",
      "\n",
      "Complaint: Your service is down, and I urgently need a feature to batch process files\n",
      "Golden Answer: ['Service Outage', 'Feature Request']\n",
      "Model Response: Service Outage, Feature Request\n",
      "\n",
      "Complaint: The graphics card is making weird noises\n",
      "Golden Answer: ['Hardware Malfunction']\n",
      "Model Response: Hardware Malfunction\n",
      "\n",
      "Complaint: My keyboard just totally stopped working out of nowhere\n",
      "Golden Answer: ['Hardware Malfunction']\n",
      "Model Response: Hardware Malfunction\n",
      "\n",
      "Complaint: Whenever I open your app, my phone gets really slow\n",
      "Golden Answer: ['Software Bug']\n",
      "Model Response: Hardware Malfunction\n",
      "\n",
      "Complaint: Can you make the interface more user-friendly? I always get lost in the menus\n",
      "Golden Answer: ['Feature Request', 'User Error']\n",
      "Model Response: Feature Request\n",
      "\n",
      "Complaint: The cloud storage isn't syncing and I can't access my files from other devices\n",
      "Golden Answer: ['Software Bug', 'Service Outage']\n",
      "Model Response: Software Bug, Service Outage\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.85"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_prompt(basic_prompt, eval_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## An improved prompt\n",
    "Our initial prompt resulted in an 85% accuracy score.  Let's make some changes to the prompt and rerun the evaluation, hopefully resulting in a better score.\n",
    "\n",
    "The following prompt incorporates an expanded explanation of the categories, as well as 9 example input/output pairs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def improved_prompt(complaint):\n",
    "    return f\"\"\"\n",
    "    You are an AI assistant specializing in customer support issue classification. Your task is to analyze customer complaints and categorize them into one or more of the following categories:\n",
    "\n",
    "    1. Software Bug: Issues related to software not functioning as intended.\n",
    "    2. Hardware Malfunction: Problems with physical devices or components.\n",
    "    3. User Error: Difficulties arising from user misunderstanding or misuse.\n",
    "    4. Feature Request: Suggestions for new functionalities or improvements.\n",
    "    5. Service Outage: System-wide issues affecting service availability.\n",
    "\n",
    "    Important Guidelines:\n",
    "    - A complaint may fall into multiple categories. If so, list all that apply but try to prioritize picking a single category when possible.\n",
    "\n",
    "    Examples:\n",
    "    1. Complaint: \"The app crashes when I try to save my progress.\"\n",
    "    Classification: Software Bug\n",
    "\n",
    "    2. Complaint: \"My keyboard isn't working after I spilled coffee on it.\"\n",
    "    Classification: Hardware Malfunction\n",
    "\n",
    "    3. Complaint: \"I can't find the login button on your website.\"\n",
    "    Classification: User Error\n",
    "\n",
    "    4. Complaint: \"It would be great if your app had a dark mode.\"\n",
    "    Classification: Feature Request\n",
    "\n",
    "    5. Complaint: \"None of your services are loading for me or my colleagues.\"\n",
    "    Classification: Service Outage\n",
    "\n",
    "    6. Complaint \"Complaint: The app breaks every time I try to change my profile picture\"\n",
    "    Classification: Software Bug\n",
    "\n",
    "    7. Complaint \"The app is acting buggy on my phone and it seems like your website is down, so I'm completely stuck!\"\n",
    "    Classification: Software Bug, Service Outage\n",
    "\n",
    "    8. Complaint: \"Your software makes my computer super laggy and awful, I hate it!\"\n",
    "    Classification: Software Bug\n",
    "\n",
    "    9. Complaint: \"Your dumb app always breaks when I try to do anything with images.\"\n",
    "    Classification: 'Software Bug'\n",
    "\n",
    "    Now, please classify the following customer complaint:\n",
    "\n",
    "    <complaint>{complaint}</complaint>\n",
    "\n",
    "    Only respond with the appropriate categories and nothing else.\n",
    "    Classification:\n",
    "    \"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run the evaluation with our improved prompt:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 100.00%\n",
      "\n",
      "Complaint: The app crashes every time I try to upload a photo\n",
      "Golden Answer: ['Software Bug']\n",
      "Model Response: Software Bug\n",
      "\n",
      "Complaint: My printer isn't recognized by my computer\n",
      "Golden Answer: ['Hardware Malfunction']\n",
      "Model Response: Hardware Malfunction\n",
      "\n",
      "Complaint: I can't figure out how to change my password\n",
      "Golden Answer: ['User Error']\n",
      "Model Response: User Error\n",
      "\n",
      "Complaint: The website is completely down, I can't access any pages\n",
      "Golden Answer: ['Service Outage']\n",
      "Model Response: Service Outage\n",
      "\n",
      "Complaint: It would be great if the app had a dark mode option\n",
      "Golden Answer: ['Feature Request']\n",
      "Model Response: Feature Request\n",
      "\n",
      "Complaint: The software keeps freezing when I try to save large files\n",
      "Golden Answer: ['Software Bug']\n",
      "Model Response: Software Bug\n",
      "\n",
      "Complaint: My wireless mouse isn't working, even with new batteries\n",
      "Golden Answer: ['Hardware Malfunction']\n",
      "Model Response: Hardware Malfunction\n",
      "\n",
      "Complaint: I accidentally deleted some important files, can you help me recover them?\n",
      "Golden Answer: ['User Error']\n",
      "Model Response: User Error\n",
      "\n",
      "Complaint: None of your servers are responding, is there an outage?\n",
      "Golden Answer: ['Service Outage']\n",
      "Model Response: Service Outage\n",
      "\n",
      "Complaint: Could you add a feature to export data in CSV format?\n",
      "Golden Answer: ['Feature Request']\n",
      "Model Response: Feature Request\n",
      "\n",
      "Complaint: The app is crashing and my phone is overheating\n",
      "Golden Answer: ['Software Bug', 'Hardware Malfunction']\n",
      "Model Response: Software Bug, Hardware Malfunction\n",
      "\n",
      "Complaint: I can't remember my password!\n",
      "Golden Answer: ['User Error']\n",
      "Model Response: User Error\n",
      "\n",
      "Complaint: The new update broke something and the app no longer works for me\n",
      "Golden Answer: ['Software Bug']\n",
      "Model Response: Software Bug\n",
      "\n",
      "Complaint: I think I installed something incorrectly, now my computer won't start at all\n",
      "Golden Answer: ['User Error', 'Hardware Malfunction']\n",
      "Model Response: Hardware Malfunction, User Error\n",
      "\n",
      "Complaint: Your service is down, and I urgently need a feature to batch process files\n",
      "Golden Answer: ['Service Outage', 'Feature Request']\n",
      "Model Response: Service Outage, Feature Request\n",
      "\n",
      "Complaint: The graphics card is making weird noises\n",
      "Golden Answer: ['Hardware Malfunction']\n",
      "Model Response: Hardware Malfunction\n",
      "\n",
      "Complaint: My keyboard just totally stopped working out of nowhere\n",
      "Golden Answer: ['Hardware Malfunction']\n",
      "Model Response: Hardware Malfunction\n",
      "\n",
      "Complaint: Whenever I open your app, my phone gets really slow\n",
      "Golden Answer: ['Software Bug']\n",
      "Model Response: Software Bug\n",
      "\n",
      "Complaint: Can you make the interface more user-friendly? I always get lost in the menus\n",
      "Golden Answer: ['Feature Request', 'User Error']\n",
      "Model Response: User Error, Feature Request\n",
      "\n",
      "Complaint: The cloud storage isn't syncing and I can't access my files from other devices\n",
      "Golden Answer: ['Software Bug', 'Service Outage']\n",
      "Model Response: Software Bug, Service Outage\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_prompt(improved_prompt, eval_data)"
   ]
  },
  {
   "attachments": {
    "process.png": {
     "image/png": "[encoded data, 223264 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We got 100% accuracy with the newer, improved prompt! \n",
    "\n",
    "Again, we're following the standard prompt + eval loop outlined in this diagram:\n",
    "\n",
    "![process.png](attachment:process.png)\n",
    "\n",
    "**Please keep in mind that this is a very simple evaluation, using a very small dataset.  This lesson aims to illustrate the general process of code-graded evaluations, but it is not meant as a canonical example of a production-scale evaluation!**\n",
    "\n",
    "This approach works, but it's a bit laborious to write all the evaluation logic from scratch, and it's difficult to compare results side-by-side.  What if we used a tool that generated nicely-formatted results with charts and graphs and made it easy to run an evaluation across multiple models?  In the next lesson, we'll see just that! Next up, we'll take a look at an evaluation framework that makes it easy to write repeatable, scalable evaluations for production use-cases."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py311",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
