{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Promptfoo: classification evaluations\n",
    "\n",
    "**Note: This lesson lives in a folder that contains relevant code files. Download the entire folder if you want to follow along and run the evaluation yourself**\n",
    "\n",
    "\n",
    "In an earlier lesson, we evaluated prompts to classify customer complains like: \n",
    "\n",
    "> Whenever I open your app, my phone gets really slow\n",
    "\n",
    "and \n",
    "\n",
    "> I can't figure out how to change my password\n",
    "\n",
    "into five different categories:\n",
    "- Software Bug\n",
    "- Hardware Malfunction\n",
    "- User Error\n",
    "- Service Outage\n",
    "- Feature Request\n",
    "\n",
    "In this lesson we'll port this prompt evaluation over to promptfoo, making it easier to batch-run and compare our results. \n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initializing promptfoo\n",
    "\n",
    "The first step is to initialize promptfoo using the command:\n",
    "\n",
    "```bash\n",
    "npx promptfoo@latest init\n",
    "```\n",
    "\n",
    "As we saw in the previous lesson, this creates a `promptfooconfig.yaml` file.  We can delete the existing contents. \n",
    "\n",
    "Next, we'll configure our provider.  Add the following to `promptfooconfig.yaml`:\n",
    "\n",
    "\n",
    "```yaml\n",
    "description: \"Complaint Classification Eval\"\n",
    "  \n",
    "providers:\n",
    "  - \"anthropic:messages:claude-3-haiku-20240307\"\n",
    "```\n",
    "\n",
    "We'll use Claude 3 Haiku to save on API costs, since we'll be running this evaluation multiple times throughout this lesson.\n",
    "\n",
    "**Ensure you have set the `ANTHROPIC_API_KEY` environment variable. You can set an environment variable by running this command in your terminal:**\n",
    "\n",
    "```bash\n",
    "export ANTHROPIC_API_KEY=your_api_key_here\n",
    "```\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing our prompts\n",
    "\n",
    "Next, we'll collect our prompts and make sure promptfoo knows about them.  We'll follow the same pattern we saw in the previous video: \n",
    "\n",
    "- We'll make each prompt a Python function.\n",
    "- Each prompt function will return a prompt string.\n",
    "- All of our prompt functions will live in a `prompts.py` file.  \n",
    "\n",
    "Create a new file called `prompts.py` and add the following prompt functions to the file.  There are the same two prompts we wrote in the original complaint classification lesson:\n",
    "\n",
    "\n",
    "```py\n",
    "def basic_prompt(complaint):\n",
    "    return f\"\"\"\n",
    "    Classify the following customer complaint into one or more of these categories: \n",
    "    Software Bug, Hardware Malfunction, User Error, Feature Request, or Service Outage.\n",
    "    Only respond with the classification.\n",
    "\n",
    "    Complaint: {complaint}\n",
    "\n",
    "    Classification:\n",
    "    \"\"\"\n",
    "\n",
    "def improved_prompt(complaint):\n",
    "    return f\"\"\"\n",
    "    You are an AI assistant specializing in customer support issue classification. Your task is to analyze customer complaints and categorize them into one or more of the following categories:\n",
    "\n",
    "    1. Software Bug: Issues related to software not functioning as intended.\n",
    "    2. Hardware Malfunction: Problems with physical devices or components.\n",
    "    3. User Error: Difficulties arising from user misunderstanding or misuse.\n",
    "    4. Feature Request: Suggestions for new functionalities or improvements.\n",
    "    5. Service Outage: System-wide issues affecting service availability.\n",
    "\n",
    "    Important Guidelines:\n",
    "    - A complaint may fall into multiple categories. If so, list all that apply but try to prioritize picking a single category when possible.\n",
    "\n",
    "    Examples:\n",
    "    1. Complaint: \"The app crashes when I try to save my progress.\"\n",
    "    Classification: Software Bug\n",
    "\n",
    "    2. Complaint: \"My keyboard isn't working after I spilled coffee on it.\"\n",
    "    Classification: Hardware Malfunction\n",
    "\n",
    "    3. Complaint: \"I can't find the login button on your website.\"\n",
    "    Classification: User Error\n",
    "\n",
    "    4. Complaint: \"It would be great if your app had a dark mode.\"\n",
    "    Classification: Feature Request\n",
    "\n",
    "    5. Complaint: \"None of your services are loading for me or my colleagues.\"\n",
    "    Classification: Service Outage\n",
    "\n",
    "    6. Complaint \"Complaint: The app breaks every time I try to change my profile picture\"\n",
    "    Classification: Software Bug\n",
    "\n",
    "    7. Complaint \"The app is acting buggy on my phone and it seems like your website is down, so I'm completely stuck!\"\n",
    "    Classification: Software Bug, Service Outage\n",
    "\n",
    "    8. Complaint: \"Your software makes my computer super laggy and awful, I hate it!\"\n",
    "    Classification: Software Bug\n",
    "\n",
    "    9. Complaint: \"Your dumb app always breaks when I try to do anything with images.\"\n",
    "    Classification: 'Software Bug'\n",
    "\n",
    "    Now, please classify the following customer complaint:\n",
    "\n",
    "    <complaint>{complaint}</complaint>\n",
    "\n",
    "    Only respond with the appropriate categories and nothing else.\n",
    "    Classification:\n",
    "    \"\"\"\n",
    "```\n",
    "\n",
    "\n",
    "Next, we need to tell promptfoo we want to use these two prompts.  Update the `promptfooconfig.yaml` file: \n",
    "\n",
    "```yaml\n",
    "description: \"Complaint Classification Eval\"\n",
    "\n",
    "prompts:\n",
    "  - prompts.py:basic_prompt\n",
    "  - prompts.py:improved_prompt\n",
    "  \n",
    "providers:\n",
    "  - \"anthropic:messages:claude-3-haiku-20240307\"\n",
    "```\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing our evaluation test set\n",
    "\n",
    "The final step is to get our evaluation data set into a shape that plays well with promptfoo.  As a reminder, this is what our original `eval_data` Python list from the earlier lesson looked like:\n",
    "\n",
    "```py\n",
    "eval_data = [\n",
    "    {\n",
    "        \"complaint\": \"The app crashes every time I try to upload a photo\",\n",
    "        \"golden_answer\": [\"Software Bug\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"My printer isn't recognized by my computer\",\n",
    "        \"golden_answer\": [\"Hardware Malfunction\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"I can't figure out how to change my password\",\n",
    "        \"golden_answer\": [\"User Error\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"The website is completely down, I can't access any pages\",\n",
    "        \"golden_answer\": [\"Service Outage\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"It would be great if the app had a dark mode option\",\n",
    "        \"golden_answer\": [\"Feature Request\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"The software keeps freezing when I try to save large files\",\n",
    "        \"golden_answer\": [\"Software Bug\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"My wireless mouse isn't working, even with new batteries\",\n",
    "        \"golden_answer\": [\"Hardware Malfunction\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"I accidentally deleted some important files, can you help me recover them?\",\n",
    "        \"golden_answer\": [\"User Error\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"None of your servers are responding, is there an outage?\",\n",
    "        \"golden_answer\": [\"Service Outage\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"Could you add a feature to export data in CSV format?\",\n",
    "        \"golden_answer\": [\"Feature Request\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"The app is crashing and my phone is overheating\",\n",
    "        \"golden_answer\": [\"Software Bug\", \"Hardware Malfunction\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"I can't remember my password!\",\n",
    "        \"golden_answer\": [\"User Error\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"The new update broke something and the app no longer works for me\",\n",
    "        \"golden_answer\": [\"Software Bug\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"I think I installed something incorrectly, now my computer won't start at all\",\n",
    "        \"golden_answer\": [\"User Error\", \"Hardware Malfunction\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"Your service is down, and I urgently need a feature to batch process files\",\n",
    "        \"golden_answer\": [\"Service Outage\", \"Feature Request\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"The graphics card is making weird noises\",\n",
    "        \"golden_answer\": [\"Hardware Malfunction\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"My keyboard just totally stopped working out of nowhere\",\n",
    "        \"golden_answer\": [\"Hardware Malfunction\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"Whenever I open your app, my phone gets really slow\",\n",
    "        \"golden_answer\": [\"Software Bug\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"Can you make the interface more user-friendly? I always get lost in the menus\",\n",
    "        \"golden_answer\": [\"Feature Request\", \"User Error\"]\n",
    "    },\n",
    "    {\n",
    "        \"complaint\": \"The cloud storage isn't syncing and I can't access my files from other devices\",\n",
    "        \"golden_answer\": [\"Software Bug\", \"Service Outage\"]\n",
    "    }\n",
    "]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we did in the previous lesson, we'll turn our data set into a CSV file.  The key difference here is that our evaluation logic is no longer a simple exact-match.  In order to grade this evaluation, we want promptfoo to make sure that each model output includes the correct classification(s)\n",
    "\n",
    "For example, given this row from the dataset:\n",
    "\n",
    "```py\n",
    "{\n",
    "    \"complaint\": \"The cloud storage isn't syncing and I can't access my files from other devices\",\n",
    "    \"golden_answer\": [\"Software Bug\", \"Service Outage\"]\n",
    "}\n",
    "```\n",
    "\n",
    "We'll write a prompt that takes this input `complaint`:\n",
    "\n",
    ">The cloud storage isn't syncing and I can't access my files from other devices\n",
    "\n",
    "For the above example, we need promptfoo to ensure that the model's response includes \"Software Bug\" and \"Service Outage\". We can't do an exact match. What if the model switches the output order of the two classifications?  Thankfully, promptfoo includes a bunch of built-in assertions that we can take advantage of.  These assertions include things like: \n",
    "\n",
    "* `contains` - output contains substring\n",
    "* `contains-all` - output contains all list of substrings\n",
    "* `contains-any` - output contains any of the listed substrings\n",
    "* `contains-json` - output contains valid json (optional json schema validation)\n",
    "* `contains-sql` - output contains valid sql\n",
    "* `contains-xml`- output contains valid xml\n",
    "* `equals` - output matches exactly\n",
    "* `icontains` - output contains substring, case insensitive\n",
    "* `icontains-all` - output contains all list of substrings, case insensitive\n",
    "* `icontains-any` - output contains any of the listed substrings, case insensitive\n",
    "* `regex`\t- output matches regex\n",
    "* many others\n",
    "\n",
    "[View an exhaustive list of built-in metrics here.](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/)\n",
    "\n",
    "For our use-case, we'll use `contains-all` to ensure that a given output contains all of the appropriate classification tags.  \n",
    "\n",
    "One way of loading and structuring a promptfoo evaluation dataset is through CSV.  As we saw previously, we can provide a special CSV column name called `__expected` to specify grading logic.  In this column we can use any of the above built-in assertions, including `contains-all`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make a new file called `dataset.csv` and paste this code into it: \n",
    "\n",
    "```csv\n",
    "complaint,__expected\n",
    "The app crashes every time I try to upload a photo,contains-all:Software Bug\n",
    "My printer isn't recognized by my computer,contains-all:Hardware Malfunction\n",
    "I can't figure out how to change my password,contains-all:User Error\n",
    "The website is completely down I can't access any pages,contains-all:Service Outage\n",
    "It would be great if the app had a dark mode option,contains-all:Feature Request\n",
    "The software keeps freezing when I try to save large files,contains-all:Software Bug\n",
    "My wireless mouse isn't working even with new batteries,contains-all:Hardware Malfunction\n",
    "I accidentally deleted some important files can you help me recover them?,contains-all:User Error\n",
    "None of your servers are responding is there an outage?,contains-all:Service Outage\n",
    "Could you add a feature to export data in CSV format?,contains-all:Feature Request\n",
    "\"The app is crashing and my phone is overheating\",\"contains-all:Software Bug,Hardware Malfunction\"\n",
    "I can't remember my password!,contains-all:User Error\n",
    "The new update broke something and the app no longer works for me,contains-all:Software Bug\n",
    "\"I think I installed something incorrectly now my computer won't start at all\",\"contains-all:User Error,Hardware Malfunction\"\n",
    "\"Your service is down and I urgently need a feature to batch process files\",\"contains-all:Service Outage,Feature Request\"\n",
    "The graphics card is making weird noises,contains-all:Hardware Malfunction\n",
    "My keyboard just totally stopped working out of nowhere,contains-all:Hardware Malfunction\n",
    "Whenever I open your app my phone gets really slow,contains-all:Software Bug\n",
    "Can you make the interface more user-friendly? I always get lost in the menus,\"contains-all:Feature Request,User Error\"\n",
    "The cloud storage isn't syncing and I can't access my files from other devices,\"contains-all:Software Bug,Service Outage\"\n",
    "```\n",
    "\n",
    "Our CSV contains two columns:\n",
    "- `complaint` - the actual input complains\n",
    "- `__expected` - contains a `contains-all` assertion\n",
    "\n",
    "Take a look at one of the rows, like this one:\n",
    "\n",
    "> \"Your service is down and I urgently need a feature to batch process files\",\"contains-all:Service Outage,Feature Request\"\n",
    "\n",
    "This row of the dataset specifies that given the input \"Your service is down and I urgently need a feature to batch process files,\" we want promptfoo to check the model's output and ensure it includes both \"Service Outage\" and \"Feature Request\"\n",
    "\n",
    "\n",
    "The final step here is to update our `promptfooconfig.yaml` file to include the tests we just wrote.  The file should now look like this:\n",
    "\n",
    "```yaml\n",
    "description: \"Complaint Classification Eval\"\n",
    "\n",
    "prompts:\n",
    "  - prompts.py:basic_prompt\n",
    "  - prompts.py:improved_prompt\n",
    "  \n",
    "providers:\n",
    "  - \"anthropic:messages:claude-3-haiku-20240307\"\n",
    "\n",
    "tests: dataset.csv\n",
    "\n",
    "```\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running the evaluation\n",
    "\n",
    "To run the evaluation, we'll use the same command we've seen before:\n",
    "\n",
    "```bash\n",
    "npx promptfoo@latest eval\n",
    "```"
   ]
  },
  {
   "attachments": {
    "eval_output.png": {
     "image/png": "[encoded data, 629624 characters]"
    },
    "output_row.png": {
     "image/png": "[encoded data, 52640 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the output we got the first time we ran the above eval: \n",
    "\n",
    "![eval_output.png](attachment:eval_output.png)\n",
    "\n",
    "It's clear that the improved prompt that includes examples performs better than our initial basic prompt.  The examples help the model understand the types of situations we may want the output to include multiple categories assigned to it:\n",
    "\n",
    "![output_row.png](attachment:output_row.png)"
   ]
  },
  {
   "attachments": {
    "web_view.png": {
     "image/png": "[encoded data, 692208 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As always, we can also open up an interactive web of the evaluation results using this command:\n",
    "\n",
    "```bash\n",
    "npx promptfoo@latest view\n",
    "```\n",
    "\n",
    "![web_view.png](attachment:web_view.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that our `basic_prompt` got 80% right, while the `improved_prompt` scored 100%.\n",
    "\n",
    "**As always, please remember that we are using small, educational datasets that are not representative of real-world evaluations.  We always recommend at least 100 rows in an evaluation dataset.**\n",
    "\n",
    "Next up, we'll take a look at writing custom grading logic in promptfoo!"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
