{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Anthropic workbench evaluations\n",
    "This lesson will show you how to use Anthropic Workbench to run your own human-graded evaluations.  This is an extremely easy-to-use and visual interface for quickly prototyping prompts and running human-graded evaluations.  While we generally recommend using a more scale-able approach for production evaluations, the Anthropic Workbench is a great place to start with human-graded evaluations before moving to more rigorous code-graded or model-graded evals.\n",
    "\n",
    "In this lesson we'll see how to use the Workbench to test prompts, run simple evaluations, and compare prompt versions.\n",
    "\n",
    "---"
   ]
  },
  {
   "attachments": {
    "empty_workbench.png": {
     "image/png": "[encoded data, 916748 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Anthropic workbench\n",
    "\n",
    "[Anthropic's workbench](https://console.anthropic.com/workbench/) is a great place to quickly prototype prompts and run human-graded evaluations.  This is what the workbench looks like when we first load it: \n",
    "\n",
    "![empty_workbench.png](attachment:empty_workbench.png)\n",
    "\n",
    "On the left side we can enter a prompt.  Let's imagine we're working on a code-translation application and want to write the best possible prompt to use the Anthropic API to translate code from any coding language into Python.  Here's an initial attempt at a prompt:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "You are a skilled programmer tasked with translating code from one programming language to Python. Your goal is to produce an accurate and idiomatic Python translation of the provided source code.\n",
    "\n",
    "Here is the source code to translate:\n",
    "\n",
    "<source_code>\n",
    "{{SOURCE_CODE}}\n",
    "</source_code>\n",
    "\n",
    "The source code is written in the following language:\n",
    "\n",
    "<source_language>\n",
    "{{SOURCE_LANGUAGE}}\n",
    "</source_language>\n",
    "\n",
    "Please translate this code to Python\n",
    "```\n",
    "\n",
    "Notice the `{{SOURCE_CODE}}` and `{{SOURCE_LANGUAGE}}` variables, which we will later replace with dynamic values."
   ]
  },
  {
   "attachments": {
    "workbench_with_prompt.png": {
     "image/png": "[encoded data, 1096232 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can put this prompt into the left side of the workbench: \n",
    "\n",
    "![workbench_with_prompt.png](attachment:workbench_with_prompt.png)"
   ]
  },
  {
   "attachments": {
    "adding_variables.png": {
     "image/png": "[encoded data, 260304 characters]"
    },
    "variables_button.png": {
     "image/png": "[encoded data, 121448 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we can set test values for our variables by clicking on the variables (`{ }`) button:\n",
    "\n",
    "![variables_button.png](attachment:variables_button.png)\n",
    "\n",
    "This will open a dialog, asking us to input values for the `{{SOURCE_CODE}}` and `{{SOURCE_LANGUAGE}}` variables:\n",
    "\n",
    "![adding_variables.png](attachment:adding_variables.png)\n"
   ]
  },
  {
   "attachments": {
    "first_output.png": {
     "image/png": "[encoded data, 472840 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we can hit run and see the resulting output from the model:\n",
    "\n",
    "![first_output.png](attachment:first_output.png)"
   ]
  },
  {
   "attachments": {
    "evaluate_button.png": {
     "image/png": "[encoded data, 46176 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Workbench evaluations\n",
    "\n",
    "Testing our prompt with one set of variables at a time is a good place to start, but the Workbench also comes with a built-in evaluation tool to help us run prompts against multiple inputs.  To switch over to the evaluate view, click the \"Evaluate\" toggle button at the top: \n",
    "\n",
    "![evaluate_button.png](attachment:evaluate_button.png)"
   ]
  },
  {
   "attachments": {
    "evaluate1.png": {
     "image/png": "[encoded data, 716312 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This opens the evaluate view, with our initial result pre-populated:\n",
    "\n",
    "![evaluate1.png](attachment:evaluate1.png)"
   ]
  },
  {
   "attachments": {
    "evaluate2.png": {
     "image/png": "[encoded data, 584872 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we can click the \"Add Row\" button to add some new test cases.  Let's add in two new test cases: some Ruby code and some C# code: \n",
    "\n",
    "![evaluate2.png](attachment:evaluate2.png)\n"
   ]
  },
  {
   "attachments": {
    "run_remaining.png": {
     "image/png": "[encoded data, 65312 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we can either click the individual \"Run\" buttons next to each test case, or we can click the orange \"Run Remaining\" button to click all remaining un-run test cases:\n",
    "\n",
    "![run_remaining.png](attachment:run_remaining.png)\n",
    "\n",
    "Let's click the Run Remaining button and take a look at our model responses:"
   ]
  },
  {
   "attachments": {
    "evaluate3.png": {
     "image/png": "[encoded data, 709184 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This are the results we got:\n",
    "\n",
    "![evaluate3.png](attachment:evaluate3.png)"
   ]
  },
  {
   "attachments": {
    "score.png": {
     "image/png": "[encoded data, 316564 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--- \n",
    "\n",
    "## Human grading\n",
    "\n",
    "Now it's time to take a close look at the model outputs and give them scores. In the right column, we have the option of assigning a score to each output: \n",
    "\n",
    "![score.png](attachment:score.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a closer look at some of the outputs.\n",
    "\n",
    "### JavaScript test case results\n",
    "\n",
    "This is the result from our JavaScript translation test case:\n",
    "\n",
    "```\n",
    "Here's the Python translation of the provided JavaScript code:\n",
    "\n",
    "chicken_count = 99\n",
    "\n",
    "\n",
    "In Python:\n",
    "1. We don't use the `const` keyword to declare constants. By convention, constants in Python are typically written in all uppercase, but this isn't enforced by the language.\n",
    "2. We don't need to use any keyword (like `let` or `var` in JavaScript) to declare variables.\n",
    "3. The variable name uses snake_case, which is already consistent with Python naming conventions.\n",
    "\n",
    "If you want to emphasize that this is intended to be a constant (i.e., a value that shouldn't be changed), you could use the all-uppercase convention:\n",
    "\n",
    "\n",
    "CHICKEN_COUNT = 99\n",
    "\n",
    "\n",
    "Both versions are valid Python code, with the uppercase version signaling to other developers that this value is intended to be constant.\n",
    "```\n",
    "\n",
    "### Ruby test case results\n",
    "\n",
    "\n",
    "This is the result from our Ruby translation test case:\n",
    "\n",
    "```\n",
    "Certainly! Here's the Python translation of the provided Ruby code:\n",
    "\n",
    "\n",
    "class Person:\n",
    "    def __init__(self, name, age):\n",
    "        self.name = name\n",
    "        self.age = age\n",
    "\n",
    "    def birthday(self):\n",
    "        self.age += 1\n",
    "\n",
    "    def introduce(self):\n",
    "        print(f\"Hello, I'm {self.name} and I'm {self.age} years old.\")\n",
    "\n",
    "    @classmethod\n",
    "    def create_family(cls, members):\n",
    "        return [cls(name, age) for name, age in members]\n",
    "\n",
    "# Create a family\n",
    "family = Person.create_family([\n",
    "    [\"Alice\", 35],\n",
    "    [\"Bob\", 40],\n",
    "    [\"Charlie\", 12]\n",
    "])\n",
    "\n",
    "# Introduce family members\n",
    "for person in family:\n",
    "    person.introduce()\n",
    "\n",
    "# Celebrate Charlie's birthday\n",
    "charlie = next(person for person in family if person.name == \"Charlie\")\n",
    "charlie.birthday()\n",
    "charlie.introduce()\n",
    "\n",
    "\n",
    "Here are the key changes and explanations:\n",
    "\n",
    "1. In Python, we don't need to explicitly declare attribute readers. The attributes are accessible by default.\n",
    "\n",
    "2. The `initialize` method in Ruby becomes `__init__` in Python.\n",
    "\n",
    "3. The `birthday!` method is renamed to `birthday` in Python, as the exclamation mark is not typically used in Python method names.\n",
    "\n",
    "4. The `self.` prefix is used explicitly in Python method definitions and when accessing instance attributes.\n",
    "\n",
    "5. String interpolation uses f-strings in Python (e.g., `f\"Hello, I'm {self.name}\"`).\n",
    "\n",
    "6. The `create_family` class method is defined using the `@classmethod` decorator in Python.\n",
    "\n",
    "7. List comprehension is used instead of `map` for creating the family members.\n",
    "\n",
    "8. The `each` method with `&:introduce` is replaced by a simple `for` loop in Python.\n",
    "\n",
    "9. The `find` method is replaced with `next()` and a generator expression to find Charlie.\n",
    "\n",
    "This Python code maintains the functionality of the original Ruby code while adhering to Python's syntax and conventions.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scoring the prompts\n",
    "\n",
    "All of our current outputs are doing a decent job of translating the code, however there are some key problems:\n",
    "\n",
    "* We don't need the annoying preambles like \"Certainly! Here's the Python translation of the provided Ruby code.\" There are just a waste of output tokens! \n",
    "* The current format is really difficult to parse programmatically.  How would we write code to separately extract the translated Python code?\n",
    "* We don't need the long explanations at the end of the output.  For our use case, we just want the translated code.\n",
    "\n",
    "Let's go ahead and grade the outputs.  We'll assign each of them a value of 3 out of 5."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--- \n",
    "\n",
    "## Updating the prompt\n",
    "\n",
    "The next step is to make some changes to our prompt and run the evaluation again! Let's update our prompt to reflect the problems we previously identified.\n",
    "\n",
    "```\n",
    "You are a skilled programmer tasked with translating code from one programming language to Python. Your goal is to produce an accurate and idiomatic Python translation of the provided source code.\n",
    "\n",
    "Here is the source code to translate:\n",
    "\n",
    "<source_code>\n",
    "{{SOURCE_CODE}}\n",
    "</source_code>\n",
    "\n",
    "The source code is written in the following language:\n",
    "\n",
    "<source_language>\n",
    "{{SOURCE_LANGUAGE}}\n",
    "</source_language>\n",
    "\n",
    "Please translate this code to Python.\n",
    "Format your response as follows:\n",
    "\n",
    "<python_code>\n",
    "Your Python translation here\n",
    "</python_code>\n",
    "\n",
    "Only output the <python_code> tags without any other text content\n",
    "```\n"
   ]
  },
  {
   "attachments": {
    "updated_prompt.png": {
     "image/png": "[encoded data, 1393024 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we switch back to the \"Prompt\" view, we can update the prompt in the interface:\n",
    "\n",
    "![updated_prompt.png](attachment:updated_prompt.png)\n",
    "\n",
    "The green highlighted text showcases the addition we made to the prompt"
   ]
  },
  {
   "attachments": {
    "updated_response.png": {
     "image/png": "[encoded data, 41008 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we click the orange \"Run\" button to test our new prompt.  This is the new result:\n",
    "\n",
    "![updated_response.png](attachment:updated_response.png)\n",
    "\n",
    "This is exactly what we were hoping for! The response doesn't include a preamble or extended code explanation."
   ]
  },
  {
   "attachments": {
    "evaluate4.png": {
     "image/png": "[encoded data, 936668 characters]"
    },
    "evaluate5.png": {
     "image/png": "[encoded data, 714776 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Re-running the evaluation\n",
    "Next, we can switch back to the evaluation view:\n",
    "\n",
    "![evaluate4.png](attachment:evaluate4.png)\n",
    "\n",
    "Notice in the top left we see \"v2\" indicating that this is our second version of the prompt.  Let's click \"Run Remaining\" and take a look at the outputs for the other test cases:\n",
    "\n",
    "![evaluate5.png](attachment:evaluate5.png)\n",
    "\n"
   ]
  },
  {
   "attachments": {
    "evaluate6.png": {
     "image/png": "[encoded data, 970784 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The new outputs look great! They all skip any preamble and explanatory text and only include `<python_code>` tags that contain the translated Python code.  Let's go ahead and mark these as all scoring 5/5!\n",
    "\n",
    "![evaluate6.png](attachment:evaluate6.png)"
   ]
  },
  {
   "attachments": {
    "comparison.png": {
     "image/png": "[encoded data, 608148 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Comparing results\n",
    "\n",
    "Now that we've tried two different prompts, we can compare the results side by side. We can click on the \"+ Add Comparison\" button on the top right and select a previous version of our prompt (v1) to compare our v2 results with.  This will showcase the model outputs and human-graded scores for both prompts side by side:\n",
    "\n",
    "![comparison.png](attachment:comparison.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly, we can see that our v2 prompt works much better for our specific use-case!\n",
    "\n",
    "The Workbench and its evaluation tool are a great place to prototype prompts quickly and compare results side-by-side.  It's an ideal environment to start your evaluation journey, before moving to a more robust solution.  In upcoming lessons, we'll see how to automate larger-scale code-graded and model-graded lessons."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
