488 KiB
488 KiB
In [22]:
from dotenv import load_dotenv
from anthropic import Anthropic
load_dotenv()
client = Anthropic()Claude models
The Claude Python SDK supports multiple models, each with different capabilities and performance characteristics. This visualization compares cost vs. speed across Claude 3 and 3.5 models, showcasing the range in tradeoffs between cost and intelligence:
When choosing a model, there a few important factors to consider:
- The model's latency (how fast is it?)
- The model's capabilities (how smart is it?)
- The model's cost (how expensive is it?)
In [3]:
import time
def compare_model_speeds():
models = ["claude-3-5-sonnet-20240620","claude-3-opus-20240229", "claude-3-sonnet-20240229", "claude-3-haiku-20240307"]
task = "Explain the concept of photosynthesis in a concise paragraph."
for model in models:
start_time = time.time()
response = client.messages.create(
model=model,
max_tokens=500,
messages=[{"role": "user", "content": task}]
)
end_time = time.time()
execution_time = end_time - start_time
tokens = response.usage.output_tokens
time_per_token = execution_time/tokens
print(f"Model: {model}")
print(f"Response: {response.content[0].text}")
print(f"Generated Tokens: {tokens}")
print(f"Execution Time: {execution_time:.2f} seconds")
print(f"Time Per Token: {time_per_token:.2f} seconds\n")In [4]:
compare_model_speeds()Model: claude-3-5-sonnet-20240620 Response: Photosynthesis is the process by which plants, algae, and some bacteria convert light energy into chemical energy. These organisms use sunlight, water, and carbon dioxide to produce glucose (a type of sugar) and oxygen. The process occurs primarily in the chloroplasts of plant cells, where chlorophyll, a green pigment, absorbs light energy. This energy is then used to drive a series of chemical reactions that ultimately result in the production of glucose, which serves as food for the plant and can be stored for later use. Oxygen is released as a byproduct of this process, making photosynthesis crucial for maintaining Earth's atmosphere and supporting life on the planet. Generated Tokens: 146 Execution Time: 2.56 seconds Time Per Token: 0.02 seconds Model: claude-3-opus-20240229 Response: Photosynthesis is the process by which green plants and some other organisms use sunlight to synthesize nutrients from carbon dioxide and water. In plants, photosynthesis occurs mainly within the leaves. This process involves the green pigment chlorophyll and generates oxygen as a byproduct. The energy from light is used to convert water, carbon dioxide, and minerals into oxygen and energy-rich organic compounds such as glucose, which are used by the plant for growth and reproduction. Photosynthesis is essential for life on Earth as it provides food for plants, which in turn provide food for other organisms, and it also releases oxygen into the atmosphere, which most living creatures need to survive. Generated Tokens: 146 Execution Time: 7.32 seconds Time Per Token: 0.05 seconds Model: claude-3-sonnet-20240229 Response: Photosynthesis is the process by which plants and certain other organisms convert light energy from the sun into chemical energy in the form of glucose (sugar). During this process, carbon dioxide from the air and water from the soil are absorbed by the plant's leaves. With the aid of chlorophyll (a green pigment) and sunlight, these raw materials are transformed into oxygen, which is released into the atmosphere, and glucose, which serves as the plant's food source and building block for growth and development. Generated Tokens: 108 Execution Time: 2.64 seconds Time Per Token: 0.02 seconds Model: claude-3-haiku-20240307 Response: Photosynthesis is the process by which plants, algae, and some bacteria convert light energy from the sun into chemical energy in the form of carbohydrates. During this process, carbon dioxide and water, with the aid of chlorophyll, are converted into glucose and oxygen through a series of chemical reactions driven by the energy from sunlight. The glucose produced is used by the plant as a source of energy, while the oxygen is released into the atmosphere. This process is essential for the survival of most life on Earth, as it provides the primary source of food and oxygen for many organisms. Generated Tokens: 126 Execution Time: 1.09 seconds Time Per Token: 0.01 seconds
The above example is a simple illustration of model speed differences, but it's not a very rigorous demonstration. Here's a plot we generated by providing all 3 models the same input prompt 50 times and averaging the response time for each model. To ensure a "fair" comparison, we prompted the models to generate extremely long outputs, and then we cut off all model responses at exactly the same number of tokens using max_tokens (we cover this in the next lesson).
In [19]:
def compare_model_capabilities():
models = ["claude-3-5-sonnet-20240620", "claude-3-opus-20240229", "claude-3-sonnet-20240229", "claude-3-haiku-20240307"]
task = """
What is the geometric monthly fecal coliform mean of a distribution system with the following FC
counts: 24, 15, 7, 16, 31 and 23? The result will be inputted into a NPDES DMR, therefore, round
to the nearest whole number. Respond only with a number and nothing else.
"""
for model in models:
answers = []
for attempt in range(7):
response = client.messages.create(
model=model,
max_tokens=1000,
messages=[{"role": "user", "content": task}]
)
answers.append(response.content[0].text)
print(f"Model: {model}")
print(f"Answers: ", answers)In [20]:
compare_model_capabilities()Model: claude-3-5-sonnet-20240620 Answers: ['18', '18', '18', '18', '18', '18', '18'] Model: claude-3-opus-20240229 Answers: ['18', '18', '18', '18', '18', '18', '18'] Model: claude-3-sonnet-20240229 Answers: ['18', '17', '16', '17', '19', '18', '18'] Model: claude-3-haiku-20240307 Answers: ['17', '17', '18', '17', '17', '17', '18']