5.8 MiB
5.8 MiB
In [1]:
from dotenv import load_dotenv
from anthropic import Anthropic
#load environment variable
load_dotenv()
#automatically looks for an "ANTHROPIC_API_KEY" environment variable
client = Anthropic()In [3]:
response = client.messages.create(
messages=[
{
"role": "user",
"content": "Write me an essay about macaws and clay licks in the Amazon",
}
],
model="claude-3-haiku-20240307",
max_tokens=800,
temperature=0,
)
print("We have a response back!")
print("========================")
print(response.content[0].text)We have a response back! ======================== Here is an essay about macaws and clay licks in the Amazon: Macaws and Clay Licks in the Amazon Deep within the lush, verdant rainforests of the Amazon basin, a remarkable natural phenomenon takes place. Flashes of vibrant color dart through the canopy, as large, magnificent parrots known as macaws congregate at special sites called clay licks. These clay licks, or "collpas" as they are known locally, are essential to the survival and well-being of macaws and other Amazonian wildlife. Macaws are some of the most striking and iconic birds of the Amazon. With their vividly-hued plumage, hooked beaks, and long, tapered tails, these large parrots are a sight to behold as they soar through the treetops. The most well-known species include the scarlet macaw, the blue-and-gold macaw, and the green-winged macaw, each adorned in a stunning array of reds, blues, greens, and golds. These magnificent birds play a crucial role in the Amazon ecosystem, acting as important seed dispersers as they feed on a variety of fruits and nuts. However, macaws face a unique dietary challenge - they require essential minerals and nutrients that are often lacking in the fruits and seeds that make up their primary diet. This is where the clay licks come into play. Clay licks are exposed deposits of mineral-rich clay that attract hundreds, sometimes thousands, of macaws and other parrots, as well as other Amazonian wildlife such as tapirs, peccaries, and monkeys. The clay contains high concentrations of sodium, calcium, and other vital minerals that help macaws and other animals to regulate their digestive systems, neutralize toxins, and maintain overall health. Visiting a clay lick is a truly awe-inspiring experience. As the first rays of dawn break through the forest canopy, the air fills with the raucous squawks and screeches of macaws as they begin to gather at the lick. Slowly, the birds will descend from the treetops, cautiously approaching the clay bank and taking turns nibbling at the mineral-rich soil. The sheer number of these vibrant birds, their colors contrasting against the earthy tones of the clay, creates a mesmerizing natural spectacle. The importance of clay licks to the survival of macaws and other Amazonian wildlife cannot be overstated. These unique geological formations are essential feeding grounds that help sustain the delicate balance of the rainforest ecosystem. As deforestation and habitat loss continue to threaten the Amazon, the protection and preservation of these clay licks has become increasingly crucial to the long-term conservation of macaws and the broader biodiversity of this remarkable region.
This works fine, but it's important to remember that with this approach, we only get content back from the API once all of the content has been generated. Try running the above cell again, and you'll see that nothing is printed out until the entire response is printed all at once. This is fine in many situations, but it can lead to bad user experiences if you're building an application that forces a user to wait around for an entire response to be generated.
Enter streaming!
Streaming enables us to write applications that receive content as it is generated by the model, rather than having to wait for the entire response to be generated. This is how apps like claude.ai work. As the model generates responses, that content is streamed to a user's browser and displayed:
In [4]:
stream = client.messages.create(
messages=[
{
"role": "user",
"content": "Write me a 3 word sentence, without a preamble. Just give me 3 words",
}
],
model="claude-3-haiku-20240307",
max_tokens=100,
temperature=0,
stream=True,
)In [5]:
streamOut [5]:
<anthropic.Stream at 0x111e5ec10>
In [6]:
for event in stream:
print(event)MessageStartEvent(message=Message(id='msg_01EZHjA6qmf6y8VWMZSeBmN5', content=[], model='claude-3-haiku-20240307', role='assistant', stop_reason=None, stop_sequence=None, type='message', usage=Usage(input_tokens=30, output_tokens=2)), type='message_start') ContentBlockStartEvent(content_block=ContentBlock(text='', type='text'), index=0, type='content_block_start') ContentBlockDeltaEvent(delta=TextDelta(text='Cats', type='text_delta'), index=0, type='content_block_delta') ContentBlockDeltaEvent(delta=TextDelta(text=' me', type='text_delta'), index=0, type='content_block_delta') ContentBlockDeltaEvent(delta=TextDelta(text='ow lou', type='text_delta'), index=0, type='content_block_delta') ContentBlockDeltaEvent(delta=TextDelta(text='dly.', type='text_delta'), index=0, type='content_block_delta') ContentBlockStopEvent(index=0, type='content_block_stop') MessageDeltaEvent(delta=Delta(stop_reason='end_turn', stop_sequence=None), type='message_delta', usage=MessageDeltaUsage(output_tokens=10)) MessageStopEvent(type='message_stop')
Each stream contains a series of events in the following order:
- MessageStartEvent - A message with empty content
- Series of content blocks - Each of which contains:
- A ContentBlockStartEvent
- One or more ContentBlockDeltaEvents
- A ContentBlockStopEvent
- One or more MessageDeltaEvents which indicate top-level changes to the final message
- A final MessageStopEvent
In the above response, there was only a single content block. This diagram shows all the events associated with it:
In [7]:
stream = client.messages.create(
messages=[
{
"role": "user",
"content": "Write me a 3 word sentence, without a preamble. Just give me 3 words",
}
],
model="claude-3-haiku-20240307",
max_tokens=100,
temperature=0,
stream=True,
)
for event in stream:
if event.type == "content_block_delta":
print(event.delta.text)Cats me ow lou dly.
In [8]:
stream = client.messages.create(
messages=[
{
"role": "user",
"content": "Write me a 3 word sentence, without a preamble. Just give me 3 words",
}
],
model="claude-3-haiku-20240307",
max_tokens=100,
temperature=0,
stream=True,
)
for event in stream:
if event.type == "content_block_delta":
print(event.delta.text, flush=True, end="")Cats meow loudly.
In [9]:
stream = client.messages.create(
messages=[
{
"role": "user",
"content": "How do large language models work?",
}
],
model="claude-3-haiku-20240307",
max_tokens=1000,
temperature=0,
stream=True,
)
for event in stream:
if event.type == "content_block_delta":
print(event.delta.text, flush=True, end="")Large language models like myself work by using deep learning neural networks that are trained on massive amounts of text data. The key aspects are: 1. Neural network architecture - We use large, multi-layer neural networks with many parameters that can learn complex patterns in language. 2. Training data - We are trained on huge corpora of text from the internet, books, articles, and other sources. This allows us to learn the statistical patterns and structures of language. 3. Self-supervised learning - During training, the model learns to predict the next word in a sequence of text, without any explicit labels. This allows it to learn general language understanding. 4. Transfer learning - The knowledge gained during this pre-training can then be fine-tuned for specific tasks like question answering, summarization, translation, etc. 5. Attention mechanisms - Advanced models like transformers use attention to dynamically focus on the most relevant parts of the input when generating output. The end result is a model that can understand and generate human-like text by leveraging the patterns and structures it has learned from its training data. Of course, the details get quite technical, but that's the high-level overview of how large language models work. Let me know if you have any other questions!
As we've seen, the ContentBlockDeltaEvents contain the text content generated by the model. There are many other events, we need to care about those? Yes! Here's one quick example:
If we want to access information about our token usage, we'll need to look in two places:
MessageStartEventcontains our input(prompt) token usage informationMessageDeltaEventcontains information on how many output tokens were generated
Let's update our code from above to print out how many tokens are used in our prompt and how many tokens the model generated:
In [42]:
stream = client.messages.create(
messages=[
{
"role": "user",
"content": "How do large language models work?",
}
],
model="claude-3-haiku-20240307",
max_tokens=1000,
temperature=0,
stream=True,
)
for event in stream:
if event.type == "message_start":
input_tokens = event.message.usage.input_tokens
print("MESSAGE START EVENT", flush=True)
print(f"Input tokens used: {input_tokens}", flush=True)
print("========================")
elif event.type == "content_block_delta":
print(event.delta.text, flush=True, end="")
elif event.type == "message_delta":
output_tokens = event.usage.output_tokens
print("\n========================", flush=True)
print("MESSAGE DELTA EVENT", flush=True)
print(f"Output tokens used: {output_tokens}", flush=True)
MESSAGE START EVENT Input tokens used: 14 ======================== Large language models like myself work by using deep learning neural networks that are trained on massive amounts of text data. The key aspects are: 1. Neural network architecture - We use large, multi-layer neural networks with many parameters that can learn complex patterns in language. 2. Training data - We are trained on huge corpora of text from the internet, books, articles, and other sources. This allows us to learn the statistical patterns and structures of language. 3. Self-supervised learning - During training, the model learns to predict the next word in a sequence of text, without any explicit labels. This allows it to learn general language understanding. 4. Transfer learning - The knowledge gained during this pre-training can then be fine-tuned for specific tasks like question answering, summarization, translation, etc. 5. Attention mechanisms - Advanced models like transformers use attention to dynamically focus on the most relevant parts of the input when generating output. The end result is a model that can understand and generate human-like text, drawing upon its broad knowledge of language and the world. But the inner workings are highly complex and not fully understood. There's still a lot to learn about how these large language models work. ======================== MESSAGE DELTA EVENT Output tokens used: 258
In [4]:
import time
def measure_non_streaming_ttft():
start_time = time.time()
response = client.messages.create(
max_tokens=500,
messages=[
{
"role": "user",
"content": "Write mme a long essay explaining the history of the American Revolution",
}
],
temperature=0,
model="claude-3-haiku-20240307",
)
response_time = time.time() - start_time
print(f"Time to receive first token: {response_time:.3f} seconds")
print(f"Time to recieve complete response: {response_time:.3f} seconds")
print(f"Total tokens generated: {response.usage.output_tokens}")
print(response.content[0].text)In [50]:
measure_non_streaming_ttft()Time to receive first token: 4.194 seconds Time to recieve complete response: 4.194 seconds Total tokens generated: 500 Here is a long essay explaining the history of the American Revolution: The American Revolution was a pivotal event in the history of the United States, marking the country's transition from a collection of British colonies to an independent nation. The roots of the revolution can be traced back to the French and Indian War, which was fought between Britain and France from 1754 to 1763. This conflict, which was part of a larger global war, resulted in the British gaining control of much of North America, including the French colonies. However, the war also left Britain with a significant debt, which it sought to recoup by imposing a series of taxes and regulations on its American colonies. One of the first major events that led to the American Revolution was the Stamp Act, which was passed by the British Parliament in 1765. This act required all printed materials in the colonies, including newspapers, pamphlets, bills, legal documents, licenses, almanacs, dice, and playing cards, to carry an embossed revenue stamp. The colonists were outraged by this tax, which they saw as a violation of their rights as British subjects. They argued that they were not represented in the British Parliament and therefore should not be subject to taxation without their consent. In response to the Stamp Act, the colonists organized a series of protests and boycotts, which eventually led to the repeal of the act in 1766. However, this was just the beginning of a series of increasingly contentious conflicts between the colonies and the British government. In 1767, the Townshend Acts were passed, which imposed new taxes on a variety of goods imported to the colonies, including glass, paint, lead, paper, and tea. The colonists responded with further protests and boycotts, and in 1770, a group of protesters in Boston were fired upon by British soldiers, resulting in the deaths of five civilians in an event known as the Boston Massacre. This incident further inflamed tensions between the colonies and the British government, and in 1773, the British East India Company was granted a monopoly on the tea trade in the colonies. In response, a group of colonists in Boston, disguised as Native Americans, boarded a British ship and dumped hundreds of chests of tea into the harbor, an event known as the Boston Tea Party. The British government responded to the Boston Tea Party
In [57]:
def measure_streaming_ttft():
start_time = time.time()
stream = client.messages.create(
max_tokens=500,
messages=[
{
"role": "user",
"content": "Write mme a long essay explaining the history of the American Revolution",
}
],
temperature=0,
model="claude-3-haiku-20240307",
stream=True
)
have_received_first_token = False
for event in stream:
if event.type == "content_block_delta":
if not have_received_first_token:
ttft = time.time() - start_time
have_received_first_token = True
print(event.delta.text, flush=True, end="")
elif event.type == "message_delta":
output_tokens = event.usage.output_tokens
total_time = time.time() - start_time
print(f"\nTime to receive first token: {ttft:.3f} seconds", flush=True)
print(f"Time to recieve complete response: {total_time:.3f} seconds", flush=True)
print(f"Total tokens generated: {output_tokens}", flush=True)
In [58]:
measure_streaming_ttft()Here is a long essay explaining the history of the American Revolution: The American Revolution was a pivotal event in the history of the United States, marking the country's transition from a collection of British colonies to an independent nation. The roots of the revolution can be traced back to the French and Indian War, which was fought between Britain and France from 1754 to 1763. This conflict, which was part of a larger global war, resulted in the British gaining control of much of North America, including the French colonies. However, the war also left Britain with a significant debt, which it sought to recoup by imposing a series of taxes and regulations on its American colonies. One of the first major events that led to the American Revolution was the Stamp Act, which was passed by the British Parliament in 1765. This act required all printed materials in the colonies, including newspapers, pamphlets, bills, legal documents, licenses, almanacs, dice, and playing cards, to carry an embossed revenue stamp. The colonists were outraged by this tax, which they saw as a violation of their rights as British subjects. They argued that they were not represented in the British Parliament and therefore should not be subject to taxation without their consent. In response to the Stamp Act, the colonists organized a series of protests and boycotts, which eventually led to the repeal of the act in 1766. However, this was just the beginning of a series of increasingly contentious conflicts between the colonies and the British government. In 1767, the Townshend Acts were passed, which imposed new taxes on a variety of goods imported to the colonies, including glass, paint, lead, paper, and tea. The colonists responded with further protests and boycotts, and in 1770, a group of protesters in Boston were fired upon by British soldiers, resulting in the deaths of five civilians in an event known as the Boston Massacre. This incident further inflamed tensions between the colonies and the British government, and in 1773, the British East India Company was granted a monopoly on the tea trade in the colonies. In response, a group of colonists in Boston, disguised as Native Americans, boarded a British ship and dumped hundreds of chests of tea into the harbor, an event known as the Boston Tea Party. The British government responded to the Boston Tea Party Time to receive first token: 0.492 seconds Time to recieve complete response: 4.274 seconds Total tokens generated: 500
In [2]:
def compare_ttft():
def measure_streaming_ttft():
start_time = time.time()
stream = client.messages.create(
max_tokens=1000,
messages=[
{
"role": "user",
"content": "Write mme a very very long essay explaining the history of the American Revolution",
}
],
temperature=0,
model="claude-3-opus-20240229",
stream=True
)
have_received_first_token = False
for event in stream:
if event.type == "content_block_delta":
if not have_received_first_token:
ttft = time.time() - start_time
have_received_first_token = True
elif event.type == "message_delta":
output_tokens = event.usage.output_tokens
total_time = time.time() - start_time
return (ttft, output_tokens)
def measure_non_streaming_ttft():
start_time = time.time()
response = client.messages.create(
max_tokens=1000,
messages=[
{
"role": "user",
"content": "Write mme a very very long essay explaining the history of the American Revolution",
}
],
temperature=0,
model="claude-3-opus-20240229"
)
ttft = time.time() - start_time
return (ttft, response.usage.output_tokens)
streaming_ttft, streaming_tokens = measure_streaming_ttft()
non_streaming_ttft, non_streaming_tokens = measure_non_streaming_ttft()
print("OPUS STREAMING")
print(f"Time to first token: {streaming_ttft}")
print(f"Tokens generated: {streaming_tokens}")
print("#########################################################")
print("OPUS NON STREAMING")
print(f"Time to first token: {non_streaming_ttft}")
print(f"Tokens generated: {non_streaming_tokens}")
In [5]:
# DO NOT RUN THIS! It takes over a minute to run and generates around 2000 tokens with Opus!
compare_ttft()OPUS STREAMING Time to first token: 1.8863098621368408 Tokens generated: 997 ######################################################### OPUS NON STREAMING Time to first token: 47.03177309036255 Tokens generated: 998
In [79]:
from anthropic import AsyncAnthropic
client = AsyncAnthropic()
async def streaming_with_helpers():
async with client.messages.stream(
max_tokens=1024,
messages=[
{
"role": "user",
"content": "Write me sonnet about orchids",
}
],
model="claude-3-opus-20240229",
) as stream:
async for text in stream.text_stream:
print(text, end="", flush=True)
final_message = await stream.get_final_message()
print("\n\nSTREAMING IS DONE. HERE IS THE FINAL ACCUMULATED MESSAGE: ")
print(final_message.to_json())
await streaming_with_helpers()In gardens fair, where beauty reigns supreme,
The orchid stands, a queen among the blooms,
Her delicate petals, like a lovely dream,
Adorned in nature's most exquisite plumes.
With colors ranging from pure white to bold,
And patterns intricate, a work of art,
Each blossom tells a story, bright and old,
Of evolution's path, a world apart.
From rainforests dense to mountain peaks so high,
The orchid thrives, a testament to grace,
Her beauty captivates the wandering eye,
And in our hearts, she finds a cherished place.
Oh, orchid fair, your splendor knows no bounds,
Forever in our gardens and hearts you'll be found.
STREAMING IS DONE. HERE IS THE FINAL ACCUMULATED MESSAGE:
{
"id": "msg_018x1nZcs3sfq15zKaS4z4gD",
"content": [
{
"text": "In gardens fair, where beauty reigns supreme,\nThe orchid stands, a queen among the blooms,\nHer delicate petals, like a lovely dream,\nAdorned in nature's most exquisite plumes.\n\nWith colors ranging from pure white to bold,\nAnd patterns intricate, a work of art,\nEach blossom tells a story, bright and old,\nOf evolution's path, a world apart.\n\nFrom rainforests dense to mountain peaks so high,\nThe orchid thrives, a testament to grace,\nHer beauty captivates the wandering eye,\nAnd in our hearts, she finds a cherished place.\n\nOh, orchid fair, your splendor knows no bounds,\nForever in our gardens and hearts you'll be found.",
"type": "text"
}
],
"model": "claude-3-opus-20240229",
"role": "assistant",
"stop_reason": "end_turn",
"stop_sequence": null,
"type": "message",
"usage": {
"input_tokens": 14,
"output_tokens": 171
}
}
In [94]:
from anthropic import AsyncAnthropic, AsyncMessageStream
client = AsyncAnthropic()
green = '\033[32m'
reset = '\033[0m'
class MyStream(AsyncMessageStream):
async def on_text(self, text, snapshot):
# This runs only on text delta stream messages
print(green + text + reset, flush=True) #model generated content is printed in green
async def on_stream_event(self, event):
# This runs on any stream event
print("on_event fired:", event.type)
async def streaming_events_demo():
async with client.messages.stream(
max_tokens=1024,
messages=[
{
"role": "user",
"content": "Generate a 5-word poem",
}
],
model="claude-3-opus-20240229",
event_handler=MyStream,
) as stream:
# Get the final accumulated message, after the stream is exhausted
message = await stream.get_final_message()
print("accumulated final message: ", message.to_json())
await streaming_events_demo()on_event fired: message_start
on_event fired: content_block_start
on_event fired: content_block_delta
[32mWhis[0m
on_event fired: content_block_delta
[32mpers[0m
on_event fired: content_block_delta
[32m dance[0m
on_event fired: content_block_delta
[32m,[0m
on_event fired: content_block_delta
[32m secrets[0m
on_event fired: content_block_delta
[32m unf[0m
on_event fired: content_block_delta
[32mol[0m
on_event fired: content_block_delta
[32md,[0m
on_event fired: content_block_delta
[32m love[0m
on_event fired: content_block_delta
[32m.[0m
on_event fired: content_block_stop
on_event fired: message_delta
on_event fired: message_stop
accumulated final message: {
"id": "msg_014G44rr3M14DzadHXPn9Xaj",
"content": [
{
"text": "Whispers dance, secrets unfold, love.",
"type": "text"
}
],
"model": "claude-3-opus-20240229",
"role": "assistant",
"stop_reason": "end_turn",
"stop_sequence": null,
"type": "message",
"usage": {
"input_tokens": 14,
"output_tokens": 14
}
}
Exercise
Write a simple Claude chatbot that uses streaming. The following gif illustrates how it should work. Please note that the color-coding of the output is completely optional and mostly helps to make the gif readable/watchable:
In [ ]:
from anthropic import Anthropic
# Initialize the Anthropic client
client = Anthropic()
# ANSI color codes
BLUE = "\033[94m"
GREEN = "\033[92m"
RESET = "\033[0m"
def chat_with_claude():
print("Welcome to the Claude Chatbot!")
print("Type 'quit' to exit the chat.")
conversation = []
while True:
user_input = input(f"{BLUE}You: {RESET}")
if user_input.lower() == 'quit':
print("Goodbye!")
break
conversation.append({"role": "user", "content": user_input})
print(f"{GREEN}Claude: {RESET}", end="", flush=True)
stream = client.messages.create(
model="claude-3-haiku-20240307",
max_tokens=1000,
messages=conversation,
stream=True
)
assistant_response = ""
for chunk in stream:
if chunk.type == "content_block_delta":
content = chunk.delta.text
print(f"{GREEN}{content}{RESET}", end="", flush=True)
assistant_response += content
print() # New line after the complete response
conversation.append({"role": "assistant", "content": assistant_response})
if __name__ == "__main__":
chat_with_claude()