courses/anthropic_api_fundamentals/04_vision.ipynb

Prompting with images

Vision capabilities

When providing images to Claude, we have to write an image content block. Here's an example:

message = {
        "role": "user",
        "content": [
            {
                    "image": {
                        "format": 'png',
                        "source": {
                            "bytes": image
                        }
                    }
            }
        ]
    }

This diagram explains the important pieces of information that are required when providing Claude with an image:

The content in our message is set to a dictionary containing the following properties:

• format - the image media type. We currently support image/jpeg, image/png, image/gif, and image/webp media types.
• image - the actual image data itself

Image only prompting

Most often, we'll want to provide some text alongside images in our prompt, but it's perfectly acceptable to only provide an image. Let's try it! We've included a handful of images for this lesson in the prompting_images folder. Let's start by looking at one of these images using Python:

In [ ]:

from IPython.display import Image
Image(filename='./prompting_images/uh_oh.png') 

Wikimedia Commons, CC-BY-SA

Now, let's work on providing this image to Claude. The first step is to get the base64 encoded image data string that we send to the model. The code might look a bit complex, but it boils down to the following steps:

1. Open the file in "read binary" mode.
2. Read the entire binary contents of the file as a bytes object.

In [ ]:

# opens the image file in "read binary" mode
with open("./prompting_images/uh_oh.png", "rb") as image_file:

    #reads the contents of the image as a bytes object
    binary_data = image_file.read() 

We can take a look at the resulting base64_string variable, but it's not going to make a lot of sense to us humans. Let's read the first 100 characters:

In [ ]:

binary_data

Now that we have our image data in a string, the next step is to properly format our messages list that we'll send to Claude:

In [ ]:

message = {
        "role": "user",
        "content": [
            {
                    "image": {
                        "format": 'png',
                        "source": {
                            "bytes": binary_data
                        }
                    }
            }
        ]
    }

The final step is to send our messages list off to Claude and see what kind of response we get!

In [ ]:

import boto3

bedrock_client = boto3.client(service_name='bedrock-runtime', region_name="us-west-2")
model_id = "anthropic.claude-3-5-sonnet-20241022-v2:0"

messages = [{
        "role": "user",
        "content": [
            {
                    "image": {
                        "format": 'png',
                        "source": {
                            "bytes": binary_data
                        }
                    }
            }
        ]
    }]

# Send the message.
response = bedrock_client.converse(
    modelId=model_id,
    messages=messages,
)

In [ ]:

response

Claude starts describing the image, because we didn't provide any other explicit instructions.

Image and text prompts

Now let's try sending a prompt that includes both an image AND text. All we need to do is add a second block to the user's message. This block will be a simple text block.

In [ ]:

messages = [
    {
        "role": "user",
        "content": [
            
            {
                    "image": {
                        "format": 'png',
                        "source": {
                            "bytes": binary_data
                        }
                    }
            },
            {
                "text": "What could this person have done to prevent this?"
            },
        ]
    }
]

 

Let's send a request to Claude and see what happens:

In [ ]:

response = bedrock_client.converse(
    modelId=model_id,
    messages=messages,
)

response

Multiple images

We can provide multiple images to Claude by adding multiple image blocks to our content of a user message. Here's an example that includes multiple images:

messages = [
    {
        "role": "user",
        "content": [
             {
                    "image": {
                        "format": image1_media_type,
                        "source": {
                            "bytes": image1_data
                        }
                    }
            },
              {
                    "image": {
                        "format": image2_media_type,
                        "source": {
                            "bytes": image2_data
                        }
                    }
            },
              {
                    "image": {
                        "format": image3_media_type,
                        "source": {
                            "bytes": image3_data
                        }
                    }
            },
            {"text": "How are these images different?"},
        ],
    }
]
 

Building an image helper

As you work with images, especially in dynamic scripts, it can get annoying to create the image content blocks by hand. Let's write a little helper function that will generate appropriately formatted image blocks.

In [ ]:

import mimetypes

def create_image_message(image_path):
    # Open the image file in "read binary" mode
    with open(image_path, "rb") as image_file:
        # Read the contents of the image as a bytes object
        binary_data = image_file.read()
    
    # Get the MIME type of the image based on its file extension
    mime_type, _ = mimetypes.guess_type(image_path)

    sub_type = mime_type.split("/")[-1] 
    
    # Create the image block
    image_block = {
        "image": {
            "format": sub_type,
            "source": {
                "bytes": binary_data
            }
        }
    }
    
    
    return image_block

The above function takes an image path and returns a dictionary that is ready to be included in a message to Claude. It even does some logic to automatically determine the mime type of the image.

Let's try working with a new image:

In [ ]:

Image("./prompting_images/animal1.png")

Using our new image block helper function, let's send a request to Claude:

In [ ]:

messages = [
    {
        "role": "user",
        "content": [
            create_image_message("./prompting_images/animal1.png")
        ]
    }
]

bedrock_client = boto3.client(service_name='bedrock-runtime', region_name="us-west-2")
model_id = "anthropic.claude-3-5-sonnet-20241022-v2:0"

# Send the message.
response = bedrock_client.converse(
    modelId=model_id,
    messages=messages,
)

response

Let's try an example that combines text and image in the prompt:

In [ ]:

messages = [
    {
        "role": "user",
        "content": [
            create_image_message("./prompting_images/animal1.png"),
            {"text": "Where might I find this animal in the world?"}
        ]
    }
]

response = bedrock_client.converse(
    modelId=model_id,
    messages=messages,
)

response

Now let's try providing multiple images to Claude. We have 3 different animal images:

In [ ]:

from IPython.display import display
display(Image("./prompting_images/animal1.png", width=300))

In [ ]:

display(Image("./prompting_images/animal2.png", width=300))

In [ ]:

display(Image("./prompting_images/animal3.png", width=300))

Let's try passing all 3 images to Claude in a single message along with a text prompt asking, "What are these animals?"

In [ ]:

messages = [
    {
        "role": "user",
        "content": [
            create_image_message('./prompting_images/animal1.png'),
            create_image_message('./prompting_images/animal2.png'),
            create_image_message('./prompting_images/animal3.png'),
            {"text": "what are these animals?"}
        ]
    }
]

response = bedrock_client.converse(
    modelId=model_id,
    messages=messages,
)

response

This works great! However, it's important to note that if we try this with a slightly less-capable Claude model like Claude 3 Haiku, we may get worse results:

Much better!

Working with non-local images (images from URL)

Sometimes you may need to provide Claude with images that you do not have locally. There are many ways of doing this, but they all boil down to the same recipe:

• Get the image data using some sort of request library
• Encode the binary data of the image content using Base64 encoding
• Decode the encoded data from bytes to a string using UTF-8 encoding

We'll use httpx to request the image data from a URL. The URL in the example below is an image of a church with the Northern Lights in the sky above it.

In [ ]:

import httpx

image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/f/fa/Church_of_light.jpg/1599px-Church_of_light.jpg"
image_media_type = "jpeg"
image_data = httpx.get(image_url).content


messages=[
        {
            "role": "user",
            "content": [
                 {
                    "image": {
                        "format": image_media_type,
                        "source": {
                            "bytes": image_data
                        }
                    }
            },
            ],
        }
    ]


response = bedrock_client.converse(
    modelId=model_id,
    messages=messages,
)

response

Just as we did earlier, we can define a helper function to generate image blocks from URLs. Below is a very lightweight implementation of a function that expects a URL and does the following:

• uses httpx to request the image data
• determines the MIME type using very simple string manipulation. It takes the content after the last '.' character, which is not a bulletproof solution
• encodes the image data using base46 encoding and decodes the bytes into a utf-8 string
• returns a properly formatted image block, ready to go into a Claude prompt!

If we were to call get_image_dict_from_url("https://somewebsite.com/cat.png") it would return the following dictionary:

{
    "type": "image",
    "source": {
        "type": "base64",
        "media_type": "image/png",
        "data": <actual image data>
    },
}

In [ ]:

def get_image_dict_from_url(image_url):
    # Send a GET request to the image URL and retrieve the content
    response = httpx.get(image_url)
    image_content = response.content

    # Determine the media type of the image based on the URL extension
    # This is not a foolproof approach, but it generally works
    image_extension = image_url.split(".")[-1].lower()
    if image_extension == "jpg" or image_extension == "jpeg":
        image_media_type = "jpeg"
    elif image_extension == "png":
        image_media_type = "png"
    elif image_extension == "gif":
        image_media_type = "gif"
    else:
        raise ValueError("Unsupported image format")

    # Encode the image content using base6

    # Create the dictionary in the proper image block shape:
    image_dict = {
                    "image": {
                        "format": image_media_type,
                        "source": {
                            "bytes": image_content
                        }
                    }
            }
    return image_dict

Now let's try it! In the following example, we are using two image URL:

• A PNG of a firetruck
• A JPG of an emergency response helicopter

We'll pass both to Claude, alongside a text prompt asking, "What do these images have in common?"

In [ ]:

url1 = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/d0/Rincon_fire_truck.png/1600px-Rincon_fire_truck.png"
url2 = "https://upload.wikimedia.org/wikipedia/commons/thumb/b/bb/Ornge_C-GYNP.jpg/1600px-Ornge_C-GYNP.jpg"

messages=[
        {
            "role": "user",
            "content": [
                {"text": "Image 1:"},
                get_image_dict_from_url(url1),
                {"text": "Image 2:"},
                get_image_dict_from_url(url2),
                {"text": "What do these images have in common?"}
            ],
        }
    ]


response = bedrock_client.converse(
    modelId=model_id,
    messages=messages,
)

response

Claude successfully identifies that both images are of emergency response vehicles! More importantly, we've now seen how to provide Claude with images downloaded from a URL.

Vision prompting tips

Be specific

Just as with plain text prompts, we can get better results from Claude by writing specific and detailed multimodal prompts. Let's take a look at an example.

Here's an image of a group of friends. There are 8 people in the image, but 2 of them are cut off by the bounds of the image.

In [ ]:

from IPython.display import Image
Image(filename='./prompting_images/people.png') 

If we simply ask Claude, "how many people are in this image?" we'll likely get a response saying there are 7 people:

In [ ]:

messages=[
    {
        "role": "user",
        "content": [
            create_image_message("./prompting_images/people.png"),
            {"text": "How many people are in this image?"}
        ],
    }
]


response = bedrock_client.converse(
    modelId=model_id,
    messages=messages,
)

response

If we instead employ some basic prompt engineering techniques like telling Claude to think step by step, that it's an expert in counting people, and that it should pay attention to "partial" people that may be cut off in the image, we will get better results:

In [ ]:

messages=[
    {
        "role": "user",
        "content": [
            create_image_message("./prompting_images/people.png"),
            {"text": "You have perfect vision and pay great attention to detail which makes you an expert at counting objects in images. How many people are in this picture? Some of the people may be partially obscured or cut off in the image or may only have an arm visible. Please count people even if you can only see a single body part. Before providing the answer in <answer> tags, think step by step in <thinking> tags and analyze every part of the image."}
        ],
    }
]


response = bedrock_client.converse(
    modelId=model_id,
    messages=messages,
)

response

Using examples

Including examples in your prompts can help improve Claude's response quality in both text and image input prompts.

To demonstrate this, we're going to use a series of images from a slideshow presentation. Our goal is to get Claude to generate a JSON description of a slide's content. Take a look at this first image:

In [ ]:

from IPython.display import display
display(Image("./prompting_images/slide1.png", width=800))

Our goal is to get Claude to generate a JSON-formatted response that includes the slide's background color, title, body text, and description of the image. The JSON for the above image might look like this:

{
    "background": "#F2E0BD",
    "title": "Haiku",
    "body": "Our most powerful model, delivering state-of-the-art performance on highly complex tasks and demonstrating fluency and human-like understanding",
    "image": "The image shows a simple line drawing of a human head in profile view, facing to the right. The head is depicted using thick black lines against a pale yellow background. Inside the outline of the head, there appears to be a white, spoked wheel or starburst pattern, suggesting a visualization of mental activity or thought processes. The overall style is minimalist and symbolic rather than realistic."
}

This is a great use-case for including examples in our prompt to coach Claude on exactly the type of response we want it to generate. For reference, here are two other slide images:

In [ ]:

display(Image("./prompting_images/slide2.png", width=800))

In [ ]:

display(Image("./prompting_images/slide3.png", width=800))

To do this, we'll take advantage of the conversation message format to provide Claude with an example of a previous input and corresponding output:

In [ ]:

def generate_slide_json(image_path):

    slide1_response = """{
        "background": "#F2E0BD",
        "title": "Haiku",
        "body": "Our most powerful model, delivering state-of-the-art performance on highly complex tasks and demonstrating fluency and human-like understanding",
        "image": "The image shows a simple line drawing of a human head in profile view, facing to the right. The head is depicted using thick black lines against a pale yellow background. Inside the outline of the head, there appears to be a white, spoked wheel or starburst pattern, suggesting a visualization of mental activity or thought processes. The overall style is minimalist and symbolic rather than realistic."
    }"""

    messages = [
        {
            "role": "user",
            "content": [
                create_image_message("./prompting_images/slide1.png"),
                {"text": "Generate a JSON representation of this slide.  It should include the background color, title, body text, and image description"}
            ],
        },
        {
            "role": "assistant",
            "content": slide1_response
        },
        {
            "role": "user",
            "content": [
                create_image_message(image_path),
                {"text": "Generate a JSON representation of this slide.  It should include the background color, title, body text, and image description"}
            ],
        },
    ]

response = bedrock_client.converse(
    modelId=model_id,
    messages=messages,
)

response

In [ ]:

display(Image("./prompting_images/slide2.png", width=800))
generate_slide_json("./prompting_images/slide2.png")

In [ ]:

display(Image("./prompting_images/slide3.png", width=800))
generate_slide_json("./prompting_images/slide3.png")

---

Exercise

For this exercise, we'd like you to use Claude to transcribe and summarize an Anthropic research paper. In the images folder, you'll find research_paper folder that contains 5 screenshots of a research paper. To help you out, we've provided all 5 image URLs in a list:

In [ ]:

research_paper_pages = [
    "./images/research_paper/page1.png",
    "./images/research_paper/page2.png",
    "./images/research_paper/page3.png",
    "./images/research_paper/page4.png",
    "./images/research_paper/page5.png"
    ]

Let's take a look at the first image:

In [ ]:

Image(research_paper_pages[0])

Your task

Your task is to use Claude to do the following:

• Transcribe the text in each of the 5 research paper images
• Combine the text from each image into one large transcription
• Provide the entire transription to Claude and ask for a non-technical summary of the entire paper.

An example output might look something like this:

This paper explores a new type of attack on large language models (LLMs) like ChatGPT, called "Many-shot Jailbreaking" (MSJ). As LLMs have recently gained the ability to process much longer inputs, this attack takes advantage of that by showing the AI hundreds of examples of harmful or undesirable behavior. The researchers found that this method becomes increasingly effective as more examples are given, following a predictable pattern.

The study tested MSJ on several popular AI models and found it could make them produce harmful content they were originally designed to avoid. This includes things like violent or sexual content, deception, and discrimination. The researchers also discovered that larger AI models tend to be more susceptible to this type of attack, which is concerning as AI technology continues to advance.

The paper also looked at potential ways to defend against MSJ attacks. They found that current methods of training AI to be safe and ethical (like supervised learning and reinforcement learning) can help somewhat, but don't fully solve the problem. The researchers suggest that new approaches may be needed to make AI models truly resistant to these kinds of attacks. They emphasize the importance of continued research in this area to ensure AI systems remain safe and reliable as they become more powerful and widely used.

To get the best results, we advise asking Claude to summarize each page in a separate request rather than providing all 5 images and asking for a single transcription of the entire paper.

---