Tool calling

Lecture 21

Dr. Benjamin Soltoff

Cornell University
INFO 4940/5940 - Fall 2025

November 11, 2025

Announcements

Announcements

TODO

Learning objectives

• Identify the value of calling external tools from LLMs
• Define and call tools using {ellmer} and chatlas
• Annotate tools with metadata to improve user experience
• Implement tools that interact with Shiny apps

Tool calling

A robot wearing a tool belt, selects a tool from a wall of futuristic space tools. A sign above the wall reads “Tools”.

Recall: How do LLMs work?

1. You write some words

2. The LLM writes some more words

3. You use those words

On their own, can LLMs… access the internet? send an email? interact with the world?

Let’s try it

library(ellmer)

chat <- chat("openai/gpt-4.1-nano")

chat$chat("What's the weather like in Ithaca, NY?")
chat$chat("What day is it?")

import chatlas

chat = chatlas.Chat(model="openai/gpt-4.1-nano")
chat.chat("What's the weather like in Ithaca, NY?")
chat.chat("What day is it?")

Run these in the terminal.

Ask: how could we get the model to know these answers? Let’s start with the easiest one: what day is it?

Answer: we could just tell it in the system prompt.

What about the weather? The model can’t know that because it’s not connected to the internet.

Tools

a.k.a. functions, tool calling or function calling

• Bring real-time or up-to-date information to the model

• Let the model interact with the world

Tools are extra capabilities that you give to the LLM, like the ability to search the posit::conf(2025) schedule. These capabilities are things that the model can’t do on its own. The goal is let the model pull in real-time, or up-to-date, or high-quality information as needed, depending on the conversation.

They can also give the model new abilities to take action on behalf of the user.

Chatbot Systems

How do tool calls work?

What should I wear to campus tomorrow?

So how do we connect the LLM to the code that this data scientist has written (or could write) to give the model new abilities?

Human in the loop

TODO host and run on Posit Connect Cloud

👨‍💻 _demos/18_manual-tools/app.R

Wait… I can write code!

library(ellmer)

ellmer::create_tool_def(weathR::point_forecast, verbose = TRUE)

tool(
  weathR::point_forecast,
  name = "point_forecast",
  description = "Retrieve point forecast meteorological data for a given latitude and longitude.",
  arguments = list(
    lat = type_number("Latitude for the forecast location."),
    lon = type_number("Longitude for the forecast location."),
    timezone = type_string("The nominal timezone for the forecast. One of OlsonNames() or '-1' for 
local time. Defaults to '-1'.", required = FALSE),
    dir_numeric = type_boolean("TRUE for numeric wind directions, FALSE for character directions. 
Defaults to FALSE.", required = FALSE)
  )
)

Wait… I can write code!

get_weather <- tool(
  \(lat, lon) weathR::point_forecast(lat, lon),
  name = "point_forecast",
  description = "Get forecast data for a specific latitude and longitude.",
  arguments = list(
    lat = type_number("Latitude of the location."),
    lon = type_number("Longitude of the location.")
  )
)

Wait… I can write code!

get_weather(lat = 42.4397, lon = -76.4953)

Simple feature collection with 156 features and 8 fields
Geometry type: POINT
Dimension:     XY
Bounding box:  xmin: -76.4953 ymin: 42.4397 xmax: -76.4953 ymax: 42.4397
Geodetic CRS:  WGS 84
First 10 features:
                      time temp  dewpoint humidity p_rain wind_speed wind_dir
1  2025-11-10 11:00:00 EST   34  0.000000       92     57         10       NW
2  2025-11-10 12:00:00 EST   35  0.000000       89     52         10       NW
3  2025-11-10 13:00:00 EST   36  0.000000       85     47         10       NW
4  2025-11-10 14:00:00 EST   35 -1.111111       82     45         10       NW
5  2025-11-10 15:00:00 EST   35 -1.666667       78     44         12       NW
6  2025-11-10 16:00:00 EST   34 -1.666667       82     42         12       NW
7  2025-11-10 17:00:00 EST   31 -2.777778       85     36         10       NW
8  2025-11-10 18:00:00 EST   31 -3.333333       81     30         10        W
9  2025-11-10 19:00:00 EST   30 -3.888889       81     25          8        W
10 2025-11-10 20:00:00 EST   28 -4.444444       85     23          8        W
                        skies                 geometry
1         Chance Snow Showers POINT (-76.4953 42.4397)
2         Chance Snow Showers POINT (-76.4953 42.4397)
3         Chance Snow Showers POINT (-76.4953 42.4397)
4         Chance Snow Showers POINT (-76.4953 42.4397)
5         Chance Snow Showers POINT (-76.4953 42.4397)
6  Slight Chance Snow Showers POINT (-76.4953 42.4397)
7  Slight Chance Snow Showers POINT (-76.4953 42.4397)
8  Slight Chance Snow Showers POINT (-76.4953 42.4397)
9         Chance Snow Showers POINT (-76.4953 42.4397)
10        Chance Snow Showers POINT (-76.4953 42.4397)

Wait… I can write code!

chat <- chat_openai(model = "gpt-4.1-nano", echo = "output")

# Register the tool with the chatbot
chat$register_tool(get_weather)

chat$chat("What should I wear to class tomorrow in Ithaca, NY?")
◯ [tool call] point_forecast(lat = 42.4534, lon = -76.4862)
#> [{"time":"2025-11-10 11:00:00 EST","temp":33,"dewpoint":-1.1111,"humidity":89,"p_rain":57,"wind_speed":…
#> The weather forecast for tomorrow in Ithaca, NY indicates cold temperatures around the low 30s to high 20s 
#> Fahrenheit, with a high chance of snow showers and rain showers throughout the day. It will be windy with wind 
#> speeds around 8-16 mph.
#> 
#> Given these conditions, I recommend wearing warm and waterproof clothing. A good outfit would include:
#> - A warm, insulated waterproof coat
#> - Layers like thermal shirts and sweaters
#> - Waterproof gloves and a hat
#> - Waterproof boots to keep your feet dry and warm
#> - An umbrella might also be useful in case of rain
#> 
#> Make sure to stay warm and dry while attending your class!

Zooming out again, the conversation starts with a user sending a message to the chatbot.

In this case, the user asks the model whether they can go to the pool today. (I live in Atlanta, GA.)

How much does the model know about the weather in Atlanta, GA?

What’s the weather like today in Atlanta, GA?

I’m unable to provide real-time weather updates. For the latest weather forecast in Atlanta, GA, please check a trusted weather service such as the National Weather Service, Weather.com, or a weather app on your device.

Go ahead and guess

Sure! Since it’s July in Atlanta, GA, you can generally expect hot and humid weather. Temperatures might range from the mid-80s to mid-90s Fahrenheit (around 29-35°C). There could also be some afternoon thunderstorms or showers, which are common during summer months in Georgia. Keep a fan or air conditioning handy if you’re indoors, and stay hydrated!

Now imagine we’ve told the LLM that it has access to a tool called get_weather(). Knowing that it 1) needs to know the weather in Atlanta and 2) it doesn’t know the weather in Atlanta, GA; the model decides to use the weather tool.

It sends back a message that includes a tool call, basically instructions to call the get_weather() function with the argument 30303. Notice that the model does know that Atlanta, GA is in the 30303 ZIP code.

The programmer can then take that message from the LLM with the instructions to call the tool and they go off and run the code that that satisfies the tool request, in this case querying the weather API for the weather in 30303 because that has been a stable fact for a long time.

The weather API sends back some data:

• The forecast is mostly sunny
• High of 98ºF
• Low of 78ºF

The programmer sends that data back to the LLM. Note that it’s often in a pretty raw format, like JSON. Not the kind of thing you’d want to read directly unless you’re the kind of person who likes reading computer data formats.

The LLM doesn’t mind JSON though, so it reads the data and generates a response that incorporates the data from the tool call.

Recap: Tool definitions in R

tool_get_weather <- tool(
  tool_fn,
  description = "How and when to use the tool",
  arguments = list(
    .... = type_string(),
    .... = type_integer(),
    .... = type_enum(
      c("choice1", "choice2"),
      required = FALSE
    )
  )
)

Recap: Tool definitions in Python

def get_weather(lat, lon):
    return NWS.GetCurrentForecast(lat, lon)

Recap: Tool definitions in Python

def get_weather(lat: float, lon: float):
    return NWS.GetCurrentForecast(lat, lon)

Recap: Tool definitions in Python

def get_weather(lat: float, lon: float):
    """
    Get forecast data for a specific latitude and longitude.

    Parameters
    ----------
    lat : str
        Latitude of the location.
    lon : str
        Longitude of the location.
    """
    return NWS.GetCurrentForecast(lat, lon)

Recap: Registering tools in Python

import chatlas

chat = chatlas.ChatAuto("openai/gpt-4.1-nano")
chat.register_tool(get_weather)

chat.chat("What should I wear to class tomorrow in Ithaca, NY?")

⌨️ 20_quiz-game-2

1. I’ve given you a function that plays a sound when called.

2. Your job: teach the model to play sounds in the Quiz Show game we made last week.

06:00

Follow up question: why send a string back to the LLM that says the sound was played?

Tool UI

A robot is standing in front of a display case, pointing at a holographic interface showing various tools and options. The robot appears to be selecting a tool from the interface.

Tool Annotations

• Provide hints to clients about tool behavior

• Help humans and AI systems make informed decisions about tool usage

• All properties are hints only - not guaranteed to be accurate

• MCP Tool Annotations Spec

Key Properties

Property	Description
title	Human-readable tool name
readOnlyHint	Tool doesn’t modify environment
destructiveHint	May perform destructive updates
idempotentHint	Repeated calls with same arguments return same
openWorldHint	Interacts with external entities vs. closed domain

Tool Annotations in R

tool_get_weather <- tool(
  get_weather,
  # description, arguments, ...
  annotations = tool_annotations(
    title = "Get Weather",
    readOnlyHint = TRUE,
    icon = fontawesome::fa_i("cloud-sun")
  )
)

Tool Annotations in Python

chat.register_tool(
    get_weather,
    annotations={
        "title": "Get Weather",
        "readOnlyHint": True,
        "extra": {
            "icon": faicons.fa_i("cloud-sun"),
        },
    },
)

⌨️ 21_quiz-game-3

1. Add tool annotations to our play_sound tools.

2. Pick an icon from FontAwesome free icons

08:00

What if we want to decide title and icon dynamically in the tool?

ellmer::ContentToolResult

get_weather <- function(lat, lon) {
  weathR::point_forecast(lat, lon)
}

ellmer::ContentToolResult

get_weather <- function(lat, lon) {
  forecast <- weathR::point_forecast(lat, lon)

  ContentToolResult(
    forecast
  )
}

ellmer::ContentToolResult

get_weather <- function(lat, lon) {
  forecast <- weathR::point_forecast(lat, lon)

  ContentToolResult(
    forecast,
    extra = list(
      display = list(
        title = paste("Weather for", lat, lon),
        icon = fontawesome::fa_i("cloud-sun")
      )
    )
  )
}

ellmer::ContentToolResult

get_weather <- function(lat, lon, location_name) {
  forecast <- weathR::point_forecast(lat, lon)

  ContentToolResult(
    forecast,
    extra = list(
      display = list(
        title = paste("Weather for", location_name),
        icon = fontawesome::fa_i("cloud-sun")
      )
    )
  )
}

chatlas.ContentToolResult

def get_weather(lat: float, lon: float, location_name: str):
    forecast = NWS.GetCurrentForecast(lat, lon)

    return ContentToolResult(
        forecast,
        extra={
            "display": {
                "title": f"Weather for {location_name}",
                "icon": faicons.fa_i("cloud-sun"),
            }
        },
    )

⌨️ 22_quiz-game-4

1. Pick an icon and a title for each action in the Quiz show app.

2. Return a ContentToolResult from each tool that sets the icon and title.

3. Play the game again to see the icons and titles in action.

08:00

Tools in Shiny

• Define the tool function inside the server

• You can update reactive values in the tool function!

• You can read reactive values, if you isolate() reads. (Be careful!)

⌨️ 23_quiz-game-5

I’ve updated the app to show the score in value boxes.
(And turned off the sounds.)

1. In the system prompt, tell the model when to use update_score.

2. Read update_score(). It should update a reactive value in the app when called. Finish implementing it.

3. Complete the tool definition.

4. Add tool annotations (optional, but fun!)

08:00

Wrap-up

Recap

• Tool calling lets LLMs access real-time information and interact with the world
• We can define tools in R and Python using {ellmer} and chatlas
• Tool annotations improve the user experience by providing hints about tool behavior
• Tools can interact with Shiny apps to update reactive values

Acknowledgments

• Materials derived in part from Programming with LLMs and licensed under a Creative Commons Attribution 4.0 International (CC BY) License.