Retrieval Augmented Generation (RAG)

Lecture 20

Dr. Benjamin Soltoff

Cornell University
INFO 4940/5940 - Fall 2025

November 6, 2025

Announcements

Announcements

TODO

Learning objectives

TODO

⌨️ 15_coding-assistant

Instructions

1. Use Claude 3.5 Sonnet to write a function that gets the weather. The first time, use Claude on its own.

2. Do some basic research for Claude about how to use a specific package to get the weather.

3. How does Claude do with the same task now?

06:00

Augmented Generation

Retrieval-Augmented Generation (RAG)

RAG is a technique to automatically find and add relevant context to your prompts, rather than needing you to go out and copy-pasting it into the prompt yourself.

In a RAG system, you have a whole pile of documents. They’re not all immediately relevant to every question you might ask, but there’s a lot of useful information in this pile of document.

When you talk to the LLM, the RAG system can take your prompt, search for documents that seem related to your prompt and then add those documents to your prompt. Then the LLM sees both your prompt and these extra documents, and it uses both when generating a response.

In the end, you get a much better response that’s more likely to be correct and to include accurate information.

Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation (RAG)

How do we find relevant documents?

Answer: word vector embeddings → turn words into vectors

🤴 - 🧔‍♂️ + 💁‍♀️ = ❓

🤴 - 🧔‍♂️ = 👑
👑 + 💁‍♀️ = ❓

🤴 - 🧔‍♂️ = 👑
👑 + 💁‍♀️ = 👸

How do we actually find the most relevant documents?

Word vector embeddings are a way to represent words as dense, real-valued vectors in a continuous space.

The idea is to capture semantic and syntactic relationships between words such that geometrical relationships in the vector space correspond to linguistic relationships in the language.

Each word is mapped to a fixed-length vector with real-valued components. Distances and directions in this space encode similarity and relational structure. Words with similar meanings or usage tend to have similar vectors (small cosine distance, or Euclidean distance). Beyond meaning, embeddings can capture grammatical or functional similarities (e.g., verbs in present tense vs. past tense, or plural vs. singular forms).

OpenAI: text-embedding-3-small

text_embedding_3_small("dplyr::left_join")
#> [-0.0384574,  0.00796838,  0.04896307, ..., -0.01687562, 0.00051399,  0.01020856]

text_embedding_3_small("LEFT JOIN")
#> [-0.0114895,  0.01873610,  0.04436858, ...,  0.0055124, 0.01100459, -0.00588281],

text_embedding_3_small("suitcase")
#> [ 0.01323017, -0.00844115, -0.02530578, ..., -0.00054488, -0.0285338, -0.02933492]

Two ways that users encounter RAG

1. Every prompt you send gets passed through a RAG system and is augmented

2. The LLM can decide when to call the RAG system

In R…

In Python…

• LlamaIndex

• pinecone

• chromadb

⌨️ 16_rag

Instructions

Follow the steps in the 16_rag exercise, which are roughly:

1. First, you’ll create a vector database from a set of documents:

   • In R: R for Data Science (R4DS)
   • In Python: The Polars Cookbook

2. Test out the vector database with a simple query.

3. Attach a retrieval tool to a chat client and try it in a Shiny app.

15:00

Wrap-up

Recap

TODO