AE 16: Classifying legislative texts with LLMs
Suggested answers
Application exercise
Answers
Load packages
Warning
If you have not already completed the pre-class preparation to set up your API key, do this now.
Classifying legislative policy attention
# import data
leg <- read_parquet(file = "data/legislation.parquet")Our goal is to classify these legislative bill descriptions into one of the Comparative Agendas Project’s major policy categories.
# A tibble: 20 × 2
policy policy_lab
<dbl> <fct>
1 1 Macroeconomics
2 2 Civil rights, minority issues, civil liberties
3 3 Health
4 4 Agriculture
5 5 Labor and employment
6 6 Education
7 7 Environment
8 8 Energy
9 9 Immigration
10 10 Transportation
11 12 Law, crime, family issues
12 13 Social welfare
13 14 Community development and housing issues
14 15 Banking, finance, and domestic commerce
15 16 Defense
16 17 Space, technology, and communications
17 18 Foreign trade
18 19 International affairs and foreign aid
19 20 Government operations
20 21 Public lands and water management We previously attempted to complete this task by training shallow and deep learning models to predict the policy topic. In this AE we will use a large language model (LLM) to classify these descriptions into one of the policy categories.
Sub-sample and partition data
We start by ensuring we only classify distinct legislative descriptions. This eliminates duplicates and ensures a more efficient and reliable classification process.
Note
For our purposes today (and to avoid excessive costs), we will only classify a small sub-sample of the data. In a real-world scenario, you would start with a small sub-sample but eventually label all the observations.
set.seed(521)
leg_lite <- leg |>
distinct(description, policy) |>
slice_sample(n = 500L)
leg_lite# A tibble: 500 × 2
description policy
<chr> <dbl>
1 A bill to designate certain Federal land within the Monongahela Natio… 21
2 A bill to establish a Judicial Service Commission 12
3 A bill to provide more effective price, pro. duction adjustment, and … 4
4 To direct the Administrator of General Services to establish a small … 15
5 To cut off Federal benefits for conviction of rioting and prohibiting… 12
6 A bill to bring employees of agricultural stabilization and conservat… 20
7 A bill relating to the employment by certain Indians of legal counsel. 21
8 A bill to amend the Internet Tax Freedom Act to extend the moratorium… 17
9 To amend the Library Services and Construction Act, and for other pur… 6
10 To prohibit discrimination on account of sex in the payment of wages … 2
# ℹ 490 more rows# distribution of sample
leg_lite |>
count(policy)# A tibble: 20 × 2
policy n
<dbl> <int>
1 1 20
2 2 13
3 3 30
4 4 24
5 5 11
6 6 20
7 7 22
8 8 18
9 9 10
10 10 31
11 12 30
12 13 28
13 14 18
14 15 25
15 16 36
16 17 8
17 18 25
18 19 15
19 20 50
20 21 66Next, we divide the bills into training/test sets to evaluate the LLMs’ performance.
Construct an initial classifier
Create a basic classifier function
Demonstration: To classify this data, we write a custom function that wraps llm_message(). This function sends each legislative description to an LLM and prompts it to assign one of the pre-defined policy codes.
classify_policy <- function(description) {
# output what the model is currently doing to the console
str_glue("Classifying: {description}\n") |> message()
# generate the prompt
prompt <- str_glue("
Classify this legislative description from the U.S. Congress: {description}
Pick one of the following numerical codes from this list.
Respond only with the code!
1 = Macroeconomics
2 = Civil rights, minority issues, civil liberties
3 = Health
4 = Agriculture
5 = Labor and employment
6 = Education
7 = Environment
8 = Energy
9 = Immigration
10 = Transportation
12 = Law, crime, family issues
13 = Social welfare
14 = Community development and housing issues
15 = Banking, finance, and domestic commerce
16 = Defense
17 = Space, technology, and communications
18 = Foreign trade
19 = International affairs and foreign aid
20 = Government operations
21 = Public lands and water management")
# list of valid codes as strings
valid_codes <- as.character(cap_codes$policy)
# attempt to classify the description
classification <- tryCatch(
{
# get the assistant's reply
assistant_reply <- llm_message(prompt) |>
openai(.model = "gpt-4o-mini", .temperature = 0) |>
last_reply() |>
str_squish()
# validate the assistant's reply
if (assistant_reply %in% valid_codes) {
as.integer(assistant_reply)
} else {
# if the reply is not a valid code, set code 98
98L
}
},
error = function(e) {
# if there's an error with the model, set code 97
97L
}
)
# output a tibble
return(
tibble(
description = description,
.pred = classification
)
)
}
How this function works
Prompt generation: The function creates a detailed prompt that instructs the model to classify the legislation into one of the CAP codes. This structured prompt ensures the model provides clear and accurate outputs. We use
str_glue()to add the function input (i.e. the legislative description) into the prompt with{description}.-
Calling the LLM: The model (in this case, GPT-4o-mini) is called using the
openai()function with the.temperatureparameter set to 0, ensuring deterministic (non-random) output. The model’s reply is retrieved and cleaned usingstr_squish()to remove unnecessary spaces:llm_message(prompt) |> openai(.model = "gpt-4o-mini", .temperature = 0) |> last_reply() |> str_squish() Error handling: The function uses
tryCatchto handle any errors during the classification process. If the API call fails, the function returns CAP code97to flag the issue.Validation of model output: The function checks whether the model’s response is a valid CAP code. If the response is invalid, it assigns code
98. This validation ensures that any unexpected outputs are handled appropriately.Returning the result: The function outputs a
tibblecontaining the original legislative description and the model’s classification or error code, providing a clean and structured result.
Let’s test the function on a single observation:
classify_policy("To amend the Internal Revenue Code of 1986 to provide for the treatment of certain direct primary care service arrangements as medical care.")# A tibble: 1 × 2
description .pred
<chr> <int>
1 To amend the Internal Revenue Code of 1986 to provide for the treatment… 3Your turn: Apply the classifier to the entire training set iteratively using an appropriate purrr::map_*() function, and collapse the results into a single tibble for further analysis.
tic()
leg_train_pred <- leg_train$description |>
map(.f = classify_policy, .progress = TRUE) |>
list_rbind()
toc()166.664 sec elapsedEvaluate the performance
Your turn: Examine the performance of these predictions. What are appropriate metrics to use? How does the classifier perform?
# add labels for API-generated codes
cap_codes_api <- cap_codes |>
bind_rows(
tribble(
~policy, ~policy_lab,
97, "API-connection failure",
98, "Invalid response",
99, "Missing (no clear policy)"
)
)
# combine predictions with true values
leg_train_pred_labels <- leg_train |>
bind_cols(.pred = leg_train_pred$.pred) |>
# convert truth and estimates to factors for evaluating performance
mutate(across(
.cols = c(policy, .pred),
.fns = \(x) factor(
x,
levels = cap_codes_api$policy,
labels = cap_codes_api$policy_lab
)
))
leg_train_pred_labels# A tibble: 400 × 3
description policy .pred
<chr> <fct> <fct>
1 To authorize the Secretary of Agriculture to develop and carry … Publi… Agri…
2 A bill to reauthorize provisions in the Native American Housing… Publi… Comm…
3 A bill to amend part A of title II of the Higher Education Act … Educa… Educ…
4 To cut off Federal benefits for conviction of rioting and prohi… Law, … Law,…
5 A bill to support the price of milk at 90 percentum of the part… Agric… Agri…
6 To amend the U.S. Housing Act of 1937 to authorize the construc… Commu… Comm…
7 A bill to amend the Internal Revenue Code of 1986 to extend for… Macro… Bank…
8 A bill to amend the Internal Revenue Code of 1954 to treat bank… Banki… Bank…
9 A bill to authorize grants for the Navajo Community College, an… Educa… Educ…
10 To amend the Internal Revenue Code of 1986 to provide tax incen… Educa… Educ…
# ℹ 390 more rows# choose class-based metrics
llm_metrics <- metric_set(accuracy, j_index, sensitivity, specificity)
leg_train_pred_labels |>
llm_metrics(truth = policy, estimate = .pred)# A tibble: 4 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 accuracy multiclass 0.66
2 j_index macro 0.673
3 sensitivity macro 0.691
4 specificity macro 0.984# confusion matrix
leg_train_pred_labels |>
conf_mat(truth = policy, estimate = .pred) |>
autoplot(type = "heatmap") +
scale_fill_continuous_sequential() +
theme(axis.text.x = element_text(angle = 45, hjust = 1))
Add response here. TODO
Evaluate multiple classifiers
Demonstration: To test different prompts and models systematically we need to allow for a more flexible classifier function that can handle different prompts or models. For this we take the prompt-building logic out of the function and allow for different api-functions and models as function arguments:
# numeric code list for reuse
cap_code_list <- c("
1 = Macroeconomics
2 = Civil rights, minority issues, civil liberties
3 = Health
4 = Agriculture
5 = Labor and employment
6 = Education
7 = Environment
8 = Energy
9 = Immigration
10 = Transportation
12 = Law, crime, family issues
13 = Social welfare
14 = Community development and housing issues
15 = Banking, finance, and domestic commerce
16 = Defense
17 = Space, technology, and communications
18 = Foreign trade
19 = International affairs and foreign aid
20 = Government operations
21 = Public lands and water management
")
# classification function that accepts a prompt, api_function, and model,
# as well as the true value to pass through as arguments
classify_policy_compare <- function(description,
policy,
prompt,
prompt_id,
api_function,
model) {
# print (message) what the model is currently doing to the console
str_glue("Classifying: {model} - {prompt_id} - {description}\n") |> message()
# list of valid codes as strings
valid_codes <- as.character(cap_codes$policy)
# attempt to classify the description
classification <- tryCatch(
{
# get the assistant's reply using the dynamically provided API function and model
assistant_reply <- llm_message(prompt) |>
api_function(.model = model, .temperature = 0) |>
last_reply() |>
str_squish()
# validate the assistant's reply
if (assistant_reply %in% valid_codes) {
as.integer(assistant_reply)
} else {
98L # return 98 for invalid responses
}
},
error = function(e) {
97L # return 97 in case of an error (e.g., API failure)
}
)
# Return a tibble containing the original occupation description and classification result
return(
tibble(
description = description,
.pred = classification,
.truth = policy,
model = model,
prompt_id = prompt_id
)
)
}Defining the prompt and model grid
We’ll define a set of prompts and models that we want to test. This will allow us to apply the classifier across different configurations and compare results. Here’s how the prompts and models are set up:
- Prompts:
- Prompt 1: A detailed prompt explaining the purpose of this classification task and what policy attention means.
- Prompt 2: Explicitly ask it to avoid making guesses by returning a special code (99) when the LLM is unsure.
- Prompt 3: A shorter, more concise version to test whether the model performs similarly with less detailed instructions.
- Models:1
1 This would also be an appropriate time to evaluate models from other providers, such as Anthropic, Mistral, or open models running locally via Ollama.
- GPT 4o-mini
- GPT 4o
We set up a grid combining all the prompts and models. The expand_grid() function is a useful tool here to create every possible combination of prompts and models, which we will use to evaluate the classifier.
Your turn: Develop appropriate prompts that meet the requirements above.
prompts <- tibble(
prompt =
c( # original prompt
"Classify this legislative description from the U.S. Congress: {description}
Your goal is correctly identify what kind of policy this legislation deals with,
based on a classification scheme developed by academic researchers that labels
policy attention based on the topic of the legislation.
Pick one of the following numerical codes from this list.
Respond only with the code!
{cap_code_list}",
# more explanation about the task
"Classify this legislative description from the U.S. Congress: {description}
Your goal is correctly identify what kind of policy this legislation deals with,
based on a classification scheme developed by academic researchers that labels
policy attention based on the topic of the legislation.
Pick one of the following numerical codes from this list.
Respond only with the code!
{cap_code_list}
If you are unsure as to the correct code, reply with just 99",
# shorter prompt
"Classify this legislative description: {description}.
Respond only with one of the following codes:
{cap_code_list}"
),
prompt_id = 1:3
)
grid <- expand_grid(
leg_train,
prompts,
model = c("gpt-4o-mini", "gpt-4o")
) |>
arrange(model) |>
rowwise() |> # glue together prompts and occupation row-by-row
mutate(prompt = str_glue(prompt)) |>
ungroup() |> # ungroup after the rowwise operation
select(description, policy, prompt, prompt_id, model)Generate predictions
Demonstration: To run the classification across the entire grid, we use pmap() from the {purrr} package, which allows us to iterate over multiple arguments simultaneously. Each combination of legislative description, prompt, and model is passed into the classify_policy_compare() function, and the results are concatenated into a single tibble:
grid_results <- grid |>
pmap(classify_policy_compare, api_function = openai) |>
list_rbind()Assess performance
Your turn: Evaluate the performance of the classifiers across different prompts and models. How does each perform?
grid_factors |>
mutate(prompt_id = factor(prompt_id,
labels = c(
"Detailed", "No guessing",
"Shortest instructions"
)
)) |>
group_by(prompt_id, model) |>
llm_metrics(truth = .truth, estimate = .pred) |>
ggplot(mapping = aes(x = prompt_id, y = .estimate, fill = model)) +
geom_col(position = "dodge") +
scale_fill_discrete_qualitative() +
scale_x_discrete(labels = label_wrap(width = 15)) +
facet_wrap(facets = vars(.metric)) +
labs(
title = "Accuracy by prompt and model",
x = "Prompt",
y = "Metric value",
fill = "Model"
) +
theme(legend.position = "top")
Add response here. TODO
Which policy topics cause the most confusion?
grid_factors |>
filter(prompt_id == 1, model == "gpt-4o-mini") |>
conf_mat(truth = .truth, estimate = .pred) |>
autoplot(type = "heatmap") +
scale_fill_continuous_sequential() +
theme(axis.text.x = element_text(angle = 45, hjust = 1))
Add response here. TODO analysis of results
Multistep chain-of-thought prompting
For this we need a major change in the classification function, because we send two messages to the model. A first one elicits a reasoning step, the second one asks for the final code, based on the answer to the first message.
classify_policy_cot <- function(description,
policy,
prompt,
prompt_id,
api_function,
model,
stream = FALSE) {
# Output what the model is currently doing to the console
str_glue("Classifying with CoT: {model} - {description}\n") |> message()
# Step 1: Ask the model to think through the problem
prompt_reasoning <- str_glue('
Think about which of the following policy topic codes would best describe this legislative description from the U.S. Congress: "{description}"
{cap_code_list}
Explain your reasoning for the 3 top candidate codes step by step. Then evaluate which seems best.
')
reasoning_response <- tryCatch(
{
conversation <<- llm_message(prompt_reasoning) |>
api_function(.model = model, .temperature = 0, .stream = stream)
conversation |>
last_reply()
},
error = function(e) {
conversation <<- llm_message("Please classify this policy topic: {description}")
"Error in reasoning step."
}
)
# Step 2: Ask the model to provide the final answer
prompt_final <- str_glue("
Based on your reasoning, which code do you pick? Answer only with a numerical code!
")
final_response <- tryCatch(
{
conversation |>
llm_message(prompt_final) |>
api_function(.model = model, .temperature = 0, .stream = stream) |>
last_reply() |>
str_squish()
},
error = function(e) {
"97"
}
)
# Validate the model's final response
valid_codes <- as.character(cap_codes$policy)
classification <- if (final_response %in% valid_codes) {
as.integer(final_response)
} else {
98L # Return 98 for invalid responses
}
# Return a tibble containing the original occupation description and classification result
tibble(
description = description,
.pred = classification,
.truth = policy,
model = str_glue("{model}_cot"),
reasoning = reasoning_response,
final_response = final_response
)
}Let’s run this function with GPT 4o:
results_cot <- grid |>
filter(model == "gpt-4o", prompt_id == 1) |>
select(-prompt, -prompt_id) |>
pmap(classify_policy_cot, api_function = openai, stream = FALSE) |>
list_rbind()Your turn: Evaluate the performance of the chain-of-thought method compared to the earlier classifiers. How does it perform?
results_cot |>
mutate(prompt_id = 4) |>
bind_rows(grid_results) |>
mutate(across(
.cols = c(.pred, .truth),
.fns = \(x) factor(x, levels = cap_codes_api$policy, labels = cap_codes_api$policy_lab)
)) |>
mutate(prompt_id = factor(prompt_id,
labels = c(
"Detailed", "No guessing",
"Shortest instructions", "Chain-of-thought"
)
)) |>
group_by(prompt_id, model) |>
accuracy(truth = .truth, estimate = .pred) |>
ggplot(mapping = aes(x = prompt_id, y = .estimate, fill = model)) +
geom_col(position = "dodge") +
scale_fill_discrete_qualitative() +
scale_y_continuous(labels = label_percent()) +
labs(
title = "Accuracy by prompt and model",
x = "Prompt",
y = "Accuracy",
fill = "Model"
) +
theme(legend.position = "top")
results_cot |>
mutate(across(
.cols = c(.pred, .truth),
.fns = \(x) factor(x, levels = cap_codes_api$policy, labels = cap_codes_api$policy_lab)
)) |>
conf_mat(truth = .truth, estimate = .pred) |>
autoplot(type = "heatmap") +
scale_fill_continuous_sequential() +
theme(axis.text.x = element_text(angle = 45, hjust = 1))
Add response here. TODO
Acknowledgments
- Materials derived in part from Classifying Texts with {tidyllm} and licensed under the MIT License.
Additional resources
Session information
sessioninfo::session_info()─ Session info ───────────────────────────────────────────────────────────────
setting value
version R version 4.4.1 (2024-06-14)
os macOS Sonoma 14.6.1
system aarch64, darwin20
ui X11
language (EN)
collate en_US.UTF-8
ctype en_US.UTF-8
tz America/New_York
date 2024-11-04
pandoc 3.4 @ /usr/local/bin/ (via rmarkdown)
─ Packages ───────────────────────────────────────────────────────────────────
package * version date (UTC) lib source
arrow * 17.0.0 2024-09-18 [1] https://apache.r-universe.dev (R 4.4.1)
askpass 1.2.1 2024-10-04 [1] CRAN (R 4.4.1)
assertthat 0.2.1 2019-03-21 [1] CRAN (R 4.3.0)
backports 1.5.0 2024-05-23 [1] CRAN (R 4.4.0)
base64enc 0.1-3 2015-07-28 [1] CRAN (R 4.3.0)
bit 4.0.5 2022-11-15 [1] CRAN (R 4.3.0)
bit64 4.0.5 2020-08-30 [1] CRAN (R 4.3.0)
broom * 1.0.6 2024-05-17 [1] CRAN (R 4.4.0)
class 7.3-22 2023-05-03 [1] CRAN (R 4.4.0)
cli 3.6.3 2024-06-21 [1] CRAN (R 4.4.0)
codetools 0.2-20 2024-03-31 [1] CRAN (R 4.4.1)
colorspace * 2.1-1 2024-07-26 [1] CRAN (R 4.4.0)
data.table 1.15.4 2024-03-30 [1] CRAN (R 4.3.1)
dials * 1.3.0 2024-07-30 [1] CRAN (R 4.4.0)
DiceDesign 1.10 2023-12-07 [1] CRAN (R 4.3.1)
digest 0.6.35 2024-03-11 [1] CRAN (R 4.3.1)
dplyr * 1.1.4 2023-11-17 [1] CRAN (R 4.3.1)
evaluate 0.24.0 2024-06-10 [1] CRAN (R 4.4.0)
fansi 1.0.6 2023-12-08 [1] CRAN (R 4.3.1)
farver 2.1.2 2024-05-13 [1] CRAN (R 4.3.3)
fastmap 1.2.0 2024-05-15 [1] CRAN (R 4.4.0)
forcats * 1.0.0 2023-01-29 [1] CRAN (R 4.3.0)
foreach 1.5.2 2022-02-02 [1] CRAN (R 4.3.0)
furrr 0.3.1 2022-08-15 [1] CRAN (R 4.3.0)
future 1.33.2 2024-03-26 [1] CRAN (R 4.3.1)
future.apply 1.11.2 2024-03-28 [1] CRAN (R 4.3.1)
generics 0.1.3 2022-07-05 [1] CRAN (R 4.3.0)
ggplot2 * 3.5.1 2024-04-23 [1] CRAN (R 4.3.1)
globals 0.16.3 2024-03-08 [1] CRAN (R 4.3.1)
glue 1.8.0 2024-09-30 [1] CRAN (R 4.4.1)
gower 1.0.1 2022-12-22 [1] CRAN (R 4.3.0)
GPfit 1.0-8 2019-02-08 [1] CRAN (R 4.3.0)
gtable 0.3.5 2024-04-22 [1] CRAN (R 4.3.1)
hardhat 1.4.0 2024-06-02 [1] CRAN (R 4.4.0)
here 1.0.1 2020-12-13 [1] CRAN (R 4.3.0)
hms 1.1.3 2023-03-21 [1] CRAN (R 4.3.0)
htmltools 0.5.8.1 2024-04-04 [1] CRAN (R 4.3.1)
htmlwidgets 1.6.4 2023-12-06 [1] CRAN (R 4.3.1)
infer * 1.0.7 2024-03-25 [1] CRAN (R 4.3.1)
ipred 0.9-14 2023-03-09 [1] CRAN (R 4.3.0)
iterators 1.0.14 2022-02-05 [1] CRAN (R 4.3.0)
jsonlite 1.8.9 2024-09-20 [1] CRAN (R 4.4.1)
knitr 1.47 2024-05-29 [1] CRAN (R 4.4.0)
labeling 0.4.3 2023-08-29 [1] CRAN (R 4.3.0)
lattice 0.22-6 2024-03-20 [1] CRAN (R 4.4.0)
lava 1.8.0 2024-03-05 [1] CRAN (R 4.3.1)
lhs 1.1.6 2022-12-17 [1] CRAN (R 4.3.0)
lifecycle 1.0.4 2023-11-07 [1] CRAN (R 4.3.1)
listenv 0.9.1 2024-01-29 [1] CRAN (R 4.3.1)
lubridate * 1.9.3 2023-09-27 [1] CRAN (R 4.3.1)
magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.3.0)
MASS 7.3-61 2024-06-13 [1] CRAN (R 4.4.0)
Matrix 1.7-0 2024-03-22 [1] CRAN (R 4.4.0)
modeldata * 1.4.0 2024-06-19 [1] CRAN (R 4.4.0)
munsell 0.5.1 2024-04-01 [1] CRAN (R 4.3.1)
nnet 7.3-19 2023-05-03 [1] CRAN (R 4.4.0)
parallelly 1.37.1 2024-02-29 [1] CRAN (R 4.3.1)
parsnip * 1.2.1 2024-03-22 [1] CRAN (R 4.3.1)
pdftools 3.4.1 2024-09-20 [1] CRAN (R 4.4.1)
pillar 1.9.0 2023-03-22 [1] CRAN (R 4.3.0)
pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.3.0)
prodlim 2023.08.28 2023-08-28 [1] CRAN (R 4.3.0)
purrr * 1.0.2 2023-08-10 [1] CRAN (R 4.3.0)
qpdf 1.3.4 2024-10-04 [1] CRAN (R 4.4.1)
R6 2.5.1 2021-08-19 [1] CRAN (R 4.3.0)
Rcpp 1.0.13 2024-07-17 [1] CRAN (R 4.4.0)
readr * 2.1.5 2024-01-10 [1] CRAN (R 4.3.1)
recipes * 1.0.10 2024-02-18 [1] CRAN (R 4.3.1)
rlang 1.1.4 2024-06-04 [1] CRAN (R 4.3.3)
rmarkdown 2.27 2024-05-17 [1] CRAN (R 4.4.0)
rpart 4.1.23 2023-12-05 [1] CRAN (R 4.4.0)
rprojroot 2.0.4 2023-11-05 [1] CRAN (R 4.3.1)
rsample * 1.2.1 2024-03-25 [1] CRAN (R 4.3.1)
rstudioapi 0.17.0 2024-10-16 [1] CRAN (R 4.4.1)
scales * 1.3.0 2023-11-28 [1] CRAN (R 4.4.0)
sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.3.0)
stringi 1.8.4 2024-05-06 [1] CRAN (R 4.3.1)
stringr * 1.5.1 2023-11-14 [1] CRAN (R 4.3.1)
survival 3.7-0 2024-06-05 [1] CRAN (R 4.4.0)
tibble * 3.2.1 2023-03-20 [1] CRAN (R 4.3.0)
tictoc * 1.2.1 2024-03-18 [1] CRAN (R 4.4.0)
tidyllm * 0.1.9 2024-10-28 [1] Github (edubruell/tidyllm@8c31704)
tidymodels * 1.2.0 2024-03-25 [1] CRAN (R 4.3.1)
tidyr * 1.3.1 2024-01-24 [1] CRAN (R 4.3.1)
tidyselect 1.2.1 2024-03-11 [1] CRAN (R 4.3.1)
tidyverse * 2.0.0 2023-02-22 [1] CRAN (R 4.3.0)
timechange 0.3.0 2024-01-18 [1] CRAN (R 4.3.1)
timeDate 4032.109 2023-12-14 [1] CRAN (R 4.3.1)
tune * 1.2.1 2024-04-18 [1] CRAN (R 4.3.1)
tzdb 0.4.0 2023-05-12 [1] CRAN (R 4.3.0)
utf8 1.2.4 2023-10-22 [1] CRAN (R 4.3.1)
vctrs 0.6.5 2023-12-01 [1] CRAN (R 4.3.1)
withr 3.0.2 2024-10-28 [1] CRAN (R 4.4.1)
workflows * 1.1.4 2024-02-19 [1] CRAN (R 4.3.1)
workflowsets * 1.1.0 2024-03-21 [1] CRAN (R 4.3.1)
xfun 0.45 2024-06-16 [1] CRAN (R 4.4.0)
yaml 2.3.10 2024-07-26 [1] CRAN (R 4.4.0)
yardstick * 1.3.1 2024-03-21 [1] CRAN (R 4.3.1)
[1] /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/library
──────────────────────────────────────────────────────────────────────────────