Documenting models

Lecture 19

Dr. Benjamin Soltoff

Cornell University
INFO 4940/5940 - Fall 2024

November 12, 2024

Announcements

Announcements

  • Homework 04
  • Project exploration

Learning objectives

  • Identify audiences for model documentation
  • Define a model card
  • Describe the major sections of a model card
  • Utilize a model card for a foundational LLM

Documenting models

Why document models?

  • ML practitioners
  • Model developers
  • Software developers
  • Policymakers
  • Organizations
  • ML-knowledgable individuals
  • Impacted individuals

Model cards

A screenshot of a model card called 'Smiling Detection in Images'.

Source: “Model Cards for Model Reporting”

Major sections

  • Model details
  • Intended use
  • Factors
  • Metrics
  • Evaluation data
  • Training data
  • Quantitative analyses
  • Ethical considerations
  • Caveats and recommendations

Model details

  • Person or organization developing the model
  • Model date
  • Model version
  • Model type
  • More information
  • Citation
  • License
  • Contact info

Intended use

  • Primary intended uses
  • Primary intended users
  • Out-of-scope uses

Factors

  • Groups
  • Instrumentation
  • Environment
  • Relevant factors
  • Evaluation factors

Metrics

  • Model performance measures
  • Decision thresholds
  • Approaches to uncertainty and variability

Evaluation data

  • Datasets
  • Motivations
  • Preprocessing

Training data

Same info if possible. If not, summary statistics are good.

Quantitative analyses

  • Disaggregated based on chosen factors
  • Model performance
  • Unitary/intersectional results

Ethical considerations

  • Sensitive data
  • Human life
  • Mitigation strategies
  • Risks and harms
  • Use cases

Caveats and recommendations

  • Is further testing necessary?
  • Were relevant groups excluded from the evaluation?
  • Additional recommendations for model use?

Application exercise

Documenting published LLMs

Use the Llama 3.2 model card on Hugging Face to answer the questions on the handout.

20:00

Wrap-up

Recap

  • Model cards are a way to document models for various audiences
  • They contain information such as model details, intended use, and evaluation criteria