Report

Your team will report on your model development process. This should include a detailed description of your data, your models, and your results. The report should be written in the report.qmd file in your project repo.

Audience

The target audience for this report is a senior-level executive with some data science background (but nothing very extensive). They are looking for a high-level overview of your project, the results, and the implications of those results. They are not interested in the nitty-gritty details of your code or the exact specifics of your model tuning process.

WarningRemember your audience

Your audience is not a data scientist. They do not care about the code you used to generate your models. In fact, it is best that no code is visible in the final output document. There should be no raw text output from R or Python. Everything should be appropriately formatted as a table or figure.

Structure

The report should contain information on:

• Objective(s). State the problem(s) you are solving clearly.

• Data description. Your data description should be about your analysis-ready data.

  This should be inspired by the format presented in Gebru et al, 2018. Answer any relevant questions from sections 3.1-3.5 of the Gebru et al article, especially the following questions:

  • What are the observations (rows) and the attributes (columns)?
  • Why was this dataset created?
  • Who funded the creation of the dataset?
  • What processes might have influenced what data was observed and recorded and what was not?
  • What preprocessing was done, and how did the data come to be in the form that you are using?
  • If people are involved, were they aware of the data collection and if so, what purpose did they expect the data to be used for?

• Resampling strategy. All teams are expected to partition their data into training/test sets using an appropriate strategy. Many teams will further partition the training set using a resampling strategy. Document your resampling strategy here and justify the approach you chose.

• Overview of modeling strategies. Provide an overview of the modeling strategies you plan to use. This should include a description of the models you used, preprocessing or feature engineering steps, tuning parameters, and the evaluation metrics you plan to use to compare models.

• Model comparison. Report on the models you evaluated. This should at least include a null model, as well as the modeling strategies from above. Include any relevant evaluation metrics and techniques we have learned in this class. Present this information in a clear and effective manner.

• Model evaluation. Evaluate the performance of your final model. This should include a discussion of the evaluation metrics you used and an explanation of how the model works (e.g. feature importance, partial dependence plots, observation-specific explanations).

• Recommendations. Provide recommendations based on your model. This could include a discussion of the model’s limitations, potential improvements, or how the model could be used in practice.

Evaluation criteria

Category	Less developed projects	Typical projects	More developed projects
Objective(s)	Objective is not clearly stated or significantly limits potential analysis.	Clearly states the objective(s), which have moderate potential for relevant impact.	Clearly states complex objective(s) that leads to significant potential for relevant impact.
Data description	Simple description of some aspects of the dataset, little consideration for sources. The description is missing answers to applicable questions detailed in the “Datasheets for Datasets” paper.	Answers all relevant questions in the “Datasheets for Datasets” paper.	All expectations of typical projects + credits and values data sources.
Resampling strategy	Does not use resampling methods (or an inappropriate method) to ensure robust model evaluation.	Selects an appropriate resampling strategy.	All expectations of typical projects + provides a thorough justification for the resampling strategy.
Overview of modeling strategies	Includes only simplistic models. Does not demonstrate understanding of the types of models covered in the course. Feature engineering steps are non-existent. Does not select evaluation metrics or metrics are not appropriate to the objective(s) + models.	Identifies several modeling strategies which could generate a high-performance model. Documents relevant feature engineering steps to be evaluate for specific modeling strategies. Steps are selectively applied to appropriate models. Evaluation metrics are appropriate for the objective(s) + models.	All expectations of typical projects + provides thorough explanation for the models/feature engineering/metrics. Shows care in selecting their methods.
Model comparison	Results are presented in a disjointed or uninterpretable manner.	Reports the results of their modeling strategies. Results are presented in a clear and interpretable manner. Utilizes tables and/or figures to effectively communicate with the audience.	All expectations of typical projects + reports results in an effective and concise narrative. Tables and/or figures show care and deliberation in their design.
Model evaluation	Does not fit a single model using the entire training set. No documentation for why the team selected this model. Performance metrics from the test set are missing. Explanation of how the model works is missing or incomplete.	Fits a single model based on the results of the model comparison. Provides written explanation for why they selected this specific model. Reports performance metrics using the test set. Explains how the model works using appropriate techniques (e.g. feature importance scores, partial dependence plots, observation-specific explanations).	All expectations of typical projects + reports evaluation in an effective and concise narrative. Tables and/or figures show care and deliberation in their design.
Recommendations	Recommendations are unclear or unsupported by the performance metrics.	Provides a clear set of recommendations regarding how to use the model. Identifies limitations and potential improvements in future iterations of the model.	All expectations of typical projects + recommendations are actionable. It is clear how this model will be used in a production environment.