Project description

Modified

December 22, 2024

Important dates

Important

The details will be updated as the project date approaches.

Introduction

TL;DR: Solve a problem using machine learning.

This is intentionally vague – part of the challenge is to design a project that showcases best your interests and strengths. However we expect you to solve some sort of problem using machine learning (broadly construed). This could be a prediction problem, a classification problem, a clustering problem, etc. The problem could be in any domain, and could involve any type of data. The project should be a substantial piece of work, and should involve a significant amount of data and computation.

The project is very open ended. Neatness, coherency, and clarity will count. All components of the project must be reproducible.

Deliverables

The four primary deliverables for the final project are

  1. A project proposal with three ideas.
  2. A deployed product that is publicly accessible.
  3. A model card that documents the model.
  4. A final report to your “client” that documents the model development process.

There will be additional submissions throughout the semester to facilitate completion of the final product and presentation.

Teams

Projects will be completed in teams of 3-4 students. Every team member should be involved in all aspects of planning and executing the project. Each team member should make an equal contribution to all parts of the project. The scope of your project is based on the number of contributing team members on your project. If you have 4 contributing team members, we will expect a larger project than a team of 3 contributing team members.

The course staff will assign students to teams. To facilitate this process, we will provide a short survey identifying study and communication habits. Once teams are assigned, they cannot be changed.

Team conflicts

Conflict is a healthy part of any team relationship. If your team doesn’t have conflict, then your team members are likely not communicating their issues with each other. Use your team contract (written at the beginning of the project) to help keep your team dynamic healthy.

When you have conflict, you should follow this procedure:

  1. Refer to the team contract and follow it to address the conflict.

  2. If you resolve the conflict without issue, great! Otherwise, update the team contract and try to resolve the conflict yourselves.

  3. If your team is unable to resolve your conflict, please contact soltoffbc@cornell.edu and explain your situation.

    We’ll ask to meet with all the group members and figure out how we can work together to move forward.

  4. Please do not avoid confrontation if you have conflict. If there’s a conflict, the best way to handle it is to bring it into the open and address it.

Project grade adjustments

Remember, do not do the work for a slacking team member. This only rewards their bad behavior. Simply leave their work unfinished. (We will not increase your grade during adjustments for doing more than your fair share.)

Your team will initially receive a final grade assuming that all team members contributed to your project. If you have a 5-person team, but only 3 persons contributed, your team will likely receive a lower grade initially because only 3 persons worth of effort exists for a 5-person project. About a week after the initial project grades are released, adjustments will be made to each individual team member’s group project grade.

We use your project’s Git history (to view the contributions of each team member) and the peer evaluations to adjust each team members’ grades. Both adjustments to increase or decrease your grade are possible based on each individual’s contributions.

For example, if you have a 4-person team, but only 3 contributing members, the 3 contributing members may have their grades increased to reflect the effort of only 3 contributing members. The non-contributing member will likely have their grade decreased significantly.

Warning

I am serious about every member of the team equitably contributing to the project. Students who fail to contribute equitably may receive up to a 100% deduction on their project grade.

Please be patient for the grade adjustments. The adjustments take time to do them fairly. Please know that the instructor handles this entire process himself, and I take it very seriously. If you think your initial group project grade is unfair, please wait for your grade adjustment before you contact us.

The slacking team member

Please do not cover for a slacking/freeloading team member. Please do not do their work for them! This only rewards their bad behavior. Simply leave their work unfinished. (We will not increase your grade during adjustments for doing more than your fair share.)

Remember, we have your Git history. We can see who contributes to the project and who doesn’t. If a team member rarely commits to Git and only makes very small commits, we can see that they did not contribute their fair share.

All students should make their project contributions through their own GitHub account. Do not commit changes to the repository from another team member’s GitHub account. Your Git history should reflect your individual contributions to the project.

Proposal

There are two main purposes of the project proposal:

  • To help you think about the project early, so you can get a head start on finding data, reading relevant literature, thinking about the questions you wish to answer, etc.
  • To ensure that the topic you wish to analyze, methods you plan to use, and the scope of your analysis are feasible and will allow you to be successful for this project.

Identify 3 problems you’re interested in potentially solving for the project. These problems should be solved using an ML model and real-world data. If you’re unsure where to find data, you can use the list of potential data sources in the Tips + Resources section as a starting point. It may also help to think of topics you’re interested in investigating and find datasets on those topics.

Write the proposal in the proposal.qmd file in your project repo.

Important

You must use one of the topics in the proposal for the final project, unless instructed otherwise when given feedback.

Criteria for datasets

The datasets should meet the following criteria:

  • At least 1000 observations
  • At least 8 columns
  • At least 6 of the columns must be useful and unique explanatory variables.
    • Identifier variables such as “name”, “social security number”, etc. are not useful explanatory variables.
    • If you have multiple columns with the same information (e.g. “state abbreviation” and “state name”), then they are not unique explanatory variables.
  • You may not use data that has previously been used in any course materials, or any derivation of data that has been used in course materials.
Warning

You may not use data from a secondary data archive. In plainest terms, do not use datasets you find from Kaggle or the UCI Machine Learning Repository. Your data should come from your own collection process (e.g. API or web scraping) or the primary source (e.g. government agency, research group, etc.).

Please ask a member of the course staff if you’re unsure whether your dataset meets the criteria.

Questions for project mentor

Note

Include specific, relevant questions you have for the project mentor about your proposed topics. These questions should be about the feasibility of the project, the quality of the data, the potential for interesting analysis, etc.

Resources for datasets

You can find data wherever you like, but here are some recommendations to get you started. You shouldn’t feel constrained to datasets that are already in a tidy format, you can start with data that needs cleaning and tidying, scrape data off the web, or collect your own data.

Proposal components

For each topic, include the following:

Problem

What is the problem you will solve?

For each topic, include the following:

  • A well formulated objective(s).
  • Statement on why this topic is important.
  • Identify the types of variables you will use. Categorical? Quantitative?
  • How will you make the model usable? An interactive web application a la Shiny? A deployable API?

Introduction and data

For each dataset:

  • Identify the source of the data.

  • State when and how it was originally collected (by the original data curator, not necessarily how you found the data).

  • Write a brief description of the observations.

  • Address ethical concerns about the data, if any.

Glimpse of data

For each dataset (if one is provided):

  • Place the file containing your data in the data folder of the project repo.
  • Use the skimr::skim() function to provide a glimpse of the dataset.

Exploration

Settle on a single topic and state your objective(s) clearly. You will carry out most of your data collection and cleaning, compute some relevant summary statistics, and show some plots of your data as applicable to your objective(s).

Write up your explanation in the explore.qmd file in your project repo. It should include the following sections:

  • Objective(s). State the problem(s) you are solving clearly.
  • Data collection and cleaning. Have an initial draft of your data cleaning appendix. Document every step that takes your raw data file(s) and turns it into the analysis-ready data set that you would submit with your final project. Include text narrative describing your data collection (downloading, scraping, surveys, etc) and any additional data curation/cleaning (merging data frames, filtering, transformations of variables, etc). Include code for data curation/cleaning, but not collection.1
  • Data description. Have an initial draft of your data description section. Your data description should be about your analysis-ready data.
  • Data limitations. Identify any potential problems with your dataset.
  • Exploratory data analysis. Perform an (initial) exploratory data analysis.
  • Questions for reviewers. List specific questions for your project mentor to answer in giving you feedback on this phase.

1 If you have written code to collect your data (e.g. using an API or web scraping), store this in a separate .qmd file or .R script in the repo.

Warning

Thorough EDA requires substantial review and analysis of your data. You should not expect to complete this phase in a single day. You should expect to iterate through 20-30 charts, sets of summary statistics, etc., to get a good understanding of your data.

Visualizations are not expected to look perfect at this point since they are mainly intended for you and your team members. Standard expectations for visualizations (e.g. clearly labeled charts and axes, optimized color palettes) are not necessary at the EDA stage.

  • Questions for reviewers. List specific questions for your project mentor to answer in giving you feedback on this phase.

Draft

The purpose of the draft is to give you an opportunity to get early feedback on your analysis. Therefore, the draft will focus primarily on the exploratory analysis and initial drafts of the final product(s).

Write the draft write-up in the report.qmd file in your project repo. This should document your modeling strategies to date. At minimum, you are expected to include:

  • Objective(s). State the problem(s) you are solving clearly.
  • Data description. Your data description should be about your analysis-ready data.
  • Decisions based on EDA. Based on your exploratory analysis, explicitly identify the decisions you made about your data (e.g. what to exclude, what to include, what to transform, etc). These could be things you do to preprocess the data before partitioning into training/test set, or possible feature engineering steps you will evaluate in the modeling phase.
  • Resampling strategy. All teams are expected to partition their data into training/test sets using an appropriate strategy. Many teams will further partition the training set using a resampling strategy. Document your resampling strategy here and justify the approach you chose.
  • Overview of modeling strategies. Provide an overview of the modeling strategies you plan to use. This should include a brief description of the models you plan to use, potential preprocessing or feature engineering steps, tuning parameters, and the evaluation metrics you plan to use to compare models.
  • Initial results. Report any initial results you have. This should at least include a null model, as well as any of the modeling strategies from above that you have already tested. Include any relevant evaluation metrics and techniques we have learned in this class.
You do not need to fit all the models in the report.qmd file

You may fit all the models in a separate R script or Quarto file and save/export any appropriate model objects so you can report relevant metrics or create visualizations/tables to report on the models’ performance. You should include the results of the models in the report.qmd file.

Standard R objects can be saved to disk using readr::write_rds() or save().

# tune some complex models
tune_rf_res <- tune_grid(...)
tune_lgbm_res <- tune_grid(...)

# save a single object
write_rds(tune_rf_res, file = "output/tune_rf_res.rds")
write_rds(tune_lgbm_res, file = "output/tune_lgbm_res.rds")

# save together
save(tune_rf_res, tune_lgbm_res, file = "output/tune_results.RData")

For model fit() objects, you will likely want to butcher() the object to reduce its overall size (otherwise the file size may be hundreds of megabytes.)

# fit the best rf model
best_rf <- fit_best(tune_rf_res)
best_rf_lite <- butcher(best_rf)

Read the documentation for {butcher} for more information.

Deployed model

All teams will publish their model for public usage. This might could be an interactive web application where users can generate custom predictions from the model, a deployable API allowing developers to interface with the model to generate predictions, or some other form.

Note

We will learn how to publish models using APIs in this course. At this time I do not expect to cover Shiny applications in the course, but your team is free to learn how to use Shiny or a similar tool independently for the project.

Make it easy to evaluate your deployment

Your deployed model should be easy to evaluate.

  • If you deploy via a Shiny app, include a link to the published app in the README.md file in your project repo.
  • If you deploy via an API, include an R script in your repo that demonstrates how to use the API to generate predictions for your test set. We should be able to source that R script and it just works.

Model card

The model card is a document that accompanies a trained machine learning model. It provides a summary of the model’s performance and limitations, as well as the context in which it was trained and used. The model card should be written in the model-card.qmd file in your project repo using the template we learned in class and practiced for homework.

Report

Your team will report on your model development process. This should include a detailed description of your data, your models, and your results. The report should be written in the report.qmd file in your project repo.

Audience

The target audience for this report is a senior-level executive with some data science background (but nothing very extensive). They are looking for a high-level overview of your project, the results, and the implications of those results. They are not interested in the nitty-gritty details of your code or the exact specifics of your model tuning process.

Remember your audience

Your audience is not a data scientist. They do not care about the code you used to generate your models. In fact, it is best that no code is visible in the final output document. There should be no raw text output from R. Everything should be appropriately formatted as a table or figure.

Structure

The report should contain information on:

  • Objective(s). State the problem(s) you are solving clearly.
  • Data description. Your data description should be about your analysis-ready data.
  • Resampling strategy. All teams are expected to partition their data into training/test sets using an appropriate strategy. Many teams will further partition the training set using a resampling strategy. Document your resampling strategy here and justify the approach you chose.
  • Overview of modeling strategies. Provide an overview of the modeling strategies you plan to use. This should include a description of the models you used, preprocessing or feature engineering steps, tuning parameters, and the evaluation metrics you plan to use to compare models.
  • Model comparison. Report on the models you evaluated. This should at least include a null model, as well as the modeling strategies from above. Include any relevant evaluation metrics and techniques we have learned in this class. Present this information in a clear and effective manner.
  • Model evaluation. Evaluate the performance of your final model. This should include a discussion of the evaluation metrics you used and an explanation of how the model works (e.g. feature importance, partial dependence plots, observation-specific explanations).
  • Recommendations. Provide recommendations based on your model. This could include a discussion of the model’s limitations, potential improvements, or how the model could be used in practice.
Confused about what this should look like?

I loathe to provide examples to students since they have a tendency to simply mimic the provided example without much critical thinking. However, I understand that this might feel a little vague. This is an example report drafted for a consulting client. Feel free to review it in terms of the structure, style, and presentation. Keep in mind your report need not look identical (or even that similar) to this example as long as you cover the necessary elements.

Reproducibility + organization

All written work should be reproducible, and the GitHub repo should be neatly organized.

  • Points for reproducibility + organization will be based on the reproducibility of the entire repository and the organization of the project GitHub repo.
  • The repo should be neatly organized as described above, there should be no extraneous files, all text in the README should be easily readable.

Teamwork

Every team member should make an equal contribution to all parts of the project. Every team member should have an equal experience designing, coding, testing, etc.

At the completion of the project, you will be asked to fill out a survey where you rate the contribution and teamwork of each team member by assigning a contribution percentage for each team member. Working as a team is every team member’s responsibility.

If you are suggesting that an individual did less than half the expected contribution given your team size (e.g., for a team of four students, if a student contributed less than 12.5% of the total effort), please provide some explanation. If any individual gets an average peer score indicating that they underperformed on the project, we will conduct further analysis and their overall project grade may be adjusted accordingly.

Overall grading

Evaluation criteria

Total 150 pts
Project proposal 10 pts
Exploration 15 pts
Draft 20 pts
Deployed model 40 pts
Report 25 pts
Model card 25 pts
Reproducibility + organization 15 pts

Project proposal

Category Less developed projects Typical projects More developed projects
Dataset ideas Fewer than three topics are included.

Topic ideas are vague and impossible or excessively difficult to collect.		 Three topic ideas are included and all or most datasets could feasibly be collected or accessed by the end of the semester.

Each dataset is described alongside a note about availability with a source cited.		 Three topic ideas are included and all or most datasets could feasibly be collected or accessed by the end of the semester.

Each dataset is described alongside a note about availability with (possibly multiple) sources cited.

Each dataset could reasonably be part of a data science project, driven by an interesting research question.
Questions for reviewers The questions for reviewers are vague or unclear. The questions for reviewers are specific to the datasets and are based on group discussions between team members. The questions for reviewers are specific to the datasets and are based on group discussions between team members.

Questions for reviewers look toward the next stage of the project.

Exploration

Category Less developed projects Typical projects More developed projects
Objective(s) Objective is not clearly stated or significantly limits potential analysis. Clearly states the objective(s), which have moderate potential for relevant impact. Clearly states complex objective(s) that leads to significant potential for relevant impact.
Data cleaning Data is minimally cleaned, with little documentation and description of the steps undertaken.

Completes all necessary data cleaning for subsequent analyses.

Describes cleaning steps with some detail.

Completes all necessary data cleaning for subsequent analyses.

Describes all cleaning steps in full detail, so that the reader has an excellent grasp of how the raw data was transformed into the analysis-ready dataset.
Data description

Simple description of some aspects of the dataset, little consideration for sources.

The description is missing answers to applicable questions detailed in the “Datasheets for Datasets” paper.

 Answers all relevant questions in the “Datasheets for Datasets” paper. All expectations of typical projects + credits and values data sources.
Data limitations

The limitations are not explained in depth.

There is no mention of how these limitations may affect the meaning of results.

 Identifies potential harms and data gaps, and describes how these could affect the meaning of results. Creatively identifies potential harms and data gaps, and describes how these could affect the meaning of results, and the impact of results on people. It is evident that significant thought has been put into the limitations of the collected data.
Exploratory data analysis

Motivation for choice of analysis methods is unclear.

Does not justify decisions to either confirm / update research questions and data description.

Sufficient plots (20-30) and summary statistics to identify typical values in single variables and connections between pairs of variables.

Uses exploratory analysis to confirm/update research questions and data description.

 All expectations of typical projects + analysis methods are carefully chosen to identify important characteristics of data.

Draft

Category Less developed projects Typical projects More developed projects
Objectives Objective is not clearly stated or significantly limits potential analysis. Clearly states the objective(s), which have moderate potential for relevant impact. Clearly states complex objective(s) that leads to significant potential for relevant impact.
Data description

Simple description of some aspects of the dataset, little consideration for sources.

The description is missing answers to applicable questions detailed in the “Datasheets for Datasets” paper.

 Answers all relevant questions in the “Datasheets for Datasets” paper. All expectations of typical projects + credits and values data sources.
Decisions based on EDA

Identifies minimal actions taken during the modeling stage based on the results of the EDA.

Actions are unlikely to effect predictions.

 Identifies concrete actions taken during the modeling stage based on the results of the EDA. All expectations of typical projects + actions demonstrate deliberate and careful analysis of the exploratory analysis.
Resampling strategy Does not use resampling methods (or an inappropriate method) to ensure robust model evaluation. Selects an appropriate resampling strategy. All expectations of typical projects + provides a thorough justification for the resampling strategy.
Modeling strategies

Includes only simplistic models. Does not demonstrate understanding of the types of models covered in the course.

Feature engineering steps are non-existent.

Does not select evaluation metrics or metrics are not appropriate to the objective(s) + models.

Identifies several modeling strategies which could generate a high-performance model.

Documents relevant feature engineering steps to be evaluate for specific modeling strategies. Steps are selectively applied to appropriate models.

Evaluation metrics are appropriate for the objective(s) + models.

 All expectations of typical projects + provides thorough explanation for the models/feature engineering/metrics. Shows care in selecting their methods.
Initial results

Only reports results of null model.

Results are presented in a disjointed or uninterpretable manner.

Reports the results of some (but not all) of their modeling strategies.

Results are presented in a clear and interpretable manner.

Reports the results of the majority of their modeling strategies.

Results are effectively communicated through the use of tables and/or figures.

Deployed model

Evaluation criteria differ based on the deployment approach. Teams using a deployment method other than publishing via API will be evaluated a bit more generously given the higher degree of difficulty.

Deployed via API

Category Less developed projects Typical projects
Deployment API is not deployed successfully to a public URL. Documentation is inaccessible. API is deployed successfully to a public URL. Documentation is accessible.
Performance API does not successfully generate predictions. API successfully used to generate predictions for observations.
Plumber reproducibility plumber.R script does not successfully run. plumber.R script successfully runs.
Docker reproducibility Required Docker artifacts are missing from the repo. Docker container fails to successfully build and/or run. All required Docker artifacts are included in the repo. Docker container successfully builds and run.

Deployed using a Shiny app

Category Less developed projects Typical projects More developed projects
Deployment App is not deployed successfully to a public URL. API is deployed successfully to a public URL. N/A
Design + visualization The design/visualization are inappropriate for the data and/or audience. Charts are hard to read. Design is non-intuitive or hard to access. The design/visualizations are appropriate and easy to read. All expectations of typical projects + consistently adheres to best practices for high-quality, accessible data visualizations. Includes but not limited to deliberate choice of color palette, appropriate font size, intuitive labeling.
Functionality App is incomplete or broken. Code generates numerous errors or warnings which cause usability problems. App does not provide value to the audience. App provides sufficient value to the intended audience. No errors or warnings present in final version. App is complete. All expectations of typical projects + app is intricate and exceptionally designed. Showcases significant technical skills beyond those directly taught in the course.
Code Code is not efficient, hard to read, and/or lacks appropriate formatting or consistency. Clearly looks like multiple individuals wrote it instead of a single, cohesive approach. Code is efficient, easy to read, and properly formatted. All expectations of typical projects + code is thoroughly documented with comments. Adheres to consistent style guide. Clearly reusable and/or extensible for outside developers.

Model card

Category Less developed projects Typical projects More developed projects
Model details Lacks basic information regarding the model version, type, and other relevant details. Answers basic questions regarding the model version, type, and other relevant details. All expectations of typical projects + questions are answered thoroughly.
Intended use Unclear why the model was created and/or how it should be used. Explains why the model was created and how it should and should not be used. All expectations of typical projects + questions are answered thoroughly.
Factors

Does not identify relevant factors for contextualizing the model.

Identified factors are artificial or superfluous.

 Identifies relevant aspects or factors that influence how users contextualize the model. All expectations of typical projects + demonstrates serious thought and consideration to relevant factors.
Metrics and data sets

Metrics are unstated or inappropriate for the model.

Unclear what data was used to train and/or evaluate the model.

Clearly states the metrics used to evaluate the model.

Identifies the data used to train the model.

Identifies the data used to evaluate the model.

 All expectations of typical projects + questions are answered thoroughly.
Quantitative analyses

Doesn’t report overall or disaggregated metrics.

Metrics are reported in a shoddy or unclear manner.

Reports overall metrics of model performance using the evaluation data.

Metrics are disaggregated based on relevant factors previously identified.

All expectations of typical projects + effectively uses tables/figures to communicate results of analyses.

Figures and tables are properly labeled.

Figures are aesthetically pleasing and clearly interpretable.
Ethical considerations Fails to seriously address ethical considerations. Identifies ethical considerations that went into model development. All expectations of typical projects + demonstrates serious thought and consideration to relevant considerations.
Caveats & recommendations

Caveats are superficial or meaningless.

Recommendations for how the model should be used to make decisions are missing or unclear.

Describes appropriate caveats or remaining concerns to using the model.

Articulates recommendations for how the model should be used to make decisions.

 All expectations of typical projects + demonstrates serious thought and consideration to relevant caveats and recommendations.

Report

Category Less developed projects Typical projects More developed projects
Objective(s) Objective is not clearly stated or significantly limits potential analysis. Clearly states the objective(s), which have moderate potential for relevant impact. Clearly states complex objective(s) that leads to significant potential for relevant impact.
Data description

Simple description of some aspects of the dataset, little consideration for sources.

The description is missing answers to applicable questions detailed in the “Datasheets for Datasets” paper.

 Answers all relevant questions in the “Datasheets for Datasets” paper. All expectations of typical projects + credits and values data sources.
Resampling strategy Does not use resampling methods (or an inappropriate method) to ensure robust model evaluation. Selects an appropriate resampling strategy. All expectations of typical projects + provides a thorough justification for the resampling strategy.
Overview of modeling strategies

Includes only simplistic models. Does not demonstrate understanding of the types of models covered in the course.

Feature engineering steps are non-existent.

Does not select evaluation metrics or metrics are not appropriate to the objective(s) + models.

Identifies several modeling strategies which could generate a high-performance model.

Documents relevant feature engineering steps to be evaluate for specific modeling strategies. Steps are selectively applied to appropriate models.

Evaluation metrics are appropriate for the objective(s) + models.

 All expectations of typical projects + provides thorough explanation for the models/feature engineering/metrics. Shows care in selecting their methods.
Model comparison Results are presented in a disjointed or uninterpretable manner.

Reports the results of their modeling strategies.

Results are presented in a clear and interpretable manner.

Utilizes tables and/or figures to effectively communicate with the audience.

All expectations of typical projects + reports results in an effective and concise narrative.

Tables and/or figures show care and deliberation in their design.
Model evaluation

Does not fit a single model using the entire training set.

No documentation for why the team selected this model.

Performance metrics from the test set are missing.

Explanation of how the model works is missing or incomplete.

Fits a single model based on the results of the model comparison. Provides written explanation for why they selected this specific model.

Reports performance metrics using the test set.

Explains how the model works using appropriate techniques (e.g. feature importance scores, partial dependence plots, observation-specific explanations).

All expectations of typical projects + reports evaluation in an effective and concise narrative.

Tables and/or figures show care and deliberation in their design.
Recommendations Recommendations are unclear or unsupported by the performance metrics.

Provides a clear set of recommendations regarding how to use the model.

Identifies limitations and potential improvements in future iterations of the model.

 All expectations of typical projects + recommendations are actionable. It is clear how this model will be used in a production environment.

Reproducibility + organization

Category Less developed projects Typical projects
Reproducibility Required files are missing. Quarto files do not render successfully (except for if a package needs to be installed). All required files are provided. Project files (e.g. Quarto, Shiny apps, R scripts) render without issues and reproduce the necessary outputs.
Data documentation Codebook is missing. No local copies of data files. All datasets are stored in a data folder, a codebook is provided, and a local copy of the data file is used in the code where needed.
File readability Documents lack a clear structure. There are extraneous materials in the repo and/or files are not clearly organized. Documents (Quarto files and R scripts) are well structured and easy to follow. No extraneous materials.
Issues Issues have been left open, or are closed mostly without specific commits addressing them. All issues are closed, mostly with specific commits addressing them.

Late work policy

There is no late work accepted on this project. Be sure to turn in your work early to avoid any technological mishaps.