Introduction to Foundation Models

Foundation Model Definitions¶

Look at different definitions of foundation models.

1. What do the definitions have in common?

2. What is different between the definitions?

3. Why are foundation models “incomplete”?

Significance of Foundation Models¶

1. What is emergence? Give an example for foundation models.

2. What is homogeneization? Give an example for foundation models.

3. Why does emergence and homogeneization make foundation models significant?

4. What risks do emergence and homogeneization pose when compared to a standard deep learning approach?

Foundation Model Components¶

1. What made foundation models possible?

2. What is self-supervised learning? Give an example outside of the foundation models domain.

3. What is transfer learning? Give an example outside of the foundation models domain.

4. How do foundation models relate to self-supervised and transfer learning?

Homework¶

1. Finish this exercise sheet and review the solutions! The sheets are not graded but solving them helps a lot to prepare for the exam. We provide solutions to the exercises on Ilias. Write your own solutions down FIRST (best by hand!) and review our solutions only AFTER you finished the exercises.

2. Read “Section 1: Introduction” from On the Opportunities and Risks of Foundation Models, Bommasani et al. 2021

3. Read The Annotated Transformer, an annotated version of the “Attention is All You Need” paper.

4. BONUS: Implement a transformer from scratch based on the assignment from Stanford’s CS336. Yes, this is a lot, but it is rewarding and since next week is free you might have the time. Next exercise, in the first 20 minutes, we can review this assignment.