The Metaverse: What, How, Why, and When

Ever since Neal Stephenson coined the term ‘Metaverse’ in his influential 1992 novel ’Snow Crash’, the word has been stretched to encompass all manners of shared virtual worlds, and with the recent business focus, the search for meaning behind this term has reached epic proportion.  Rather than trying to project it down, in this class we’re blowing it open to expose the evolving many sides of the Metaverse from different angles.  Things are most exciting before we really know what they are, and this is such a moment.  In this class, we’ll attempt to cover different components of what we see the ‘Metaverse,’ tracing it from its inception and prior incarnations, exploring its exciting intersection with the Internet of Things, and looking at its future.  The class will be in two components – one a paired lecture/symposium, with two noted practitioners, researchers, influencers, and authorities describing a particular topic in a 2 hour session with some time for Q&A and discussion.  Students taking the class for credit will be required to complete a series of projects, which will be handled in a separate recitation section.  We request that all interested students come to the first lecture and sign up there.


Main Lecturers / Conveners: Dr. Valentin Heun, Prof. Joe Paradiso, Prof. Hiroshi Ishii

TA’s: D. Pillis (, Cathy Fang (

Class Times: Meet in E15-341, Thursdays from 2:30-5 PM, Recitation Sessions Thursdays from 5-6 PM


Credit: Students taking the class for credit (12 units) will attend the lecture and recitation sections and complete 4 projects during the term, one each inspired by the lectures and ‘Chapter’ themes (Presence, Convergence, Artificial vs. Real, and New ‘Metaverse’ Realities) the projects can build and concatenate into one project that will be presented at the end of the term.

Class Size: Lectures will be open to the ML community and the for-credit class size will be limited to 15-20 due to the overhead in administering projects.  Some guests will present over zoom, while others will present in person (e.g., those who are local).  Class members will have first preference in speaker Q&A and some physical sessions may be reserved for the class.


Additional Speakers To Be Announced

Readings for each week in the full syllabus to be distributed in class


Chapter 1: Introduction

Week 1 

Feb. 9
Lecture: Welcome, orientation, edge of tomorrow

Recitation: Intro to VR/AR Workflows
Assignment: Project 1 Pitch

Week 2

Feb. 16
Lecture: Escape from Flatland: Why is Spatial Context important?
Primary Speakers: Valentin Heun, Hiroshi Ishii

Suggested Readings:

  • Weiser, Mark. “The computer for the 21st century.” ACM SIGMOBILE mobile comuting and communications review3.3 (1999): 3-11.
    B. Moggridge and B. Atkinson, “The Desktop Metaphor,” in Designinginteractions,vol. 14, MIT press Cambridge, MA, 2007, p. 53.
    R. A. Bolt, “Put-that-there”: Voice and gesture at the graphics interface, vol. 14, no. 3. ACM, 1980.
    H. Ishii, D. Lakatos, L. Bonanni, and J.-B. Labrune, “Radical atoms: beyond tangible bits, toward transformable materials,” interactions,vol. 19, no. 1, pp. 38-51, 2012.
  • Greenwold, Simon. “Spatial computing.” Massachusetts Institute of Technology, Master (2003).
  •  L. B. Rosenberg, “The use of Virtual Fixtures to Enhance Operator Performance in Time Delayed Teleoperation.,” DTIC Document, 1993.
  • S. Feiner, B. Macintyre, and D. Seligmann, “Knowledge-based augmented reality,” Commun. ACM, vol. 36, no. 7, pp. 53-62, 1993.
  • U. Brandes and M. Erlhoff, My desk is my castle: exploring personalizationcultures.
  • Walter de Gruyter, 2012.
  • J. Underkoffler and H. Ishii, “Urp: A luminous-tangible workbench for urban planning and design,” in Proceedings of the SIGCHI con/erence on Human factors in computing systems: the CHI is the limit, 1999, pp. 386-393.
  • Heun, Valentin Markus Josef. The reality editor: an open and universal tool for understanding and controlling the physical world. Diss. Massachusetts Institute of Technology, 2017.
  • Lifton, J. and Paradiso, J.A., “Dual Reality: Merging the Real and Virtual,” in Proceedings of the First International ICST Conference on Facets of Virtual Environments (FaVE), Berlin, Germany, 27-29 July 2009, Springer LNICST 33, pp. 12-28.
  • Lifton, J., Laibowitz, M., Harry, D., Gong, N.W., Mittal, M. and Paradiso, J.A., “Metaphor and Manifestation – Cross Reality with Ubiquitous Sensor/Actuator Networks,” IEEE Pervasive Computing Magazine, July-September 2009 (vol. 8 no. 3), pp. 24-33.
  • Dublon, G., Pardue, L. S., Mayton, B., Swartz, N., Joliat, N., Hurst, P. and Paradiso, J.A., “DoppelLab: Tools for Exploring and Harnessing Multimodal Sensor Network Data,” in IEEE SENSORS 2011, Limerick Ireland, Oct. 28-31, 2011, pp. 1612-1615.
  • Dublon, G., Paradiso, J. A., “How a Sensor-Filled World Will Change Human Consciousness,” Scientific American, July 2014, pp. 36-41.
  • Mayton, B., Dublon, G., Russell, S., Lynch, E.F., Haddad, D.D., Ramasubramanian, V., Duhart, C., Davenport, G., and Paradiso, J.A., “The Networked Sensory Landscape: Capturing and Experiencing Ecological Change Across Scales”, Presence: Teleoperators and Virtual Environments, 26(2), MIT Press, 2017.

Recitation: Materials, Lighting, 3D Design Tools
Project 1 Pitch Review: Team Assignment

Chapter 2: Presence (Three Classes)

Week 3

Feb. 23
Lecture: From AR to the Metaverse and Beyond
Primary Speakers: Avi Bar Zeev, Steve Feiner

Suggested Readings:

Feb. 24
Recitation: Unity for Programming Interactions, Introduction to Spatial Awareness, Inside Out/Outside In Tracking & Experimental Motion Capture Overview
Project 1 Work Session

Week 4

March 2
Lecture: Scalable Presence, varying perspective, and situated experiences
Primary Speakers:

Karthik Bala
CEO at Velan Studios, Inc.

Dr. Shunichi Kasahara
Researcher at SonyCSL

Suggested Readings:


Digital Twins, Sensors for 3D Volumetric Capture
Project 1 Work Session

Week 5

March 9
Lecture: IoT, Robots and the Metaverse


Mark Benson
Head of Samsung SmartThings

Cody Paige, Don Haddad

Week 6

March 16
Lecture: Avatars and presence in the Metaverse

Timmu Tõke
CEO at Ready Player Me

Athena Demos
CEO of Big Rock Creative 

Suggested Readings:

  • Beth Coleman, “Hello Avatar”, MIT Press


Project 1 Review/Presentation
Review and Critique by Hiroshi Ishii (3 weeks for project 1)

Chapter 3: Convergence (Four Classes)

Week 6

March. 23
Lecture: Making the physical world digital


Michael Naimark, Nikolaos Vlavianos

Suggested Readings:


Tools for Immersive Experience Design
Project 2 Work Session

Week 7

March. 30
Lecture: Metaverse Platforms and Experiences


Jinha Lee
Co-Founder of Spatial

Marko Tempest
Creative Technologist, Strategy Consultant, Keynote Speaker

Week 8 

April 13
Lecture: Sensemaking of the world: Machine Learning / Tracking People

Emilio Maggio
Director of Engineer at PTC Vuforia

Jason Saragih
Director Research Scientist at Meta


Tracking 2.0, Computer Vision for Virtual Environments
Project 2 Review / Presentation
Review and Critique by Hiroshi Ishii

Chapter 4: Artificial 

Week 9

April 20
Lecture: Virtual Realities: We Have All Been Here Before


Jon Radoff, Jaron Lanier

Suggested Readings:


Final Project Pitches and work time
(Five weeks for the Final Project)

Week 10

April 27
Lecture: Synthesizing Reality


Andreas Blattmann & Robin Rombach
Co-Author of “High-resolution image synthesis with latent diffusion models”
(Stable Diffusion)

Matthew Tancik
Co-Author of NeRF and NeRF Studio

Suggested Readings:

Final Project Work Session

Chapter 5: New Realities

Week 11

May 4
Lecture: Digital Assets and Digital Economies behind the X-Verse


Gabriel René
CEO at & Executive Director at Spatial Web Foundation


Suggested Readings:

  • Nakamoto, Satoshi, and A. Bitcoin. “A peer-to-peer electronic cash system.” Bitcoin.–URL: https://bitcoin. org/bitcoin. pdf 4 (2008): 2.

Recitation: Final Project Work Session
Review and Critique by Hiroshi Ishii

Week 12

May 11
Lecture: Being connected in the Metaverse


Matthew Ball, Neal Stephenson

Recitation: Final Project Work Session