Ever since Neal Stephenson coined the term ‘Metaverse’ in his influential 1992 novel ’Snow Crash’, the word has been stretched to encompass all manners of shared virtual worlds, and with the recent business focus, the search for meaning behind this term has reached epic proportion. Rather than trying to project it down, in this class we’re blowing it open to expose the evolving many sides of the Metaverse from different angles. Things are most exciting before we really know what they are, and this is such a moment. In this class, we’ll attempt to cover different components of what we see the ‘Metaverse,’ tracing it from its inception and prior incarnations, exploring its exciting intersection with the Internet of Things, and looking at its future. The class will be in two components – one a paired lecture/symposium, with two noted practitioners, researchers, influencers, and authorities describing a particular topic in a 2 hour session with some time for Q&A and discussion. Students taking the class for credit will be required to complete a series of projects, which will be handled in a separate recitation section. We request that all interested students come to the first lecture and sign up there.
Main Lecturers / Conveners: Dr. Valentin Heun, Prof. Joe Paradiso, Prof. Hiroshi Ishii
TA’s: D. Pillis (pillis@media.mit.edu), Cathy Fang (catfang@media.mit.edu)
Class Times: Meet in E15-341, Thursdays from 2:30-5 PM, Recitation Sessions Thursdays from 5-6 PM
Zoom: https://mit.zoom.us/j/99744779878
Credit: Students taking the class for credit (12 units) will attend the lecture and recitation sections and complete 4 projects during the term, one each inspired by the lectures and ‘Chapter’ themes (Presence, Convergence, Artificial vs. Real, and New ‘Metaverse’ Realities) the projects can build and concatenate into one project that will be presented at the end of the term.
Class Size: Lectures will be open to the ML community and the for-credit class size will be limited to 15-20 due to the overhead in administering projects. Some guests will present over zoom, while others will present in person (e.g., those who are local). Class members will have first preference in speaker Q&A and some physical sessions may be reserved for the class.
Additional Speakers To Be Announced
Readings for each week in the full syllabus to be distributed in class
Chapter 1: Introduction
Week 1
Feb. 9
Lecture: Welcome, orientation, edge of tomorrow
Recitation: Intro to VR/AR Workflows
Assignment: Project 1 Pitch
Week 2
Feb. 16
Lecture: Escape from Flatland: Why is Spatial Context important?
Primary Speakers: Valentin Heun, Hiroshi Ishii
Suggested Readings:
- https://www.youtube.com/watch?v=B6rKUf9DWRI
- Weiser, Mark. “The computer for the 21st century.” ACM SIGMOBILE mobile comuting and communications review3.3 (1999): 3-11.
B. Moggridge and B. Atkinson, “The Desktop Metaphor,” in Designinginteractions,vol. 14, MIT press Cambridge, MA, 2007, p. 53.
R. A. Bolt, “Put-that-there”: Voice and gesture at the graphics interface, vol. 14, no. 3. ACM, 1980.
H. Ishii, D. Lakatos, L. Bonanni, and J.-B. Labrune, “Radical atoms: beyond tangible bits, toward transformable materials,” interactions,vol. 19, no. 1, pp. 38-51, 2012. - Greenwold, Simon. “Spatial computing.” Massachusetts Institute of Technology, Master (2003).
- L. B. Rosenberg, “The use of Virtual Fixtures to Enhance Operator Performance in Time Delayed Teleoperation.,” DTIC Document, 1993.
- S. Feiner, B. Macintyre, and D. Seligmann, “Knowledge-based augmented reality,” Commun. ACM, vol. 36, no. 7, pp. 53-62, 1993.
- U. Brandes and M. Erlhoff, My desk is my castle: exploring personalizationcultures.
- Walter de Gruyter, 2012.
- J. Underkoffler and H. Ishii, “Urp: A luminous-tangible workbench for urban planning and design,” in Proceedings of the SIGCHI con/erence on Human factors in computing systems: the CHI is the limit, 1999, pp. 386-393.
- Heun, Valentin Markus Josef. The reality editor: an open and universal tool for understanding and controlling the physical world. Diss. Massachusetts Institute of Technology, 2017.
- Lifton, J. and Paradiso, J.A., “Dual Reality: Merging the Real and Virtual,” in Proceedings of the First International ICST Conference on Facets of Virtual Environments (FaVE), Berlin, Germany, 27-29 July 2009, Springer LNICST 33, pp. 12-28.
- Lifton, J., Laibowitz, M., Harry, D., Gong, N.W., Mittal, M. and Paradiso, J.A., “Metaphor and Manifestation – Cross Reality with Ubiquitous Sensor/Actuator Networks,” IEEE Pervasive Computing Magazine, July-September 2009 (vol. 8 no. 3), pp. 24-33.
- Dublon, G., Pardue, L. S., Mayton, B., Swartz, N., Joliat, N., Hurst, P. and Paradiso, J.A., “DoppelLab: Tools for Exploring and Harnessing Multimodal Sensor Network Data,” in IEEE SENSORS 2011, Limerick Ireland, Oct. 28-31, 2011, pp. 1612-1615.
- Dublon, G., Paradiso, J. A., “How a Sensor-Filled World Will Change Human Consciousness,” Scientific American, July 2014, pp. 36-41.
- Mayton, B., Dublon, G., Russell, S., Lynch, E.F., Haddad, D.D., Ramasubramanian, V., Duhart, C., Davenport, G., and Paradiso, J.A., “The Networked Sensory Landscape: Capturing and Experiencing Ecological Change Across Scales”, Presence: Teleoperators and Virtual Environments, 26(2), MIT Press, 2017.
Recitation: Materials, Lighting, 3D Design Tools
Project 1 Pitch Review: Team Assignment
Chapter 2: Presence (Three Classes)
Week 3
Feb. 23
Lecture: From AR to the Metaverse and Beyond
Primary Speakers: Avi Bar Zeev, Steve Feiner
Suggested Readings:
- https://github.com/M3-org/awesome-metaverse
- https://github.com/pmndrs/react-xr
- https://hello.vrchat.com
- https://developer.nvidia.com/nvidia-omniverse-platform
- https://github.com/omigroup/omigroup
- Mann, S., Yuan, Y., Furness, T., Paradiso, J., and Coughlin, T., “Beyond the Metaverse: XV (eXtended meta/uni/Verse),” presented at IEEE CEC 2022. (https://www.researchgate.net/publication/366321241_Beyond_the_Metaverse_XV_eXtended_metauniVerse ).
- P. Milgram and F. Kishino, “A Taxonomy of Mixed Reality Visual Displays,” IEICE Trans. Information and Systems, vol. E77-D, no. 12, 1994, pp. 1321–1329.
- Azuma, R.T., “A Survey of Augmented Reality,” Presence: Teleoperators and Virtual Environments 6, 4 (August 1997), 355-385.
Feb. 24
Recitation: Unity for Programming Interactions, Introduction to Spatial Awareness, Inside Out/Outside In Tracking & Experimental Motion Capture Overview
Project 1 Work Session
Week 4
March 2
Lecture: Scalable Presence, varying perspective, and situated experiences
Primary Speakers:
Karthik Bala
CEO at Velan Studios, Inc.
Dr. Shunichi Kasahara
Researcher at SonyCSL
Suggested Readings:
- Shunichi Kasahara and Jun Rekimoto. 2014. JackIn: integrating first-person view with out-of-body vision generation for human-human augmentation. In Proceedings of the 5th Augmented Human International Conference (AH ’14). ACM, New York, NY, USA, , Article 46 , 8 pages. DOI=http://dx.doi.org/10.1145/2582051.2582097
- https://shunichikasahara.com/publications/
- https://www.youtube.com/watch?v=A7O7jzRfZ4k
- https://www.linkedin.com/posts/ajjames_just-a-goldfish-driving-its-robotic-fish-ugcPost-6831114591720763392-QR_2
- Rekimoto, Jun. “Traxion: a tactile interaction device with virtual force sensation.” ACM SIGGRAPH 2014 Emerging Technologies. 2014. 1-1.
- Higuchi, Keita, and Jun Rekimoto. “Flying head: a head motion synchronization mechanism for unmanned aerial vehicle control.” CHI’13 Extended Abstracts on Human Factors in Computing Systems. 2013. 2029-2038.
Recitation:
Digital Twins, Sensors for 3D Volumetric Capture
Project 1 Work Session
Week 5
March 9
Lecture: IoT, Robots and the Metaverse
Speakers:
Mark Benson
Head of Samsung SmartThings
Cody Paige, Don Haddad
Week 6
March 16
Lecture: Avatars and presence in the Metaverse
Speaker:
Timmu Tõke
CEO at Ready Player Me
Athena Demos
CEO of Big Rock Creative
Suggested Readings:
- Beth Coleman, “Hello Avatar”, MIT Press
Recitation:
Project 1 Review/Presentation
Review and Critique by Hiroshi Ishii (3 weeks for project 1)
Chapter 3: Convergence (Four Classes)
Week 6
March. 23
Lecture: Making the physical world digital
Speaker:
Michael Naimark, Nikolaos Vlavianos
Suggested Readings:
Recitation:
Tools for Immersive Experience Design
Project 2 Work Session
Week 7
March. 30
Lecture: Metaverse Platforms and Experiences
Speaker:
Jinha Lee
Co-Founder of Spatial
Marko Tempest
Creative Technologist, Strategy Consultant, Keynote Speaker
Week 8
April 13
Lecture: Sensemaking of the world: Machine Learning / Tracking People
Speaker:
Emilio Maggio
Director of Engineer at PTC Vuforia
Jason Saragih
Director Research Scientist at Meta
Recitation:
Tracking 2.0, Computer Vision for Virtual Environments
Project 2 Review / Presentation
Review and Critique by Hiroshi Ishii
Chapter 4: Artificial
Week 9
April 20
Lecture: Virtual Realities: We Have All Been Here Before
Speaker:
Jon Radoff, Jaron Lanier
Suggested Readings:
- https://arxiv.org/pdf/2206.03189.pdf
- https://hubs.mozilla.com
- https://decentraland.org
- https://secondlife.com
- https://somniumspace.com
- https://www.sandbox.game
- https://dreamwave.tech/secret-sky-2021
Recitation:
Final Project Pitches and work time
(Five weeks for the Final Project)
Week 10
April 27
Lecture: Synthesizing Reality
Speaker:
Andreas Blattmann & Robin Rombach
Co-Author of “High-resolution image synthesis with latent diffusion models”
(Stable Diffusion)
Matthew Tancik
Co-Author of NeRF and NeRF Studio
Suggested Readings:
- https://docs.nerf.studio/en/latest/index.html#
- https://github.com/awesome-NeRF/awesome-NeRF
- https://openai.com/dall-e-2/
- https://openai.com/blog/chatgpt/
Recitation:
Final Project Work Session
Chapter 5: New Realities
Week 11
May 4
Lecture: Digital Assets and Digital Economies behind the X-Verse
Speaker:
Gabriel René
CEO at Verses.io & Executive Director at Spatial Web Foundation
Suggested Readings:
- Nakamoto, Satoshi, and A. Bitcoin. “A peer-to-peer electronic cash system.” Bitcoin.–URL: https://bitcoin. org/bitcoin. pdf 4 (2008): 2.
Recitation: Final Project Work Session
Review and Critique by Hiroshi Ishii
Week 12
May 11
Lecture: Being connected in the Metaverse
Speaker:
Matthew Ball, Neal Stephenson
Recitation: Final Project Work Session