Seminar: Scaling AI at the Speed of Light: Rethinking Compute, Memory, and Networks Through Optics

I attended my first seminar talk today at Cambridge, for the Computer Lab Systems Research Group by Dr Paolo Costa of Microsoft Research. Paolo introduced quite a fascinating idea and here’s my recollection of it (mistakes all mine). One of the biggest bottlenecks for inference workloads on GPUs is High Bandwidth Memory to load weights for processing by tensor cores. HBM has been rapidly increasing in price due to scarcity (hyperscalers are consuming HBM at an insatiable rate), even impacting the consumer market as manufacturing resources are diverted to more valuable lines. It’s difficult to physically scale the layers of HBM up, partially due to lower manufacturing yield and partially because cooling the system quickly becomes a bottleneck. Paolo’s team proposed the use of microLEDs to augment existing copper wiring in data centres, to allow for memory banks to be placed at a greater distance whilst preserving communication throughput. MicroLEDs have the advantage of being cheaper…