CNCF October Edition

My first time at Verda office in Helsinki. This is just the building across the road from the Oodi library. I had not given it a second look before today. I mean, it was just another building on my way to Oodi. Now, it's Verda's offices! Svitlana had the intro slides. Intro slides Intro about the host Serverless inference on Kubernetes Intro slide first talk The problem - Inference traffic comes in bursts - idle GPU is wasted money. Scaling to zero causes cold starts. Serverless inference means user provides the image and they get an endpoint. Serverless architecture Two types - Real-time sync and async Batch Jobs When nothing is running, KEDA scales to zero. The endpoint, queue and metadata remain alive. Queue system (NATS) holds requests until a replica is ready. Cold starts First we check if GPU is available.Then you pull the container image - images are large, so it’s a slow pull. Kubernetes prefers nodes that already have the image available. So that’s good.Loading models takes…