Giving AI Agents a Way to Control the Physical World

In late 2024 Anthropic released the Model Context Protocol (MCP) spec, an open standard for LLMs to interact with and share data with external systems. A couple of months later a team and I spent a weekend building a robotic, AI-powered tiki bar at South Park Commons. The system talked to visitors, understood their preferences and current mood, and invented a customized drink for each of them. It then mixed up to eight ingredients in a cocktail and robotically served the beverage. Under the hood it ran surgical-grade peristaltic pumps and a sequence of servos, lights, and sensors. We won the hackathon, received an Anthropic sponsorship, and our robot got hired to mix drinks for Jensen Huang.The interesting part is how we built it. Rather than a classical control system governed by a state machine with pre-defined drink recipes, we wanted to lean into LLM reasoning to drive the sequence. So we gave the model three things: observability into current system state (e.g. ingredient status:…