In today's rapidly evolving tech landscape, the story of Datadog's production migration is a fascinating glimpse into the potential and challenges of AI-assisted engineering. Datadog's journey, as shared by engineer Arnold Wakim, offers a unique perspective on how AI can revolutionize critical systems while also highlighting the human touch still required in this process.
The Challenge: Scaling Stream Router
Datadog's Stream Router, an API managing routing in the metrics pipeline, faced a significant challenge as its routing table expanded. The key-value (KV) database model, while initially effective, began to slow down dramatically due to transaction size limits. This slowdown, with operations taking up to 45 minutes, was a clear indication that a change was necessary.
Redesigned Schema and AI Refactoring
The Datadog team's solution involved a careful redesign of the schema to reflect relationships between domain entities, replacing complex logic in the application code with foreign keys. This new schema needed to be implemented across the entire codebase, a task that Datadog turned to AI for assistance.
AI-Assisted Refactoring with Claude and Cursor
Datadog used Claude and Cursor to accelerate a test-driven refactoring process. For each method, they provided the old implementation, the new schema, and a failing test, allowing the models to generate a first pass. The tests then determined the correctness of the generated code.
Key Success Factors
According to Wakim, three key elements contributed to the successful migration: strong code modularity, a comprehensive test suite, and a parallel infrastructure. The modularity of the new Stream Router allowed it to implement the same API on PostgreSQL without changes elsewhere. The test suite provided a clear pass/fail criterion for every AI-generated change, and the parallel infrastructure ensured continuous comparison and immediate alerts for any divergences.
Migration Phases and Results
The migration process was divided into three phases: describing the intent of key functions with Claude, crafting prompts to fix failing tests, and deploying to production using a blue/green approach. The results were impressive: operation times dropped from 45 minutes to a second, latencies decreased significantly, and data storage became up to 40x smaller. PostgreSQL and DuckDB simplified relationship handling and improved query efficiency, leading to reduced CPU and memory usage and a 90% decrease in database costs.
Where AI Fell Short
However, Wakim also noted areas where the AI approach fell short. Higher-level prompts were less effective, and Claude tended to generate correct but suboptimal queries, requiring human input for niche optimizations. Additionally, high token consumption was a challenge, driven by the use of full test output dumps and the iterative loop of test output, code context, and schema information.
Conclusion: The Role of Human Expertise
In my opinion, Datadog's story underscores the importance of human expertise in AI-assisted engineering. While AI can accelerate processes and generate code, it is the human touch that adds the necessary nuance and optimization. The strength of Datadog's test suite, which determined how much they could trust AI-generated code, is a testament to the critical role of human oversight and interpretation in this field. This migration is a step towards a future where AI and human engineers collaborate, each bringing their unique strengths to the table.
