AI Home Control: Ex-OpenAI & Tesla Engineer Uses ‘Dobby’ to Automate His House

Andrej Karpathy, a prominent figure in the field of artificial intelligence and co-founder of OpenAI, has described a surprisingly domestic application of the technology: an AI-powered system he calls “Dobby” that manages his home, including controlling his pool and spa. This setup, revealed in a recent interview on the “No Priors” podcast, highlights the rapidly evolving capabilities of AI agents and their potential integration into everyday life.

Karpathy, who also previously served as the director of AI at Tesla and recently founded Eureka Labs, an AI education platform, explained that Dobby operates as a “claw” – a term gaining traction to describe autonomous AI agents capable of performing actions on a user’s behalf. Unlike traditional large language models that generate text-based responses, these AI claws can directly interact with applications and internet-connected devices. As reported by Fortune, Karpathy has been increasingly reliant on such agents, to the point where he estimates he hasn’t written a line of code since December.

How ‘Dobby’ Works: Beyond Text-Based Interaction

The core difference between Dobby and earlier AI systems lies in its ability to act, not just respond. Traditional chatbots, even sophisticated ones, primarily deliver information or generate text. AI claws, still, leverage frameworks like OpenClaw – recently acquired by OpenAI in February – to directly manipulate digital interfaces. This means Dobby can control lights, HVAC systems, shades, security systems, and even manage the pool and spa without requiring direct human input. The system operates by connecting to devices on Karpathy’s local network, adding a layer of security by preventing direct internet access to these systems.

Karpathy detailed one practical application: delivery tracking. Dobby sends him a WhatsApp message with an image from an external camera when a delivery truck arrives, alerting him to latest mail. This functionality alone replaced six separate apps on his phone, demonstrating the potential for consolidation and simplification offered by these agents. The system’s capabilities are further enhanced by recent hardware upgrades, including a DGX Station powered by a GB300 superchip, gifted to Karpathy by Nvidia CEO Jensen Huang. Karpathy details his work and the evolution of these systems on his website, karpathy.ai, including educational resources on large language models.

The Rise of AI Claws and Agentic AI

Dobby is an example of a broader trend toward “agentic AI.” This concept moves beyond simply building models that can perform specific tasks (like image recognition or language translation) to creating agents that can autonomously pursue goals. These agents can break down complex tasks into smaller steps, utilize various tools, and adapt to changing circumstances. OpenClaw, the framework underpinning Dobby’s functionality, has been instrumental in popularizing this approach. The open-source nature of OpenClaw has allowed a community of developers to contribute to its development and explore its potential applications.

The acquisition of OpenClaw by OpenAI signals the growing importance of agentic AI within the industry. Silicon Valley companies are now actively competing to develop similar technologies, recognizing their potential to revolutionize how we interact with technology. This competition is driving rapid innovation and pushing the boundaries of what’s possible with AI.

Impact on Daily Life and the Future of Home Automation

Karpathy’s experience with Dobby offers a glimpse into a future where AI agents seamlessly integrate into our daily routines. The ability to automate mundane tasks, such as controlling home appliances and managing deliveries, can free up valuable time and reduce cognitive load. However, the widespread adoption of AI claws also raises important questions about security, privacy, and control.

While Karpathy emphasizes the security benefits of Dobby operating on a local network, the potential for vulnerabilities remains. As these systems develop into more sophisticated and interconnected, they could become targets for malicious actors. The increasing reliance on AI agents raises concerns about data privacy and the potential for misuse of personal information.

Navigating the “Psychosis” of Rapid AI Advancement

Karpathy himself has expressed a sense of unease about the rapid pace of development in the field. He described feeling a “psychosis” as he observed his own coding workload being increasingly taken over by AI agents. This sentiment reflects a broader concern among AI researchers about the potential for these technologies to outpace our understanding and control.

The shift in Karpathy’s workflow – from writing 80% of his code to writing almost none – underscores the transformative power of agentic AI. It also highlights the need for ongoing research and development to ensure that these technologies are aligned with human values and goals.

What Comes Next: Refinement, Security, and Broader Access

The development of AI claws is still in its early stages. Future research will likely focus on improving the reliability, security, and robustness of these systems. Addressing potential vulnerabilities and ensuring data privacy will be crucial for building trust and fostering widespread adoption. Efforts will be needed to develop these technologies more accessible to a wider range of users, beyond those with technical expertise.

As AI claws become more sophisticated, One can expect to see them integrated into a wider range of applications, from healthcare and education to transportation and manufacturing. The potential benefits are enormous, but realizing these benefits will require careful planning, responsible development, and ongoing dialogue about the ethical and societal implications of this transformative technology. The evolution of systems like Dobby, and the ongoing work of researchers like Karpathy, will be central to shaping that future.