Agents to Do the things Claude Can't
Beyond Claude: AI Features in Marimo Notebooks and Bespoke Agents
This post explores how Marimo notebooks enable reactive programming, dynamic AI interactions, and customizable agents through Python. All of which are capabilities that go beyond what Claude Code can do.
Introduction to Marimo's Next-Generation Features
I recently had the pleasure of chatting with Vincent, who demo'd some things with Marimo, a next-generation notebook. I've been following Vincent for a long time [01:28] because his work isn't just interesting, it's practical. I've even been using Marimo in my recent talks [01:34] across various topics. So, I was super excited to dive into the AI features. Vincent's also created Bespoken [00:29], a way to create agents that do exactly what you want and scoped for particular tasks. Often there's a tendency to try to find a tool that does everything [00:36], but the truth is, tailored solutions often perform much better
Bespoken aims to be the exact opposite of a one-size-fits-all solution [01:12]. The goal is to give you an LM that starts with almost nothing but can be configured precisely to your needs.
Interactive UI Elements and Data Visualization
One of the standout features of Marimo is its reactive environment [03:23]. This means that changes to one cell automatically trigger updates in dependent cells, making it incredibly intuitive to work with dynamic data. As Vincent demonstrates [03:28]:
If you change A to 19 [03:45], any cell that depends on A will automatically re-run. This is because each cell "listens" to its parents, defined by the variables they use [04:00]. This eliminates the need to worry about the order in which cells are executed [04:27]
This idea can be extended to Marimo Widgets as well, for example, you can add a slider [04:36] and link it to a variable [04:53]
Sliding it around dynamically updates the value of A, and that updates the value shown for B. I [05:53] know in this example, it kind of looks like it's a little bit of a toy, but I use this all the time. If I'm working on a RAG application, I might have 10-30 samples and a slider for changing the number [06:06].
Marimo allows you to insert custom widgets [06:52] from libraries like anywidget. The text editor installs non-existent libraries [07:01] into the environment for you. After installing anywidget, Vincent shows off a drawing widget where you can draw data that is then displayed and updated automatically in a dataframe below [07:14].
AI Integration and Model Selection
Marimo brings powerful AI capabilities right into the notebook [05:30]. A chat sidebar allows you to select from various models and interact with your data in context [05:36].
Vincent points out [08:12] that you can connect to external data sources like Postgres or MotherDuck. Dataframes can be added as data sources [08:28], allowing you to run SQL queries directly against them [08:45].
But here's where it gets really interesting: the LLM can now "talk" to these dataframes [09:08]. You can ask it to perform calculations, generate code, and more [09:17]. Marimo injects the schema and first few rows of the dataframe into the prompt [09:32], giving the LLM the context it needs to work effectively. As Vincent notes [09:57]:
It's really nice to have an LLM that has access to the variables in memory.
This is where Marimo starts to shine compared to something like Claude [10:03]. While Claude can interact with files, it doesn't always have access to the runtime variables and data structures within your notebook either in a side chat or to add or modify a cell.
Bespoke Agents and Custom Tools
Vincent introduces Bespoken [27:47], a tool for creating highly customizable AI agents.
What would be the simplest ever API I could come up with such that it's like almost 80 to 100% configurable with just a couple of simple Python functions?
This is different from an "everything tool" [00:36] - it allows you to specify constraints so it only edits what you want, adds your own shortcuts, and overall configure the agent with some simple Python functions [27:33].
You can add slash commands that are Python functions which run [28:14], and there's also a UI library [28:20] so the python functions can ask for input from the user.
With Bespoken, you can create agents tailored to specific tasks, like a technical teacher [30:08] using the Socratic method [29:34].
Another great example is an agent that can code with you [33:12], with only limited access to files.
Vincent stresses [36:20] the importance of constraints and customization, allowing you to build your own library of agents for very specific tasks. He imagines an API where you can have different modes [36:07], such as brainstorming, writing, planning, and editing.
AI Safety and Prompt Injection
It's important to address the risks associated with AI, especially prompt injection [45:41]. Even with constraints, there's still a possibility for things to go wrong. As Vincent points out [46:23]:
Every talk I give about LLMs needs to end with there is also lots of dangerous stuff here still, right? Like we got to be mindful of that and constraints help, but it's a best it's a remedy. It's not a solution for that problem.
Combining tools increases this risk [46:33], as one tool might convince another to take actions you didn't intend. Isaac recounts an incident where private information about a confidential project was published in a GitHub issue by an agent on an open-source project and remained undiscovered by the project's owners for weeks [47:12].
He also warns about LLMs' tendency to blindly follow information they find on the web [47:45], even if it's intentionally misleading. It's crucial to be aware of these vulnerabilities and implement appropriate safety measures.
Key Takeaways
Marimo offers reactive programming capabilities that go beyond traditional notebooks.
Custom agents can be created with specific constraints and tools using Bespokein.
The platform supports multiple AI models with easy switching between providers.
Safety and control are prioritized through constrained tools and specific access permissions.
Q&A
Q [51:32]: When you're doing this AI, you can tag the dynamic variables. You showed that. What other context can it see in the, like when you're, when you're running it in a cell or in the chat window, you know, we're both, can it see the whole notebook or just cells above it?
A [52:06]: You can use Python modules and pass variables directly to the AI, so essentially any Python variable is passable. However, data sets and data sources are treated a bit differently. When a Python object is received, we need to convert it into a format suitable for a prompt. For datasets, we have special handling that works well. We're currently moving quickly on these AI tools, so there are still many open questions. For example, should we allow users to customize how variables are converted into prompt content? Or would that introduce too many risks? It's possible we’ll make changes to how prompts work with different types of data sets, but in general, if you pass a Python variable, we’ll do our best to include it in the prompt in a sensible way.
Q [57:01]: How does the AI in Marimo handle images and graphs? For example, if you have a graph or image in an output cell, can it see and respond to those?
A [57:09]: Not exactly. The AI can access images or charts if you assign them to a variable—like an Altair chart, for example—so it knows you’ve created a chart. However, even if you had an LLM that could look at the image, it would likely still make mistakes, because LLMs aren’t good at interpreting charts.
We can confirm that a variable is an Altair chart, and some of that context can be passed along. I haven’t tried it myself, but that’s how I expect it to work. Interpreting a chart directly is a different challenge. One feature we might add is the ability to upload an image in the sidebar and then ask the AI to generate a widget or take some action based on it. That would be a legitimate use case. But I want to be careful not to overpromise—just because the AI can see your chart doesn’t mean it will make the right decision. I’m cautious about that.