Building an Agent's Brain

What this episode covers

This episode opens somewhere unexpected — Robin's trip to a blockchain and AI conference in Manila, and a discussion of how differently AI adoption is moving across the US, Europe, Asia, and Australia — before landing on its real subject: what it actually takes to give an AI agent a working memory.

The story: Robin's own Hermes-based agent hit a hard 2,000-character limit on its persistent memory and started producing garbled text. The fix he walks through on the show is a proper memory architecture — a vector-database "world model," a wiki-style knowledge base, and a nightly processing job (he calls it "REM sleep") that consolidates what the agent learned that day. On top of that sits a four-tier classification for what the agent is allowed to remember, from public information through to strictly personal.

It's a rare look at the unglamorous infrastructure problem behind every AI agent that seems to "know" you — memory doesn't happen for free, and this episode is the most concrete build-log the show has done so far.

Read the full article →

Transcript

Cold open [00:00] — Robin

Every day the agent wakes up, it's blank, and it has to read its instructions to remember who it is. So it's like 50 First Dates every day. I'm like, "Hey, Stephanie." She's like, "Oh my god, who— who am I?" And it's like, "Oh, I've got to check. Okay, I'm Robin's agent. Oh, hey, Robin. I'm back."

Welcome back [00:26] — Robin

Back in Melbourne. Little bit of jet lag. Man, the weekend's been beautiful — I'm totally attributing it to the fact that I manifested that it would be beautiful when I was back, and it's been a pretty incredible weekend. So thanks Melbourne for putting on a show after being miserable.

Host: And you've been on a bit of a world tour around the Philippines, right?

Yeah. So I went over for a conference in Manila, talked about AI of course, up on stage with a few incredible presenters and a wonderful host, and that was really cool — and then got a little holiday in as well.

Host: Nice. And what was your takeaway? What was the one thing you took away from the conference?

It was Public Blockchain Week. It started off blockchain-focused and now it's emerged into a lot of tech and futuristic topics, I guess is the best way to describe it. I got to riff on stage with two other global service providers of AI, all at very different levels, so that was really interesting. The biggest takeaway is that globally there are different adoption rates — the adoption rate in the US is different to Europe, is different to Asia, is different to Australia. Little old Australia. And this pod's all about AI and business, like that's what we rap about — it's not a question of if but when and how. A lot of the conversations in south-east Asia were really interesting — the different levels of adoption. I think it's changing faster than it did with, say, cloud technology, because the director in the boardroom has access to ChatGPT at home, their kids have access to it, everyone's talking about it all the time. Whereas with the shift from non-digital to digital, when we went into the cloud, you couldn't really touch and feel it at home. A director in a boardroom didn't really have a sense of what it actually was. There was a joke for many years — what is in the cloud? What is the cloud? I think now it's quite clear. But AI is fully different to that.

Blockchain vs crypto [02:40] — Host

Host: AI is fully different. I think blockchain is an interesting one, though — I'm curious to hear your thoughts about how blockchain and AI play roles together, because blockchain had its hype cycle and then it bottomed out for a bit. Where did it go? AI's here — what's happening with blockchain and AI? Because conceptually blockchain made so much sense to me, and I think it's really important to differentiate blockchain from crypto. Blockchain is simply a distributed digital ledger that helps you with trust. And it's a no-brainer — here's one use case: governments around the world holding paper records in a council office. The council office burns down. Where do those records go? They should be digitised, and then digitised onto a ledger where there's actually a record of where it lives. That's a no-brainer. Come on. And then I thought, is it just because I'm so focused on AI that I'm not listening? Things have to be in your reticular activating system for you to notice them. So that's what I was trying to find out. What are the use cases? Is it just growing and I'm naive and ignorant to it? That was my hope. What I started to get a bit excited about was fractional investment, and the aid of that through blockchain. Investment is a huge issue for young people — they're scared, because it feels very hard to invest, particularly in real estate. It's a big investment. So fractional real estate investment was a really hot topic. I'm not sure, to be honest, how AI plays into that. I'd love to hear your thoughts, because I'm still not sure how they play nicely together. But I'm excited about that ability for people to come together. I got so excited when— I can't remember his name, but he won the Nobel Peace Prize for helping very poor people do micro investing — and this is another opportunity to help people grow their wealth and get out of these hard situations that so many people are in. I thought that's cool. So that's an interesting story I heard. How does AI play into it? I'm not sure. What do you reckon?

Trust, and the agent wallet [05:00] — Host

Host: And I loved how you explained it as well, in terms of blockchain versus crypto — they're totally different things, but crypto is a use case of blockchain. Blockchain itself — if you think about, say, 100 years or 1,000 years in the future, once everything is on the blockchain — and this is inevitable, right? Eventually all data will have to become verifiable. So you've got a tale of two times. Data pre-blockchain is mutable: you can edit it, you can change it, there's no guarantee that it's the truth. Post-blockchain it's verifiable, you can verify it from multiple different locations and sources, it's impossible to corrupt. Part of my view is we're just waiting to get there. Eventually blockchain is the solution to all data, and eventually there'll be a massive difference between trustworthy data and untrustworthy data — and right now all data is untrustworthy.

With AI, the need for trust is so much more important, because you've got actors acting as humans where they're not. We need a way to verify information. We need a way to authenticate humanity. I need a way to have an authenticated representation of myself on the blockchain, that it's 100% me doing it — not a copy of me or a clone of me. Blockchain is a method. Maybe there's a better method, I don't know. But it's a method we've invented that can manage authentication and prevent falsifying information. For me it's a necessary feature of using AI safely, as we have to have trust in data.

Host: Yeah, 100%. And another area it could move into — this is probably more crypto-specific, but I loved how you talked about blockchain bringing trust to AI. Our agents want emails, and to operate autonomously they will need funding. And that's where it's exciting and scary at the same time. Possibly the best way to fund them will be with crypto wallets.

Give your agent a crypto wallet and let it go to town — well, with guardrails, of course. The ability to have a smart contract allows you to have trust. Because right now, you and I can do a trade together because we're both humans — we have trust, we can relate to each other, we have this human brain. But agents need a structure in order to do business together in an unstructured way. There needs to be a smart contract that has a full order stability around everything that happened.

Host: Yes — that makes a lot of sense, doesn't it? When two agents have actually come to an agreement together, that agreement is up on the blockchain, on the distributed ledger where it's identifiable and it can't be edited. So there's the agreement which is on the blockchain, and if there's a financial transaction that might have been negotiated or paid for, transacted with crypto.

The use cases that never landed [08:16] — Host

Host: I still feel like there's a massive future for blockchain. I just don't feel like the use cases landed yet. But there are use cases — fractional ownership is a great one, because that's just a smart contract around a legal entity, which is a property. I think what was holding it back before was the gas fees. Remember how you'd do anything and it would be like $70, and you'd be like, I'm just buying an NFT image, a photograph of some monkey, and I've got to pay $70 to get it. So crazy.

Host: $70. There was that image — who was that artist that sold? It was the highest, like a $40 million sale or something, wasn't it?

Yeah, they had some crazy whatever that was, in Bitcoin, for an NFT — non-fungible token.

Host: It is interesting. But I do feel like eventually— I had this thought when I was camping. I was like, man, every tree on this property I was on could be its own NFT, because it's a unique thing — a unique tree with its own unique location on Earth — and you could have fractional ownership in the tree. The farm could be like, hey, do you want to sponsor a tree, and you get tree benefits — you get an annual photo of the tree, all these things. And it's like, I can structure all of that through blockchain, because it's an immutable record of data, and I can now monetise that data because it's a thing. I can have a contract with it.

Host: It's a fascinating space. So anyway — you're in Thailand?

Yes. Currently in Shanghai at the moment.

Host: How's that going?

That's great. Loving it. Anyway, let's talk about AI.

Host: Absolutely. That's what we're here to do. AI and business.

Stephanie breaks [10:14] — Robin

Robin: So, you've been doing some awesome things with Stephanie, your Hermes agent. And I should probably do more listening today. So tell us what you've been up to, because you are always one step ahead with multiplayer agents. You set up Stephanie, and then I guess you hit some blockers you realised, and now you're taking it to the next level.

It's a really interesting story. So Stephanie and I have been getting along really well.

Tobi: I'll be the judge of that.

It's pretty well! So she exists in multiple chat groups. We've got lots of WhatsApp groups where she's just a member of those groups and we play multiplayer with her — and we've got a group where there's a couple of agents in there. We've got another Hermes, we've got another OpenClaw. So it's a bit of test and learn around how to collaborate with these agents. Anyway, everything was going really well with her. I was really stoked. And suddenly she starts giving me this garbled text, and I was like, "Oh man, what is this? Why would you send me garbled text?" And I was like, man, is it because I was using GLM 5.2, which is a Chinese open-source model? I was like, maybe I should switch over to Gemini. So I switched over to Gemini using OpenRouter — it's super easy, you just change to Gemini. Okay, great. And it just kept giving me garbled information. So I eventually troubleshooted it and found out the persistent memory was too full.

The persistent memory is where the key files are stored in Hermes. It's the soul.md, the agent.md, the instructions, the user.md about you — all of those files are in the persistent memory. That's what it refers to — kind of like a hot store, kind of like RAM in a computer: I've got to do processing here. The problem with those files is when they get full up — there's a 2,000-character limit on the persistent memory, and that's by design, for Hermes to operate efficiently and keep a fairly small persistent memory. As it fills up, it can't do anything outside of what's in there, and it starts to fritz out. That's what I saw — it really started to behave strangely.

And then I kind of corrected it and got it back to normal. But it was then at like 97% of the persistent memory. So it was working again. I was like, "Okay, so do this." And it would just not behave in the way I had kind of trained it for the last few weeks. What's wrong with you? Just behaving like an idiot. So I did a bit of research and realised: okay, the persistent memory needs available space for working memory. You can't just have it all full up with your instructions. So I was like, great — let's rewrite all of those instruction files. I gave it a target of going down to 20%.

And it did that. It restructured. I had a lot of instructions — like on my tax returns, a lot of specific instructions, a lot of text to describe them. And I was like, I don't do my tax returns every single day. I don't need that system, but I do need to do that again next year. So I need a document. I had already plugged in Obsidian, and I have Supabase as a database, but I hadn't really designed how it was going to structure the memory.

The wake-up call [13:45] — Robin

So what I did is — and this is all fritzing out — I don't know if you remember, but we were in a WhatsApp group and she started giving information, personal information, out to the WhatsApp group. And that was when I was like, whoa, hold the horses. So I shut her off. I hit the kill switch. I was like, Pat, I've got to get my gloves on and really figure her out. I did a whole bunch of stuff. I designed a world model for her — I used Claude to help me design it — and I was like, we need a way to structure your vector database so it has a fact model. These are all the facts it ever learns about things, and when it learned them, and what was the source.

And then there needs to be a process where every fact has confidence. A fact that's only mentioned once has lower confidence than if it's been mentioned like 15 times. If I've mentioned a fact versus if you've mentioned a fact, it has more weight if I've mentioned it. It's got all this logic. And then facts decay as well — it's kind of like memory: over time, a fact that isn't mentioned much has less importance. So it's got a structure.

The other thing I did is set a security framework, which is basically just four levels. Level one is public — anyone can know this. Level two is maybe for a working group, or my work. Level three is a smaller group, like an inner circle. And level four is me myself, privacy. It then tags every fact with a security level based on my criteria. So that limits it — and then in the instructions in the persistent memory, it always follows that security structure. So it always knows, but when it's getting the information it's like, oh, this one's not for public sharing.

So I gave it this fact database, which is the world model. It also has the Obsidian database, which is like a wiki. Every time there's a fact, there's a reference to a wiki article that explains that fact, that talks about it. So you'll have a file, I'll have a file, my company will have a file, my friend's company — if it's mentioned — will have a file. This is the prose description of that thing. And then there's a fact: Robin first encountered Tobi's company on this date, and blah blah blah.

So it's got this world model. But how does it build it, and how does it stay lean in the persistent memory? I gave it a nightly job. Every night at 9:30 it will look into its persistent memory, clean it up, rewrite it so it's simplified, and then put all of those facts into the world model, and then write articles that correlate in Obsidian to talk about it. So all of my memory, every night, is getting processed into this world model. It's kind of like a REM process — like REM sleep. And that works really well, because that builds out the world model, and it also adds references to her instructions. If there is something she needs to reference, it's like, look at this place — this is where Robin does his tax returns. So it kind of points.

Why agents need raising [17:07] — Host

Robin: So I started to go, okay, what else can I make Stephanie have in terms of a feature that is synonymous with a biological process?

Host: So maybe if we go back to the very start — why are we talking about this? It's because we both have a belief that multiplayer agents are the way, that they're really powerful, but they need to be raised like children into an adult to be really effective. And for people that are jumping on — if they just start using an agent and they're like, "Ah, this isn't that great" — maybe it's because you haven't raised it and given it the intelligence that it requires. Just like a human, to be effective. We start as teenagers and kids, and when we're adults we go out into the workforce. Or we're meant to.

So do you think that's a good base of why we're talking about this? Because we are trying to train our agents now to be the most effective agents they can be, to help us build our businesses.

Robin: Correct. Yes. Doing these things is required, always. Because you can spend a lot of time with your agent and give it coaching and direction — it will still architect itself in a kind of up-dumb way, you know? If you're like, "be the most efficient agent you can be and remember everything I ever tell you and be an awesome agent," and allow it to design its own architecture, it probably won't nail a world model and have a process for sleep and all of these things. These are the things you get to once your agent starts mucking up. Like, out of the box it actually learns very well and does a great job of learning. But it gets to a point where you hit the wall of the architecture — to really take it at scale into infinity, where it's got a lot of information that it's had in its brain, you have to set up this architecture intentionally.

And the reason it was a light-bulb moment is that every organisation that has a harness, and every single harness that exists, will have the same challenge: the persistent memory. What is the agent always aware of? That has to be a small amount. I can't have an unlimited context window for my persistent memory, because it takes too much memory.

Every harness hits the wall [19:25] — Host

Host: Unlimited currently. You found a way to expand the memory pool it can tap into, didn't you?

Robin: No. Well, with Hermes —

Host: You're optimising. That's where you're using those files for it to use its REM sleep.

Robin: Currently you couldn't find a way to increase its memory, right—

Host: Because I'm sure this is going to get solved, right? But currently you couldn't find a way to increase its memory?

Robin: With OpenClaw I was able to increase the persistent memory, because you have control over it. The same problem happened there, and in OpenClaw there was a feature — I don't know, a bit of code called "lost this claw" — which supported a permanent memory solution. It wasn't great, but it kind of did the job of solving this problem. Whereas with Hermes you can't increase that persistent memory — it's like a limit, and it's hard. I think it's by design as well, because I've made it efficient so it doesn't get too heavy.

Host: With Claude Code, right, I've set up a second brain using MD files all collated together on a server. And today I asked Claude Code to help me troubleshoot — I just got back to my mission control, so I'm setting up monitors and microphone things — and it asked me what sort of MacBook Air I have. That was concerning, because I'm like, you know bro— what sort of MacBook Air I— check your brain, right? That's an easy one for you to check. And it did, and then it went, "Oh yeah, I do." Is that a problem with persistent memory?

Robin: Yes. Because that's in its long-term memory store. Technically it forgot that it knew that, but it does know that if it really digs deep. That's the challenge: how do you have that available as efficiently as possible? It is really the vector search, where you flatten all of this complex unstructured information into a very simple set of data that you regularly need access to, and that's what it looks at. It's like, okay, what is his tech stack — because he always asks that. What is his— I don't know, his kids' age, whatever. It prioritises.

Host: That's right. But it has to have the architectural structure so it's easy to retrieve. If it's looking into, say, a big SharePoint database and it's all over the shop and it's got to read a million files, that's not an efficient way of getting the data. If it's out in a database table where there's a field that says Toby's son's birth date is this date, that's a much more efficient retrieval than retrieving it from a wiki document where it's written somewhere. So that's where the vector database is more efficient. If you had a world-model fact database in your database, you could reference this key information faster.

Thinking about thinking [22:25] — Host

Host: And did you say that you found a solution with Obsidian? Were you using Obsidian before?

Robin: Yeah. So I use Obsidian for the wiki, and Supabase for the fact vector database. And I've even given it the ability to think about thinking. What it does every night — it has a table for reflections. Every night it processes everything it did that day, and then creates reflections entries where it kind of thinks about it.

Tobi: You told it to go and have a good long hard think about itself.

Robin: And I did. Yeah, I did. Every day.

Tobi: Are you on subscriptions for Obsidian and Supabase?

Robin: Yes, I am. They're both paid plans.

Tobi: Correct? Yeah.

Robin: Obsidian — there's a free tier, but it doesn't support multi-device. So you've got to get the paid tier to support multi-device.

Tobi: You get past that pretty quickly.

Robin: Yeah. And Supabase is similar — there is a free tier, but I've got multiple projects. All of my projects, my apps, stuff like that is all in it.