I gave my AIs a shared brain (and only leaked the keys once)

If you read my AI Stack page, you've met the crew. Kimi runs the long jobs. Claude writes the careful code. GPT drafts. Gemini reads entire codebases for breakfast. On paper it's a dream team.

In practice it's a dream team where every member has amnesia, and none of them have met.

Tell Claude on Monday that my server runs AlmaLinux with DirectAdmin and OpenLiteSpeed. Tuesday, ask Codex to fix something on that same server, and Codex looks at me like a golden retriever being shown a card trick. Wednesday, ChatGPT confidently suggests an Apache config. For a server that does not run Apache.

I wasn't running an AI team. I was running eight first days at a new job, every single day.

01The problem: memory lives in eight different houses

Most AI tools now have some kind of "memory." The catch is where it lives. Claude's memory sits with Anthropic. ChatGPT's sits with OpenAI. Gemini's sits with Google. Each one is a walled garden with a very nice wall.

• They can't share. A decision made in one chat doesn't exist anywhere else.
• I repeat myself constantly. Same projects, same server, same "please don't use Apache."
• The data isn't mine. It's on someone else's servers, in a format I can't really search, back up, or move.

I mentioned this to Claude, half complaining, half hoping. It came back with a plan that was almost annoyingly sensible.

02The idea: one brain, on my own server

Don't fight eight memory systems. Build one, host it on my own VPS, and plug every AI into it through MCP (Model Context Protocol). MCP is the open standard that lets AI tools connect to outside tools and data, basically USB-C for AI. Anything that speaks MCP can plug in. Anything that doesn't can use a plain REST API.

So now the crew roster looks like this, and every one of them reads from the same notebook:

• Claudeweb, desktop, mobile
• Claude Codethe terminal one
• ChatGPTvia connectors
• CodexCLI
• GeminiCLI
• KimiREST API
• GLMREST API
• QwenREST API

Each agent gets its own key, so every memory is signed by whoever wrote it. If ChatGPT ever saves something weird, I'll know exactly who to give a stern look.

03The build: Claude wrote it, Claude Code shipped it

Here's the part I still find a little funny. Claude designed and wrote the whole thing in a regular claude.ai chat: the server, database, admin CLI, tests, the systemd service, even the backup script. It handed me a zip file like a contractor dropping off blueprints.

Then Claude Code, running on the server itself, took over as the inspector. Before running a single line, it read every file to make sure nothing sketchy was hiding in there. That's either reassuring or deeply ironic, depending on how you feel about AIs auditing other AIs. Then it:

1. Built the project and ran the tests. Eight out of eight passed.
2. Installed it as a background service that survives reboots.
3. Found my existing memory subdomain, noticed my wildcard SSL certificate already covered it, and wired up the web server proxy.
4. Fixed a couple of bugs along the way, including one it spotted just by reading the code.
5. Pushed it all to GitHub with documentation.

Under the hood it's simple on purpose:

The agents get seven tools: remember, recall, list_recent, list_projects, get_memory, update_memory, and forget. That's the whole personality. No vector database, no Kubernetes, no seventeen microservices. Just one file I can back up with a single command, which is the most underrated feature in all of software.

04The bloopers (because of course there were bloopers)

I promised myself this blog would be honest, so here's the gag reel.

1. The keys that unlocked nothing. I generated eight shiny access keys. None of them worked. The admin tool wasn't reading its config file, so it quietly saved every key into a completely different database inside the project folder. Eight keys, cut for a door that didn't exist. Claude Code tracked it down and fixed the tool so it can't happen again.
2. I pasted all eight keys into an AI chat. The README said, in bold, "Don't paste keys into any AI chat." I pasted all eight keys into an AI chat. Luckily, see blooper #1: they were useless. Never has a bug been so well timed.
3. I ran the key script twice. Claude Code wrote a script that makes every key at once and hides them in a private file. I ran it. Then, a little too eagerly, I ran it again, which replaced every key I'd just made. The first file became a souvenir.
4. I typed the error message as a command. The server asked rm: remove regular file…? I copied that exact line and pasted it back as a command. Bash replied rm:: command not found, which is the computer version of a slow blink.

To Claude Code's credit, it never once said "I told you so." It just quietly added a "Lessons learned" section to the README. Which is somehow worse.

05Day one

The very first memory in the database was written by Claude Code. It was a note describing how the memory server itself is set up, so the next agent that needs to fix it already knows where everything lives.

The first thing my shared brain remembered was how to fix itself. I've hired people with worse instincts.

Now when I start a task, the AI calls recall, loads what the rest of the crew already knows, and skips the twenty-minute "so here's my setup" monologue. When we make a real decision, it gets saved with remember, under the right project, signed by whoever made it.

Is it perfect? No. Every key can read everything, because it's built for one person, not a company. And memory is only as good as what gets saved, so you still need decent habits. But for the first time, my AI crew works like a team instead of eight brilliant strangers who keep meeting each other for the first time.

06Steal this

The whole thing is open source. If you've got a cheap VPS and a few AI tools that don't talk to each other, you can run your own:

# clone, build, test
git clone https://github.com/williamlodge/ai-memory.git
cd ai-memory && npm ci && npm run build && npm test
# → 8 PASS lines, hopefully

# connect Claude Code
claude mcp add --transport http --scope user memory \
  https://your-domain/mcp --header "Authorization: Bearer <key>"

Want to plug in your own models? The setup guide has a step-by-step part for Claude, Claude Code, ChatGPT, Codex, Gemini, Kimi, GLM, and Qwen.

The README covers the full setup, how to connect every major AI tool, the security model, and yes, every blooper above, written up as lessons learned so you can make entirely new mistakes instead.

Just don't paste your keys into a chat. Trust me on that one.