Baochun Li — blog

Defensible Moat and OpenAI

Baochun Li — Wed, 08 Apr 2026 00:00:00 GMT

I have read, in its entirety and with interest, an article written more than a year ago in November 2024, titled “Building LLMs is probably not going be a brilliant business,” by Cal Paterson. Paterson argued that, just like airlines, OpenAI may not have the moat it needed to justify its 800+ billion valuation, not to mention the kind of defensible moat that Buffett was looking for. Apparently, John Gruber of Daring Fireball agreed.

And Google agreed, too. In fact, in its leaked internal document, Google claimed that open source AI will outcompete OpenAI (and itself). Written in May 2023, the document can have quite a bit of truth in it today, given how Z.AI’s GLM 5.1 performs compared to Opus 4.6 and GPT 5.4, as well as the fact that this 745B model can be deployed locally.

Autoresearch

Baochun Li — Sun, 22 Mar 2026 00:00:00 GMT

It has been two weeks since Andrej Karpathy released Autoresearch.

It has a simple idea: give an AI agent an environment where it knows what is the benchmark that it should run and optimize on, and ask it to repeatedly take actions to optimize the project for this particular benchmark. Experiment runs on the benchmark can be used to keep or discard the optimization, and optimizations that are kept will accumulate over time.

Surprisingly, such a simple idea turns out to be extremely effective. As Kaparthy proclaimed:

…in any case no one could tell if that’s right or wrong as the “code” is now a self-modifying binary that has grown beyond human comprehension.

The key is to define a precise benchmark that can be used to evaluate any solutions to a problem, so that an AI agent — or multiple collaborating agents — can run this benchmark to decide whether an idea should be kept or discarded. Naturally, since this requirement is not too exacting, quite a large number of projects have spun up, including my own experiments trying the idea on the Days discrete-event network simulator, improving performance by over 25%. Autoresearch doesn’t really care about what you wish to optimize, as long as some precise benchmark is defined.

This requirement, however, was not really satisfied in many academic research papers. Often, it is difficult to read between the lines to see what a paper is trying to optimize for. A paper can go on for 10 pages, yet there is not a single prescribed benchmark that can precisely capture the problem that it wishes to solve, and how the paper advances the state-of-the-art on this benchmark. In my own words, these papers are not autoresearch-friendly.

Here are some noteworthy autoresearch projects over the past two weeks:

—

Shopify’s CEO, Tobi Lütke, announced that David Cortés and he implemented Autoresearch as a Pi extension, pi-autoresearch, in about 2500 lines of TypeScript code.

My own experiments in Days used this extension, and it worked extremely well. Without any prompts and with only /autoresearch, it would automatically dig into the codebase to find the most suitable benchmark to optimize for. After I provided a specific benchmark, it would switch to the one I asked for in my explicit prompt. For the initial benchmark that included a routing protocol implementation, the agent got a bit too eager and coded a custom routing implementation for FatTree topologies only, which breaks the routing mechanism when the topology is not a FatTree. Overall, however, autoresearch saved about 25% in runtime performance on this particular benchmark, which is quite a bit given that the codebase has already gone through many rounds of optimizations in the past.

Since its inception, the pi-autoresearch extension has been evolving. Two noteworthy improvements have been added in the past three days: a confidence score has been added, and additional actionable side information on why an optimization is discarded has been recorded.

—

Nous Research used its open-source Hermes agent to write a novel using autoresearch. The benchmark, in the context of autoresearch, is reader_panel.py, which uses four different personas from Claude Opus 4.6 — the editor, the genre reader, the writer, and the first reader — to review the novel. It also runs review.py, which also uses Claude Opus 4.6 with the following dual-persona prompt:

Read the below novel, “{title}”. Review it first as a literary critic (like a newspaper book review) and then as a professor of fiction. In the later review, give specific, actionable suggestions for any defects you find. Be fair but honest. You don’t have to find defects.

I like the inclusion of you don’t have to find defects in the prompt. Strictly speaking, these reviews are not really a precise benchmark, as Karpathy mentioned:

Not exactly verifiable but might still work quite well given some effort.

Though they may be helpful to edit the writeup, similar to discarding an experiment, the loop can certainly iterate confidently towards mediocre results. Still, this is a worthy experiment towards writing anything, not just fiction.

—

Autoresearching Apple’s LLM in a Flash to run Qwen 397B locally, by Dan Woods, is a mind-boggling autoresearched advance towards running large models off SSDs on Macs. With freshly coded Objective-C, the AI agent can improve the performance of running a Qwen 3.5 397B MoE model on a MacBook Pro to around 6 token/second, which is extremely impressive. This also showcases the immense power of autoresearch and of AI agents in general, given the right context for them to get started working.

Converting PDFs with Apple Silicon GPU Acceleration

Baochun Li — Sat, 21 Mar 2026 00:00:00 GMT

Typically, PDF to Markdown converters either do not do a very good job converting mathematical formulas to LaTeX, or require an NVIDIA GPU to run a Transformer model. After quite a bit of work, I have discovered a way of converting PDF files, with all mathematical formulas converted to LaTeX, and using Apple silicon GPUs for acceleration.

First, create a Python virtual environment and install docling and docling[vlm]. One way to do it is to quickly create a new file pyproject.toml:

[project]
name = "pdf-convert"
version = "0.1.0"
description = "Setting up the virtual environment for converting PDFs with Apple Silicon GPUs."
requires-python = ">=3.13"
dependencies = [
    "docling",
    "docling[vlm]",
]

and then run uv sync and source .venv/bin/activate. After setting up the environment, the launch command I used was:

docling --enrich-formula --pipeline vlm --vlm-model granite_docling file.pdf

This runs the Granite Docling model, with 258M parameters, on the Apple Silicon GPUs with MLX. The conversion process may take a while, but the results look excellent. I have added my setup above to a git repository so that I can use it more easily.

Successfully running Qwen 3.5 27B on my NVIDIA RTX 4090 (using 21 GB of CUDA memory)

Baochun Li — Fri, 20 Mar 2026 00:00:00 GMT

I have successfully started the Qwen 3.5 27B model, with 4-bit quantization, on my GPU server with a NVIDIA RTX 4090.

The launch command I used was:

./build/bin/llama-server -m Qwen3.5-27B-Q4_K_M.gguf -ngl 99 -c 262144 -np 1 -fa on --cache-type-k q4_0 --cache-type-v q4_0 --host 0.0.0.0

To use the model, I am running it with the Hermes agent. My setup was inspired by Sudo Su’s blog, which showed how capable the Qwen 3.5 9B model is when paired with the Hermes agent.

With the 27B model, I am using around 21 GB of CUDA memory. It is hard to imagine having access to GPT-5 level intelligence on such modest GPU hardware! I have switched my email triage system to Hermes and QWen 3.5 27B as well.

After the Prompt, Who Still Learns to Program?

Baochun Li — Sat, 14 Mar 2026 00:00:00 GMT

During my jog today, I listened to the New York Times Magazine article Coding After Coders: The End of Computer Programming as We Know It. Read by James Patrick Cronin, it was an engaging 38-minute story about how coding evolved, from writing assembly to prompting Claude Code. Clive Thompson interviewed over 70 software developers, some optimistic, and some worried that A.I. is atomizing the work force.

Don’t get uppity at work — we could replace you with a bot.

I feel that A.I., as we know it, would forever change not only how software is to be developed, but also how computer science talent is to be educated. Fewer students will have any software development experience beyond prompting an A.I. agent, and when something is not optimally designed or breaks, no one will be able to fix them by hand.

How to win a best paper award, written by Nicholas Carlini, is worth a quick read. Most of the ideas here were great, for example:

The majority of my papers that have received best paper awards were rejected at least once before they got in. In one case, a paper of mine was rejected four times first.

And also:

Once you’ve read everything, the second step is to forget it all. The reason is simple: everything that’s already been done has already been done. If you constrain yourself to thinking only about what’s been done, you’ll never come up with something clever and new.

In a similar spirit, I recall Professor Jane W.S. Liu once said to her Ph.D. students: “Do not read more than 15 papers in your Ph.D. — you want to become a world-class researcher, not a mediocre one.”

If I were to briefly summarize the best writing advice I’ve received, it would be to listen to how your writing sounds spoken out loud, and try to make it understandable. I used to do this by reading my papers out loud to force myself to hear every word; I still do this sometimes, but now I also use text-to-speech systems to read the words back to me. You’ll notice things you’d never have caught yourself.

This reminds me of the NSF panel summary sessions, where panelists are required to read their panel summaries out loud to the entire panel. Apparently, many issues in writing can be caught by just listening to spoken words.

But some of the other ideas seem to be quite dated in the A.I. era. For example, one doesn’t really need to proofread the work — the agents will gladly take over the job. Conducting many experiments are no longer harder than conducting a few: the agents will automatically conduct these for you.

My Multi-Agent Setup using Linear

Baochun Li — Thu, 12 Mar 2026 00:00:00 GMT

I installed OpenAI’s Symphony, by simply directing Codex to its GitHub repository, as recommended by Symphony’s README:

Launch codex in your repo, give it the URL to the Symphony repo, and ask it to set things up for you.

Before I run Symphony, I will need to first create the issues corresponding to the task I would like to complete. I used the following prompt:

Right now, there is no UI for a reviewer/TPC member to enter a review (or a meta review or a Area Chair review). The review form can already be configured by the chair under administration, but it did not surface to the reviewers or TPC members or Area Chairs. Each entry in the table in their list of “My Assignments” does not really have a button that they can use to view (and to review) the paper.

Use your frontend design skills, design such a UI for reviewers/Area Chairs/TPC members to enter reviews/meta reviews/Area Chair reviews, based on the configured templates from the ‘review form’ configuration under Administration.

Write a comprehensive plan carefully and break this into issues in project ‘reviewsdue’ in Linear. Scope each issue to one atomic task that can be completed with one agent and scoped to one reviewable PR. Include acceptance criteria in each issue. For each issue, add a follow-up ‘review’ issue that have acceptance criteria like:

reviewer reads the full diff, not just the summary

verifies behavior against the acceptance criteria of the issue

checks tests are adequate and not just passing narrowly

records concrete findings or explicitly states no findings

only closes after review comments are addressed

This provides a deliberate “implementation issue” followed by a separate “review issue” workflow, which is better than trusting the implementation issue alone. Set blocking relationships where order matters; later issues should be blocked by both its upstream implementation issue and their review gate. Use red/green test-driven development (TDD) when implementing each issue.

The first two paragraphs described the task I wish Codex to work on, and the later paragraphs instructed Codex to use Linear, and to create both implementation issues and their review gates.

Symphony runs correctly, but I found that it is using a lot of tokens for simple features, and does not allow me to see what is going on in each of the active sessions it launched. For one issue, it couldn’t finish running the agent after over an hour — something must be broken inside the agent.

So instead of depending on Symphony to pull the issues, I entered the following prompt:

Now that you have the Linear issues, pull each of these issues, respect dependency requirements, and then spawn subagents (in parallel, if needed) to resolve these issues. Once each agent is finished, commit with a detailed commit message following the PR instructions when pushing (but don’t need to PR, just commit). Respect instructions in WORKFLOW.md when pulling Linear issues. When all the issues have been completed, PR the entire feature with a detailed description.

This did the trick: it worked in a faster and perhaps more token-efficient way as using Symphony; but even more importantly, there is more transparency: I can read what the agents did in their sessions.

There is one major advantage of using this workflow and Linear: after each issue is completed, Codex would add results of running the agents as a comment in the issue itself, providing me with a central repository to log the history of all the work completed by the agents in the project. I think this is more important than the speed of running multiple agents, since I can read these comments and get a sense of what’s going on, reducing the cognitive debt when managing the project.

Computer Engineering in the Next 10 Years

Baochun Li — Mon, 09 Mar 2026 00:00:00 GMT

My student asked me a question over Telegram: “How would computer engineering evolve over the next 10 years?”

Here is my answer, with thinking set to “low”:

The immense power of AI agents, distributed on billions of user devices, will be a norm, not an exception. Personalizing these agents will become cheaper and more accessible than ever, making it feel like living in prehistoric times if these agents are suddenly unavailable. Hardware advances will make these agents faster than ever, in both the cloud and user devices. Agents will be used to improve themselves, and to improve all disciplines in computer engineering in general. Though these agents will still not be very creative, they can help us create new ideas with accelerated speeds. Software will still be as relevant as ever, given that its design requires a lot of creativity.

Pages, Window Chrome, and Liquid Glass

Baochun Li — Sun, 08 Mar 2026 00:00:00 GMT

The Window Chrome of Our Discontent is a great, well-crafted piece. I have been using Pages since its inception (around 2005), and in terms of chrome, I prefer having some vibrant colours in the first and second design, rather than the plain, greyscale design in the later versions. The worst UI design is, of course, the Liquid Glass design in macOS Tahoe, which I refuse to upgrade to. I am currently still on macOS 14.8. Interestingly, because the latest release of Pages requires at least macOS 15, I am not allowed to upgrade to it. I consider this a feature as it serves as a guardrail against inadvertent upgrades. Oh, and I love it when Nick Heer, the author of this article, used “A.I.” for artificial intelligence, rather than “AI”.

Rechecking the Days Codebase with GPT 5.4 and GPT 5.4 Pro

Baochun Li — Fri, 06 Mar 2026 00:00:00 GMT

Here’s my past experience: it is a good idea to ask a new model, such as the just-released GPT 5.4, to review the codebase again. So I fired up Codex, and asked it to review the ring all-reduce implementation in Days, my Rust-powered discrete-event network simulator. It quickly wrote the following review:

Findings

High: ring flow IDs are reserved incorrectly during config parsing. The parser advances by the configured flow_count in src/flows/collective.rs:316 and src/ flows/collective.rs:372, but runtime expansion emits 2 * n * (n - 1) flows and only then rewrites collective.flow_count in src/topos/topo.rs:676. A 4-rank ring reserves 4 IDs and later consumes 24, so any later flow or collective can collide.

High: the dependency graph is not a real ring all-reduce schedule. Expansion only serializes each rank’s own outgoing steps in src/topos/topo.rs:605 and src/ topos/topo.rs:643. But the chunk mapping in src/topos/topo.rs:1269 means later steps should depend on receiving a chunk from the previous rank, not on the same sender’s previous transmit finishing. The model captures local serialization, not cross-rank data availability.

High: the TCP path does not actually propagate or reduce data around the ring. Each directed link gets its own fresh zero-filled AppDataSource in src/topos/topo.rs:1281 backed by src/flows/app_source.rs:275, and flows just read byte slices from that local buffer in src/topos/topo.rs:1289. No receive-side statefeeds the next hop, and no reduction happens anywhere. This is not an all-reduce implementation; it is a pre-sliced traffic generator.

High: handle-backed TCP ring flows silently drop any chunk smaller than the fixed 512-byte MSS. pull_from_appsource() can buffer a short chunk in src/flows/tcp_source.rs:321, but send_packet() only transmits when a full MSS fits in src/flows/tcp_source.rs:770 and src/flows/tcp_source.rs:804. stop_run() then treats that flow as complete in src/flows/source.rs:439. The bundled example at configs/collective_tcp.toml:39 uses size = 512 with 4 ranks, so each ring chunk is 128 bytes and every hop can complete without sending a packet.

High: mixed broadcast + ring TCP collectives can lose the broadcast app-source actor because temporary ownership maps mix collective IDs and host IDs. Broadcast stores its source under collective.id at src/topos/topo.rs:1246, ring later stores under src at src/topos/topo.rs:1307, and collective IDs start from zero in src/lib.rs:80. In the bundled config, the first broadcast and host 0 coexist in configs/collective_tcp.toml:21 and configs/collective_tcp.toml:39, so the broadcast source can be overwritten and dropped before registration.

Medium: explicit ring paths are validated but ignored. Parsing accepts them in src/flows/collective.rs:201, but ring expansion hardcodes dst from sources in src/topos/topo.rs:609 and builds every ring flow with path: None in src/topos/topo.rs:619. So a user can believe they pinned ring routes while runtime silently falls back to normal routing.

Medium: coverage is far too narrow for this feature. The only ring-specific test is src/topos/topo.rs:1434, and it only checks same-rank serialization. It does not exercise config parsing, ID allocation, runtime app-source wiring, chunk sizes below MSS, or explicit paths.

And then summarized with the following verdict:

The main assumption behind the review is that RingAllReduce is intended to model an actual reduce-scatter + allgather collective, as the docs claim, rather than just generate ring-shaped traffic. Under that assumption, the implementation is currently incorrect in both semantics and some concrete runtime behavior.

That’s a pretty thorough and critical review! I then asked GPT 5.4 Pro to see if it can confirm that these are correct and if it has anything to add. GPT 5.4 Pro confirmed that everything is correct, and found three more suggestions. Then I asked Codex to fix these issues one by one, and reviewed the results.

Harmony and Harness Engineering

Baochun Li — Thu, 05 Mar 2026 00:00:00 GMT

On February 11, OpenAI posted a new blog post titled Harness engineering: leveraging Codex in an agent-first world. I have not posted it in this website at the time, because I felt that it was pretty difficult to reproduce.

Yesterday, OpenAI open-sourced Symphony — OpenAI really loved music as it has another open-source repository called Harmony. Symphony includes a reference implementation of managing the work that agents need to get done, implemented with Elixir, a dynamic, functional language for building scalable and maintainable applications.

I should allocate a bit of time to study both the blog and Symphony.

Donald Knuth on Claude

Baochun Li — Wed, 04 Mar 2026 00:00:00 GMT

In his article titled Claude’s Cycles, Prof. Donald Knuth, at age 88, has carefully documented how Claude Opus 4.6 solved a problem that he has been working on for several weeks.

Shock! Shock! I learned yesterday that an open problem I’d been working on for several weeks had just been solved by Claude Opus 4.6 — Anthropic’s hybrid reasoning model that had been released three weeks earlier! It seems that I’ll have to revise my opinions about “generative AI” one of these days. What a joy it is to learn not only that my conjecture has a nice solution but also to celebrate this dramatic advance in automatic deduction and creative problem solving.

Open-source PDF to Markdown with Marker

Baochun Li — Mon, 02 Mar 2026 00:00:00 GMT

I discovered a best-in-class open-source PDF to Markdown convertor: Marker. On my NVIDIA RTX 4090 server, it converts a PDF in about 30 seconds to a minute, and the results are spectacular. I used uv venv to create a virtual environment, activated it, and installed Marker within the environment using pip install. It does require an NVIDIA GPU server, but the output quality is worth it.

Email Triage System with Codex

Baochun Li — Thu, 26 Feb 2026 00:00:00 GMT

I just used Codex to implement a new email triage system. It uses JMAP to access my Fastmail account using an API token, and automatically triages inbound emails into high, medium, or low priority levels, and archives the medium and low-priority emails. For high priority emails, it will also automatically use Codex to draft responses. It runs on my Linux server every 15 minutes, and everything is configurable.

I find the following prompt useful for reviewing a large codebase:

I want you to sort of randomly explore the code files in this project, choosing code files to deeply investigate and understand and trace their functionality and execution flows through the related code files which they import or which they are imported by. Once you understand the purpose of the code in the larger context of the workflows, I want you to do a super careful, methodical, and critical check with “fresh eyes” to find any obvious bugs, problems, errors, issues, silly mistakes, etc. and then systematically and meticulously and intelligently correct them. Be sure to comply with ALL rules in AGENTS.md and ensure that any code you write or revise conforms to the best practice guides referenced in the AGENTS.md file.

Building a Linear Walkthrough of a Codebase

Baochun Li — Tue, 24 Feb 2026 00:00:00 GMT

I used the following prompt excerpted from Simon Willison’s excellent commentary to build a linear walkthrough of Nextmini, a fairly complex codebase built in Rust. Unsurprisingly, Codex launched several subagents to scout different parts of the codebase without any additional hints on spawning subagents.

Read the source and then plan a linear walkthrough of the code that explains how it all works in detail.

Then run “uvx showboat --help” to learn showboat - use showboat to create a walkthrough.md file in the repo and build the walkthrough in there, using showboat note for commentary and showboat exec plus sed or grep or cat or whatever you need to include snippets of code you are talking about.

The command-line utility, showboat, doesn’t need a skill, since showboat --help is so comprehensive that the agent can understand how to use it just by reading the help.

The entire session takes 9 minutes 41 seconds to complete with GPT 5.3 codex xhigh, which is suprisingly fast to me.

Agentic Engineering Patterns

Baochun Li — Mon, 23 Feb 2026 00:00:00 GMT

I have read Simon Willison’s Agentic Engineering Patterns, and started to wonder why good content like this can be read for free over the Internet. I have never previously heard about the red/green test-driven development (TDD), but it feels so powerful that I must give it a try soon on one of my open-source projects.

My Own Extension for the Pi Coding Agent

Baochun Li — Tue, 17 Feb 2026 00:00:00 GMT

Today, I wrote my own extension, called collaborating agents, for the Pi coding agent — which the famous (or infamous) OpenClaw is based upon. It allows me to easily work with multiple agents that can collaborate with one another by sending and receiving messages, and to allow an orchestrator agent to spawn multiple subagents.

My idea is inspired by Jeffrey Emanuel’s Agentic Coding Flywheel, and in particular its MCP Agent Mail. Spawning multiple agents is definitely helpful from a context engineering point of view, but they need to be able to communicate with one another, and to reserve and release files so that conflicts can be avoided.

Jeffrey Emanuel’s system can be effective, but it is way too complex for me to use. In contrast, my new Pi extension is designed for agents to be easily spawned and to talk to one another, but with only around 3400 lines of TypeScript code. I wrote the extension with less than a day of work, written by working with Pi itself. It may need a bit more fine-tuning to be battle-tested, but it is very usable already.

To install this collaborating agents extension and its included skill, install Pi first:

bun install -g @mariozechner/pi-coding-agent

And then choose one of the installation options:

To install the extension as an npm package, run:

pi install npm:@baochunli/pi-collaborating-agents

To install it from the git repository, run:

pi install https://github.com/baochunli/pi-collaborating-agents

To update the extension and skill to the latest release:

pi update

To remove the extension and skill:

pi remove npm:@baochunli/pi-collaborating-agents

or (if installed from a git repository):

pi remove https://github.com/baochunli/pi-collaborating-agents

Token Anxiety and Cognitive Debt

Baochun Li — Mon, 16 Feb 2026 00:00:00 GMT

I read two pieces today that somehow connect: token anxiety and cognitive debt. I enjoyed reading the following paragraph:

I replaced Netflix with Claude Code. I lie in bed thinking about what I can spin up before I fall asleep, what can run while I’m unconscious. Reading a novel feels indulgent now. Watching a movie without a laptop open feels wasteful. This voice in my head that says “something could be running right now” just doesn’t shut off. I’m not even building a company. I’m just addicted to building my random ideas.

As tokens becoming less expensive ($1 an hour for the new MiniMax 2.5), I think it is more and more addictive and cognitive debt will become widespread — not only on code, but also on research papers as they are more and more written by AI.

Codex is finally able to draw reasonably well

Baochun Li — Sun, 15 Feb 2026 00:00:00 GMT

I discovered an effective tool called mcp_excalidraw — which combines the powers of an MCP server and a skill — to get codex to draw figures using Excalidraw reasonably well.

The difference between this tool and other alternatives is that, with this tool, Codex can see the figures by capturing screenshots using MCP calls. It is a bit of additional work to set up, but well worth it. I followed the following steps.

First, I cloned the git repo:

git clone git@github.com:yctimlin/mcp_excalidraw.git

Then I copied the skill to my own skills folder:

cp -R mcp_excalidraw/skills/excalidraw-skill ~/.agents/skills

Then I installed the MCP server:

codex mcp add excalidraw \
  --env EXPRESS_SERVER_URL=http://localhost:3000 \
  --env ENABLE_CANVAS_SYNC=true \
  -- node /Users/bli/Playground/mcp_excalidraw/dist/index.js

Finally, I started the local web server (which the git repo did not mention):

cd mcp_excalidraw
npm install
npm run build
HOST=0.0.0.0 PORT=3000 npm run canvas

In Codex, I just need to say: “Use the Excalidraw skill, draw…”. Here is an example figure drawn by Codex (with only minor adjustments by me).

When I don’t need to draw figures (which is most of the time), I would just remove it for better efficiency:

codex mcp remove excalidraw

Migrated Days and Nextmini websites to TanStack Start

Baochun Li — Sat, 14 Feb 2026 00:00:00 GMT

Today, I have migrated both Days and Nextmini websites from Next.js to TanStack Start.

It is a surprise that Codex only needs a few minutes to migrate each project. Since their respective documentation websites are already using Fumadocs and TanStack Start, this is a natural transition and a more seamless fit. As a new web framework, TanStack Start feels simpler and much faster than Next.js, and will be my choice for new projects going forward.

Subagent-Friendly Planning Rules

Baochun Li — Fri, 13 Feb 2026 00:00:00 GMT

Following this suggestion, I added the following sections to AGENTS.md to make sure that codex always writes subagent-friendly plans and uses subagents more proactively. They worked well.

## Additional Agent Operating Rules

### Context7

- ALWAYS proactively use Context7 when I need library/API documentation, code generation, setup or configuration steps without me having to explicitly ask.
- External libraries/docs/frameworks should be guided by Context7.

### Planning

- All plans MUST include a dependency graph.
- Every task in a plan must declare `depends_on: []` using explicit task IDs such as `T1`, `T2`.

### Execution

- Complete all tasks from a plan without stopping for permission between steps. Use best judgment, keep moving.
- Only stop to ask when a step is destructive/irreversible or there is a genuine blocker.

### Subagents

- Spawn subagents automatically when:
  - Parallelizable work (e.g., install + verify, npm test + typecheck, unblocked tasks from plan)
  - Long-running or blocking tasks where a worker can run independently.
  - Isolation for risky changes or checks
  - Code review would be helpful
- If you're launching subagents for parallelization, add this robust context to your prompt:
  - **Context**: Share plan file location and info if available
  - **Dependencies**: What work/files are completed? Any dependencies?
  - **Related tasks**: Any adjacent tasks, files, or agents?
  - **Exact task**: Description, file paths/names, acceptance criteria
  - **Validation**: How to validate work if possible.
  - **Constraints**: Risks, gotchas, things to avoid
  - **Be thorough**: Provide ANY/ALL context that will aid success.
- ALWAYS wait for all subagents to complete before yielding.

### Bugs

- Add a regression test when it is appropriate for bug-related changes.

iOS Codex Workflow with Moshi and GPT 5.3 Codex Spark

Baochun Li — Thu, 12 Feb 2026 00:00:00 GMT

Only after a day, I discovered a better way to use codex from my iPhone: use the Moshi app to connect into my computer via the Tailscale network. It is simpler, faster, and more secure than Agentboard.

GPT 5.3 Codex Spark has been released and it is super fast — much faster than the regular 5.3 codex. This will be good enough for, say, fixing the “must-fix” items quickly after a code review.

Oracle Testing, Something Big, and Something Small

Baochun Li — Wed, 11 Feb 2026 00:00:00 GMT

Configurancy, written by Electric, emphasized what we should do when writing code is cheap.

What I find interesting is the concept of oracle testing, where the oracle (Postgres in their example) is the spec that the codebase needs to satisfy. The moral of the story is that we need specs and conformance suites, not just simple unit test cases.

Something Big Is Happening

Something Big Is Happening, written by Matt Shumer, is fascinating and long, but a must read.

The era of manual coding is over, and perhaps soon, “vibe coding” will become “vibe research” in general, where not only programming, but also research, will be produced with almost 100% assistance by AI agents. This will allow us to try out new ideas, and to find out which works and which doesn’t, with unprecedented velocity.

I like one piece of advice from this article:

“Spend one hour a day experimenting with AI. Not passively reading about it. Using it.”

243-Line MicroGPT by Karpathy

In sharp contrast with his NanoChat project and its DeepWiki documentation, the 243 lines of pure, dependency-free Python, MicroGPT, reminds me of his online videos on building an autograd engine. Surely someone will post a detailed tutorial soon, explaining these 243 lines line-by-line.

On a side note, the DeepWiki MCP definitely sounds very interesting and may be better than Context7. I have added it to my codex MCP setup with the command:
codex mcp add deepwiki --url https://mcp.deepwiki.com/mcp

A Language for Agents

Baochun Li — Tue, 10 Feb 2026 00:00:00 GMT

Quick tip on how to get the current model served by codex:

RUST_LOG='codex_api::sse::responses=trace' codex exec --sandbox read-only --model gpt-5.3-codex 'ping' 2>&1 | grep -m1 'SSE event: {"type":"response.created"' | sed 's/^.*SSE event: //' | jq -r '.response.model'

Before running this command, I need to enter a trusted directory first.

Agentboard and iOS Codex Access

There are quite a few projects that are designed to help access agentic CLI tools, such as codex, from an iOS device. I have researched several of them, and the best is Agentboard. It uses Tailscale to seamlessly visit a home computer or a server via just a web browser on the phone, and connect into any live codex sessions there. So far, Agentboard offers the best experience of connecting to a codex session from my phone, and is much better than other alternatives, such as happy.engineering, tgterm, or directly using Openclaw with Telegram.

A Language for Agents

A Language for Agents, by Armin Ronacher, is an excellent (but long) essay about the future of programming languages in an agentic world.

Though it lobbied for a new language only for agents, it also made a strong case for using Rust and TypeScript (and perhaps also Go) as “the agent’s language,” but not nearly as much for Python, which is not strongly typed.

In my personal opinion, we can just settle for TypeScript and Rust as the programming languages of choice going forward when starting greenfield projects, and only use Python for machine learning. To reduce the cognitive load, we shouldn’t be learning programming languages beyond TypeScript (as the first language) and Rust (as the advanced, performance-oriented alternative) for general-purpose programming, and perhaps Python for its ecosystem in machine learning.

Redesigned Personal Website with a Minimal Writing Workflow

Baochun Li — Sun, 08 Feb 2026 00:00:00 GMT

I redesigned my personal website on the flight from Doha back to Toronto. Last year’s design took me a day and used a content management framework called Quartz, while the new design took me only about two hours in codex, and used only Astro as a lightweight framework. I used the plan mode in codex to produce a plan first before implementation. I have to say, codex completely changed how a website is to be designed: the old days of manually coding websites are gone.

The new website allows creating a new post with a CLI tool: post, for which I asked codex to create a skill for itself to use. In order to post a new entry, all I need to do is to activate the skill and tell codex what I wish to say.

tiny-llm and Practical PyTorch Learning Prerequisites

Baochun Li — Tue, 29 Apr 2025 00:00:00 GMT

tiny-llm Exactly what I wished for. It also contains links to two existing PyTorch related courses to machine learning from Carnegie Mellon University, to be used as prerequisites for this course.

Arc Browser and the Modern shadcn/ui Tooling Stack

Baochun Li — Wed, 02 Apr 2025 00:00:00 GMT

Arc — My new browser of choice. I love the fact that bookmarks are organized on the side panel, rather than clustered at the top of the window. Split windows and spaces are also quite nice, and it’s cool to read about using Swift to build the UI. The “Little Arc” windows are kind-of cute, too.

Vaul — What a beautiful and simple design! Enjoyed reading the designer [Emil Kowalski]’s website. His other creation, Sonner, is also something I use — its documentation is a thing of beauty, including its use of the marvellous Berkeley Mono typeface.

shadcn/ui — The best UI component distribution mechanism and library out there. It’s fully compatible with Tailwind CSS 4.1, and tweakcn can be used to customize it.

bazza/ui — Best data table filters, based on shadcn/ui.

Circle — A dashboard template based on shadcn/ui, that allows components to be dragged and dropped across columns, as in Kanban.

Dashboards in shadcn UI Kit — A pretty good dashboard, but not without minor issues when adapting to narrower windows.

shadcnblocks — Hundreds of useful blocks (for a flat fee). Very useful to have such a large selection.

Supabase UI Library — Distributed using the shadcn CLI.

Dice UI — Unstyled UI component library based on the latest Tailwind CSS 4 and distributed using the shadcn CLI. It includes a nice Linear-like table filter and sorting.

Silk — Native‑like swipeable sheets on the web, and more advanced than Vaul. 299 Euro for small businesses with fewer than 5 employees.

Evaluating Eleventy as a Lightweight Static Site Generator

Baochun Li — Tue, 01 Apr 2025 00:00:00 GMT

Eleventy appears to be a pretty simple static website generator that is worth exploring. A competitor to Hugo.

How I Use LLMs: Key Notes from Andrej Karpathy

Baochun Li — Tue, 11 Mar 2025 00:00:00 GMT

How I use LLMs by Andrej Karpathy — A must watch.

Panasonic S1R II and Early Claude Code Impressions

Baochun Li — Tue, 25 Feb 2025 00:00:00 GMT

Panasonic S1R II — With the Sigma 28-105 f/2.8, this would be my dream camera. It is just slightly heavier than my Panasonic S5 IIx (1.57 lb vs. 1.45 lb body only).

Claude Code — I joined the waitlist last night and received the invitation today.

I gave it a try on one of my ongoing projects and it was pretty costly (costing $0.40 to simply set up a basic understanding in CLAUDE. d). I also tried CodeBuff and it does seem to fare better, but not magically solving the issues I experienced.

Morale of the story: Know what you are doing and do not try to use AI blindly. It will mess up the codebase to a point where it is taking longer to rescue the code than to work on the project manually without AI from the beginning.

Ultra-Scale LLM Training Playbook and Streaming DiLoCo

Baochun Li — Wed, 19 Feb 2025 00:00:00 GMT

The Ultra-Scale Playbook: Training LLMs on GPU Clusters — Amazing, and finally we have a 100-page open-source online book on how models are trained with multiple GPUs, with reproducible source code.

Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch — Latest paper from DeepMind about efficient geographically distributed training with overlapped communication.

Crafted UI, Fumadocs, and Design System References

Baochun Li — Tue, 18 Feb 2025 00:00:00 GMT

Crafted — What a great looking set of open-source, hand-crafted UI templates based on shadcn/ui!

Fumadocs — Better Auth’s documentation is built with this excellent documentation framework based on Next.js.

The website of Gwern Branwen — Beautiful design, with Adobe Source Serif Pro as the main serif font choice. I couldn’t believe that the entire site infrastructure is open source, and constantly being updated.

Syntax: Tailwind’s documentation template — A bit pricy, but good quality.

Remix Icons — Used by the Grok website.

Better Auth, Origin UI, and Open Research Data Tools

Baochun Li — Sun, 16 Feb 2025 00:00:00 GMT

Better Auth — A new authentication library that is feature-complete and easy-to-use. Compared to Lucia, which advocates a copy-and-paste approach, this library requires less intimate knowledge about authentication, and its plug-in system implies that it doesn’t sacrifice on extensibility. It feels more like an automatic with manual overrides in cars, rather than a manual transmission. My choice going forward.

Origin UI — What an excellent set of UI components based on shadcn/ui! The number of variants for each UI category is mind-boggling.

Top 10 Email Deliverability Tips — Resend’s tips on improving the deliverability of outbound emails.

Simon Willison’s take on MLX — Simon Willison (finally) added MLX as a new plugin to his LLM CLI utility, llm. His experiences with MLX were very positive:

This is really good software. This small team at Apple appear to be almost single-handedly giving NVIDIA’s CUDA a run for their money!

Semantic Scholar — Unlike Google Scholar, Semantic Scholar provides an open REST API to obtain metadata about papers and their authors, forming an academic graph. Pretty cool and I didn’t know about it before.

Zed supports edit prediction with its open-source Zeta model — The blog post that introduces Zeta was pretty comprehensive and covered a lot of grounds, including their deployment with Baseten with latency minimized.

OpenAlex — A fully open catalog of the global research system. The world becomes a better place with the dedication and hard work of people behind these efforts at OurResearch. It is also part of in the recently released Common Corpus 2, a second version of the Common Corpus.

A Minimal GRPO Implementation from First Principles

Baochun Li — Sat, 15 Feb 2025 00:00:00 GMT

Andriy Burkov’s minimalist implementation of GRPO from scratch — Rather than using a library such as Hugging Face’s TRL, it would always be a good idea to read a minimalist, back-to-square-one implementation of the GRPO reinforcement learning algorithm.

Transformer Lab: MLX Fine-Tuning Workspace on Mac

Baochun Li — Fri, 14 Feb 2025 00:00:00 GMT

Transformer Lab — a free, open-source LLM workspace that prepares a custom dataset and fine-tunes a model using MLX on the Mac (or of course, using a GPU-powered computer or in the cloud). Deep Gandhi offered a quick step-by-step guide for using MLX to fine-tune a model. It’s open-source with the MIT license, and the tech stack for building its UI seems to be Electron and React. This is first UI I found that can fine-tune models using MLX.

Lucia’s New Authentication Design and Practical Tradeoffs

Baochun Li — Tue, 11 Feb 2025 00:00:00 GMT

Lucia — Lucia, the authentication library, has adopted the design of cutting and pasting code, just like shadcn/ui, rather than implementing a library to encapsulate the details. This should work well with authentication, and reflects the design principle of working with simpler libraries rather than all-in-ones. In this case, the new Lucia design uses Arctic and Oslo, but all session and cookie management code need to be written (cut and pasted).

From 0 to Production: Notes on Theo’s Modern React Tutorial

Baochun Li — Sun, 09 Feb 2025 00:00:00 GMT

From 0 to Production — The Modern React Tutorial — Theo released it last year, and I always wanted to learn from this marathon tutorial. It covers all of the modern frameworks, Next.js, shadcn/ui, and TypeScript. I will find some time to finish it.

Unsloth GRPO, S1-Style Scaling, and RL Learning Resources

Baochun Li — Sat, 08 Feb 2025 00:00:00 GMT

Unsloth.ai’s GRPO — it seems that the Unsloth implementation of GRPO uses less GPU memory, and it supports QLoRA and LoRA.

S1-style test-time scaling with MLX — Awni Hannun, the primary architect of MLX, posted a simple implementation of S1-style test-time scaling using DeepSeek R1 distilled models locally, with only 138 lines of Python code. Simplicity at its best.

Spinning Up in Deep RL — Excellent introduction to deep reinforcement learning, with a sufficient amount of math but skips unnecessary formalism. It comes with PyTorch implementations for the algorithms. As it stated in its introduction:

However, while there are many resources to help people quickly ramp up on deep learning, deep reinforcement learning is more challenging to break into. To begin with, a student of deep RL needs to have some background in math, coding, and regular deep learning. Beyond that, they need both a high-level view of the field—an awareness of what topics are studied in it, why they matter, and what’s been done already—and careful instruction on how to connect algorithm theory to algorithm code.

The high-level view is hard to come by because of how new the field is. There is not yet a standard deep RL textbook, so most of the knowledge is locked up in either papers or lecture series, which can take a long time to parse and digest. And learning to implement deep RL algorithms is typically painful, because either

the paper that publishes an algorithm omits or inadvertently obscures key design details,

or widely-public implementations of an algorithm are hard to read, hiding how the code lines up with the algorithm.

Connecting algorithm theory to algorithm code is what’s sorely missing in many other online books or resources, especially in reinforcement learning. Many used Jupyter notebooks, which lead to horrible ways of learning from source code.

AI Peer Review with LLMs and S1 Test-Time Scaling

Baochun Li — Thu, 06 Feb 2025 00:00:00 GMT

DOGE: Make AI Conferences Great Again — Zeyuan (Allen) Zhu wrote a very interesting piece on using LLMs as arbitrators in the reviewer-author discussions and the paper review process. Zhu is one of the co-authors of the 2021 LoRA paper, which with over 11000 citations became the de facto standard in parameter-efficient fine-tuning, and widely used throughout the entire machine learning community.

In the paper, one surprising fact was that the 2021 LoRA paper was initially rejected by NeurIPS 2021, even after author rebuttal. I believe this shows clear evidence that the paper review system is broken, at least in the ML/AI community, making Zhu’s proposal of using LLMs to improve the fairness of the review process more interesting.

P.S. It looks like the widely cited paper, “Distilling the Knowledge in a Neural Network”, co-authored by Geoffrey Hinton and Jeff Dean, was also rejected by NeurIPS 2014, and later appeared in the NeurIPS 2014 Deep Learning Workshop. It has since received over 23000 citations.

s1: Simple test-time scaling — Stanford University showed in this paper that, by fine-tuning the Qwen2.5-32B-Instruct model with a curated high-quality dataset of only 1000 samples, and by appending wait to force the model to think longer, a 32B model can perform as well as o1-preview. It is the simplest way to do test-time scaling over the total number of thinking tokens, but it appears that it works well.

Interestingly, the paper stated:

The concurrently released r1-32B shows stronger performance than s1-32B while also only using SFT (DeepSeek-AI et al., 2025). However, it is trained on 800 × more reasoning samples. It is an open question whether one can achieve their performance with just 1,000 samples.

While DeepSeek r1-32B was indeed trained with far more reasoning samples, once the training is complete, it doesn’t need to perform test-time compute, which degrades user experience in terms of waiting time.

Karpathy’s LLM Deep Dive and MLX Rust Ecosystem Links

Baochun Li — Wed, 05 Feb 2025 00:00:00 GMT

Deep Dive into LLMs like ChatGPT — Andrej Karpathy continues his top-notch hours-long education on large language models with a new episode today. I am also keeping an eye on his new venture, Eureka Labs, which hopefully will eventually arrive with genuinely helpful educational content on all things machine learning.

Deep Learning — a long list of 26 white board lectures on deep learning, taught by Professor Bryce, Davidson College.

mlx-rs — Rust bindings for Apple’s MLX machine learning library on Apple silicon. Two of my favourite technologies are Rust and MLX, and this one has a bit of both.

GRPO on Apple MLX and Minimal-R1 Scaling Insights

Baochun Li — Mon, 03 Feb 2025 00:00:00 GMT

GRPO will soon be added to Apple MLX — The PR now works, using about 32 GB of memory when training Qwen2.5-0.5B.

Minimal-R1 — Another excellent reproduction of DeepSeek R1 with GRPO, using only a 8xH100 server. It addresses the issue of scalability in Hugging Face’s Open-R1 when generating long completions. What makes it stand out is that it doesn’t depend on TRL, and has its own GRPO implementation. It dedicated one GPU for vLLM generation, and one GPU for the reference model.

Kevin Bryan shares his view on OpenAI Deep Research — Kevin Bryan from the University of Toronto shares his early experiences with OpenAI’s Deep Research. He is extremely upbeat about it, even sharing a paper that Deep Research (a.k.a. the o3 model with web search capabilities) wrote in 15 minutes, as well as another paper that is more theoretical.

Here are some interesting quotes of what Prof. Bryan said:

Nick Pretnar asks: Can it simultaneously write a paper + model code, estimate/calibrate such model, discern which results are relevant to discuss then present such results in a way humans can understand?

Kevin Bryan: That’s beyond current capabilities. But the proof of concept is pretty clear. At this point, it’s by far most useful as a complement — you should be writing your code with Cursor plus frontier models, having AI supplement and check analysis, having AI check proof accuracy, etc.

This is what I would call a human-in-the-loop approach to academic research. But of course, when abused, the landscape of academic research papers can have lots of mediocre AI-generated content in the near-term future.

WTF happened in 1971? — 1971 is indeed a special year, it was when Elon Musk, Marc Andreessen, Ma Huateng, Liu Yunhao, and I were born.

Simple GRPO Implementations and DeepSeek FAQ Highlights

Baochun Li — Sun, 02 Feb 2025 00:00:00 GMT

Another simple DeepSeek R1 reproduction — This reproduction of GRPO has one distinct feature: it is exceedingly simple and quite elegant. To run it on the Mac, I only need to do a few minor changes, such as removing the quantization using bitsandbytes, which only works for CUDA. I have also used the following pyproject.toml:

[project]
name = "grpo"
version = "0.1.0"
description = "DeepSeek R1 reproduction using small models"
readme = "README.md"
requires-python = ">3.11, <=3.12"
dependencies = [
    "torch",
    "accelerate",
    "transformers",
    "datasets",
    "tqdm",
    "wandb"
]

and the following command:

uv run R1ZeroTrain.py

Out of the several DeepSeek R1 reproductions, this is my favourite. Not only it is simple enough and does not depend on any external RL library (such as TRL and veRL), it shows some of the nice features in GRPO. Obviously, due to its simplicity, its GRPO implementation is not complete and may need more work. But this is an educational codebase, and the author even posted a YouTube video, which I will try to find some time to watch.

OpenAI releases Deep Research — ChatGPT Pro users who pay $200 a month get 100 Deep Research questions per month. No coding examples in the introduction.

DeepSeek R1 reproduction now runs on my Mac — With a slight modification to train.py to turn off the usage of flash attention 2, I got the DeepSeek R1’s GRPO reproduction on small models with GSM8K running on my Mac, with the following pyproject.toml:

[project]
name = "grpo"
version = "0.1.0"
description = "DeepSeek R1 reproduction using small models"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
  "peft>=0.14.0",
  "torch>=2.6.0",
  "torchvision>=0.21.0",
  "transformers>=4.48.2",
  "trl>=0.14.0",
  "wandb>=0.19.5",
]

and the command:

uv run train.py

On my late-2021 M1 Max 64GB MacBook Pro, it runs at around 8.6 times slower than the NVIDIA RTX 4090, completing each step of RL in about 403 seconds, rather than 47 seconds on the 4090. Memory usage is up to 58 GB.

Interestingly, on my server with 3 NVIDIA RTX A4500 GPUs (each with 20 GB of CUDA memory), each step takes around 193 seconds, about 4x slower than the 4090. Out of a total of 60 GB CUDA memory, 23 GB is utilized[^1]. At least for this training session, the M1 Max (without using flash attention 2) is only roughly 2x slower than 3 A4500s.

DeepSeek FAQ — I have long admired Ben Thompson’s writing style in terms of its clarity, and this article on DeepSeek is no exception. It is indeed a long read, but the time is worth it. I enjoyed reading about the DeepSeek V2, which very few others mentioned:

Let’s work backwards: what was the V2 model, and why was it important?

The DeepSeek-V2 model introduced two important breakthroughs: DeepSeekMoE and DeepSeekMLA. The “MoE” in DeepSeekMoE refers to “mixture of experts”. Some models, like GPT-3.5, activate the entire model during both training and inference; it turns out, however, that not every part of the model is necessary for the topic at hand. MoE splits the model into multiple “experts” and only activates the ones that are necessary; GPT-4 was a MoE model that was believed to have 16 experts with approximately 110 billion parameters each.

DeepSeekMoE, as implemented in V2, introduced important innovations on this concept, including differentiating between more finely-grained specialized experts, and shared experts with more generalized capabilities. Critically, DeepSeekMoE also introduced new approaches to load-balancing and routing during training; traditionally MoE increased communications overhead in training in exchange for efficient inference, but DeepSeek’s approach made training more efficient as well.

DeepSeekMLA was an even bigger breakthrough. One of the biggest limitations on inference is the sheer amount of memory required: you both need to load the model into memory and also load the entire context window. Context windows are particularly expensive in terms of memory, as every token requires both a key and corresponding value; DeepSeekMLA, or multi-head latent attention, makes it possible to compress the key-value store, dramatically decreasing memory usage during inference.

And it concludes with an upbeat note on competition:

China is also a big winner, in ways that I suspect will only become apparent over time. Not only does the country have access to DeepSeek, but I suspect that DeepSeek’s relative success to America’s leading AI labs will result in a further unleashing of Chinese innovation as they realize they can compete.

That leaves America, and a choice we have to make. We could, for very logical reasons, double down on defensive measures, like massively expanding the chip ban and imposing a permission-based regulatory regime on chips and semiconductor equipment that mirrors the E.U.’s approach to tech; alternatively, we could realize that we have real competition, and actually give ourself permission to compete. Stop wringing our hands, stop campaigning for regulations — indeed, go the other way, and cut out all of the cruft in our companies that has nothing to do with winning. If we choose to compete we can still win, and, if we do, we will have a Chinese company to thank.

[^1]: This is with vLLM turned off. With it turned on, the server with 3 A4500s always ran out of CUDA memory, for reasons that are still unknown, at this point, to me.

Reproducing DeepSeek R1 GRPO on Consumer Hardware

Baochun Li — Sat, 01 Feb 2025 00:00:00 GMT

Fourth attempt on reproducing DeepSeek R1’s GRPO on small models — The ~~third~~ fourth time is the charm. I can successfully run this repo, without activating vLLM (keep vllm=true uncommented in the source code), on a single NVIDIA RTX 4090 with 24 GB CUDA memory, training the Qwen2.5-Math-1.5B model with the gsm8k dataset.

I used the following pyproject.toml:

[project]
name = "grpo"
version = "0.1.0"
description = "DeepSeek R1 reproduction using small models"
readme = "README.md"
requires-python = ">=3.11, <=3.12"
dependencies = [
    "torch",
    "transformers",
    "datasets",
    "peft",
    "wandb",
    "vllm",
    "trl",
    "flash-attn",
]

[tool.uv]
no-build-isolation-package = ["flash-attn"]

[tool.uv.sources]
torch = [
  { index = "pytorch-cu121", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
]
torchvision = [
  { index = "pytorch-cu121", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
]

[[tool.uv.index]]
name = "pytorch-cu121"
url = "https://download.pytorch.org/whl/cu121"
explicit = true

And the following command to run the repo:

uv run train.py

I obtained the following result after around 6 hours and over 450 steps:

Third attempt on reproducing DeepSeek R1’s GRPO on small models — Will Brown’s GRPO reproduction uses the openai/gsm8k dataset with 7470 samples, rather than the Countdown Game dataset in the two previous attempts — TinyZero and Mini-R1 — which is much more meaningful. It has been shown by others that even the small Qwen2.5-0.5B model can be trained from 41.6% to 51% on the gsm8k test set. I will try to reproduce this result some time, but for now it ran out of CUDA memory for a single NVIDIA RTX A4500 with 20 GB of CUDA memory, even for training the Qwen2.5-0.5B model.

Home server at $2000 for DeepSeek R1 at 4-bit quantization — $2000 home server, running the DeepSeek R1 671b model at 4-bit quantization and 3.5-4 tokens per second.

NVIDIA hosts DeepSeek R1 — much slower than Lambda Labs.

OpenAI o3-mini — On ChatGPT Plus, the rate limits are 150 messages per day for o3-mini-medium, and 50 messages per week for o3-mini-high. The latter is designed to be the strongest model on coding.

Running DeepSeek R1 on Lambda Labs and Notes on Ghostty

Baochun Li — Fri, 31 Jan 2025 00:00:00 GMT

Lambda Labs hosts DeepSeek R1 — the dashboard is simple, nice to look at, free to use, and pretty fast when generating tokens. Overall, an excellent user experience. The DeepSeek Llama 3.3 70B is also available, and it is much faster: reasoning is done in 9 seconds for my question What are the axioms of probability theory?, as opposed to 69 seconds with DeepSeek R1 671B.

Ghostty — version 1.1.0 is available with lots of updates and bug fixes. The best terminal emulator becomes even better.

Fine-Tuning Open LLMs in 2025 with Hugging Face and Mini-R1

Baochun Li — Thu, 30 Jan 2025 00:00:00 GMT

How to fine-tune open LLMs in 2025 with Hugging Face — Philipp Schmid a Technical Lead at Hugging Face, posted this article on fine-tuning LLMs using Hugging Face tools, without using the Unsloth API. I find it comprehensive and I will need to give it a try myself.

Mini-R1 — Philipp Schmid also posted this interesting reproduction of DeepSeek R1’s RL training. Similar to TinyZero, it used the Countdown Game as the task, but the article is much better written.

Mini-R1 used Hugging Face’s own TRL, designed to train transformer language models with RL in the post-training phase, which Hugging Face introduced in its smol course. To support multi-GPU training, it used DeepSpeed. In contrast, TinyZero used ByteDance’s veRL for both RL and distributed training, which doesn’t have either TRL or DeepSpeed in its dependencies.

veRL is based on HybridFlow, a University of Hong Kong/ByteDance paper published in EuroSys 2025, co-authored by Prof. Chuan Wu from the University of Hong Kong. I will allocate some time to study this paper in greater detail.

Microsoft added DeepSeek R1 to GitHub Models — I tried it with a simple question and not only the inference speed is astonishingly low, errors occurred before completing the answer. It is unusable at this point.

DeepSeek, Export Controls, and Open-Weight AI Debates

Baochun Li — Wed, 29 Jan 2025 00:00:00 GMT

On DeepSeek and Export Controls — Dario Amodei, Anthropic’s CEO, wrote a fairly long editorial on DeepSeek. However, it doesn’t mention at all the fact that DeepSeek’s models are open-weight models under a permissive MIT license, while Anthropic and OpenAI remained closed-weight models with no transparency on the technologies they used for both training and inference. At one point, Amodei mentioned that both DeepSeek and OpenAI o1 used RL, and used this to imply that DeepSeek’s use of RL to train R1-Zero is not so innovative. But we don’t know how OpenAI used RL to train o1, except that o1 “uses a chain of thought when attempting to solve a problem,” and that reinforcement learning has been used to train it[^1]. It could be the case that DeepSeek’s use of RL for train-time compute is very different from o1, and the fact that its affiliated technical report goes into sufficient technical detail on GRPO that makes it fully reproducible is much more noteworthy.

DeepSeekMath Paper Explained — Yannic Kilcher gave this one-hour explanation of the DeepSeekMath paper. I watched the first five minutes and minute 30 and beyond on GRPO. His explanations of GRPO are top-notch. The final five minutes, on Section 5.2.2 (“Why RL Works”), is insightful and worth tuning into.

Complete hardware for the full DeepSeek R1 at Q8 quantization, at $6000 — The fact that this CPU-only server can generate at 6-8 tokens per second — the same as human reading speed — shows the very substantial advantage of Mixture-of-Experts (MOE) models when running in CPU-only home servers, as compared to dense models such as the Llama 3.1 405B. Assembling such a server is non-trivial and not for the faint of heart, but it certainly has been proven possible.

[^1]: Learning to reason with LLMs, OpenAI, September 12, 2024.

The Illustrated DeepSeek-R1: A Clear Visual Walkthrough

Baochun Li — Tue, 28 Jan 2025 00:00:00 GMT

The Illustrated DeepSeek-R1 — Jay Alammar, the author of O’Reilly’s Hands-On Large Language Models, wrote a short piece on explaining DeepSeek R1 at a high level. I found it easy to read and the illustrations are pleasing to the eye.

Qwen 2.5 7B 1M Local Testing and RL Survey Notes

Baochun Li — Mon, 27 Jan 2025 00:00:00 GMT

Qwen 2.5 7B 1M — I have just tried Qwen’s latest local model, the 7B 1M, locally in LM Studio 0.3.8 (Build 4). I loaded an entire PhD thesis into the model, and LM Studio gleefully chose inject-full-content as its content injection strategy, rather than retrieval, which uses — the notoriously useless, in my humble opinion — RAG. This was not feasible before using a previous model, such as the DeepSeek R1 Distill Qwen 7B, with a context length of 128K.

It took 38 minutes to successfully inject the PhD thesis (with 166 pages), and fans in my MacBook Pro M1 Max 64GB memory were blowing at full speed. Once injected, it generates 2 output tokens per second, and once the content is injected, the next question only needs 21 seconds to the first token. So asking this model to read an entire PhD thesis works on a local Mac, but one would have to be a bit more patient. LM Studio reports that 20 GB of RAM has been used after the model is loaded, with the context length set to 256K.

Overall, this is indeed a very useful model for local use.

P.S. Of course, if data privacy is not a concern, one can also use the 14B 1M model available on Qwen chat. I tried it and it takes about 2 minutes to inject the entire PhD thesis and answer the first question. It’s interesting to observe that the time to first token for the second question is not much faster, taking about a minute and a half. The quality of the summaries is quite solid, but the language is not much easier to understand than the original thesis. This implies that if the original document is not well written, the summaries will not be too helpful either.

Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models — a recently updated (v3) survey of reinforced reasoning with LLMs from Tsinghua University. After a quick read, I felt it is already somewhat out of date, despite the fact that it is last updated a few days ago. The DeepSeek R1 technical report has not been cited yet, for example. The paper spent quite a bit of space talking about the Process Reward Model (PRM):

Process Reward Model (PRM) based Reinforcement Learning represents a significant advancement in LLM reasoning, emphasizing the evaluation of intermediate steps rather than solely focusing on end-state outcomes.

While discussing PRMs, it did include a brief mention of GRPO, with a citation to the DeepSeekMath paper that introduced it originally back in February 2024. The paper also spent quite some more space discussing the use of Monte Carlo Tree Search (MCTS).

However, the DeepSeek R1 technical report rendered both PRM and MCTS unsuccessful, at least with DeepSeek’s own attempts:

PRM is a reasonable method to guide the model toward better approaches for solving reasoning tasks (Lightman et al., 2023; Uesato et al., 2022; Wang et al., 2023). However, in practice, PRM has three main limitations that may hinder its ultimate success. First, it is challenging to explicitly define a fine-grain step in general reasoning. Second, determining whether the current intermediate step is correct is a challenging task. Automated annotation using models may not yield satisfactory results, while manual annotation is not conducive to scaling up. Third, once a model-based PRM is introduced, it inevitably leads to reward hacking (Gao et al., 2022), and retraining the reward model needs additional training resources and it complicates the whole training pipeline. In conclusion, while PRM demonstrates a good ability to rerank the top-N responses generated by the model or assist in guided search (Snell et al., 2024), its advantages are limited compared to the additional computational overhead it introduces during the large-scale reinforcement learning process in our experiments.

Inspired by AlphaGo (Silver et al., 2017b) and AlphaZero (Silver et al., 2017a), we explored using Monte Carlo Tree Search (MCTS) to enhance test-time compute scalability. This approach involves breaking answers into smaller parts to allow the model to explore the solution space systematically. To facilitate this, we prompt the model to generate multiple tags that correspond to specific reasoning steps necessary for the search. For training, we first use collected prompts to find answers via MCTS guided by a pre-trained value model. Subsequently, we use the resulting question-answer pairs to train both the actor model and the value model, iteratively refining the process.

However, this approach encounters several challenges when scaling up the training. First, unlike chess, where the search space is relatively well-defined, token generation presents an exponentially larger search space. To address this, we set a maximum extension limit for each node, but this can lead to the model getting stuck in local optima. Second, the value model directly influences the quality of generation since it guides each step of the search process. Training a fine-grained value model is inherently difficult, which makes it challenging for the model to iteratively improve. While AlphaGo’s core success relied on training a value model to progressively enhance its performance, this principle proves difficult to replicate in our setup due to the complexities of token generation.

Nvidia, DeepSeek, and RL Reasoning: Long-Form Analysis Notes

Baochun Li — Sun, 26 Jan 2025 00:00:00 GMT

The Short Case for Nvidia Stock — I spent less than an hour reading a pretty substantial portion of this article. It’s so good that I will need to allocate some time to read it again. The entire article, and especially the DeepSeek portion of it, is highly recommended, even if one is not interested in investing. It’s a detailed outlook for the entire AI industry.

As I am reading it the second time, the article covered tech that I have been following quite closely as well:

It mentioned how Cerebras solved its yield problem, while I have read its CePO test-time compute strategies;
It mentioned Groq, and I have tried its excellent and speedy inference service with a free account;
It mentioned George Hotz’s Tiny Corp. and its tinygrad, which I have been closely following on X. Back in the day, George Hotz was famous for jailbreaking the original iPhone as a teenager;
It mentioned MLX, which, as the article said, provides a PyTorch-like API that can run efficiently on Apple Silicon, showing how abstraction layers can enable AI workloads to run on completely different architectures. MLX is particularly interesting as it supports distributed computation — both training and inference — across multiple Macs. And its main contributor, Awni Hannun, mentioned today that DeepSeek R1 can run with 4-bit quantization across three 192 GB M2 Ultra Mac Studios at 12 tokens per second, requiring a minimum of 450 GB GPU memory;
And of course, it covered DeepSeek R1 in sufficient technical detail.

Wow, what a gem as a long-form read!

P.S.

Chamath Palihapitiya also thought the article is very good:

With R1, DeepSeek essentially cracked one of the holy grails of AI: getting models to reason step-by-step without relying on massive supervised datasets. Their DeepSeek-R1-Zero experiment showed something remarkable: using pure reinforcement learning with carefully crafted reward functions, they managed to get models to develop sophisticated reasoning capabilities completely autonomously. This wasn’t just about solving problems— the model organically learned to generate long chains of thought, self-verify its work, and allocate more computation time to harder problems.

The technical breakthrough here was their novel approach to reward modeling. Rather than using complex neural reward models that can lead to “reward hacking” (where the model finds bogus ways to boost their rewards that don’t actually lead to better real-world model performance), they developed a clever rule-based system that combines accuracy rewards (verifying final answers) with format rewards (encouraging structured thinking). This simpler approach turned out to be more robust and scalable than the process-based reward models that others have tried.

With extraordinary prescience, NVidia stock is down by over 14% around 11 a.m. the next morning, after this article is written. And the tech-heavy Nasdaq Composite falls 2.5%.
Simon Willison likes it too, calling it “Long, excellent piece by Jeffrey Emanuel capturing the current state of the AI/LLM industry.” —

The real joy of this article is the way it describes technical details of modern LLMs in a relatively accessible manner. I love this description of the inference-scaling tricks used by O1 and R1, compared to traditional transformers.

7B Model and 8K Examples: Emerging Reasoning with Reinforcement Learning is Both Effective and Efficient — Interesting. DeepSeek R1’s RL training techniques can be successfully applied to smaller models as well, at least for simple math datasets.

DeepSeek R1 for Everyone and DeepSeek V3 101 — With a brief read, these look promising as an easy read to understand some of the technical details of DeepSeek R1 and V3.

“Agents” still haven’t really happened yet — “If you tell me that you are building “agents”, you’ve conveyed almost no information to me at all. Without reading your mind I have no way of telling which of the dozens of possible definitions you are talking about.”

Open-R1 and TinyZero: Early DeepSeek R1 Reproductions

Baochun Li — Sat, 25 Jan 2025 00:00:00 GMT

Open-R1 — Hugging Face started to reproduce DeepSeek R1 in the open, and discussed the R1 technical report in a recorded YouTube video.

TinyZero — a reproduction of DeepSeek R1 Zero in countdown and multiplication tasks.

What I’ve Been Reading

Baochun Li — Fri, 24 Jan 2025 00:00:00 GMT

This website is a space for storing — and sharing, if anyone cares about these — some of the websites, code repositories, and tweets that I have read. They are mostly about technology, but not necessarily tech that is currently in the spotlight. They are stored here because I thought they are worthy of preserving for a longer period of time.

I have two relatively quick ways of preserving quality content. I can bookmark a link in a web browser or app; it is painless, but those bookmarks are too easy to misplace or forget about, and pretty difficult to search for after accumulating a larger number over time. For longevity, I can also place the link in a personal note (such as iOS Notes or Obsidian), which takes more effort and time, and thus the motivation for doing so in the long run is a bit questionable: why would anyone diligently copy and paste links to a personal note every time something interesting comes up?

The beauty of sharing links publicly, besides the nature of sharing itself, is to add a slice of motivation to the cocktail: it motivates me to do the work for copying and pasting. It also motivates me to add a bit of commentary, which records what I have been thinking about when reading the content linked to. Some called such a publicly shared website that contains links and commentaries a digital garden or a microblog, terms that I don’t quite like. Simon Willison and John Gruber called such a style a link blog. I will simply call it “What I’ve been reading.”