Over the past two years, I spent my spare time working on ReviewsDue, a reimagined web-based system for managing the entire workflow for running academic conferences — from submitting papers to optimizing the technical sessions.
It’s hard to imagine that I registered this domain name, reviewsdue.com, over four and half years ago, and started to work on the project — by hand — around two years ago. Now that we are in the deep waters of AI, and having worked on it extensively over the past year, I am finally able to complete it.
ReviewsDue reflects my vision on what an ideal conference management system should be like: simple to use, eye candy to look at, yet there is no shortage of the raw power and flexibility when the demand comes. It is designed with modern technologies for web design and deployment, all the way from mobile-friendly tables down to traffic load balancing across origin servers.
At this time, ReviewsDue has over one million lines of TypeScript and Python source code. Even with a self-hosted runner, around a dozen concurrent “shards,” and lots of optimization, each CI run still takes about 10 minutes. In the past, this kind of software scale is almost impossible to work on without a sizeable team of dedicated developers, designers, and testers; now, I can afford to work alone with my agents in my spare time.
DeepSeek recently released a blog (in Chinese) and prototype code documenting their own experiences shepherding multiple agents, using what they called a super-agent. In the article, they mentioned that that human attention will not be able to scale beyond 3-5 sessions. Well, I believe that this number is just one session, so I have my own design of a super-agent — another buzzword that I don’t quite like — or simply an orchestrating agent. I only have enough focus and time to talk to my only orchestrating agent, which is Claude Code running Claude Opus 5.5 at high effort (I still believe Claude Code is the best agent at this time).
My orchestrating agent manages a complex workflow: up to a dozen or so subagents, including internal Opus, Sonnet, or Haiku subagents, external codex agents using Astra, Sol, and Terra models, and Muse Code agents using the Muse Wonder Full model. With Sheprd (formerly herdr-gpui) and Herdr, Claude knows how to work with external agents. I find it most efficient to gradually evolve the workflow with trial-and-error over a long period of time with my orchestrating agent; and the project-agnostic portion of such a workflow must be saved in a skill, so that my other projects can use it as well. I open-sourced the skill, called orchestrate.
With over a million lines of source code, everything feels uncharted territory to me. As examples:
- Continuous integration (CI) must be turned on from early stages of the project, so that bugs can be caught by tens of thousands of unit and integration tests before they are merged into the main branch.
- All commits, pull requests, and merges should be performed by agents, for the sake of improving the workflow efficiency. 852 pull requests have been merged over the past three weeks alone, which is around 41 a day.
- The advantage of working with agents is that they can work at night as well. However, I let the agent know that there should be no questions pausing progress over night, and all decisions should be made on my behalf, and summarized for my review in the morning.
- It’s a good idea to use computer use for live testing over multiple browsers and user accounts, and the prompts for the testing agents should also be written by the same orchestrating agent.
- Continuous improvement is important: deploy often, and use load balanced servers so that if one server is draining requests for its restart, the other server can take over, and the website will not have any downtime.
After over two years of design, development, and testing, by hand and using agents, ReviewsDue is now in production, serving its first conference. I plan to improve and maintain this project continuously over the years and decades to come, with the hope that more conferences will start using it in the future.