Back to my AI setup that I previously said lived, like Harry Potter, in the cupboard under the stairs.
It has evolved in a couple of weeks.
Now it has a hierarchy. Agent Number One:
(1) is the agent I speak to the most → and speech is the primary mode of input with text or generated artifacts in return;
(2) and manages a fleet1 of agents that often have specific roles.
However, since then, I’ve recruited agents to red-team Number One’s plan (including Agent 1b); counselor agents; and an agent to specifically provide human feedback to the entire stack.
All of this has made me a manager of agents. But it looked familiar.
It looked like a teaching hospital.
Managing AIs?
Yes a teaching hospital. When I was a PGY 1 I supervised a medical student. When I was a PGY2 I had 2 interns2 and a bunch of medical students.
But medicine pretends NOT to like the word manager. We aren’t the managers - they are the blue blazers3, not us. In Canada we even removed it from the curriculum. The Royal College’s CanMEDS framework included a role called Manager for nearly two decades, and then in 2015 we renamed it Leader, partly because “manager” sounded too administrative, too middle-management, and maybe insufficiently aspirational.
But I think we cancelled the word manager about ten years before every physician was going to need it back.
The rest of the economy is not waiting for us
This is not a niche prediction. We are all going to be AI agent managers. There will be legions of AIs doing autonomous tasks and physicians will have to manage them, similar but not identical to managing a human workforce. Last year’s Microsoft’s Work Trend Index surveyed 31,000 workers and identified what every employee is about to become: the agent boss. The AI agent manager will be assigning them tasks, reviewing their output, deciding what gets delegated at all. Leaders in that survey expected their teams to be training and managing agents within five years. Jensen Huang put it even more bluntly at CES 2025 more than a year ago now: the IT department of every company is going to become the HR department of AI agents. Onboarding them. Training them on the local vocabulary. Likely even keeping them in line.
And medicine’s AI fleet is already forming
The agents are not hypothetical anymore. This spring Nature reported DeepRare, a multi-agent system for rare disease diagnosis that coordinates more than 40 tools, generates a ranked differential, and shows its reasoning chain - expert reviewers agreed with 95.4% of those chains. For research, in Science this month was Biomni, a general-purpose biomedical research agent that composes its own workflows across 25 domains and runs them. A Mayo Clinic scoping review of agentic AI in healthcare described seven studies of agentic AI systems across different domains, some of which were deployed.
NEJM Catalyst is already arguing about how to PAY for this! It’s coming! and their framing is familiar: AI that will “deliver clinician-grade care under the direction of a clinician.” Under the direction of a clinician. That job description is “manager”.
So what does managing agents actually mean?
I have been thinking about what I actually DO with this fleet, hour to hour. It is not coding. It is sort of execution. I do three things.
I brief and give context: I tell an agent what I want it to know for the task, what the task is, what the constraints are, and what “finished” looks like. The quality of what comes back depends almost entirely on how well I posed the task. Without context, without a goal, or without the boundaries - it often is a terrible result!
I check. Not everything. That would defeat the purpose, and honestly there isn’t time. An attending doesn’t repeat every physical exam on rounds; they pick the ones that matter and takes the rest on trust. I now make that same choice dozens of times a day. Which agent outputs do I read closely, which do I skim, which do I hand to Number 1 to attack before I ever see them. Getting that calibration right is a lot of guesswork and akin to the supervision of medical trainees, nobody taught me this either.
I decide how much rope. Competency based medical education has a beautiful word for this: entrustment. We built a whole framework for deciding when a trainee can do a task with less supervision, and the wisdom in it is that trust is never a global verdict. It is task by task. Full autonomy on the history but direct observation on the family meeting. I run the same judgement on my agents now. Number One can send a calendar invite on its own; but it absolutely cannot send an email in my name4.
So we need to teach this
I learned this by practice, over three months of working with agents relying on two decades of mainly teaching hospital experience, and also watching people who were good at it and occasionally being burned by people who were not. Experiential management I think sort of works when the things we managed were other humans who could push back, fill in our gaps, and tell us when our instructions made no sense.
Agents will not do that always, at least right now. They may cheerfully execute a bad note. The potential cost of “untrained management” i.e. errors is about to go up.
So I think we should teach how to manage overtly and mandatorily in medical school and residency. Teach briefing the way we eventually taught handover and checklists. Teach “checking” as its own competency: how to sample another’s work, what to verify personally, how to give feedback that changes the next draft rather than just grading the last one. And make entrustment of AI explicit: if a resident is going to decide a course of action based on an AI: they should be able to say exactly what they have entrusted, on what evidence, and what they still need to verify every single time. As I’ve written about before, I think that is an assessable skill.
And frankly the trainees might be the easy part - they are significantly more able with AI. It is the mid-career attendings, me included, who are managing fleets right now with no training at all. Faculty development has to catch up to this, and the honest starting point is admitting that trainee supervision was always management, and that we have been credentialing all management in medicine purely on vibes.
Back to the agent under the cupboard
So maybe this is what the last couple of weeks have been teaching me. The cupboard under the stairs, it turns out, is very similar to a teaching hospital. The hierarchy, the red-teamers, the feedback agent wasn’t computer science. It felt like I was the senior resident on medicine again, running a clinical teaching unit.
Only this time around we need to teach the next generation how to do the management.
(Gary Gallagher - a shining example - one of the best managers I have ever had)
Rob Sargeant, one of my colleagues in internal medicine when I was at St. Michael’s, was an inspirational young physician who passed away far too soon a few years ago. He would often joke about terms for groups of physicians, akin to groups of animals (a herd of elephants, a pride of lions, a parliament of owls, etc). This was inspired by teaching hospitals where on rounds, you often saw the attending followed by an army of housestaff, going around the hospital, so it certainly was a group that was travelling. Rob of course insisted that it had to be a “fleet of gastroenterologists”.
And yes, I realize I’ve become a blue blazer.
Instead it has its own e-mail: sammybot at samirgrover.com