How tech vendors use AI agents within their own operations gives customers some useful pointers. For one thing, it's a good way of testing whether their advocacy for the tech is just a marketing ploy or founded on practical experience. If they have truly embraced it, then what they've learned along the way as an informed early adopter can provide valuable lessons for the rest of us. I recently caught up with Olivia Nottebohm, COO of Box, to find out how the enterprise content management vendor has been reshaping its own operations to make best use of agentic AI.

There were several surprises in what she said, which contradict some commonly held views about the impact of AI. For example, Box doesn't agree that AI agents will mean it employs fewer people — it is actually creating new roles to manage its own roll-out of agents and it sees AI as an aid to growth rather than a reason to cut back. She tells me:

In the near term, what we at Box are seeing is actually new roles being introduced, which is fascinating. It's the opposite of what the meme is out there, which is there's this compression, there's this narrowing, there's this loss of jobs.

Instead, what we're seeing is, ‘No, actually, because we use AI to drive more growth, what we need to do is not compress the org structure, but rather add roles that we've never had before'... That's been really invigorating because it means that there's new opportunities for our Boxers, there's new opportunity to bring in different talent.

So for us, it's been more of an expansive mindset than a contractionary approach to the talent pool and our org structure.

The ten new job titles include tech-centric roles such as AI program manager, AI architect, forward-deployed engineer for AI, and AI platform specialist. But they also include more business-focused roles that shed light on how Box is deploying agents internally. One example is the AI workflow designer, a role that helps figure out the best way to apply agents to automate business processes. She explains:

We have someone who is working right now on workflows that go across go-to-market teams — so marketing, sales, to customer success. Their day job is literally designing the workflow, with AI powering it, that will simplify some of the work, the workflows that we do today, so those processes.

That for us is a full-time job, because they have to deeply understand the processes that exist today. They need to deeply understand AI, and then they need to be able to redesign that in a way that takes AI into account and is actually not just taking the same process and 'automating' it, but redesigning the process entirely, because you know the capabilities that AI can accomplish.

An allied role is the AI program manager, who helps to put agents into production and then tracks and manages their ongoing performance. This role has more of a technical element as it's responsible for writing the right instructions for the agent, making sure it uses the appropriate models in the right way, and for ensuring effective integrations into other data sources. Once in production, the role continues to optimize and manage those agents, while tracking the usage, adoption, and business value that they generate. This centrally resourced role works in tandem with leaders within a given business function that also play their part in AI management, bringing their functional expertise to bear and also being a role model for how colleagues should use the agent.

One agent that Nottebohm cites as a "tangible and pragmatic" example of AI's business impact is helping salespeople prepare a case for customers to upgrade to Box's highest subscription level, Enterprise Advanced. By collating information that would previously have taken hours to do manually, it frees up that time that can be used more effectively when speaking to customers. She explains:

That agent has been given the instructions of everything Enterprise Advanced can do, from a capabilities perspective. It's also been given a set of business realities of what would constitute impactful business ROI. And then of course it's accessing all the information we have about that company in our unstructured data that's within Box.

It provides an incredibly tailored view, specific to that company, specific to the stakeholders we already know, specific to the realities of that business, for how you can leverage our tier that unlocks AI and workflows, all of that.

Another way in which Box goes against the grain of conventional wisdom is in its rejection of personal AI assistants. In contrast to the investment in enterprise agents that take on processes used by teams or across different functions, Box has no budget for personal AI assistants and doesn't encourage them. "I think that's failure mode," Nottebohm tells me. She explains:

What we're finding, and I think it's now starting to be understood more broadly, is that productivity agents — like some of these other companies that offer things that help you optimize your calendar or write a better email — you'll get some benefit from that. But really, the big unlock is from what I would call enterprise-grade agents... We actually at Box have not put a lot of investment into the personal agents. In fact, we don't have any personal agents — we haven't deployed the standard ones that you would think about.

We're really focused on Box agents. I can always go into Box AI and ask a question of my content. That's standard. But in terms of enterprise-grade deployable agents — there's over 40 of them, so it's not like we just have two or three — but they're role-specific and they do a certain task. It's repeatable, and we know that it meets the quality bar that we require from that role... You have the [Box] platform as the anchor. You have the security that's built into the platform. You have the governance and the permissioning, all of that.

I think where people get stuck is, they build these personal agents, and then they try to send them out into the wild across all of these other data sources, and the interoperability of the agents, I think, is far from where the vision is still.

Focusing on task automation — while humans, not agents, still take the important decisions — is where Box is able to see a return on investment in its use of agents. Often these are new tasks and processes that weren't cost-effective to take on in the past, but have now become viable with the right models. Nottebohm sees this happening at customers, too:

Customers have started to say, 'Okay, I can deploy XYZ model, which is costs is now reasonable, and I can get to 97% accuracy, which is greater than human accuracy, and I can do it within a latency that makes sense and all of that. And so all of a sudden, now I can do workflows that either I just never had a human do because it was too expensive, or I had humans doing it, but they couldn't do it at a scale that I would like to do it to really grow my business. 

So I think you see unlocks in two dimensions. One is the scalability of it now, but also the ability to do workflows that were never considered with humans in the first place. A lot of that has to do with not only the advancements of the tech itself, but the cost curve of when those models are available and at what price, frankly. 

I think we saw this time of, it was cool to spend as much as possible on AI. That's not a long-term stable state of enterprise IT. We know that there's a limit to IT budgets, even if you're starting to make them interchangeable with labor, because there's a finite amount of labor budget within the P&L of a company. So I do think people are being more reasonable now and starting to really understand, 'Okay, what sort of ROI do I need, what does that process look like, and then how do I track it and measure it to make sure I am actually getting that ROI?'

As an enterprise technology vendor, Box has more than its share of unstructured content within its own operations — contracts, product documentation, sales proposals, support tickets, policies, regulations, call records, message threads and email archives, to name just the most obvious examples. It has a dual incentive, therefore, to deploy agents that help it utilize that content more effectively. Not only to realize business value in its own operations, but also to learn from its experience and pass on those learnings to its customers as they contemplate their own libraries of unstructured content. Some of the key takeaways:

• Focus on repetitive tasks and processes that are helpful at a team or organizational level.
• Beware of investing in personal assistants that serve individuals but have no wider enterprise context.
• Look to automate scalable, value-driving activities that will save your staff time, or which have been too labor-intensive to do in the past.
• Instead of maxxing out on tokens, use lower-cost models that produce 'good enough' results that still beat manual alternatives.
• Create a central team with the appropriate skills to design, deploy and monitor AI-powered workflows to deliver real business value.
• Create AI 'champions' in your business teams with the functional expertise to help design effective AI workflows, and who will spearhead the roll-out within their departments and locations.

I've often felt that putting generative AI to work on unstructured data offers the best scope for return on investment, because this is precisely the kind of data and process that its probabilistic methodology excels at. And instead of trying to do what we already do with fewer people, it opens up the potential to do things that were never economically viable before, thus creating new jobs and growth rather than contracting existing operations, as Box is finding.