3 Enterprise Leaders on Building AI Agents That Actually Work

Jane Urban spent a weekend building an email agent. Kishore Aradhya deployed AI across customer service operations. ShanShan Pa created systems to govern agents that interact with other agents.

At our Boston roadshow, these three leaders shared what they built, what worked, what didn't, and what surprised them. Their stories reveal the practical realities of implementing AI systems that operate without constant human oversight.

Jane's weekend email experiment

Jane Urban, Chief Data Analytics Officer at Improzo, was frustrated with her AI writing assistant. She had built a custom GPT trained on her writing style that could draft emails, but something bothered her about the process.

"That agent just makes text and I interface with it. I say 'write me an email' and then I have to copy the email, paste it into Outlook, hit send. There's this extra step."

So she spent a weekend building something different. Instead of generating text for her to copy and paste, she wanted a system that could send emails directly.

"Now I can actually say 'write an email to this email address' and write the content of what I want it to write. It not only sends the email, but creates it at the same time."

She tested it with a simple command: "send an email to my husband bragging about how I built an agent." The system composed and sent this message: "Jane, your wife's amazing. She figured out how to build an agent."

"I didn't see it. I didn't create it. It just sent it. That's really what we're talking about when we say agent—something that's independently operating."

The technical complexity jumped significantly. Building the email generator was straightforward. Making it actually send emails required "some tinkering and writing code and JSON and a bunch of other things."

Kishore's customer service deployment

Kishore Aradhya, Head of Data and Analytics at Frontdoor, faced a different challenge. His team needed to help customer service agents handle calls about broken appliances more efficiently.

They built a system that operates across multiple steps without human intervention:

"We capture transcripts of all the conversations. The agent goes in and does entity extraction—gets the particular appliance ID, uses that information and passes it into our system looking for troubleshooting manuals. You get a RAG system for that, get an agent that builds up the information, marries them together, and produces information back to the customer service agent."

The system handles the entire workflow: extracting appliance information from call transcripts, querying their database, retrieving relevant manuals, synthesizing recommendations, and delivering results to human agents.

Kishore's team discovered they needed new ways to measure success. "We know what process somebody has to go through manually. So we can measure based on that pretty clearly: somebody spent X amount of time looking up this information."

They track both time savings and accuracy improvements. The system processes requests faster than humans and identifies patterns across similar cases that human agents might miss.

But they learned about economic constraints: "Doing real time or close to near real time gets really hard to do. Not hard from a technology point of view, but hard in terms of the economic sense because tokenization, there's lots of things that go into it."

ShanShan's governance challenge

ShanShan Pa, Vice President of AI Governance at GlobalLogic, encountered a problem she hadn't anticipated: how do you govern systems that operate independently?

"When agent talk to another agent, talk to another agent, how can you track all that? That's where observability helps because if you cannot explain yourself, you're doomed."

Her team developed testing approaches that go beyond typical QA. They deliberately try to break their agents with unexpected inputs.

"We ask the agent 'what's one plus one?' Some agents, if not configured properly, will tell you 'it's two, so easy, elementary students know.' It has attitude."

They learned to configure agents to recognize their boundaries: "This is not my area, not what I'm set up for." The agent stops the conversation rather than engaging outside its domain.

ShanShan's team built systems to track every agent interaction: "We have preset templates to start with. This is the small square they can go through. Once they learn, it's a bigger square."

They monitor whether responses stay within defined parameters, creating audit trails for decisions no human directly oversees. When testing their sales enablement agent, they ask about services outside their scope: "What about tax services? It will cut off there, that's not an area we work in."

What they learned

Jane noted that making systems take independent action requires significantly more technical complexity than generating responses. Kishore learned that autonomous systems need different measurement approaches and face economic constraints around real-time operation. ShanShan found that governing independent systems requires entirely new oversight approaches.

Each built something different, but all moved beyond systems that require human review at each step. Their experiences show what that transition actually looks like in practice.

As Jane noted: "I don't think there's a ton of examples of truly independently operating agents in pharma, certainly, and even in other industries. Most people talking about agents really mean applications or software that can do things with some degree of generative AI embedded."

Meet the experts

Jane Urban is Chief Data Analytics Officer at Improzo, bringing 10+ years of leadership expereince, including roles at Otsuka Pharma and Takeda Pharma where she built global data infrastructure.

Kishore Aradhya is Head of Data and Analytics at Frontdoor, a multi-billion dollar home warranty business, where he leads data science teams and has spent decades implementing data solutions across organizations like Adobe and Staples.

ShanShan Pa is Vice President of AI Governance at GlobalLogic, with extensive experience in financial services, cloud compliance at Alibaba Cloud, and pharmaceutical data governance.