Q & A: Scaling generative AI and agentic AI without sacrificing trust

We sat down with Dr. Lorne Rothman, Principal, Data Sciences, Canada Healthcare Lead for SAS, to peel through the second annual Data and AI Impact Report: The New Economics of Trust through a healthcare lens.

The research shows that across all industries, Canada has made significant progress in building the foundations for trustworthy AI, but most organizations are still waiting to see meaningful business results from those investments.

Q: Canada’s AI trust scores are rising, yet organizations still aren’t seeing corresponding gains in business impact or ROI. In healthcare, do you think the issue is that we’re being too cautious with AI, or that we’re still measuring success using the wrong metrics?

A: The survey didn’t specifically target healthcare, so it’s worth keeping that in mind. For Canada, results show that the Trustworthiness Index is rising, while Perceived Trust actually declined, so when we say AI trust scores are rising in Canada, we’re referring to the Trustworthiness Index, not Perceived Trust.

Focusing on generative AI and the move to more autonomous, agentic AI, I wouldn’t say healthcare is being too cautious, but it is cautious. There are lags between when a technology shows it can address a problem, when the regulatory and governance requirements are in place (trustworthiness), and when healthcare organizations are willing to test and implement it (trust).

I think a lot of healthcare AI is still in that experimentation and testing stage in Canada rather than in production.

Some of the gap between improving trustworthiness and measurable impact may simply reflect the time it takes to move from experimentation to production and gaining the necessary trust, particularly with the fairly abrupt and transformational emergence of agentic AI, before realizing benefits at scale. And in healthcare those lags may be longer than in other industries.

We also need to be careful about how business impact or ROI is defined in Canadian healthcare. The return may not be primarily financial savings. It may be freeing up clinician time, reducing administrative work, shortening wait times, improving access and quality of care, or allowing the same resources to serve more patients.

Q: The research shows people often override AI recommendations because they don’t understand how the model reached its conclusion. In a clinical setting, how much explainability is enough before a physician or care team can confidently act on an AI-driven recommendation?

A: Let’s first be clear about what we mean by explainability, because it means different things depending on the type of AI. With traditional predictive machine learning models using tabular data (columns, rows, and numbers), we’re generally talking about understanding what factors drove a prediction, both overall and for an individual patient. There are now standard approaches and algorithms for doing that.

With generative and agentic AI, explainability can mean something quite different. If an LLM or agent reaches a conclusion based on a patient’s record or other documents, I would want the clinician to be able to ask what evidence it relied on, where in the patient’s record that evidence came from, and ideally go back to the actual source patient records. If an agent takes actions, there also needs to be a record of what it did and why.

And if a predictive model is being used somewhere within that process, then the traditional forms of model explainability still apply to that component.

I don’t think there’s a single level of explainability that’s enough. It depends on what the AI is doing and what happens if it gets it wrong. As the consequences and degree of autonomy increase, I’d expect the requirements for explanation, traceability and human oversight to increase as well.

Regardless, this brings up an important point: we’re not simply letting agents loose on everything and allowing them to make fully autonomous decisions. Their access can be restricted to specific subsets of patient data, and consequential actions should require human review.

Q: The report suggests organizations are shifting focus from model development to data quality and governance. In healthcare, are we at a stage where the biggest barrier to better AI outcomes isn’t the sophistication of the algorithm, but the quality and accessibility of the underlying patient and operational data? 

A: From what I’ve seen, the data can certainly be a bigger challenge than the sophistication of the algorithm. We have good predictive modelling and AI techniques, and increasingly sophisticated ways of governing them.

The harder problem in healthcare is often the data. Is the data complete and reliable? Is it representative of the population where the model will actually be used? Can we access it? And can we bring together information that may be sitting in different systems or different organizations?

That’s particularly important in healthcare because there is so much fragmentation. You can have very good data, but if it’s sitting in separate clinical, laboratory, pharmacy and operational systems, it may still be difficult to use effectively. And you can also have data across silos that isn’t standardized (different terminology, units of measurement, ways of expressing dosages, etc.).

I think the shift toward data quality and governance makes sense. Better algorithms can’t necessarily compensate for missing or unreliable data. Interestingly, generative AI may actually help address some of the other data problems, particularly non-standardization and information that is difficult to use because it’s buried in unstructured (text) records.

So, getting the data foundation right remains important, but AI itself may also become part of the solution to some of these data challenges. In fact, we’re already seeing interest in using generative AI to help deal with some of these data standardization problems.

Q: Healthcare is one of the most regulated sectors in Canada, and compliance concerns are rising across organizations. How can healthcare leaders strike the right balance between patient safety and regulatory oversight without creating so much friction that promising AI initiatives never make it into practice?

A: I don’t think the answer is less regulation. Patient safety, privacy and oversight have to come first in healthcare. But we also don’t want the process to become so difficult that useful applications remain stuck in pilot projects.

Part of the answer is taking a risk-based approach. Not every use of AI presents the same level of risk. An AI application helping schedule appointments is very different from one influencing a diagnosis or treatment decision, or one that surfaces highly sensitive patient data. The level of oversight should reflect that.

And governance doesn’t always mean adding another layer of committees and approvals. Some of it can be built directly into the technology: controlling what data an AI can access, what actions it can take, where human approval is required, while keeping a record of what it did.

I don’t see safety and innovation as opposing objectives. Good governance should allow lower-risk applications to move relatively quickly while applying greater scrutiny where AI could directly affect patient care or privacy.

Q: The report suggests organizations are slowing their adoption of more autonomous, agentic AI until stronger governance is in place. Looking ahead three to five years, what healthcare workflow would you be comfortable handing over to an AI agent first, and which decisions should always remain in human hands?

A: I would start with workflows that are relatively low risk and where an error can be easily identified and corrected. Scheduling, routing referrals, following up on missing information and coordinating routine administrative processes are obvious candidates.

But I wouldn’t simply divide this into administrative versus clinical workflows. The risk can vary considerably depending on the use case. De-identification is a good example. It may sound like a relatively straightforward data-processing task, but an LLM has to work with identifiable patient information, and missing something could expose that information downstream. On the other hand, a seemingly higher-risk use of an LLM might be acceptable with the right human oversight. An LLM might flag a patient record as indicating a particular medical condition. But if the record has first been de-identified so it no longer contains personally identifiable information, and a physician reviews the case before clinical action is taken, risk can be reduced.

I think we need to look at risk across the entire workflow: what data an AI agent has access to, what systems it can interact with, what actions it is allowed to take, and how easily an error can be detected. Governance of agentic AI isn’t just policies and committees. It can also be built directly into the technology, for example through decision flows (a branching structure that combines deterministic rules, predictive models, LLMs and AI agents, with different decisions or actions taken depending on the patient’s path through the flow).

You can restrict what data and tools an agent can access, constrain what actions it can take, control the prompts used within an agentic decision flow so the LLM is given specific, predefined tasks and instructions, build human approval points into the decision flow, log the agent’s actions, and put guardrails around which patients can be assessed, which data can be reviewed, and where and how AI-generated conclusions can influence subsequent decisions and actions.

————————————————————————————————————————————————————————–

Are healthcare organizations and consumers using AI responsibly?

By Neil Zeidenberg

TORONTO – A session called “Scaling AI Responsibly in the Public Sector,” held in September at the SAS Innovate on Tour conference, posed several questions we should ask ourselves prior to using and trusting AI.

Speakers included Saad Rais, a senior manager of health data science at the Ontario Ministry of Health.

Asked, ‘What does responsible AI mean to you, and to whom are you responsible?’, Rais first noted that before allowing AI to access patient data, we should ask ourselves: “How will a patient’s information be used, and what exactly will be disclosed? Is the information being used to create models that train AI? How and where will that information be going – will it be staying in Canada, or will it end up in another country? Finally, how will that information be used by others?”

He asserted that before you can trust in AI, you need to safeguard the data and specify who is responsible for protecting it. That’s where governance comes in – it’s about managing the data through defined or set policies, roles and standards and determining who’s responsible for the data.

Rais added, a common concern people have is accuracy of AI. “In a recent KPMG survey, two out of three people have adopted AI, but 46 percent of those people didn’t trust the AI. And that number varied across different demographics. “People need to first trust the AI and what it’s doing before they’re truly comfortable using it.”

Those with higher adoption had higher trust in AI; those with lower adoption had lower trust.

At the same time, a group of users adopted AI without considering accuracy and 56-percent of them experienced errors in their work. Many of these users worry less about the accuracy of AI in exchange for expedience.”

Speakers agreed there needs to be a governance structure put into place – creating the rules of engagement and determining who is responsible for the data. Ideally, the data should have predictive powers, demonstrate value through better outcomes and you should have to prove the case prior to a major launch.