By Daniel H. Bendell

Quality Assurance professionals have spent decades developing methods for establishing confidence in computerized systems. We define requirements, test whether systems perform as intended, control the information they process, and establish procedures governing how their outputs are reviewed and used.

Generative artificial intelligence does not make those disciplines obsolete. But enterprise AI introduces something different between controlled information and governed output: a process of derivation and synthesis that is considerably more difficult to constrain, reproduce, and predict.

This suggests a useful way of considering enterprise AI governance: as a risk curve.

At either end of the curve are disciplines with which Quality organizations are already familiar. At the input are information governance, identity, permissions, classification, document control, and data quality. At the output are human review, approval, audit trails, procedures, and accountability.

The less familiar territory lies between them.

Inputs → GenAI Derivation and Synthesis → Outputs

Figure 1. The Enterprise AI Governance Curve

It is in the middle that generative AI retrieves, summarizes, correlates, interprets, infers, and synthesizes information into organizational insight. This is where the governance risk rises.

Validation Already Accommodates Variability

Some computerized systems are highly deterministic. A spreadsheet applying a fixed formula illustrates the simplest case: known inputs and an unchanged calculation produce a predictable result.

Traditional validation, however, is not limited to systems or processes that produce identical results.

A laboratory instrument or automated manufacturing process may exhibit expected variability. Measurements may differ slightly between runs, or a process may operate within an acceptable range rather than at one mathematically exact value. We establish confidence by characterizing that variability, defining acceptable performance, and demonstrating how the system or process operates within those expectations.

Generative AI presents a more difficult form of variability.

Consider an enterprise AI asked to review information concerning a quality event, identify relevant relationships, and summarize areas warranting further investigation.

Two responses might differ in wording, organization, or emphasis without either being incorrect. But the differences can become more consequential. One response might identify a relationship another misses. One might emphasize an approved investigation while another gives greater significance to an informal laboratory observation. Both responses might appear reasonable while directing the investigator toward different conclusions.

The problem, therefore, is not simply that generative AI may produce different answers. Traditional validation already accommodates variability.

The more difficult question is whether we can sufficiently characterize acceptable variability when the system’s function includes interpretation and inference.

The answer will necessarily depend in part on what the organization intends the AI to do and the consequences of poor performance. But as AI moves beyond retrieval and summarization toward interpretation and inference, defining acceptable performance becomes increasingly difficult.

The Enterprise AI Governance Risk Curve

At the input side of the curve, organizations have substantial governance mechanisms available.

Enterprise AI should only use information that the requesting user is authorized to access. Existing identity, authorization, classification, and information-governance controls provide the foundation.

AI does make the accuracy of those controls more consequential. Historically, a person might technically have access to information without knowing it existed or knowing where to find it. Enterprise AI largely removes that practical limitation. It can potentially discover and synthesize information from every source the user is permitted to access.

At the output side, familiar governance mechanisms reappear. AI-generated information can be subjected to human review. Decisions can require approval. Actions can remain governed by procedures. Audit trails can be retained, and people can remain accountable for consequential decisions.

It is the derivation and synthesis between those two points that presents the less familiar challenge.

The progression can be considered as:

Retrieve → Summarize → Correlate → Interpret → Infer → Recommend

As AI moves across this spectrum, it performs an increasing share of the analytical work historically performed by people. That is also where much of its value lies. If enterprise AI merely retrieved documents, it would be little more than an advanced search capability. Organizations want AI precisely because it can connect information, recognize relationships, and produce useful insights.

The governance objective therefore cannot be to eliminate derivation and synthesis. It must be to understand how to govern it.

Not All Information Is Equal

This becomes particularly important when AI synthesizes information from sources of differing authority.

A regulated enterprise may contain approved SOPs, validated laboratory results, regulatory submissions, change controls, laboratory notebooks, draft documents, emails, meeting notes, and informal Teams or Slack discussions.

A user might legitimately have access to all of them. A Quality professional would not ordinarily give all of them equal evidentiary weight.

An approved procedure carries different authority from a draft. A validated result carries different significance from an informal observation. An informal discussion may contain valuable information without having the authority of a controlled record.

Humans familiar with an organization’s quality system make these distinctions routinely. Enterprise AI raises the question of how those distinctions should influence its synthesis.

Access control determines what information AI may use. Knowledge governance must increasingly consider how AI should weigh what it is permitted to use.

This is fundamentally different from asking whether a system correctly retrieved a document or executed a calculation. Two AI analyses might both contain factually accurate information yet reach different conclusions because they assigned different significance to the available evidence.

How an organization establishes confidence in that process is a developing governance challenge.

Validation Is Necessary, but No Longer Sufficient

Traditional validation remains essential. Systems must perform as intended. Access controls must operate correctly. Information must remain protected. Appropriate controls must govern consequential outputs.

Enterprise AI does not invalidate those principles. It exposes an area between them that requires additional attention.

Organizations can take practical steps now: maintain accurate access controls, clearly distinguish controlled and authoritative information, preserve source provenance, require transparency where AI-derived analysis may influence important decisions, and retain appropriate human review and accountability.

At the same time, Quality organizations should begin asking a newer set of questions:

• Does the AI appropriately distinguish among information of differing authority?
• Can users identify the evidence supporting consequential AI-generated insights?
• How should conflicting sources be handled?
• What constitutes acceptable variability when AI performs interpretation or inference?
• Who remains accountable when AI performs analytical work previously performed by people?

The governance frameworks for these questions are still evolving. That does not make enterprise AI unusable, nor does it mean traditional validation has failed. It means that Quality organizations need to recognize where established controls are strong and where new governance thinking is required.

Existing validation and information governance practices provide an important foundation for the use of enterprise AI. Organizations have established approaches for controlling information at the input and for reviewing and governing the resulting output. Generative AI introduces additional considerations in the derivation and synthesis that occurs between those two points — particularly in how information is selected, weighted, interpreted, and combined. As enterprise AI becomes more widely used, Quality professionals and regulated organizations should begin developing a better understanding of these issues and consider how they should be addressed within their existing governance frameworks.

About the Author

Daniel H. Bendell is an independent IT consultant with more than 30 years of experience helping organizations manage, secure, and govern enterprise technology. Through Assurance Technology Management, Inc., he works with regulated and other organizations on practical information-governance, cybersecurity, and emerging-technology issues. To continue the conversation, connect with Dan Bendell on LinkedIn.

Author’s Note on AI Assistance

Generative AI was used during the development of this article to assist with organizing concepts, exploring alternative ways of expressing the author’s ideas, and preparing and refining draft language. The arguments, conclusions, and final content reflect the author’s own understanding and judgment.