This is an AHRA Quick Credit article. The corresponding post-test in the AHRA Online Institute is coming soon.

Abstract

Healthcare systems are moving fast to deploy medical imaging AI for triage, detection, workflow orchestration, and reporting support. Yet after initial go‑lives, many organizations discover gaps in governance, acceptance testing, and ongoing performance surveillance, areas where traditional picture archiving and communication systems (PACS), radiology information systems (RIS), and quality infrastructures were never designed to help.

The American College of Radiology (ACR) now offers two complementary resources to close those gaps: the ACR Recognized Center for Healthcare‑AI (ARCH‑AI), a recognition program that operationalizes best practices for safe, effective AI use; and Assess‑AI, a National Radiology Data Registry program that continuously monitors real‑world AI performance and benchmarks it against national peers. Together, these programs provide a life cycle approach to AI management spanning procurement, local validation, deployment, monitoring, and assessment.

This article also examines the governance challenges associated with AI adoption and outlines how structured oversight models can mitigate risk and support sustainable innovation in imaging departments.

Over the past several years, the number and variety of imaging AI products has grown significantly, fueled by maturing methods, clinical demand, and investment. Healthcare systems now face a growing marketplace of FDA‑cleared tools addressing triage, detection, quantification, and workflow orchestration.

However, experience across institutions shows that performance observed in premarket studies may not generalize to local populations, scanners, protocols, and workflows. In fact, published studies document variability in performance across sites and subgroups. These realities underscore the need for structured, locally accountable oversight and continuous evaluation and monitoring^1-8.

Regulatory Context

Regulatory agencies have acknowledged that AI presents unique oversight challenges. The FDA has articulated a total product life cycle framework for AI and machine learning Software as a Medical Device (SaMD). This framework emphasizes premarket evaluation, transparency, post-market monitoring, and continuous improvement⁹.

Although regulatory authorization confirms that a device met defined safety and effectiveness criteria during review, it does not guarantee uniform performance across all clinical environments. Development data sets may not reflect local patient populations or workflow conditions. Therefore, healthcare organizations remain responsible for validating performance within their own settings.

 Administrators must understand that regulatory clearance represents one component of oversight rather than a comprehensive assurance of ongoing performance. Structured institutional governance, model selection criteria, local acceptance testing, and ongoing monitoring are all required to bridge the gap between regulatory review and real-world clinical implementation. To support healthcare systems navigating these challenges, the ACR has developed two programs: ARCH-AI and Assess-AI.

Program Overviews: ARCH‑AI and Assess‑AI

ARCH‑AI

ARCH‑AI is the first national AI quality assurance recognition program for radiology facilities. It describes best practices and recognizes organizations that attest to having the appropriate governance, inventory, security, selection, acceptance testing, monitoring, and (where applicable) local model controls in place. Recognition is intended to demonstrate adherence to best practices and build trust with patients, payers, and referring clinicians.

Above, left: ACR Recognized Center for Healthcare-AI (ARCH-AI) badge awarded to organizations that demonstrate adherence to best practices for the governance, implementation, and monitoring of artificial intelligence in imaging interpretation.

Above, right: ACR Assess-AI badge indicating participation in the American College of Radiology’s Assess-AI registry for real-world performance monitoring of clinical imaging AI models.

Key criteria for recognition include:

• Establishing an interdisciplinary AI governance group.
• Maintaining a detailed inventory of AI tools and use cases.
• Ensuring security and compliance checkpoints.
• Performing structured selection and local acceptance testing.
• Monitoring performance and effectiveness.
• Contributing to Assess‑AI for benchmarking.

Governance infrastructure typically includes a multidisciplinary AI oversight committee composed of radiologists, informatics leaders, compliance officers, quality personnel, and information technology representatives. This committee defines evaluation criteria for new AI solutions, reviews vendor validation data, and establishes local testing requirements.

Standardized processes should require documentation of intended use, performance metrics, regulatory status, integration considerations, and model update pathways. Institutions benefit from maintaining formal records of validation procedures and performance thresholds prior to clinical deployment. By formalizing governance structures, ARCH-AI promotes consistency, transparency, and accountability across AI initiatives.

Healthcare systems are already familiar with the ACR framework for advancing quality through accreditation programs across imaging modalities such as CT, MRI, ultrasound, mammography, and nuclear medicine. These programs are grounded in ACR Practice Parameters and Technical Standards, which outline the training, equipment, clinical processes, and quality assurance activities necessary to support safe and effective imaging¹⁰.

As AI becomes more integrated into radiology workflows, similar guidance is needed to ensure responsible implementation. Lessons learned from ARCH-AI have also informed the development of a proposed ACR–SIIM Practice Parameter for Imaging Artificial Intelligence, being developed in collaboration with the Society for Imaging Informatics in Medicine to further formalize guidance for the safe use of AI in clinical imaging.

Assess‑AI

The National Radiology Data Registry (NRDR), is a suite of clinical quality registries that enables organizations to benchmark performance, monitor quality metrics, and support continuous improvement through national comparative data. Thousands of imaging facilities already participate in NRDR registries such as the Dose Index Registry, National Mammography Database, and Lung Cancer Screening Registry¹¹. Because many healthcare organizations already have the technical infrastructure and workflows in place to submit data to NRDR, implementing Assess-AI, ACR’s registry for post-deployment monitoring of imaging AI models, builds upon existing processes rather than requiring a completely new reporting framework.

Organizations participating in Assess-AI submit de‑identified AI result outputs, associated radiology reports, and study metadata via ACR Connect. Organizations currently participating in NRDR already have ACR Connect installed or are using its predecessor, TRIAD, which can be easily upgraded.

ACR Connect operates locally at a facility or within a private cloud and enables secure connections to systems such as RIS, PACS, and electronic health record (EHR)¹². The platform supports local data processing, distributed/federated processing, and an application-based architecture that allows additional tools and workflows to be deployed as needed.

The Assess-AI registry normalizes vendor outputs, extracts surrogate labels from reports using secure large language models prompting, computes concordance/discordance, and returns interactive dashboards for local and national benchmarking (Figure 1, below).

Figure 1. An example facility’s report shows the raw concordance and incidence values for example vendor AI Detect’s ICH model.

Assess‑AI helps facilities monitor input stability (equipment, protocol, software), track model versioning, trend concordance over time, compare to national peers, and drill into discordant cases for root‑cause review using a local re‑identification workflow and the Forensics App on ACR Connect (Figure 2, below). With the Forensics App, a reviewer can view the imaging study, the radiology report, and the clinical AI outputs to understand why the case is discordant. The reviewer can also submit feedback to Assess-AI and use that knowledge to address the AI performance if needed.

Figure 2. The Forensics App on the local facility Connect node allows a reviewer to adjudicate discordant cases.

Ensuring consistent performance is an important quality requirement. As part of acceptance testing and ongoing monitoring, analysis of subgroup performance (e.g., age, sex, scanner manufacturer) using available metadata and investigation of meaningful gaps will help identify potential inconsistencies in performance. Assess-AI provides dashboards with drill-down reports that enable transparent communication of known limitations to both end users and leadership.

ARCH‑AI and Assess‑AI Work Together

ARCH‑AI defines the organizational processes that make AI safe and sustainable; Assess‑AI supplies the longitudinal monitoring. Organizations cannot manage what they cannot measure.

Practically, this means governance policies require local acceptance testing and continuous monitoring. The Assess-AI registry provides the mechanism for meeting these requirements. The registry provides standardized measures, comparative context, and drill‑down tools. Insights from the registry feed back into training, protocol adjustments, vendor engagement, and, when needed, pause or rollback decisions^{1, 9, 13}.