An international review of AI Governance Tools and suggestions for pathways forward

Kate Kaye     Pam Dixon

December 2023

---

Download

↓ PDF The Full Report

---

Index

• Brief Summary of Report
• About the Authors
• About The World Privacy Forum
• Executive Summary: Why the World Privacy Forum Conducted This Research
• Background and Introduction
• Methodology
• Findings
• Part I. Discussion: Critical Analysis of AI Governance Tools
• Measuring AI Fairness Measures
• Use Cases in AI Fairness
• Pathways for Building an Evaluation Environment and Creating Improvements in the AI Governance Tools Ecosystem
• PART II: A Survey of AI Governance Tools and Other Notable AI Governance Efforts from around the World
• Intergovernmental Organization Toolkits and Use Cases from International and Regional Multilateral Institutions
• Regional Development Banks
• AI Governance Tools and Use Cases from National Governments and NGOs
• Appendix B: AI Governance Tools and Features Comparison Chart
• Appendix C: Some AI Governance Tools Feature Off-label, Unsuitable, or Out-of-context Uses of Measurement Methods
• Appendix D: OECD Catalog of Tools and Metrics Framework

---

Brief Summary of Report

AI systems should not be deployed without simultaneously evaluating the potential adverse impacts of such systems and mitigating their risks. Most of the world agrees about the need to take precautions against the threats posed by AI systems. Tools and techniques exist to evaluate and measure AI systems for their inclusiveness, fairness, explainability, privacy, safety and other trustworthiness issues. These tools and techniques – called here collectively AI governance tools – can improve such issues. While some AI governance tools provide reassurance to the public and to regulators, the tools too often lack meaningful oversight and quality assessments. Incomplete or ineffective AI governance tools can create a false sense of confidence, cause unintended problems, and generally undermine the promise of AI systems. This report addresses the need for improved AI governance tools.

It is the goal of this research to help gather evidence that will assist in the building of a more reliable body of AI governance tools. This report analyses, investigates, and appraises AI governance tools, including practical guidance, self assessment questionnaires, process frameworks, technical frameworks, technical code, and software disseminated in Africa, Asia, North America, Europe, South America, Australia and New Zealand. The report also analyzes existing frameworks, such as data governance and privacy, and how they integrate into the AI ecosystem. In addition to an extensive survey of AI governance tools, the research presents use cases discussing the contours of specific risks. The research and analysis for this report connects many layers of the AI ecosystem, including policy, standards, scholarly and technical literature, government regulations, and best practices.

Our work found that AI governance tools used in most regions of the world for measuring and reducing risks and negative impacts of AI could introduce novel, unintended problems or create a false sense of confidence unless accompanied by evaluation and measurement of those tools and their effectiveness and accuracy.

In this report we suggest pathways for creating a healthy AI governance tools environment, and offer suggestions for governments, multilateral organizations, and others creating or publishing AI governance tools. These suggestions include best practices taken from existing AI and other quality assessment standards and practices already in widespread use. Appropriate procedural and administrative controls include: 1) providing AI governance tool documentation and contextualization, review, audit, and other quality assurance procedures to prevent integration of inappropriate or ineffective methods in policy guidance; 2) identifying and preventing conflicts of interest; and 3) ensuring that capabilities and functionality of AI governance tools align with policy goals. If governments, multilateral institutions, and others working with or creating AI governance tools can incorporate lessons learned from other mature fields such as data governance and quality assessment, the result will establish a healthier body of AI governance tools, and over time, healthier and more trustworthy AI ecosystems.

About the Authors

(Listed alphabetically)

Pam Dixon is the founder and executive director of the World Privacy Forum, a respected nonprofit, non-partisan, public interest research group. An author and researcher, she has written influential studies in the area of identity, AI, health, and complex data ecosystems and their governance for more than 20 years. Dixon has worked extensively on data governance and privacy across multiple jurisdictions, including the US, India, Africa, Asia, the EU, and additional jurisdictions. Her field research on India’s Aadhaar identity ecosystem, peer-reviewed and published in Nature Springer, was cited in India’s landmark Aadhaar Privacy Supreme Court opinion. Dixon currently serves as the co-chair of the UN Statistics Data Governance and Legal Frameworks working group, and is co-chair of WHO’s Research, Academic, and Technical network. At OECD, Dixon is a member of the OECD.AI Network of Experts and serves in multiple expert groups, including the AI Futures group. In prior work at OECD, Dixon was part of the original AI expert group that crafted the OECD AI Principles, which were ratified in 2019. Dixon has presented her work on complex data ecosystems governance to the The National Academies of Sciences, Engineering, and Medicine and to the Royal Academies of Science. She is the author of nine books and numerous studies and articles, and she serves on the editorial board of the Journal of Technology Science, a Harvard-based publication. Dixon was named one of the most influential global experts in digital identity in 2021. Dixon received the Electronic Frontier Foundation Pioneer Award in 2021 for her ongoing oeuvre of groundbreaking research regarding privacy and data ecosystems.

Kate Kaye, deputy director of the World Privacy Forum, is a researcher, author, and award-winning journalist. She is a member of the OECD.AI Network of Experts, where she contributes to the Expert Group on AI Risk and Accountability. In addition to her extensive research and reporting on data and AI, Kate is the recipient of the Montreal AI Ethics Institute Research Internship, and was a member of UNHCR’s Hive Data Advisory Board.

Editor: Robert Gellman

Data Visualizations and Report Design: John Emerson

About The World Privacy Forum

The World Privacy Forum is a respected NGO and non-partisan public interest research group focused on conducting research and analysis in the area of privacy and complex data ecosystems and their governance, including in the areas of identity, AI, health, and others. WPF works extensively on privacy and governance across multiple jurisdictions, including the US, India, Africa, Asia, the EU, and additional jurisdictions. For more than 20 years WPF has written in-depth, influential studies, including groundbreaking research regarding systemic medical identity theft, India’s Aadhaar identity ecosystem —peer-reviewed work which was cited in the landmark Aadhaar Privacy Opinion of the Indian Supreme Court — and The Scoring of America, an early and influential report on machine learning and consumer scores. WPF co-chairs the UN Statistics Data Governance and Legal Frameworks working group, and is co-chair of the WHO Research, Academia, and Technical Constituency. At OECD, WPF researchers participate in the OECD.AI AI Expert Groups, among other activities. WPF participated as part of the first core group of AI experts that collaborated to write the OECD Recommendation on Artificial Intelligence, now widely viewed as the leading normative principles regarding AI. WPF research on complex data ecosystems governance has been presented at the National Academies of Science and the Royal Academies of Science. World Privacy Forum: https://www.worldprivacyforum.org.1

Risky Analysis: Assessing and Improving AI Governance Tools
An international review of AI Governance Tools and suggestions for pathways forward

World Privacy Forum

www.worldprivacyforum.org

© Copyright 2023 Kate Kaye, Author; Pam Dixon, Author; Robert Gellman, Editor; John Emerson, Designer.

Cover and design by John Emerson

All rights reserved.

EBook/Digital: ISBN: 978-0-9914500-2-2

Publication Date: November 2023

Nothing in this material constitutes legal advice.

This report is available free of charge at: https://www.worldprivacyforum.org/2023/12/risky-analysis-assessing-and-improving-ai-governance-tools

Updates to the report will be made at : https://www.worldprivacyforum.org/2023/12/risky-analysis-assessing-and-improving-ai-governance-tools

Acknowledgements

The World Privacy Forum extends our gratitude to the following people who reviewed and/or were interviewed for this report:

Report Reviewers

(Listed alphabetically)

Ryan Calo, Lane Powell and D. Wayne Gittinger professor, University of Washington School of Law; adjunct professor, UW’s Paul G. Allen School of Computer Science and Engineering; and faculty co-founder, UW Tech Policy Lab

Abhishek Gupta, founder and principal researcher at the Montreal AI Ethics Institute

Cristina Pombo Rivera, principal advisor and fAIr LAC coordinator, Social Sector, Inter-American Development Bank

Jason Tamara Widjaja, director of artificial intelligence and responsible AI lead, Singapore Tech Center, MSD

Report Sources

(Listed alphabetically)

Dr. Rumman Chowdhury, CEO, Humane Intelligence

Maria Paz Hermosilla Cornejo, director of GobLab UAI, in the School of Government at Adolfo Ibáñez University, Chile

Abigail Jacobs, assistant professor of information, School of Information, and assistant professor of complex systems, College of Literature, Science, and the Arts, University of Michigan

Lizzie Kumar, PhD candidate in computer science at the Brown University Center for Tech Responsibility

Tim Miller, professor in artificial intelligence at the School of Electrical Engineering and Computer Science at The University of Queensland

Luca Nannini, PhD student in the Information Technology Research PhD program at CiTIUS of the University of Santiago de Compostela, Spain

Ndapewa Onyothi Wilhelmina Nekoto, research and community lead at the Masakhane Research Foundation

Abigail Oppong, independent researcher and Masakhane Research Foundation contributor

Eike Petersen, postdoctoral researcher, DTU Compute, Technical University of Denmark

Cristina Pombo Rivera, principal advisor and fAIr LAC coordinator, Social Sector, Inter-American Development Bank

Cynthia Rudin, Earl D. McLean, Jr. professor of Computer Science and Electrical and Computer Engineering, Duke University

Ian Rutherford, statistician, United Nations Statistics Division

Jason Tamara Widjaja, director of artificial intelligence and responsible AI lead, Singapore Tech Center, MSD (known as Merck and Co. in the US and Canada).

Jane K. Winn, professor of law, University of Washington School of Law

---

Executive Summary: Why the World Privacy Forum Conducted This Research

The World Privacy Forum conducted the research, writing, and background work necessary to complete this report to address the risks posed by profound changes and advances in the AI ecosystem. These changes and advances impact people, groups of people, and communities, and require evidence-based policy responses as soon as possible. Currently, there is a meaningful lack of evidence regarding how to implement and ensure trustworthy AI; this is true for older AI systems, and it is also true for newer, more advanced AI systems.

This report is intended to begin building much-needed evidence and procedures regarding how to implement trustworthy AI by analyzing AI governance tools and their functions. This is a critically important task because AI governance tools form a pivotal component of AI systems and their lifecycle. This report defines AI governance tools as:

“Socio-technical tools for mapping, measuring, or managing AI systems and their risks in a manner that operationalizes or implements trustworthy AI.”2

This report also documents what these tools do, where they are located and used, their range of maturity, some of the specific risks they pose, the practices currently in place in relation to these tools, and initial steps to take to begin creating improvements.

AI governance tools are important as an area of focus because they sit at the implementation layer of the AI ecosystem and operate across AI system types. AI governance tools, when they function well, can assist the people, businesses, governments, and organizations implementing AI or researching AI to delve into various aspects of how AI models are functioning, and if they are performing in expected or intended ways. For example, some AI governance tools are meant to measure fairness or to “de-bias” AI systems. Some AI governance tools are meant to explain AI system outputs. And some AI governance tools are designed to measure and improve system robustness, among other tasks. However, when AI governance tools do not function well, they can exacerbate existing problems with AI systems.

The timing of this report is noteworthy. In 2007, WPF began work on an extensively researched report on machine learning and its impacts, The Scoring of America. Published in 2014, the report articulated the problems with the deep machine learning of the time and discussed why policymakers needed to address problems with bias, transparency, interpretability, fairness, and other issues. It would have been impossible to know that in just a few years, groundbreaking research introducing a new approach to AI network architecture3 would begin to evolve AI and its capabilities in novel ways.4 In a sense, WPF’s 2014 publication marked the last years of an earlier deep learning AI era. In contrast, this report, Risky Analysis—while building on WPF’s earlier AI work—sits at the start of a burgeoning new era in AI.

As such, Risky Analysis is a different kind of report. It documents the existing evidence regarding AI governance tools with an intent to begin building the larger evidentiary repository needed to create an evaluation environment that supports a transparent and healthy body of AI tools, which will in turn facilitate a healthier AI ecosystem. WPF intends to continue building on this work. For these reasons, WPF is treating this report as a living document, which WPF will update on a regular basis.

Research Conducted for This Report: Building an Evidence Base Regarding AI Governance Tools

This report surveys the international landscape of AI governance tools and provides an early evidentiary foundation that documents multiple aspects of these tools. The report focuses on AI governance tools published by multilateral organizations and by governments. The report utilizes the evidence from the survey of tools in conjunction with in-depth case studies and scholarly literature review to construct a lexicon of AI governance tool types. Based on the evidence gathered, these tool types include: practical guidance, self assessment questionnaires, process frameworks, technical frameworks, technical code, and software.

Figure 1: AI Governance Tool Types Lexicon

Practical Guidance	Includes general educational information, practical guidance, or other consideration factors
Self-assessment Questions	Includes assessment questions or detailed questionnaire
Procedural Framework	Includes process steps or suggested workflow for AI system assessments and/or improvements
Technical Framework	Includes technical methods or detailed technical process guidance or steps
Technical Code or Software	Includes technical methods, including use of specific code or software
Scoring or Classification Output	Includes criteria for determining a classification, or a mechanism for producing a quantifiable score or rating reflecting a particular aspect of an AI system

(Source: World Privacy Forum, Research: Kate Kaye, Pam Dixon. Image: John Emerson). For more information regarding methodology guiding the evaluation of AI governance tools and Finding 2—Some AI governance tools feature off-label, unsuitable, or out-of-context uses of measurement methods—see Appendix C.

• The survey of AI governance tools in this report includes tools from each region. Some examples include:
• An updated process for acquisition of public sector AI from Chile’s public procurement directorate, ChileCompra
• Self-assessment-based scoring systems from the Governments of Canada and Dubai and Kwame Nkrumah University of Science and Technology in Ghana
• Software and a technical testing framework from Singapore’s Infocomm Media Development Authority
• An AI risk management framework from the US National Institute of Standards and Technology (NIST)
• A culturally-sensitive process for reducing risk and protecting data privacy throughout the lifecycle of an algorithm from New Zealand’s Ministry for Social Development
• A vast repository of AI governance tool types from The Organization of Economic Cooperation and Development (OECD), a multilateral institution

The report also includes two detailed case studies in AI fairness and explainability and provides suggestions for how to begin building an evidence basis for an AI governance ecosystem. Going forward, AI governance tools, when fit for purpose, can help provide better health at the implementation layer of AI. AI governance tools are nascent, flexible, and when designed and applied for their intended purpose, can improve the health of the AI ecosystem. However, much more work needs to be done to build the evidentiary basis to create an evaluation environment for this ecosystem.

This report analyzes and discusses the existing governance structures in place today that are intended to protect privacy and govern large data ecosystems by facilitating effective and trustworthy management of data flows. This report discusses how data privacy and governance regulations that were installed as recently as 10 years ago no longer retain the same fit and effectiveness in some AI environments, particularly in environments where advanced versions of AI are in use. The research and analysis conducted for this report indicates that we do not yet know what will be effective replacement or evolutionary structures to protect privacy and govern data in an advanced AI era. It is essential to gather the evidence now for what will work, and to develop an AI governance ecosystem based on this and other evidence.

The report research includes analysis of what could be helpful to create improvements in the AI ecosystem. The research found that there are many unknowns regarding AI governance tools, and what standards, methods, or measurements could best be applied to create transparency and a basis for AI that is trustworthy. The research found hopeful avenues and places to start; these include use of the Plan-Do-Study/Check-Act cycle to improve management of AI governance tools and working to improve documentation of AI governance tools. The report also suggests an adaptation of an early AI governance tools framework from the OECD to provide improved gatekeeper functions for entities that publish collections of AI governance tools.

Going Forward

The research for Risky Analysis indicates that an evaluative environment in which AI governance tools can be tested, matured, and validated will require an evidentiary foundation. The work to create this foundation is just beginning and will require multistakeholder cooperation. The World Privacy Forum is committed to continuing to do the work necessary to gather and analyze the evidence that will facilitate the building of an AI governance ecosystem that is based on evidence, protects privacy and other values, is trustworthy, and provides a genuine foundation for regulatory structures and implementation practices that are fit for purpose today and for the coming AI era.

Background and Introduction

AI governance tools are important because they can map, measure, and manage complex AI governance challenges, particularly at the level of practical implementation. The tools are intended to remove bias from AI systems,5 or increase the explainability of AI systems, among other tasks. Seeking an orderly, automated way of solving complex problems in AI systems can create efficiencies. But those same efficiencies, if not well-understood and appropriately constrained, can themselves exacerbate existing problems in systems and in some cases create new ones. This is the case with AI governance tools, an important and nascent part of AI ecosystems which this report defines as:

AI Governance Tools:

Socio-technical tools for mapping, measuring, or managing AI systems and their risks in a manner that operationalizes or implements trustworthy AI.6 7

An AI governance tool can be used to evaluate, score, audit, classify, or improve an AI system, its decision outputs, or the impacts of those outputs. These tools come in many forms. This report classifies AI governance tools in the following categories: practical guidance, self assessment questionnaires, process frameworks, technical frameworks, technical code, and software.

While AI governance tools offer the promise of improving the understanding of various aspects of AI systems or their implementations, not all AI governance tools accomplish the goals of mapping, measuring, or managing AI systems and their risks, which we argue are essential features of an effective AI governance tool. Further,

given the lack of systematic guidance, procedures, or oversight for their context, use, and interpretation, AI governance tools can be utilized improperly or out of context, creating the potential for errors ranging from small to significant.

For example, AI governance tools can be used in novel or “off-label”8 ways, which can lead to meaningful errors in contextualization and interpretation. Some of the more complex AI governance tools can create additional risk by producing a rating or score that in and of itself can be subject to error or misinterpretation, especially if there is a lack of documentation and guidance for use of the tool. All told, flawed usage and interpretation can result in a gap between what people want these tools to accomplish, and what these tools actually do accomplish.

Lessons learned from earlier AI policy implementations

AI governance tools can exacerbate risk when there are gaps in controls and standards for the tools, their context of use, and other items, such as their outputs or results. The many types of AI governance tools can range from simple questionnaires all the way to software. Therefore, the manner in which some AI governance tools are used to map, measure, and manage the risks of AI systems can differ substantially. The quality of an AI governance tool and any quantifications it produces matters greatly. For instance, an AI output in the form of a score can seem deceptively simple to interpret. However, AI score outputs are historically notorious for requiring great care and proper contextualization to accurately interpret and apply.9 This is an important area to understand because challenges with today’s AI governance tool outputs, which are in some cases scores, reflect a long history of broader AI scoring approaches (of which there are many) that for decades have been studied, understood, and in some cases, regulated.10

Some AI governance tools, such as those intended to remove bias from an AI system, may provide a score or rating to indicate the prominence or presence of certain biases. Not all AI governance tools produce quantified measures or scores. However, in such cases that they do produce scores, it will be necessary to properly interpret the score, and to validate the score for the specific context in which it is used with objective criteria. Accomplishing these kinds of tasks consistently across the full body of AI governance tools and tool types requires a range of policy guidance from informal technical and policy guidance to formal legal guidance or regulation, depending on the tool being used, its output, and the context and purpose of its use.

Using examples from the classical AI machine learning context, some countries regulate AI systems that specifically impact decisions related to eligibility, including AI systems that automate decision-making associated with credit reporting. AI systems trained to analyze credit eligibility often produce a credit score as an output. As mentioned, some of these systems currently have regulations in place. This is true across multiple jurisdictions.11 Most credit scoring regulations are intended to provide transparency regarding automated decisions, error correction, and redress, among other features of credit scoring systems. In regulated scoring models, scores are evaluated for fit, accuracy, and other factors based on the evidence. Evaluative techniques and processes guide proper implementation of the scoring systems.12 There is enough history and established policy around credit scoring systems and their risks, that credit scores, their use, and their interpretation is well-understood. For example, errors or problems introduced by flaws in either the data, the analysis, or even the implementation or interpretation of the scoring can create meaningful impacts for people, groups of people, and communities.13

Credit scoring systems and regulations provide many lessons. However, there is an additional policy lesson here beyond just the importance of regulating credit scoring: many thousands of other AI scoring systems exist, and most of these are not formally regulated.14 In fact, regulated scores are rare. The Scoring of America analyzed many kinds of AI scores beyond credit scoring—from patient frailty scores to consumer prominence scores indicating purchasing power to identity scores used to quantify the validity of an identity.15

The 2014 Scoring report found that unregulated AI scores of that time operated a lot like analytical plumbing, humming along in the background of many business processes. These scores are plentiful and largely unseen, but nevertheless could have an impact on bias, fairness, privacy, transparency, and other issues. The lack of broader policy action regarding the various AI scoring systems that were not in scope of credit scoring regulation resulted in widespread and opaque use of a variety of scores spanning a range of risk levels.

Today, AI governance tools are in an intriguingly similar position in that they, too, are increasingly common and are poised to become a critical part of the evolution of the “AI analytical plumbing,” despite the fact that they are often not subject to evidence-based assessments or regulation. Even though AI governance tools are nascent and largely unregulated, they are already in widespread use across the world, and across sectors. For example, some AI governance tools are already in use in eligibility contexts, such as to measure AI systems used in relation to employment. While legal scholars who research these fields know about some of these tools, their uses, and their risks, the knowledge is not yet widespread.16

Additionally, because AI governance tools are often made available with minimal documentation and have little to no regulation, these tools exist in various stages of quality assurance. Many new AI governance tools are in development today. But their risks, and the policies that will address these risks consistently are largely not yet well-developed, or in some cases, not present at all. Given the beneficial potential of AI governance tools,17 it is worth meaningful efforts to understand more about them, how they operate in today’s AI environments, and how the AI governance tools environment can be improved while incorporating existing legal and policy guardrails from other regulatory regimes.

Addressing policy disruptions stemming from old and new forms of AI: incorporating lessons learned from the data governance and privacy domain

Human rights, privacy, and data governance laws and policies are currently in various stages of change as a result of AI. Some older AI systems that emerged in past decades have had regulatory oversight in place for many years, for example, credit scoring models were regulated beginning as early as the 1970s. However, the emergence of advanced AI models 18 is creating novel disruptions, and questions abound about what new regulations for newer models should look like. Even though existing approaches are not necessarily responsive to the changes in AI or are being bypassed, there is still much to be learned from the past history of data governance and privacy.

We begin with a discussion of terminology. Data governance and privacy are related, but they are not interchangeable. Data governance is a comprehensive approach to the entirety of data of an organization or entity that ensures the information is managed through the full data lifecycle. This can include data collection practices, data security, quality, documentation, classification, lineage, cataloging, auditing, sharing, and other aspects. Data privacy is a subset of data governance and is best defined in context as forms of protecting either personal data, or the personal data of a group of people. Privacy is often seen in terms of individual data rights, such as the right to deletion, and so forth. While the conception of privacy as an individual right is currently ascendant in terms of legislation today,19 conceptions of privacy as a group or community-based privacy right are emerging as well, and can be found, for example, in Māori approaches to privacy.20

In response to changes in technology, new opportunities, and other developments, data governance policy, laws, and institutions were introduced, developed, and adjusted mightily over the decades. These evolutions were driven by an urgent need to respond to then-radical changes in technology and governance policy. The impetus was to provide responses to a variety of emerging threats and opportunities related first to the emergence of the computer, and later, to the emergence of the Internet and subsequent factors, such as the emergence of social media platforms.

In the late 1960s, attention to data governance, data protection, and privacy began slowly, with small developments here and there around the world. Responding to the growth of personal computing, countries enacted different privacy laws beginning in the 1970s and 1980s.21 It did not take long before the differences and limits in these national laws created problems with international data flows. Europe began to address these problems, and the EU, after some significant effort, adopted a Data Protection Directive 22 in the 1990s. The shortcomings of the Directive and the challenges with its implementation resulted in its replacement by the EU General Data Protection Regulation23 which has been enforced since 2018. Many other countries around the world now follow the EU privacy model. There is no question GDPR forms a near-worldwide regulatory structure.24

See for example, Figures 1 and 2, which visualize the distribution of data governance and data protection laws across the world. Notice the distinct patterns of distribution of data protection laws through global regions over time.

Figure 2: Table of Global Privacy Laws

(Source: World Privacy Forum. Research: Pam Dixon, Kate Kaye. Data Visualization: John Emerson.)

Figure 3: Table of Global Privacy Laws, Regional Breakout

(Source: World Privacy Forum. Research: Pam Dixon, Kate Kaye. Data Visualization: John Emerson.)

For decades data policy and technical developments came about more or less at approximately the same time, albeit with some delay, and at a much slower pace. For example, at about the 20-year mark of early credit model development, credit score models were regulated.25 These first major credit system regulations developed in the 1970s, at the beginning of a series of worldwide data governance and privacy developments that then unfolded in incremental steps over decades.26

Fair Information Practices (FIPs),27 the core statement of data governance and privacy values started in 1973 in the United States, was restated by the Organization of Economic Cooperation and Development (OECD) in 1980,28 and became the basis for many privacy laws and policies around the world. Eventually, FIPs faded into the background, not because the policies were wrong, but because the general policies that served so well for so long were not specific enough to address ongoing developments in technology, industry, and government. To offer one example, FIPs did not call for privacy agencies, but countries quickly recognized the value of privacy agencies or data protection authorities, and the idea spread around the world. Data protection authorities function as enforcers of data protection and governance laws, and help guide the implementation data governance ecosystems at the ground level effectively.29

As privacy laws and institutions matured, it became clear over time that solutions which had seemed responsive in theory did not always work well in practice, or, sometimes, ideas that worked in one jurisdiction or social context did not fit in others. For example, GDPR and GDPR-like legislation, which grew from the 1995 EU Data Protection Directive, both of which focused on individual privacy rights, does not always fit well in some contexts, including Indigenous contexts, where privacy and data are often handled as community rights.30 31 Additional ideas from jurisdictions and stakeholders came along. There was a lot of experimentation, which created an evidentiary basis over time. The data governance and privacy learning curve stretched over decades, and the various stakeholders in the data ecosystems are still learning. Similarly, AI regulations have been moving along for the most part in incremental steps.32

In contrast to the long evolution of data governance and privacy laws and norms, advanced forms of AI, though in development since 2017, jumped to public awareness seemingly overnight in 2022.33 The presence of newer and more advanced AI models brought quick regulatory reactions and proposals that in some cases derogated from decades of established knowledge and lessons. Quick response is on the whole hopeful. However, responses will need to ensure there is input from all stakeholders, and ensure that existing legal and other guardrails, including existing human rights and privacy guardrails, are integrated. One lesson from data governance and privacy is history is that it takes time to understand what works and what does not.

It is worth recalling that various forms of machine learning have been used and regulated for many decades. As discussed, credit score regulations—which address data inputs, algorithms, set points, and other aspects of machine learning—have existed in some jurisdictions since the 1970s. These early forms of machine learning regulations often include well-understood governance mechanisms that are common today, such as error correction, a formal dispute process, extensive government oversight, and other forms of consumer redress. The procedural, and administrative controls used in these types of regulations were new at one time, but are now international standards. These standards, norms, and older governance models enshrined into law need to be taken into full account by those seeking to address the risks of emerging advanced AI systems.

AI governance tools hold out great promise for mapping, measuring, and managing new AI risks. Work to address how AI governance tools can be managed competently with appropriate and helpful guardrails is important, and will entail building the necessary evidence and measurement environments to facilitate this work. Without an evidentiary basis for policy, we are all just making guesses, which is not sufficient to address the actual risks the developing AI ecosystem may have.

The importance of acknowledging what we do not yet know

The more advanced forms of AI that are in place introduce novel problems and new architectures, and these may require a range of new governance approaches. Necessary responses include changes in the way privacy, human rights, and data governance are operationalized and implemented in AI and other ecosystems.

This is new territory, and sufficient evidence regarding valid and fit-for-purpose governance of these systems does not exist yet in a world filled with AI activities. We simply do not how privacy and data governance models must adjust to remain effective. There is a continuing need to construct evidence-based models of privacy and data governance and to use these as a basis to respond to the new challenges and opportunities introduced by advanced AI models.

At the same time, governments, academics, companies, and others have jumped into what appears to be a high-speed race to regulate AI. The reaction appears somewhat instantaneous, especially in comparison to the long ramp that data privacy and data governance policy experienced. It appears that the socio- technical and policy responses to AI will not have the luxury of developing slowly over decades, or perhaps even years. Regulations of varying quality, validation level, and enforceability will be coming soon. This contrasts to the environment of a decade ago when AI governance was not seen as needing immediate action.

As discussed, many of the concerns at the heart of data governance and protection policymaking – fairness, due process, discrimination, openness, rights and responsibilities of data controllers and data subjects, and limits on data use and disclosure – apply to AI activities that include processing of information about individuals, and increasingly, about groups of people and communities. Evidence built up over time shows what works and does not for these systems. However, some of the older governance models are too narrow to address the full range of issues today. Yet some of the newer AI governance models proposed today lack the deep policy knowledge and experience from recent decades.

The world is beginning to acknowledge the substantial impacts of modern AI on privacy, fairness, bias, transparency, and other values. However, there is not yet enough evidence to enable an analysis of what those full range of impacts might be, or how AI activities will develop.

It is critically important to engage in humility regarding what we do and do not know about advanced AI models.34 We must ensure that evidence gathering and better understanding precede a rush to solutions — regulation must relate to the reality on the ground. We can accomplish a better result through testing and validation of the technical and policy systems at hand. Otherwise, regulation may well be unfit for purpose and fail to accomplish the goals which are vitally important to attain.

Creating a healthier AI ecosystem and fit-for-purpose guardrails: Toward building an evidence-based AI governance tools environment to operationalize and implement trustworthy AI goals

AI governance tools hold great promise to create improvements at the implementation level. They can function as implementation interfaces for AI systems and ecosystems, and they are already in widespread distribution internationally. This report provides an international survey of the tools as they exist today, a detailed analysis of these tools and their benefits and risks, as well as how they operate in various contexts.The report suggests several concrete pathways for improving outcomes and ensuring that AI governance tools, which are intended to improve AI systems, do just that.

While this report is specifically focused on AI governance tools as an important body of tools to create improvements in AI systems, many additional pathways to improvement regarding more broadly defined AI systems exist, and these possible pathways can be experimented with. Privacy Impact Assessments (PIAs), now commonplace across the world, could be helpful. PIAs were developed in the mid-1990’s and reached maturity around 2005-2009. PIA development is still undergoing ongoing cycles of improvement.35 However, PIAs alone will not be enough to address the full range of challenges that AI presents. Additional tools will be needed.

For example, the development of ethical or trustworthy AI Impact Assessments is already underway.36 37A high-quality and verifiable AI impact assessment that evaluates impacts and validity is essential for those relying on machine learning models and AI system outputs.38 The need for assessment is especially urgent for those models and systems that support decisions about patient health; matters pertaining to employment and other eligibility-related or eligibility-adjacent decisions; law enforcement and criminal justice decisions; and other activities directly affecting the lives of individuals, groups of individuals, and communities.

Recognize, however, that assessments for privacy, human rights, certain aspects of governance, and other assessments and validation in relation to today’s AI activities are far from mature. And even the most perfect assessment tools will not be enough to address the full range of challenges through the AI lifecycle. More will be needed to address all of the aspects of the lifecycle, including implementation through AI governance tools, and this work has barely begun in regards to addressing advanced forms of AI.

Methodology

This report surveys AI governance tools with to assess how widespread they are, where they are located, and how they might—or might not—improve AI systems. This research examines two use cases in depth plus additional smaller exemplars.

To conduct the preliminary work for this report, the first part of the research sought to determine: 1) how widespread AI governance tools are; 2) where they are published geographically or virtually; 3) what entities created them; 4) the goals of the tools; and 5) the extent to which the tools met their stated objectives. To assess these questions, we sampled a limited set of AI governance tools across jurisdictions and organizations.

This research also analyzed scholarly literature that assesses the quality, functionality and applicability of AI governance tools. The analysis encompasses literature published between 2017 and 2023 that analyzes AI fairness and explainability tools. The research for this report concluded Oct. 31, 2023. Only a few minor updates were added after that time.

An extensive comparative study and analysis of similar fields to determine existing norms and standards for documentation, quality assessment, testing, ongoing monitoring, and other aspects of assessment and post-market improvement cycles was conducted, as well as multiple interviews with experts in the AI field.

This report adopts the term AI Governance Tools and introduces a lexicon of AI Governance Tool Types. The research conducted for Part II of this report made it clear that there was a need to clarify and distinguish among the items commonly labeled generically as “AI Tool” or “AI Toolkit” in the international AI policy and governance sphere.

For more information regarding methodology guiding the evaluation of AI governance tools and Finding 2: Some AI governance tools feature off-label, unsuitable, or out-of-context uses of measurement methods, see Appendix C.

Findings

1. AI governance tools are widely published and offered by governments, multilateral institutions, and other organizations.

AI governance tools exist across Africa, Asia, Europe, North America, South America, and Oceania (Australia and New Zealand), at varying levels of maturity and dispersion. Governments, multilateral organizations, academia, civil society, business, and others utilize these tools in different types of AI implementations.39 This research focuses on AI governance tools used, promoted, or cataloged primarily by governments and multilateral institutions, especially those tools that seek to implement principles of trustworthy AI.40 It remains difficult to quantify precisely how many tools exist.

2. Some AI governance tools feature off-label, unsuitable, or out-of-context uses of measurement methods.

More than 38% of AI governance tools reviewed in this report either mention, recommend, or incorporate at least one of three measures shown in scholarly literature to be problematic. These include off-label, unsuitable, or out-of-context applications when used to measure AI systems.41

3. AI Governance Tool providers and hosts have important gatekeeper and quality assurance roles.

Some collections of AI governance tools are published online. This research focused on tools and tool catalogs published by governments and multilateral organizations. These tools and tool catalogs vary significantly in size and types of offerings. Some are comprised of a listing of an AI governance tool with little to no additional information. Some tool collections go further and provide certain levels of assessment or at least a detailed description of the tools.

The OECD, for example, publishes a catalog of AI governance tools that is among the largest offered to date.42 The OECD framework for its tool catalog may become a helpful model going forward. For example, at least 12 items featured in the OECD’s Catalogue of AI Tools and Metrics either mention, recommend, or incorporate off-label measures discussed in Part I of this report, which features use cases of problematic AI fairness and explainability measures. Tool catalog hosts and publishers have important roles as gatekeepers with responsibilities to ensure tool quality and transparency.

[Hed 2} Secondary Findings

1. Standards and guidance for quality assessment and assurance of AI governance tools do not appear to be consistent across the AI ecosystem.

This research did not seek to determine as a primary goal whether quality assessments are in place for each AI governance tool or tool catalog. However, it became apparent during the research process that while some AI governance tool providers have conducted some quality assessments of those tools, some have not; if they do conduct quality assessments, AI governance tool providers do not always conduct them according to an internationally recognized standard.

Complete product labeling, documentation, provision for user feedback, requirements for testing, or provision of redress in the case of problems are important features of traditional products, but these features are not always present in AI governance tools.

[Hed 2 } Pathways for Improvements: Summary

The following is a high-level summary of the solutions and steps that will begin to address the problems and opportunities the research for this report identified. At the end of Part I of this report, a section titled Pathways for Building an Evaluation Environment and Creating Improvements discusses in detail potential pathways and solutions toward improving the AI governance tools environment.

Establishing an Evaluation Environment for AI Governance Tools

There is not enough data yet about how AI governance tools interface with specific standards. As a result, foundational work needs to be done to build an evaluative AI governance tools environment that facilitates validation, transparency, and other measurements. Establishing an evaluation environment for AI governance tools will be crucial to create a healthy AI governance tools ecosystem, and more broadly, a healthier AI ecosystem.

In considering what might help build a transparent, evaluative environment for AI governance tools, the application of international and other standards holds potential. For example, the extensive quality assurance ecosystem articulated in formal standards and norms is well-understood across many mature sectors.

Although many established standards already exist and are important to acknowledge, currently, there is limited knowledge about the functionality of these standards as applied to AI governance tools. Testing of available tools would improve understanding of how existing standards might apply, and it would also support the ecosystem based on evidence. The Plan-Do-Check (or Study)-Act cycle will be a key tool to assist in this maturation.

Establishing Baseline Requirements for Documentation and Labeling of AI Governance Tools:

The research found high variability in the documentation and labeling of AI governance tools. This suggests that developing norms regarding documentation and labeling of AI governance tools could produce meaningful levels of improvements. For example, it would be helpful if tools routinely include information about the developer, date of release, results of any validation or quality assurance testing, and instructions on the contexts in which the methods should or should not be used. A privacy and data policy is also important and should be included in the documentation of AI governance tools.

• Additional items can be provided in the documentation, for example:
• Appropriate performance metrics for validity and reliability
• Documentation should provide the suggested context for the use of an AI governance tool. AI systems are about context, which is important when it comes to applicable uses, environment, and user interactions. A concern is that tools originally designed for application in one use case or context may potentially be used in an inappropriate context or use case or “off-label” manner due to lack of guidance for the end user.
• Documentation should give end users an idea of how simple or complex it would be to utilize a given AI governance tool.
• Cost analysis for utilizing the method: How much would it cost to use the tool and validate the results?
• A data policy: A detailed data policy should be posted in conjunction with each AI governance tool. For example, if applicable, this information could include the kind of data used to create the tool, if data is collected or used in the operation of the tool, and if that information is used for further AI model training, analysis, or other purposes.
• Complaint and feedback mechanism: AI governance tools should provide a mechanism to collect feedback from users.
• Cycle of continuous improvement: Developers of AI governance tools should maintain and update the tools at a reasonable pace.
• Conflict of interest: The identities of those who financed, resourced, provided, and published AI governance tools should be made public in a prominent manner in conjunction with publication or distribution of the tool.

The crucial role of NIST and the OECD in convening stakeholders and developing an evaluative environment and multistakeholder consensus procedures for high-quality AI governance tools and catalogs

The National Institute of Standards and Technology (NIST) could play an additional role in AI by building an environment in which an evidentiary basis for the socio-technical contexts and best practices for AI governance tools could be created. WPF urges NIST to undertake this work, including developing recommendations for a process for developing, evaluating, and using AI governance tools.

The OECD could play an additional role in AI by creating a definitive best-practice framework for AI governance tool or tool catalog publishers by further developing and refining its existing work in this area. This report includes initial suggestions for this work in the Pathways for Building an Evaluation Environment and Creating Improvements discussion at the end of Part I., This builds on OECD’s existing work on a framework for AI governance tools. WPF urges the OECD to gather international stakeholders to further this work.

Measurement Modeling: A structured Approach Aligning AI Governance Tools and Policy Goals

Measurement modeling could play a positive role in improving outcomes and the quality of AI governance tools. A shorthand for understanding measurement modeling is that it is a structured method that can be used to illuminate gaps between the actual results of measurement systems and policy goals.43

When embedded in policy rules and guidance, specific methods or metrics for building more fair, accountable and transparent AI systems and gauging AI risks can have a lasting impact on the ways society comprehends AI systems and their effects on people’s lives. What and how we measure something44 not only reflects our understanding of it, but imposes frameworks or structures for our future understanding.

Measurement modeling is one approach that can assist in this process.45 For example, measurement modeling has been proposed as a method for recognizing gaps in relation to fairness gaps in computational systems. 46 47

Measurement modeling essentially asks evaluators to distinguish between what or how a metric or tool measures, and the goals of the measurement. In other words, is there proper alignment between a metric or tool and the goals of policy? Does the metric or tool actually measure for the same things the policy aims to achieve? The method might be applied when vetting or validating AI governance tools or metrics used to gauge AI fairness, for example.

Some researchers have devised an audit framework for assessing the validity and stability of specific measures such as personality testing metrics used in automated hiring systems. For instance, a socio-technical algorithmic auditing framework found that two real-world personality prediction systems showed “substantial instability with respect to key facets of measurement, and hence cannot be considered valid testing instruments.”48

This and other frameworks for assessing measurement methods could be helpful to policymakers as they inspect AI governance tools.

Going forward, ensuring alignment of AI governance tools with policy goals for trustworthy AI will be of the utmost importance. For this reason, it will be helpful to assess underlying assumptions about what the measurement mechanisms or methods used in AI governance tools actually do. This very issue is at the heart of the next section of the report, which covers detailed use cases of AI governance tools in their implementation contexts.

Part I. Discussion: Critical Analysis of AI Governance Tools

Governments from Australia to Singapore, Ghana to India, and Europe to the US, have begun to put AI principles into practice by presenting methods for measuring and improving the impacts of AI systems through a variety of AI governance tools. The overwhelming majority of AI governance tools reviewed in this report emphasize two particular governance goals among many49: 1.) fairness, or avoidance of bias and discrimination in AI-based decisions and outputs, and 2.) explainability, or interpretability of those systems.50

The impulse to operationalize AI principles by measuring and improving AI impacts is a positive one, which will ideally guide users, developers, and other actors on a path toward more beneficial and trustworthy AI systems. Measuring the world around us is one way humans make sense of it. It’s only natural that people want to quantify fairness, explainability, and other aspects of AI.

The measures established today could have lasting effects on how the impacts of AI are reflected and interpreted for years to come. Today’s measurements will form the foundation of risk scores, consumer scores,51 ratings, and other statistics we rely on to help make sense of these systems and enforce the rules and regulations addressing them. No one wants to standardize ill-suited methods or embed them in policy in ways that could introduce new problems or harms. That’s why it is so important to make sure measurement approaches align with policy goals.

Part II of this report reviews and analyzes a wide-ranging group of more than 30 AI governance tools and adjacent guidance distributed in 13 national jurisdictions across a number of regions. This section’s focus is intentionally narrowed to encompass literature that analyzes AI fairness and explainability tools. There is a wealth of relevant literature from scholars in technical and socio-technical fields published between 2017 and 2023. This rich and growing body of work investigates a variety of approaches to measuring and improving AI fairness and explainability. Put simply, it seeks to “measure the measures.”

The literature cited and reviewed in this section paints a vivid portrait of what could go wrong if AI governance tools are applied without rigorous evaluation or in inappropriate contexts. This body of literature questions some commonly-used approaches for assessing or improving AI fairness or explainability.52 It shows that AI measurement methods can lead to AI system accuracy failures, unintentional harms to individuals or groups, or manipulation of metrics to produce tainted measurement outcomes.

Scholars interviewed for this report generally agree that the mission to guide AI actors toward developing and operating more trustworthy AI systems through AI governance tools is beneficial. However, their work may not always be known to policymakers or others influencing those tools. As noted in this report’s findings, some AI governance tools mention, recommend, or incorporate off-label uses of potentially faulty or ill-suited tools that are scrutinized in the growing body of scholarly literature.53 Our intention is to point out salient areas in which the scholarly research we’ve reviewed might inform future AI governance and AI governance tools.

Measuring AI Fairness Measures

A considerable body of research has emerged in recent years highlighting the potential problems when AI actors use automated AI governance tools that promise to create systems that are more fair,54 but do so without properly assessing those methods or their applicability to a chosen purpose. For example, particularly in the past few years, scholars from around the world have raised alarms about application of metrics that do not align with specific AI fairness-related tasks, such as measuring bias in a dataset used to train an AI model or assessing the risk of unfair decisions made by an AI system.

Recent research intended to elucidate basic requirements of appropriate fairness metrics suggests that “the choice of the most appropriate metrics to consider will always be application-dependent.” This scholarly literature finds that assessment of a risk model’s fairness in itself is crucial because such models are used to inform human decision-makers.55

Governments are just beginning to recognize the need to assess methods intended to improve AI fairness. For example, as discussed in Part II of this report, Chile’s 2022 bidding and quality assurance requirements for government acquisition of AI systems stress the importance not only of evaluating the system’s impacts on equity, but of evaluating the equity metrics themselves.56

Detailed guidance for implementing those requirements states that the type of metric employed is important; it calls on the public sector entity making the purchase to determine appropriate metrics, rather than the technology vendor. In the end, the goal is for both parties to collaborate on determining the most appropriate metrics.57

Part II of this report illustrates that governments and other organizations want to put AI principles into practice, and many also want to find ways to produce quantifiable AI risk and analysis measures in the form of fairness scores or ratings. Yet, some of the literature referenced here reminds us that there are pitfalls inherent in quantifying fairness through one-size-fits-all assessments, encoded technical tools, or other quick technical fixes.

This section contains two use cases. The first spotlights an inappropriate use of metrics to automatically alleviate bias from disparate impacts of AI systems. The second highlights the use of SHAP and LIME, two related approaches intended to explain how AI systems produce particular outputs or decisions, both of which have attracted scrutiny among computer science researchers.

Use Cases in AI Fairness

The Risks of Using the US Four-Fifths Employment Rule for AI Fairness Without Appropriate Context

In the laudable mission to ensure that AI systems do not produce negative impacts on specific groups of people, an array of tools and metrics intended to remove disparate impacts from AI datasets and systems has emerged. Some of these tools use as their foundation encoded translations58 of a complex US rule: the “Four-Fifths rule.”59

The Four-Fifths Rule is well-known in the US labor recruitment field as a measure of adverse impact and fairness in hiring selection practices. Detailed in the Equal Employment Opportunity Commission Uniform Guidelines on Employee Selection Procedures of 1978,60 the rule is based on the concept that a selection rate for any race, sex or ethnic group that is less than four-fifths—or 80%—of the rate reflecting the group with the highest selection rate is evidence of adverse impact on the groups with lower selection rates. The rule has been widely applied by employers,61 lawyers,62 and social scientists63 to determine if hiring practices are lawful and if they result in disparate or adverse impacts against certain groups of people.

Employers with more than 100 employees are required to maintain information regarding disparate impact in hiring selection rates, according to the Uniform Guidelines.64 While the guidelines state that the Four-Fifths rule is “generally” regarded by federal enforcement agencies as evidence of adverse impact, it explains that in some cases, smaller differences in selection rate may constitute adverse impact, and in others, greater differences in selection rate may not constitute adverse impact. In other words, context matters.65

Despite its widespread use, legal, employment, and technical experts have cautioned against use of the Four-Fifths Rule as a singular means of assessing disparate impact.66 Many experts warn against simplistic applications of the rule, both within its historical use in US labor contexts as well as for its use in AI contexts.67

In June 2023, the chair of the U.S. Equal Employment Opportunity Commission cautioned against relying solely on meeting the 80% threshold. Calling the Four-Fifths rule “a check” and just one single standard used at the start of federal investigations, rather than the only measure used for gauging disparate impact, she said that “smaller differences in selection rates may constitute disparate impact.”68

Further, according to a U.S. Justice Department legal manual addressing disparate impact, “not every type of disparity lends itself to the use of the Four-Fifths rule, even with respect to employment decisions.”69 Legal scholars also have questioned the limits of the Four-Fifths rule, noting its failure to statistically reflect hiring disparity impact adequately.70

Despite those caveats, the Four-Fifths Rule and its 80% benchmark have been repurposed in computer code form and used in a variety of AI fairness metrics and tools.71 The rule is applied in both employment72 and non-employment contexts 73 as a means of measuring or “removing” bias or disparate impacts.74 It is also used outside of the US employment context and is encoded into AI governance tools offered in other jurisdictions. 75

In a 2019 study of 18 vendors offering algorithmic pre-employment assessments, researchers found that three vendors “explicitly mentioned the 4/5 rule” and several “claimed to test models for bias, ‘fixing’ it when it appeared.”76 A more recent review indicates this is still happening. As of August 2023, some companies providing AI software publicly mentioned the four-fifths rule as a basis for addressing disparate impact in their systems. 77

It is not known at this time how many of the entities and individuals using metrics or tools that incorporate the rule’s 80% benchmark are aware of the full background, context, and underlying rationale of the four-fifths rule as encoded in those tools. It is also unknown how many of those using the tools outside of a US employment context would continue using them if they were aware of the potential problems.

Scholarly researchers also argue that application of the rule in algorithmic recruitment systems is “coarse as it is agnostic to quality of candidates” and does not “account for uncertainties and biases in the data systematically.”78 Scholars also find that codifying the Four-Fifths Rule into AI fairness software should not be used in contexts outside hiring in the US or US labor law and compliance.79 Scholars also assert that tools intended to produce non-discriminatory AI systems that incorporate the Four-Fifths Rule may miss other important factors weighed in traditional assessments, such as which subsections of applicant groups should be measured using the rule.80

Meanwhile, some civil rights and employment lawyers argue that use of the Four-Fifths Rule rule as a test for disparate impact is unreliable in some cases81 and, particularly in relation to AI used in labor recruitment, “is not only unsupported by the case law, but it is also bad policy.” 82

Use Cases in Automating Fairness: A Compendium of Potential Risks

The movement toward establishing practices that create fairer AI outcomes is positive. However, scholarly literature reviewed for this report indicates an array of unintended consequences of applying metrics or other technical approaches to measure or improve AI fairness.

For example, attempting to de-bias AI systems by abstracting, simplifying and de-contextualizing complex concepts such as disparate impact is just one problematic approach emerging within the AI governance tool environment among many.

It is worth noting there are significant distinctions among definitions of fairness, which may complicate the efficacy of technical approaches designed according to one perception of fairness when used in other contexts.83 Some key concerns and potential problems:

“Fairness gerrymandering”:

“Fairness gerrymandering” in AI fairness tools is a term of art utilized in the scholarly literature to represent when algorithms that take fairness into account have the paradoxical effect of making their outcomes particularly unfair to one subgroup.84 Technically speaking, this occurs when “a classifier appears to be fair on each individual group, but badly violates the fairness constraint on one or more structured subgroups defined over the protected attributes (such as certain combinations of protected attribute values).”85

Fairness gerrymandering might occur if a method for achieving algorithmic fairness is applied in the context of only a small number of pre-defined groups. For example: when, in relation to two binary features corresponding to race and gender, a classifier is considered equitable if it corresponds to one combination of those binary features (such as if it corresponds to a “Black man,” or a “white woman”), but not another combination, such as “Black woman.”86 The risk is that an analytical process may result in unfairness in relation to other groups not explicitly considered. In other words, a process that creates more equitable outcomes for some groups might produce undesirable side effects for other groups.87

Abstraction traps, oversimplification, and lack of critical context:

Because “abstractions are essential to computer science, and in particular machine learning,”88 they are inherent in technical interventions that can create “abstraction traps” when used in societal contexts.89 The aforementioned abstraction of the four-fifths rule is just one example of an abstraction trap. Another such trap might result if an algorithm designed to solve a problem in one social setting, such as predicting risk of recidivism, is also used in relation to loan default. Such abstractions may “render technical interventions ineffective, inaccurate, and sometimes dangerously misguided when they enter the societal context that surrounds decision-making systems.”90

Also, in an effort to codify governance goals such as fairness and explainability, AI governance tools may lack critical context. Removing or reducing the proper context for an AI governance tool “may flatten nuance and suggest that the tools to solve complex problems lie within the confines of the kit,” or can “[abstract] away” crucial elements of the social context in which AI systems are deployed.91

For example, the NIST AI Risk Management Framework, an AI governance tool reviewed in Part II of this report, recognizes that metrics used to measure AI risk “can be oversimplified, gamed, lack critical nuance, become relied upon in unexpected ways, or fail to account for differences in affected groups and contexts.”92

Scholarly literature also addresses problems that result from application of formal mathematical models of “fair” decision-making used in policy, analyzing the potential for decision-making using algorithms to “violate at least one normatively desirable fairness principle.”93

Fairness and risk scoring model tradeoffs:

Maximizing fairness across different legally protected groups of people and also achieving maximal accuracy is a topic of intense scrutiny in the literature—because this is nearly impossible to accomplish. Research evaluating methods and metrics used to score or predict AI risk levels indicates the distribution of true risks may differ among groups, and in particular, may not be proportional to one another.

This could lead to unfair allocation or access to resources, among other potentially negative outcomes.94 This issue can apply when risk assessments are used by decision-makers across sectors. Examples include mortgage lending, employment, college admissions, child welfare, and medical diagnoses.While some risk scoring models are regulated, many others are not.

Limited applicability throughout AI life cycle:

Some AI governance tools or fairness AI auditing software may only apply during limited phases of the AI life cycle. For example, some AI fairness tools may only apply to the model training stage of AI development. While this is important, determining fairness at one life cycle stage does not mean that fairness is imbued thereafter through the AI life cycle.

For example, if models are adjusted post-deployment the fairness of their outputs can be affected negatively. In addition, some fairness tools do not support early stages of the ML development lifecycle, such as problem formulation stages.95 Also, there is a risk of AI fairness tools being applied to inappropriate use cases, misinterpreted, or misused.96

Limited applicability to third-party AI systems:

Third-party AI systems, including third-party software, hardware, and data components, among others, “may complicate risk measurement.”97 The inner workings of third-party AI systems are not always transparent to users. In addition, certain AI governance tools and metrics may not be applicable when attempting to assess third-party AI software or systems built using unstructured data, as opposed to structured data.98 99 The OECD notes the importance of understanding where, when, and what parts of an AI system are built in-house or by a third party, noting three configurations for how this might be operationalized.100

Regional contextual constraints: Off-the-shelf AI auditing software products or open-source governance tools may have been designed for use in specific countries that have high AI capacity.101 Because of this, these products do not always address concerns in all Asian, African, Caribbean, or Latin American jurisdictions, among others.

For instance, an AI governance tool may not recognize nuances among the large variety of Asian and African sub-populations, demographics, and languages. Masakhane Research Foundation, a grassroots organization based in Kilifi, Kenya, that is mentioned in Section II of this report,102 generates, curates, and annotates datasets that are inclusive of the languages people speak throughout the African continent.103

Singapore’s Generative AI Evaluation Catalogue104 also states that LLM evaluation techniques tend to be Western-centric, and should consider user demographics and cultural sensitivities. In addition, India’s Tamil Nadu State Policy for Safe and Ethical AI105 points to the importance of cultural relevance for AI governance tools. This work is deeply embedded within the regional AI and cultural contexts.

Lack of definitional consistency:

There are significant distinctions among definitions of fairness, which may complicate the efficacy of technical approaches designed according to one perception of fairness when used in other contexts.106

The difficulty of assessing fairness and privacy in AI systems:

Assessing AI model fairness may be in conflict with data minimization and data protection goals, as well as existing regulations in some circumstances. Measurement for disparate impacts against particular groups requires knowledge of sensitive attributes such as race or age, the very types of data attributes that some data governance regulations may restrict. AI fairness researchers have documented this phenomenon107 as well as introduced methods for measuring fairness while protecting sensitive data.108

It is important to note that this paradox is not present in all data ecosystems. For example, National Statistical Organizations (NSOs) operate under a derogation in most countries of the world, even where data governance legislation is present. There is a unique set of rules providing ethical guardrails for NSOs in precisely these kinds of analytical circumstances.109 In addition, there are many other exemptions for conducting analysis using sensitive data; for example, public health data during a national public health emergency is often treated more leniently because data protection rules may be suspended in certain emergency situations.110

Inspecting AI Explainability

Governments, corporations, and others using AI systems, along with those affected by these systems, want to understand how these systems make predictions and decisions. Through what is referred to as explainability, explicability, or interpretability, governments and others hope to illuminate the unseen aspects of AI systems.111

AI explainability represents the capacity of an AI system to reveal how it arrived at a particular output, such as a decision, prediction or score. (For more details, see the sidebar on transparency, explainability, and interpretability.)

AI developers and practitioners are working to find ways to illuminate the inner workings of AI systems, some of which are becoming increasingly complex, such as neural networks.115 116 However, there is no consensus regarding whether it is possible to achieve genuine AI explainability or interpretability.117 Nevertheless, some scholarly literature indeed shows that AI models that are designed to be interpretable are possible, and that in some cases, such as situations involving high-stakes decisions, “interpretable models should be used if possible, rather than ‘explained’ black box models.”118

Some literature goes further still, cautioning against the emphasis on AI explainability goals, suggesting that demands for AI explainability “nurture a new kind of ‘transparency fallacy.’”119

In addition, some AI auditing experts doubt rhetoric suggesting that some AI systems are too densely complicated to be explained, and suggest that with additional transparency, “the mystery disappears.”120

The research for this report identified specific questions and concerns related to AI explainability and interpretability detailed in the literature. Here, we highlight a specific use case involving SHAP and LIME, two related approaches intended to explain how AI systems produce particular outputs or decisions, both of which have attracted scrutiny among computer science and AI researchers.

In addition, later in this section, we discuss scholarly literature addressing the limits and unintended consequences of explainable AI methods such as risk of manipulation and inappropriate applications.

SHAP and LIME: Popular but Faulty AI Explainability Metrics

In the absence of widely-adopted AI explainability standards, two approaches—SHAP and LIME—have grown in popularity, despite attracting an abundance of criticism from scholars who have found them to be unreliable methods of explaining many types of complex AI systems.121

Use of both SHAP122 and LIME123 has increased in part because they are model agnostic, meaning they can be applied to any type of model that data scientists build. An abundance of accessible and easy-to-use documentation related to the two methods has also fostered interest in them.124

The proliferation and adoption of SHAP and LIME as AI explainability methods recognized and used around the world is evident in documentation related to AI governance tools reviewed in Part II of this report. The review found that six AI governance tools from national governments reference or mention SHAP or LIME or both. In addition, a catalog of tools from a multilateral organization includes 12 items recommending SHAP and/or LIME.125

However, the applicability and efficacy of both SHAP and LIME are limited, particularly when used in an attempt to explain complex AI systems comprised of non-linear machine or deep learning models. In a typical use case, an AI practitioner might employ SHAP or LIME to explain a single instance of a model output, such as one decision or prediction, rather than the whole model. Because both methods work by approximating more complex, non-linear models (the types that are often called “black-box” models) with more straightforward linear models, they may produce misleading results.126

Short for Shapley Additive exPlanations, SHAP is based on a concept known as the the Shapley Value, introduced by Lloyd Shapley in 1951127 in the context of cooperative game theory. The Shapley Value is a method used to determine the importance or contribution of each player to an overall competition between groups.128

Today, SHAP is used for another purpose entirely: in an attempt to expose and quantify feature importance, or the importance of factors that contribute to predictions of machine learning models.129 Oftentimes, SHAP is used in the hopes of revealing how factors affect the outputs of opaque, “black box” AI systems such as deep learning models and neural networks, which are difficult to interpret.

SHAP has grown in popularity since around 2017.130 By 2020, use of SHAP for AI explainability had become widely adopted. When researchers asked people from 30 organizations in 2020 which explainability techniques they used and how, they reported that “feature importance was the most common explainability technique, and Shapley values were the most common type of feature importance explanation.”131

Why SHAP and LIME Can Produce Misleading Explanations

SHAP reflects feature importance numerically. For instance, when using SHAP to determine how certain input features affect a more straightforward linear regression model trained on a California housing dataset, the SHAP value of the median house age in a block group might be expressed as -0.22, and the SHAP value of median income as +0.92. The process would be used to add other features, such as the average number of rooms or average home occupancy, until the current model output is reached.132

Although Shapley values have been applied in the context of feature importance for decades,133 researchers have found several mathematical, practical, contextual, and epistemological problems associated with use of the method for explaining AI systems. For example, when attempting to attribute influence to a large set of features affecting AI model decisions or predictions, the approach relies on the modeler to decide which features count as “players” and which are redundant; these subjective decisions can affect the resulting explanations.134

Scholarly research also indicates that some users of SHAP may not understand how to interpret its results. A survey of data scientists using SHAP-based tools showed that many were unable to accurately describe what SHAP values or scores represented.135 The study also found that the popularity of SHAP-based tools influenced some data scientists to trust the tools even if they did not understand what they did or how to interpret their results.

In addition, research shows that use of SHAP in AI explainability tools may lead users to falsely believe they discovered a precise explanation for why or how a system produced a specific output, such as a decision or prediction. This in turn may lead to misconceptions about what SHAP values represent and the actionable information that can be gleaned from them.136

Even scholars who acknowledge benefits of using SHAP to provide insight into certain aspects of models and data suggest they “can lead to wrong conclusions if applied incorrectly,”137and argue that they can be expensive to compute.138

LIME, a similar AI explainability method that has grown in adoption, was first introduced in 2016.139Short for Local Interpretable Model-agnostic Explanations, LIME produces explanations by randomly sampling “locally” around the singular instance chosen to be explained. But its randomness is a pitfall: If LIME is used again in an attempt to explain the very same instance, its explanation will be different.140 The use of LIME for AI explainability has been criticized, and research shows the method can lead to inaccurate results,141 or be manipulated or “gamed.”142

Overall, the research indicating that there are vulnerabilities in these popular explainability measures is not reassuring; however, it is not completely unexpected. Trustworthy AI implementation is still nascent, with much work and refinement yet to come.

Risks Inherent in the Rush to Explain AI

As policymakers have pushed for ways to interpret or explain how AI systems make decisions, a growing body of scholarly literature revealing the limits and unintended consequences of explainable AI methods has emerged.

Wrong Tool for the Job?

Some researchers have found that, even if people are provided with explanations for AI decisions or predictions, they may not actually take the explanations into consideration when they make their decisions.143 For example, people might disregard AI-based recommendations and their explanations associated with a medical diagnosis because they usurp their human decision-making agency or control. Also, common explainability approaches might not provide relevant explanations.144

Explainability metrics are not always intended solely to show end users how a system made a decision or how it weighted certain factors. Explainability metrics might also be used by AI practitioners and evaluators to validate and debug models. Developers might use these metrics to help expose ways to adjust models and their data inputs in an attempt to reduce unintended outputs, such as inaccurate predictions or decisions that disfavor specific groups. In other words, explainability metrics and methods are not one-size-fits-all; rather, they are “data, task, and algorithm-specific.”145

Ultimately, improper application of some AI governance tools can create a false sense of trust and confidence in their ability to explain AI systems.146 147

The limitations of AI explainability methods also can extend to AI policy documents. Recent scholarly literature found a lack of common AI explainability-related terminology and definitions. It also found misalignments between on-the-ground, often-nascent technical research addressing explainability and policy that demands AI explainability be addressed in order to facilitate civil rights goals.148

So, the terminology is confusing, and the research itself has not reached a point where it is settled. AI governance tools in the area of explainability are still in an early phase of development. More work may be required on the technical and policy sides before even a rough consensus or middle ground emerges.

Risk of Manipulating Explanations

Recent research indicates that interpretation of deep learning predictions can be extremely fragile and manipulated relatively easily.149 Other research highlights the potential for fundamental conflicts between providers and recipients of AI explanations, warning that “the provider might manipulate the explanation for her own ends.” For example, feature importance indicated in financial or medical AI system outputs could be sensitive to random or targeted perturbation or disturbance.150

Pathways for Building an Evaluation Environment and Creating Improvements in the AI Governance Tools Ecosystem

It is the goal of this research to help gather evidence that will assist in the building of a more reliable body of AI governance tools. In working through the use cases for this report, the research and scholarly literature points to a variety of evaluation, validation, and quality weaknesses that are present in the uses of a number of AI governance tools. If we do not understand the limitations of AI governance tools, we cannot use them to establish a trustworthy ecosystem of AI systems. Indeed, use of the tools without understanding their limitations is more likely to achieve the opposite result.

• One of the most significant limitations of AI governance tools is the lack of knowledge about which contexts are and are not appropriate for the use of a particular tool. Further, even when some may be aware of the limitations of a tool, others using it may not. To cite a specific example of this from the research, challenges in using SHAP for AI explainability mentioned in this report’s case studies here in Part I are openly discussed amongst technical experts and specialized researchers; however, the problems of applying the four-fifths rule—another measurement approach for AI fairness described in detail in a Part I use case— may be less widely known or understood. This is especially true when the four-fifths rule is encoded into an AI governance tool in a way that is opaque, and then used outside of its originally intended context-sensitive use case.
• The research for this report posits several reasons why this and other breakdowns in contextual understanding, among other problems, are occurring. For example:
• AI governance tools are nascent; as such, a transparent, evaluative community basing their judgments on the evidence has yet not been fully constructed.

The scrutiny and detailed research found in the scholarly literature has not reached all AI governance tool end users, tool publishers, or regulators.

Some problems may be deeply encoded into the AI governance tools, and these problems can be very difficult to see by even careful researchers, much less by end users of the tools.

In considering what might help build a transparent, evaluative community for AI governance tools, the quality assurance system developed for international and other technical standards holds potential. The extensive body of quality assurance structures developed for standards is well-understood and used across a variety of business and professional sectors. For example, several practices are dependent upon quality assurance and quality control standardization, including engineering, medicine, nuclear facilities and activities,151 the development of drugs and therapeutics,152 and so forth. A premise of this report is that AI governance tools can learn from these existing structures of governance.

Ideally, in a matured evaluative environment, AI governance tools would be of high quality and standardized so that when end-users encounter an AI governance tool, they can use that tool in accordance with well-documented standards. At the same time, AI governance tool publishers and managers could contribute to that process by checking for the presence of quality assurance and standardization to ensure the tools they are making available to the public are trustworthy.

Drawing from existing and well-established standards and norms, this report distills a selection of administrative procedures, tools, and methods that articulate how AI governance tools could be created, documented, managed, and maintained. This research focuses on 1) what AI governance tool developers can use; and 2) what “gatekeeper” organizations that host catalogs of AI governance tools can use for quality assurance. There is still much work to be done to test, adapt and create procedures and norms specific to AI governance tools. The distillation here can provide a starting point for further work in helping to build an evaluation environment for AI governance tools.

Creating an Evaluation Environment for AI Governance Tools

This section provides ideas and suggestions about what standards, norms, guidance, and information may prove helpful as AI governance tools continue to evolve. Although many established standards already exist and are important to acknowledge, there remains limited knowledge about their functionality and trustworthiness as applied to AI governance tools. Testing of available tools improves understanding of the current capabilities and quality and encourages building the evaluative environment based on evidence. The Plan-Do-Check (or Study)-Act cycle will be a key tool to assist in this maturation.

For the nascent body of AI government tools, the focus in the beginning can reasonably be on creating a uniform system of measurement for AI governance tools, and then over time, through experimentation and evidence gathering, building the knowledge of how to improve the standards, or write new ones, if they are required. Quality assessment and management of AI governance tools eventually needs to be a routine part of the AI tools and metrics lifecycle. In time, and with work, it will hopefully be consistent across the AI ecosystem.

AI governance tool developers, end-users and researchers should consider using the existing body of evaluation and measurement standards to assess the quality of AI governance tools in a consistent way. Ideally, actors in the community using AI governance tools can, through cycles of continuous improvement, eventually coalesce around one or more standards that already exist, or work to create new standards over time based on the evidence.

For example, one pathway could be via utilizing ISO 9001 as a starting point. ISO 9001 is a significant family of standards that lays out specific-yet-flexible criteria for a quality management system.153 This group of standards is among the most widely used international quality management systems standards today. The 9001 standards are scalable, flexible, and adaptable for use with AI governance tools and the broader network of AI governance tool catalogs.

The ISO 9001:2015 is a specific member of the ISO 9001 series of standards focusing on quality management systems. The ISO 9001:2015 Quality management systems – Requirements standard defines quality as the “degree to which a set of inherent characteristics of an object fulfills requirements.” This definition could apply to many types of “objects” including products and services supplied to or created for the machine learning and AI governance tools ecosystem. There is some existing scholarly work that assesses quality aspects of machine learning systems,154 but this kind of quality assessment has not yet been systemically applied to AI governance tools.155

The implementation of a continuous improvement cycle, which takes the form of the Plan-Do-Check-Act (PDCA) cycle in the ISO 9001:2015, could be a practical and specific starting point in this standard which could be usefully applied to AI governance tools include in particular.156

Plan-Do-Check-Act: The PDCA Cycle

The ISO 9001:2015 standard opens with the Plan-Do-Check-Act cycle, a process-based approach focusing on continuous cycles of improvement and risk-based thinking. From the standard:

The PDCA cycle enables an organization to ensure that its processes are adequately resourced and managed, and that opportunities for improvement are determined and acted on.

Risk-based thinking enables an organization to determine the factors that could cause its processes and its quality management system to deviate from the planned results, to put in place preventive controls to minimize negative effects and to make maximum use of opportunities as they arise.157

ISO 9001:2015 is a flexible standard. In applying the PDCA cycle to the development or publication of AI governance tools, several implementation models are instructive.158 Fortunately, there is a meaningful body of work on implementing the PDCA cycle in various contexts.159 The Deming Cycle, also known as PDSA, or Plan-Do-Study-Act cycle, is also of relevance.160 Although the PDCA cycle and the Deming Cycle are often used interchangeably, the Deming Institute views the “study” aspect of the PDSA cycle as distinct from the “check” aspect of the PDCA cycle. The Deming Institute properly evaluates the “check” aspect as focused more on the implementation of a change, with success or failure attached.

1. A detailed history of the evolution of the PDCA and PDSA cycles describes both of their roots in the scientific method, and how the cycles differ.161 For AI governance tools, there will likely be a period of adjustment and experimentation as tool developers, publishers, and others test and fine-tune the PDCA and/or the PDSA cycle specifically for AI governance tools.

The following are some potential avenues to consider in implementing the four key aspects of the PDCA cycle:

• Plan: Determine the purpose of the AI governance tool and then assess it to see if it is fit for purpose or accomplishes the intended goals.Then test its functionality and trustworthiness: Does it respect privacy? Is interpretable? Is it secure? Has the context been well-understood and identified?

For example:

Identify and understand the use case or context in which an AI governance tool will be used. Choosing the right tool for the proper context is an important aspect of fitness for purpose, which is why it is high on this list.

-Develop the criteria for assessment, aligned with the context. For example, what are the key aspects of the AI governance tool that need to be assessed?

• -Assess the risks involved in implementing the tool. Risk analysis comprises an extremely large literature for AI as well as other fields. AI Impact Assessments,162 Data Privacy Impact Assessments,163 and Safety Impact Assessments164 are helpful tools to experiment with as the evaluation environment is built.

• Determine relevant evaluation criteria to address the stated purpose of the AI governance tool being tested. The criteria will reflect any key aspects that will be addressed in the testing. For example, is the tool accurate, does it scale, if so how much or how well, and how usable is the tool? Are the results from using the tool interpretable, and if so, how interpretable?

• Some AI governance tools will utilize benchmark datasets. Choosing benchmark datasets for testing is a domain of research in and of itself. The suggestion here is that a diverse set of benchmark datasets could be chosen, and they should be clearly representative of the types of data the tool or metric addresses.

• Identify appropriate performance metrics and cut-off points. For example, if a developer is testing tool quality, the performance metric will indicate the point after which the tool or metric ceases to be of acceptable quality for its described purpose.

2. Do: Execute the plan.

For example:

• Conduct experiments: What experiments assess the AI governance tool? In this step, the performance metrics from the Plan phase are fully tested. Ideally, the experiments will have a formal methodology that is fair and that can be made public and be evaluated. Testing should at a minimum follow the tool developer’s recommendations.

• Adversarial testing: Adversarial testing165 can gauge the quality and utility of AI governance tools. This testing can be done prior to deployment by developers, and or after release by tool or tool catalog publishers, end users or researchers. There is empirical support for adversarial testing of AI governance tools. For example, a well-known early adversarial attack on LIME and SHAP usefully revealed that the methods did not fulfill their stated purpose.166 Developing adversarial testing approaches specific to AI governance tools is another nascent area of work. Work to develop baselines and standardized approaches specifically for testing AI governance tools should be a priority.

• Analyze testing results: What do the results obtained from the experiments indicate? What patterns does the testing reveal? What strengths or weaknesses become apparent after testing? How well does the AI governance tool perform across diverse testing datasets?

3. Check/Study: Evaluate the methodology, evaluation methods, and implementation of testing. Conduct gap analysis. The evaluation result should also inform the quality and fitness of steps 1 and 2.

For example:

• Gather input about the evaluation process itself to assess whether it can be improved. For instance, how many different datasets were tested? Were a variety of different parameters tested? Were a variety of different settings used?
• Consider peer review and additional external validation: For example, involve domain experts or other researchers knowledgeable about the AI governance tool, including model evaluation where applicable, documentation and labeling evaluation including truth-in-advertising statements about the product, and evaluation of appropriate contexts and use cases.

4. Act: Improve transparency and functionality of the tool(s).

For example:

• Public dissemination of the evaluation methodology, results, and conclusions can improve transparency. This can take the form of a quality assessment report. Quality assessment reports are called “safety reports” in some fields. These reports contain “practical examples and detailed methods that can be used in support of safety standards.”167
• End users should be made aware of the evaluations in a prominent manner, and the evaluation should be readily understandable by non-expert users. What problems can users expect? What outcomes can users reliably expect to achieve?
• Ensure AI governance tools are transparent and free from conflicts of interest. For example, core pieces of information to make available to end users should provide details about how development of AI governance tools are resourced and financed, by whom, and who published them. Ensuring that there is no conflict of interest is a quality measure that is readily achievable. Ensure that this information is included in each tool or technique’s documentation. Conflict of interest statements will differ depending on the type of tool. For example, a complex checklist may have one publisher and several authors. A set of metrics, models or algorithms used to address problems such as bias may also require disclosure of the entities who have authority to modify source code for that tool.

There is a great deal more work to be completed to reliably and consistently demonstrate that AI governance tools are fit for purpose. Ideally, within the next few years, a robust body of evidence will emerge to assist tool developers, users, and researchers in their analysis and understanding of the various aspects of AI governance tools.

• Solicit ongoing feedback about the tool/product. Ensure a feedback mechanism is routinely available.
• Iterate and refine.
• Repeat the evaluation with different tools, datasets, or evaluation criteria as necessary to continue to refine and improve the product.

>Suggested Framework for Hosts, Publishers, or Managers of AI Governance Tools and Tool Catalogs

Multilateral institutions, governments, and other large entities that host AI governance tool repositories or collections are responsible for ensuring that the AI governance tools they host adhere to high product standards and are fit for purpose. In other mature product-focused realms, there are often specific laws addressing standardization and responsibilities for quality assurance. AI governance tools have not reached full maturity as a body. As a result, the quality controls for individual products, and, by extension, quality control for hosts and publishers of AI governance tools, are not yet fully developed. While this work is underway, much more needs to be done to support it.

Among the most important AI governance tool catalogs published today is at the OECD.AI Observatory,168 which publishes a large catalog of AI governance tools called the OECD Catalogue of Tools and Metrics for Trustworthy AI.169 Fortunately, the OECD has been experimenting and gathering evidence for several years in this area. In a foundational background paper, the OECD provides a framework articulating how it constructs its catalog.170 On page 15 of the paper, the OECD includes a detailed chart depicting its organizational methodology for the catalog and discusses the structure and framework for its decision making in regards to its Catalogue of Tools.171 This framework can be seen in Appendix D.

Figure 4: OECD Catalogue of Tools and Metrics Framework

Source: OECD, June 2021, https://www.oecd-ilibrary.org/docserver/008232ec-en.pdf.

The OECD Catalogue of Tools and Metrics for Trustworthy AI is the first known multilateral framework of its kind to specifically address AI governance tools.

To understand how the OECD’s current methodology could offer broader guidance for other tool catalogs and repositories, this research analyzed the OECD guidance, and compared it with product documentation standards in other areas of work, such as the ISO standards for product documentation, quality assessment and assurance, as well as consumer product safety standards.

In conducting this comparative work, this research found several areas where the OECD framework can be made more robust and provide greater quality controls for the tools listed. The structure below is adapted from the OECD’s original catalog structure, with several additions from the ISO literature and near-worldwide normative laws including data governance law and consumer product safety law. The original OECD chart is reproduced in Appendix D.

Adapted framework for collections of AI governance tools (using as a foundation the OECD framework of tools for trustworthy AI)

• The following framework utilizes the structure of the OECD framework for its Catalogue of Tools and Metrics, and adapts it to include elements that make it more robust in creating a healthy environment for AI governance tools.

1. General Information:

Tool Name

Context of Use

Description

Link to tool and documentation for the tool

Organisation

Country of Origin

Date of Publication

2. Technical Specifications:

– Type of Tool (e.g., software, guideline, standard, certification, etc.)

– Technology Platform Compatibility (e.g., platform-neutral, specific, multi-platform)

– AI System Lifecycle Stages Covered (e.g., design, data collection, processing, deployment, monitoring, etc.)

3. Target Audience:

– Intended Users or User Groups (e.g., AI developers, researchers, and a variety of end users, such as affected consumers and communities, etc.)

– Applicable Policy Areas (e.g., health, finance, transportation, etc.)

– Geographical Scope

4. Usability and Accessibility:

– Tool Usability

– Accessibility (Assesses the tool’s adherence to accessibility standards)

– Required Resources for Implementation (e.g., IT skills, domain expertise, infrastructure, cost, etc.)

5. Trustworthiness Metrics:

– Alignment with International AI Principles (e.g., transparency, interpretability, fairness, robustness, accountability, etc.)

– Ethical Considerations

– Privacy and Data Protection Measures (ensure a privacy policy is posted.)

– Security Measures

6. Performance and Effectiveness:

– Maturity of the Tool (e.g., prototype, in development, production-ready, etc.)

– Case Studies or Use Cases (if available)

– Validity and reliability of the tool in relation to fulfillment of tool and policy goals including in relation of trustworthiness metrics

– User Reviews and Ratings

7. Update and Maintenance:

– Update Process (e.g., how frequently the tool is updated)

• – Support and Maintenance Availability

8. Cost and Licensing:

– Cost of tool use and testing (if any)

– Licensing Type

• 9. Data Policy:
• This category would assess how the AI governance tool handles data from tool users. Key points might include:

User Rights: What control do users of AI governance tools have over the data involved in the lifecycle of the tool?

Does the AI governance tool train on any data? If so, is there a choice in the matter?

What is the copyright policy of the AI governance tool? For example, is there a policy that ensures all data involved in the lifecycle of the tool will continue to be governed by a policy and made transparent?

10. Transparency and Conflict of Interest Notice: This category lists applicable resources and funding sources, and/or financial interests in relation to the tool. Commercial Interests also should be noted, for instance, if the tool promotes specific commercial products or services. Affiliations, including relationships that potentially impact objectivity, including information about commercial or other entities that donated the tool for open-source use, should also be noted.

Individual Product-level Documentation for AI Governance Tools

The research conducted for this report found inconsistent documentation for AI governance tools. It was not a focus of the methodology for the report to analyze this particular aspect of AI governance tools, but it was difficult to avoid noticing it. Hosts and publishers of AI governance tools and tool catalogs could find experimentation regarding documentation to be fruitful. AI governance tool documentation deserves more discussion, given its importance, and given the inconsistent application of documentation when it is made available.

A great deal of existing work has already been done in other areas that could be helpful. For example, significant documentation standards and norms exist around consumer products, software products, and other technology products offered to the public. These norms are encapsulated in multiple ISO standards,172 as well as OECD Responsible Business Conduct principles and implementation guidance.173 174

Baseline documentation for individual AI governance tools is an area where rapid improvements may be achieved. Just as with any tool or product released to the public, developers should create and make robust documentation available. Entities publishing large collections of AI governance tools should require robust documentation prior to publication of any tool, no matter how simple or complex the tool may be.

As mentioned, many standards exist regarding product documentation, some sector-specific.175 Ideally, over time, a set of product documentation characteristics specific to AI governance tools will develop through testing, experimentation, and evidence building.

The following suggestions for creating documentation for an AI governance tool are distilled from the large body of existing work in software documentation and consumer product documentation, among other areas. These suggestions are provided here as an initial step toward further experimentation and evidence building.176

Suggestions for Transparency and Documentation of Individual AI Governance Tools: 177

Documentation will typically focus on what was done, what assumptions and constraints were present, what data was used, and other elements that a user, a developer, a researcher or tools publisher may want to know and use for validating a tool.

For example, the following is an adaptation of documentation found in the standards literature:

Title of AI Governance Tool

Introduction and Context – An overview of the AI tool or metric will often include a discussion of the purpose and background of a tool, a description of its key features, and a description of the target audience or users of the tool.

Getting Started and User Manual – This can include items like system requirements, installation guide when applicable, and use instructions. For software-based AI governance tools, a detailed guide would describe the features of the tool, instructions for use, and a guide to troubleshooting.

Socio-Technical Documentation – This documentation provides developers and technical users with information about an AI governance tool’s architecture, interfaces, APIs if applicable, as well as integrations with other AI governance tools or systems. It could also include information about intended environments or contexts of use and intended human interaction.

Tutorials and Case Studies – These can discuss relevant use cases for the AI governance tool, and proposed uses of the tool.

Changelogs/Version History – This can provide historical data regarding iterative updates and changes made to a tool, including a version history.

Support and Contact – AI governance tool providers can let users know how to get support or report product issues. Ideally, the documentation will include a feedback process and formal complaint mechanism.

Funding and Conflict of Interest Statement – This category creates transparency regarding the resourcing, funding, affiliations, and objectivity of the tool. The importance of such transparency structures cannot be overstated, given the variety of ways both conscious as well as unconscious bias can undermine the objectivity of research and tool outcomes.

Concluding Thoughts Regarding How to Build an Evaluation Environment and Create Improvements in the AI Governance Tools Ecosystem

The suggestions in this discussion of how to begin creating and building improvements for the AI governance tools ecosystem discuss a body of standards, governance techniques, and approaches that are currently available. This is by no means to say that these are the only policy tools that should be considered—far from it. Many additional strategies exist. For example, regulator-approved codes of conduct may play an important role in addressing specific AI risks in focused use-cases or even specific types of AI systems.178

This being said, this discussion highlights the rich body of strategies that, while often overlooked in the governance of AI, may be among the most important. When it comes to AI governance, it is still early; right now, we are all crossing the river by feeling the stones.

PART II: A Survey of AI Governance Tools and Other Notable AI Governance Efforts from around the World

The movement to advance responsible and trustworthy AI beyond theoretical principles toward practical implementation is upon us. This research indicates there is a keen interest in devising workable policy approaches to measuring and improving AI systems according to established AI principles, and it is happening on a worldwide scale.

Primarily in the past four years, governments, multilateral organizations, non-governmental organizations (NGOs), development banks, standards bodies, academic institutions, and public-private partnerships have taken concrete steps toward establishing AI governance tools for measuring and improving AI fairness, explainability, robustness, privacy, and more. This section of the report discusses these tools.

Some of this important work has been out of the spotlight amid more prominent pressures for enforceable AI regulations and guardrails. The AI governance tools we survey here are implementation tools: they form the interface between the goals of AI governance and how it happens in reality. These tools generally do not fall under specific AI regulations at this time, depending on the jurisdiction. However, it would be a mistake to assume that only strict regulation would afford positive change in a technical ecosystem: many AI governance tools demonstrate a genuine commitment to address goals and concerns around AI.

In some cases, these reviews include constructive critiques and warnings regarding components of AI governance tools that demand improved inspection and quality assurance. Although development of AI governance tools is nascent, this is no time to ignore due diligence for the methods and measures that will form the basis of AI governance for years to come. AI systems affect real people, groups of people, and communities every day.179 And, although sandbox policy approaches and toy technical projects have potential value, they still will need effective quality controls and assessment before dissemination.

As we detail in Part III of this report, a next step for many of the tools reviewed here will be reevaluation and possible adjustment to ensure a reliable and thriving AI governance tool ecosystem people can trust.

Highlights from Distinctive AI Governance Tools and Activities

Notable efforts to put goals for responsible AI into practice are happening in earnest around the world. Consider it a much-needed sign of unity in an often divided environment.

This landscape of AI governance tools is fertile ground. Some of the tools reviewed here represent efforts that have reached more advanced stages than others. All are important.

• Although commonalities do exist in all of these AI governance tools, the efforts are in no way entirely homogenized. They occupy a vast spectrum of nuanced approaches and ideas, reflecting the panoply of communities and cultures that dot the globe.

A few of these tools in particular stand out for a variety of reasons.

• In South America, the government of Chile has established a new approach to acquisition of AI. In an effort to incentivize technology providers to build their systems better from the start, the country’s updated technology bidding and procurement process ensures public sector agencies thoroughly assess the AI systems they seek to use.
• In India, a plan from the state of Tamil Nadu for adoption of AI-based systems proposes the use of a point-based rating system. The rating system considers commonly-mentioned factors including fairness and transparency, but it also measures something we did not see in many other AI governance tools: diversity, as well as relevance and performance of AI systems across geographies and societies. This is a key concern in India, where the world’s largest population represents several distinct cultures, languages, and customs.
• In Kenya, a project created to help ensure that genuinely inclusive AI can be built has taken root. Sometimes, AI governance means tackling foundational problems before tools are needed to improve existing AI systems. One such problem plaguing Africa is a lack of inclusive language data. Without high-quality data reflecting diverse populations, AI systems will not be fair, inclusive, or beneficial for everyone. Masakhane, which roughly translates to “We build together” in isiZulu, has launched a variety of initiatives designed to create datasets representing low-resourced African languages.
• In New Zealand, a process for identifying and reducing risk throughout the life cycle of an algorithm not only centers on personal data collection, use, or disclosure throughout all stages of algorithm governance, but it also strives for relevance to communities not typically represented in government technology policy. The process recommends use of frameworks for assessing the cultural data implications of algorithms affecting Māori communities.

In what may be considered the most technical approach to implementing AI principles among the AI governance tools reviewed in this report, Singapore has developed software and a technical testing framework for improving the fairness, explainability, and robustness of AI systems. In October 2023, it became one of the first national governments to recommend specific approaches to evaluating generative AI systems such as Large Language Models.

Notably, the research and analysis in this report does not include corporate tools, although some of the tools reviewed here do mention, recommend, or incorporate AI governance methods created by private corporations. And in general, we narrowed the scope of AI governance tools distributed by governments to national-level governments rather than state or local governments.

Still, this report is inclusive in a very deliberate way. For AI to be truly fair and equitable—this survey of AI governance tools indicates this is the most common goal of all AI principles—it must be inclusive. That means inclusive data or inclusive approaches to AI-related education, work, and development. It also means that when it comes to measuring AI’s impacts, everyone gets a seat at the table.

Some entries in this survey are not fully mature tools yet; they are efforts or programs that are adjacent to, or building toward operationalizing responsible AI. These tools were worthy of highlighting in order to ensure geographical inclusivity and to provide an equitable overview of AI governance tools activities as they exist today.

>Notes About How This Section is Organized

The AI governance tools in this section are organized using the M49 ISO standard for naming and listing regions, subregions, and nations. The M49 ISO standard is the United Nations/ISO joint standard for naming and classifying regions, subregions, and for articulating country names.180 It is considered to be the authoritative standard for naming and structuring geographical-related data.

For those unfamiliar with this standard, there are distinctions that may be unfamiliar. The major regions are as follows:

Africa
Americas
Antarctica
Asia
Europe
Oceania.

This report lists the tools in order first by the major regions. So, for example, AI governance tools from Africa are listed first, with tools from the Americas second, and so forth.

The M49 standard includes subregions, and this report includes subregions where necessary. For example: for tools from Singapore, the region is Asia and the subregion is South-eastern Asia.

The tools reviewed here appear in the following order:

• Tools from international organizations
• Tools from international standards organizations
• Tools from regional banks
• Tools from national governments, followed by tools from other organizations in countries, organized in alphabetical order by region name: subregion name: country name.

This report introduces and uses the term AI Governance Tools, which this report defines as:

AI Governance Tools:

Socio-technical tools for mapping, measuring, or managing AI systems and their risks in a manner that operationalizes or implements trustworthy AI.181

This definition encompasses the wide array of formats and methods reviewed here in Part II. There are different types of tools. The following lexicon of AI governance tool types further distinguishes differences among them.

[Possible Image] AI Governance Tool Types

• Practical Guidance – Includes general educational information, practical guidance, or other consideration factors
• Self-assessment Questions – Includes assessment questions or detailed questionnaire
• Procedural Framework – Includes process steps or suggested workflow for AI system assessments and/or improvements

Technical Framework – Includes technical methods or detailed technical process guidance or steps

Technical Code or Software – Includes technical methods, including use of specific code or software

Scoring or Classification Output – Includes criteria for determining a classification, or a mechanism for producing a quantifiable score or rating reflecting a particular aspect of an AI system

The AI Governance Tools Comparison Chart spotlights key features of select mature AI Governance Tools from national governments and multilateral organizations.

Figure 5: AI Governance Tool Types and Features Comparison Chart

(Source: World Privacy Forum, Research: Kate Kaye, Pam Dixon.)

Intergovernmental Organization Toolkits and Use Cases from International and Regional Multilateral Institutions

A range of intergovernmental organizations182 have begun working in earnest on implementing trustworthy AI. Among entities establishing ways to operationalize trustworthy AI principles, the intergovernmental organizations are especially important, as their work has a greater opportunity to become normative across multiple countries and, in some cases, multiple regions.

Organization of Economic Cooperation and Development (OECD)

The Organization of Economic Cooperation and Development (OECD) is a multilateral institution that, for much of its history, has focused on the most developed economies.183 OECD is headquartered in Paris, France. Today, it has 38 government members with five accession members (Argentina, Brazil, Bulgaria, Croatia, and Peru) and five key partners (Brazil, China, India, Indonesia, and South Africa).184

The OECD is notable for its formal multistakeholder process, which includes representatives from governments, industry, trade unions, and civil society. OECD’s Council Recommendations, such as the OECD Privacy Guidelines,185 are normative, and are also Customary International Law,186 which can be adjudicated under the auspices of the International Court of Justice.187 The OECD Privacy Guidelines also went on to form the early basis of most international privacy law as it developed in the 1980s through the 1990s and beyond.

The OECD began work in 2018 crafting the first multilateral principles for AI. The OECD AI Secretariat appointed a large group of international AI experts called the AI Expert Group, or AIGO. Along with the OECD government delegations and formal stakeholder groups, AIGO formed the core of the OECD multistakeholder process related to AI. The OECD AI Principles were adopted as a formal OECD Council Recommendation in May 2019, and were ratified by member governments.188 The OECD AI Principles were the first “soft law” AI principles to be developed.189 Since this time, the OECD has published a handbook, established a dedicated AI Working Party, launched and populated an international AI Observatory (OECD.AI), and is moving forward on multiple AI projects inside expert groups. The OECD AI Principles have been adopted normatively. The US National Institute of Standards and Technology (NIST) has incorporated OECD definitions and framework into its work,190 and the European Union also has adopted the OECD definition of an AI system in its EU AI Act proposed legislation. There are cooperative OECD efforts in the broader standards world and in legislative arenas as well.

OECD.AI Catalogue of AI Tools and Metrics to Promote Trustworthy AI

Tool Type: Catalog

One of the focus points for the OECD’s current AI work is to guide the implementation of the OECD AI Principles. There are multiple aspects to this work, and one particularly significant component has been the 2023 launch of the OECD.AI Catalogue of AI Tools and Metrics to Promote Trustworthy AI (hereafter, the Catalogue of Tools). The Catalogue of Tools was created to facilitate broader accessibility to the available tools and metrics intended to produce more trustworthy AI systems191 that respect the OECD AI Principles.192

The OECD’s Catalogue of Tools repository features a wide variety of AI governance tool types. As of September 2023, the Catalogue of Tools includes technical tools for auditing or reducing bias; technical tools for robust and secure data collection and processing; metrics to measure model performance; and metrics to measure privacy. The collection also includes metrics to evaluate portions of AI datasets, research papers detailing methods for addressing AI problems, links to software and auditing services sold by private corporations,193 194 and an open-source database for archiving real-world examples of AI failures and vulnerabilities.195 The submissions of tools and metrics featured in the Catalogue of Tools are to be vetted by the OECD Secretariat as well as partner stakeholder organizations to ensure accuracy and objectivity.196 197

As noted in the Findings of this report, multilateral institutions and other AI Governance Tool providers and hosts have an important role as quality assurance gatekeepers. The OECD’s Catalogue of Tools includes tools intended to measure and improve fairness and explainability of AI systems, some of which have drawn sharp criticism in scholarly literature reviewed here in Part I. For instance, nine items hosted in the Catalogue feature use of SHAP, and three feature use of LIME. In addition, despite scrutiny among researchers of methods that abstract and encode the US Four-Fifths Employment Rule in an attempt to measure disparate impact of AI systems, three tools featured in the OECD’s catalog include such methods.198

There are differences in opinion among researchers and practitioners regarding the effectiveness of these measures in practice when applied to explain or interpret AI or determine its potential disparate impacts on particular groups.199

United Nations Educational, Scientific, and Cultural Organization (UNESCO)

UNESCO Recommendation on the Ethics of Artificial Intelligence

In 2022, UNESCO published its global standard on AI ethics. It did so in its flagship Recommendation on the Ethics of Artificial Intelligence.200 The text may be summarized into what UNESCO calls four core values: human rights and dignity; living in peaceful, just, and interconnected societies; ensuring diversity and inclusiveness; and environment and ecosystem flourishing. The core principles include the idea of proportionality as well as a “do no harm” principle. Other core principles include the right to privacy and data protection, sustainability, and fairness and non-discrimination.

UNESCO has also produced training around its AI Recommendation. For example, its introductory online training course on AI and the Rule of Law features sections addressing “best practices that translate ethical principles into practice both in terms of the use of AI in justice systems, and in cases involving AI impacting human rights.”201 The course, aimed at members of the judiciary, includes sections on algorithmic bias and its implications for judicial decision-making and AI ethics and governance concerning judicial operators.202

UNESCO AI Ethical Impact Assessment and Report

UNESCO’s AI Ethical Impact Assessment (EIA), fully articulated and discussed in a 2023 report,203 assesses algorithms according to their alignment with the principles and guidance contained in the UNESCO Recommendation on AI. The EIA also recommends creating transparency by surfacing information about AI systems across the life cycle. The EIA is designed for procurers of AI systems, as well as those who want to evaluate whether or not use of AI is appropriate for a given problem.

World Bank Group

Risk Assessment Framework for World Bank Projects

The World Bank Group (WBG) created a Risk Assessment Framework initiative to Identity and Classify Ethical Risks from AI use in World Bank projects.204 The framework was based in part on the OECD Framework for the Classification of AI Systems.205

The WBG also has proposed an initiative to ensure World Bank operations staff have tools for establishing algorithmic accountability and identifying AI’s impact on human rights.206

International Standards Organizations

International Standards Organizations play a pivotal role when it comes to putting policy into practice and establishing norms for technology design, development, and use internationally. So, it should come as no surprise that standards bodies have begun contributing to the AI governance tool ecosystem.

This section discusses the three main international standards development organizations, or SDOs. NIST is an SDO. Although technically based in the US, NIST is unique in that it works cooperatively with multilateral organizations in its standards settings. As such, NIST functions as an international SDO.

There are many regional SDOs, including the European Committee for Electrotechnical Standardization (CENELEC), The African Organization for Standards (ARSO), and the British Standards Institution (BSI), among many others. This report, however, does not include an analysis of these or other regional SDOs.

International Organization for Standardization (ISO)

ISO is a significant international SDO, which to date has created a number of core technical standards for AI systems in addition to its overarching technical standards work.207 ISO launched its AI technical subcommittee in 2018208 to address AI computational methods, trustworthiness, and societal concerns through the development of standards and guidelines. Some key AI-related work from ISO includes a standard providing guidelines for development and use of products, systems, and services for managing AI risk;209 technical reports on AI bias measurement; and assessment of neural network robustness.210 ISO has taken a whole-of-ecosystem approach211 to its work on AI standards: its current roster of 20 AI-related standards, technical specifications, and reports thus far addresses bias, robustness, and various aspects of risk management.212

National Institute of Standards and Technology (NIST)

Artificial Intelligence Risk Management Framework and AI RMF Playbook

The National Institute of Standards and Technology is based in the US; however, in the area of AI, it has functioned in substantive ways as an international standards organization due to its collaborative work with OECD on AI terminology and other related work. NIST published its Artificial Intelligence Risk Management Framework213 and companion AI RMF Playbook in January 2023.

The framework is divided into two parts. First, it discusses how organizations can frame the risks related to AI, and then it describes in detail the four functions present in the AI life cycle—Govern, Map, Measure, and Manage—and how to address them. While the Govern function applies to all stages of an organization’s AI risk management processes, the framework document explains that the Map, Measure, and Manage functions can be applied according to specific contexts at specific stages of the AI life cycle.

The NIST AI RMF recognizes the fallibility of metrics used to measure AI risk, noting that such metrics “can be oversimplified, gamed, lack critical nuance, become relied upon in unexpected ways, or fail to account for differences in affected groups and contexts.” To address potential problems with AI measures, the framework suggests that AI metrics and effectiveness of existing controls are regularly assessed and updated, and that the AI system measurement process should involve consultation with multidisciplinary stakeholders.214

There are numerous other examples of practical implementation in the NIST framework, which has substantial overlap and harmonization with the OECD implementation work as well as the work being done on AI implementation and standardization at ISO.

As of October 30, 2023, NIST has been given substantial additional responsibilities in relation to AI governance, safeguards, and standards.215 For example, NIST is tasked with developing guidelines, standards, and best practices for AI safety and security to help ensure the development of trustworthy AI systems. As part of this, it will develop a companion resource to the AI RMF for generative AI.

Institute of Electrical and Electronic Engineers (IEEE)

GET Program for AI Ethics and Governance Standards

IEEE, a well-established international SDO, has developed the IEEE GET Program in conjunction with partners including product certification company TÜV SÜD.216 The program provides free access to seven AI ethics and governance standards as of January 2023.217

The program is designed to support efforts regarding AI ethics and governance literacy as well as AI systems governance and standardization, among other topics. The GET program is tied to IEEE’s broader global initiative regarding ethical AI, which aims to “ensure every stakeholder involved in the design and development of autonomous and intelligent systems is educated, trained, and empowered to prioritize ethical considerations so that these technologies are advanced for the benefit of humanity.” 218

Standards featured in the IEEE GET Program include guidance and procedural methods for assessing impacts of AI systems according to human well-being; procedures for achieving testable levels of transparency; standards for ethical system design and data privacy; and a set of ontologies to establish ethically driven methodologies for the design of robots and automation systems.

The GET Program’s IEEE Standard for Transparency of Autonomous Systems, for instance, includes detailed methods for gauging the level of transparency of an AI model.219

In related work, IEEE’s Global Initiative on Ethics of Autonomous and Intelligent Systems also includes a Standard for Algorithmic Bias Considerations, one of 11 IEEE ethics-related standards currently under development.220 The standard includes a development framework intended to avoid unintended, unjustified, and inappropriately differential outcomes for users.

Regional Development Banks

Region: Africa

The African Development Bank (ADB)

Financial Inclusion Grant Program

The African Development Bank launched a financial inclusion grant program of $1 million in 2021 to develop AI-enabled systems to process customer complaints in several key local languages in Ghana, Rwanda,221 and Zambia.222 However, there does not appear to be any program, methods, or tools to facilitate implementation of AI governance by ADB at this time.

Region: Americas: Latin America and the Caribbean

Inter-American Development Bank (IDB)

Region: Americas

The Inter-American Development Bank (IDB), headquartered in Washington, D.C., is one of four major regional financial sector multilateral institutions.223 In addition to lending money to countries in Latin America and the Caribbean (LAC), IDB provides member governments with technical assistance for a variety of projects intended to bring economic inclusion and growth to the region. IDB has actively assisted institutions to operationalize AI principles.224 In 2019, IDB launched a significant regional alliance for ethical and responsible use of technology, with a focus on AI. The partnership is between public and private sectors, and includes civil society and academia. Called “fAIr LAC,”225 this project seeks to assist regional governments as they navigate the ethical application of AI.

The program features a series of pilot projects; an observatory of responsible AI use cases; four regional hubs (Jalisco, Costa Rica, Colombia, and Uruguay); and frameworks and documents written in Spanish, Portuguese, and English for use by public servants, government ministers, and AI developers working for governments. The initiative has resulted in procedures for AI self-assessment by public sector and private entities, and practical manuals such as a handbook for AI project directors for incorporating ethical considerations throughout the AI life cycle.226

As part of its fAIr LAC program, the IDB has also helped countries build AI-related projects, including public online dashboards detailing information about AI-based tools used by the government of Chile.227

Region: Asia

Asia Development Bank (ADB)

Mapping poverty through data integration and AI

Programs involving AI are underway at the Asia Development Bank (ADB), including extensive work to map poverty more accurately in the Asian region through use of novel data types and AI.235 The 2020 report, Mapping poverty through data integration and Artificial Intelligence, offers detailed guidance on how to prepare data for feasibility studies, how to ethically apply a convolutional neural network to poverty prediction, and advice on how to reduce bias and other undesirable components of sensitive poverty research.

For example, the report provides three possible approaches to using big data and AI processes without contaminating the data samples with self-selection bias, including specific guidance regarding measurement to address these issues.236

The ADB’s 2020 report, AI in Social Protection — Exploring Opportunities and Mitigating Risks, co-published with Germany’s Agency for International Cooperation GIZ (Gesellschaft für Internationale Zusammenarbeit),237 provides guidance for implementing ethical approaches for AI systems used in relation to social protection programs. The guidance addresses topics including inclusive data strategies and transparent and independent review of AI systems.

Region: Europe

The European Bank for Reconstruction and Development (EBRD)

Report: Approach to Accelerating the Digital Transition

In its 2021 report outlining its approach to accelerating the digital transition,238 the European Bank for Reconstruction and Development set forth a series of pledges. The bank said it plans by 2025 to develop a legal and regulatory framework that promotes innovation, healthy competition in digital markets, and cybersecurity, while safeguarding financial stability and inclusion and ensuring diversity, the ethical use of artificial intelligence (AI), and appropriate data protection.239 Thus far, the EBRD gives every indication that it is laying the foundations to implement ethical AI, but it has not yet published specific AI governance tools.

In the report, The EBRD’s approach to accelerating the digital transition, 2021-25, the EBRD discusses digital transformation and AI, posing opportunities and risks related to issues ranging from privacy and bias problems to electronic waste. EBRD established its Digital Hub240 in January 2022 to support and coordinate the implementation of its digital approach, which is intended to enable equal access to digital technology and skills, establish robust governance practices, and provide financial and technical support to companies and governments.

AI Governance Tools and Use Cases from National Governments and NGOs

This research unambiguously found that national governments and other organizations around the world throughout Africa, the Americas, Asia, Europe, Oceania, and the United Kingdom are crafting AI governance tools. Often, these tools are part of a larger national strategy on AI, with most of these efforts focused on moving from principles to practical guidance.

Observations of AI governance tools from this research indicate that national governments and NGOs have focused on operationalizing approaches to achieving AI fairness, explainability, robustness, and data minimization, among other important goals. AI governance tools reviewed here cover a wide range of formats, including self-assessment questionnaires, procedural workflow charts, detailed rules for AI system procurement, and recommendations for oversight committees. Some feature scoring mechanisms to quantify risk or other aspects of AI systems. In more rare cases, governments and other organizations have provided detailed technical guidance and even technical software, all with the intent to assess AI systems and assuage particular problems.241

This section of the report primarily includes AI governance tools from national governments. In order to recognize efforts related to establishment of AI governance tools and practices taking place internationally, we also include here some work from NGOs and hybrid entities involving partnerships among NGOs, academia, and private industry. And, as previously mentioned, because the spectrum of AI development stages reflects a wide continuum across geographic regions, this report includes AI-related work that is building toward formal AI governance tools.

Africa: Northern Africa: Morocco

The International Artificial Intelligence Center of Morocco

AI Movement

The International Artificial Intelligence Center of Morocco, opened in November 2022,242 has as its primary objective to deliver practical, resilient, and ethically sound approaches to AI-related challenges faced by society, the environment, the market, the economy, and technology.

The Center has launched a four-tiered certificate program called the “AI Governance and Applications” executive program, which includes a significant section on responsible and ethical AI and data management.243

A partner of the Sub-regional Forums on AI set up by UNESCO, the Center is involved with implementing the UNESCO Recommendation on the Ethics of AI in the African context through conferences and other work. These initiatives are conducted in relation to UNESCO’s Global Priority Africa program.244

Africa: Eastern Africa: Kenya

Masakhane Research Foundation

Language Datasets for Africans, by Africans

AI systems require data as core inputs to learn how to recognize patterns in information and produce outputs such automated decisions and predictions. For example, datasets representing low-resourced languages are necessary to ensure that AI systems incorporating natural language processing (NLP) for products such as chatbots or mobile apps used to assist people in healthcare, finance, or social services are truly fair, inclusive, accurate, and trustworthy. However, for many low-resourced languages, high-quality datasets simply do not exist.

Work is underway to help ensure that the datasets needed to train AI models—reflecting the groups and communities who will use them or be affected by them—are built.

Masakhane Research Foundation is a grassroots organization based in Kilifi, Kenya.245 The group aims to ensure that low-resourced African language datasets exist, and that AI systems recognize African names, cultures, places, and history by conducting and strengthening NLP research in African languages: for Africans, by Africans.

Contributors to the Masakhane project are generating, curating, and annotating datasets that are inclusive of the languages people speak throughout the African continent.246

Masakhane roughly translates to “We build together” in isiZulu, one of South Africa’s 12 official languages, which include Sepedi, Sesotho, Setswana, siSwati, Tshivenda, Xitsonga, Afrikaans, English, isiNdebele, isiXhosa, and South African Sign Language.247

The organization has launched a variety of initiatives designed to create new datasets including a project to deliver open, accessible, and high-quality text and speech datasets for low-resourced East African languages from Uganda, Tanzania, and Kenya.248

Another Masakhane project is intended to build a multilingual scientific corpora of African research translated into multiple African languages. That project will include creation of the Masakhane Ethical Manifesto for use as a reference for any NLP dataset creation efforts on the African continent. The manifesto will incorporate ubu-Ntu ethics, “notable for its strong focus on enriching relationships in ways that affirm human rights and human dignity through the equitable distribution of power, and individual and communal participation in mutually beneficial goals.”249

Africa: Western Africa: Ghana

Kwame Nkrumah University of Science and Technology Responsible Artificial Intelligence Lab

FACETS Framework

Tool Type: Practical Guidance with Self-assessment Questions and Scoring Output

The Responsible Artificial Intelligence Lab (RAIL) at Kwame Nkrumah University of Science and Technology in Kumasi, Ghana250 seeks to ”improve the quality of life for all in Africa and beyond by partnering with Africa’s science and policy communities to leverage AI through high-quality research, responsible innovation, and strengthening talent.”

The lab is part of the Artificial Intelligence for Development in Africa program partnership between the Swedish International Development Agency and International Development Research Centre. It is also supported by GIZ, Germany’s Agency for International Cooperation GmbH, through its FAIR Forward initiative.

RAIL introduced in January 2023 its framework for a quantitative approach to measuring responsible AI, called FACETS (Fairness, Accountability, Confidentiality, Ethics, Transparency, and Safety).254 Based on established ISO 26000 social responsibility standards for businesses and organizations,255 as well as other related frameworks for fairness, accountability, and transparency, FACETS is intended to provide a quantitative measure of ethical AI practices incorporated throughout the development of an AI system.

The framework includes an online self-assessment questionnaire256 addressing the FACETS-related issues according to stages of an AI life cycle from origin to deployment, including data- and model-related questions. Self-assessment evaluators can respond “Yes,” “No,” or “I don’t know.”

Examples of questions featured in the FACETS assessment:

• “Did you consider inclusivity by putting people in the center from the beginning of the process (use-case definition, data collection, and system development)?”
• “Does the dataset follow acceptable standards best practices and specifications for data development like datasheets for datasets?”
• “Is the model published in a conference or journal?”
• “For Explainable AI (XAI) methods, is the explanation for the prediction shared in a decision report?”

Based on responses to the online self-assessment, the system calculates a numerical FACETS score.

Americas: Latin America and the Caribbean: South America: Chile

Ethical, Responsible, and Transparent Algorithms Project and Procurement Requirements

Chile launched its Ethical, Responsible, and Transparent Algorithms Project in 2020.257 The public-private partnership, established through funding from Inter-American Development Bank’s IDB Lab, brings together the Universidad Adolfo Ibáñez and several public sector agencies to incorporate ethical standards in the purchase, use, and reporting of AI and automated decision algorithms by government agencies, as well as in the development of AI and automated systems by technology providers.

As part of the ongoing effort, the group has conducted pilots for the ethical implementation of automated systems in public institutions, and has developed guidelines, regulations, and other methods for ethical design and purchase of algorithmic systems.

Americas: Latin America and the Caribbean: South America: Chile

Government of Chile

ChileCompra Standard Bidding Terms for Data Science and AI Projects

Tool Type: Practical Guidance

Chile’s Ethical, Responsible, and Transparent Algorithms Project has also led to new requirements for quality assurance for government agency procurement of AI and automated systems. Chile’s purchasing and public procurement directorate, ChileCompra,258 established its Standard Bidding Terms for Data Science and AI Projects in December 2022. Its Resolution No. 60 (Direccion de Compras y Contratación Publica Aprueba Formato Tipo de Bases Administrativas Para la Adquisición de Proyectos de Ciencia de Datos e Inteligencia Artificial) includes bidding terms, impact assessments, and guidance on measuring AI fairness and explainability. The requirements have been piloted with Chile’s National Health Fund (FONASA) and its Public Criminal Defender’s Office.259

The project aims to incentivize industry players to build AI systems with responsible and trustworthy AI considerations if they want to win government contracts.260 It uses these bidding and procurement requirements for public sector agencies to do that. The standard bidding conditions are promoted by ChileCompra to facilitate the participation of government suppliers in relation to large sum public sector tech acquisitions.

The AI procurement terms require that statistical equity metrics be taken into account to help conduct an ethical and responsible analysis of a system.261 The terms also state that such metrics must be considered in a public sector agency’s evaluation and choice of a model, based on the specific problem or context at hand.

ChileCompra’s AI procurement requirement documentation suggests use of Datasheets for Datasets262 to assist in detecting bias in data used to develop the system under assessment. It also suggests use of Model Cards for Model Reporting,263 as well as risk analysis of personal data processing through an impact assessment.

In addition, the requirements make note of specific measures for AI explainability including counterfactual explanation, Local Interpretable Model-Agnostic Explanations (LIME), and the What-if Tool, an open-source tool original devised by Google. The What-if Tool attempts to assess the behavior of trained machine learning models, including models used for image recognition and conversational AI. Documentation associated with the What-if Tool features use of SHAP to reveal feature importance to analyze model fairness.264

As noted in the Findings section of this report, use of LIME and SHAP for AI explainability has drawn sharp criticism in scholarly literature reviewed here in Part I.

Americas: Latin America and the Caribbean: South America: Chile

Universidad Adolfo Ibáñez, Government of Chile and IDB

Repositorio Algoritmos Públicos

Universidad Adolfo Ibáñez’s GobLab UAI is the public innovation laboratory of the research university’s School of Government. To create more transparent use of algorithmic systems, the university helped build an online dashboard detailing information about algorithmic and AI-based tools used in the public sector. The platform was built with support from The Inter-American Development Bank in conjunction with the government of Chile.265

The digital platform featured information on 75 different automated systems used by public institutions in Chile by the end of 2022, from rules-based systems to deep learning and neural network-based systems.266 Systems are categorized in the platform according to government divisions such as education, health, defense, and economic affairs.

Americas: Northern America: Canada

Government of Canada

Algorithmic Impact Assessment Tool

Tool Type: Practical Guidance with Self-assessment Questions and Scoring Output

Canada’s Algorithmic Impact Assessment Tool,267 part of Canada’s responsible use of artificial intelligence policy work, is mandatory for federal government institutions. It comes in the form of a questionnaire, and is answered by government agencies intending to use algorithmic systems. Inquiries address the purpose for AI systems, how they are designed, and how data is sourced and prepared.

The Algorithmic Impact Assessment (AIA) is designed to score the impact level of an algorithmic system according to factors including its design, algorithm, decision type, and data. Impact levels are classified from “little to no impact” to “very high impact” in relation to individual rights, health and well-being of individuals or communities, economic interests, and sustainability of an ecosystem.

Depending on impact level, Canada’s AIA process requires peer review, data bias and quality testing, data governance implementation, analysis using Canada’s “Gender-based Analysis Plus” analysis method,268 human intervention during the decision-making process, and meaningful explanation of decision results.

Canada’s AIA reports are available in Canada’s Open Government Portal.269

Americas: Northern America: United States

US federal agencies and the White House have produced multiple sets of voluntary guidance and procedural approaches to incorporating accountability, fairness, data privacy, and risk management into AI development and use in the US.

Americas: Northern America: United States

US Government Accountability Office

An Accountability Framework for Federal Agencies and Other Entities

Tool Type: Process Framework with Self-Assessment Questions

The US Government Accountability Office (GAO) published Artificial Intelligence: An Accountability Framework for Federal Agencies and Other Entities in 2021.272 The guidance provides questions and audit procedures for promoting AI accountability, ensuring data quality, producing results that are consistent with objectives, and monitoring for reliability and relevance of AI systems over time.

The GAO’s accountability framework mentions IBM’s AI Fairness 360273 as an example of “guidance on incorporating ethical principles such as fairness, accountability, transparency, and safety in AI use.”274

In addition, the GAO framework mentions Microsoft’s model drift monitoring guidance,275 276 and the use of data and model cards for data transparency purposes.277

Americas: Northern America: United States

US General Services Administration

The Artificial Intelligence Governance Toolkit

Tool Type: Practical Guidance and Process Framework with Self-Assessment Questions

The Artificial Intelligence Governance Toolkit278 provides procedural guidance from the US General Services Administration (GSA). Published in 2022, it is complementary work to the GAO report on AI. The GSA’s AI governance toolkit addresses both data privacy and data governance.

Data privacy and the substantial data collection and use that fuels AI are not always front and center in AI governance tools and guidance. In the GSA’s toolkit, they are. Therein, the process guidance and questions to be considered throughout the AI development life cycle focus on privacy-related issues informed by the US Privacy Act’s Fair Information Practice Principles.279

For instance, the GSA framework provides a series of questions addressing the origin and provenance of data used for AI, as well as whether data is sufficiently representative in order to prevent bias. It also recognizes the need for entities to minimize the amount of sensitive or personally-identifiable data they collect or process. To inspire sensitive data minimization, it asks, “Can you achieve similar/effective results with less (PII) data?”

Americas: Northern America: United States

National Institute of Standards and Technology (NIST)

Artificial Intelligence Risk Management Framework

Tool Type: Practical Guidance and Process Framework with Self-Assessment Questions

The National Institute of Standards and Technology published its Artificial Intelligence Risk Management Framework (RMF)280 and companion AI RMF Playbook in January 2023. The framework and playbook are organized according to four broad categories or functions present in the AI life cycle: Govern, Map, Measure, Manage. See more detailed discussion of this important framework in the Standards section in Part II of this report.

Americas: Northern America: United States

White House

The Blueprint for an AI Bill of Rights and the Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence

Tool Type: Practical Guidance

The Blueprint for an AI Bill of Rights281 was published by the White House Office of Science and Technology Policy in October 2022. Rather than providing prescriptive procedures, AI assessment steps, or technical method recommendations, the Blueprint for an AI Bill of Rights includes a wealth of practical guidance for design, use, and deployment of automated systems based on five principles for protecting people’s rights.

In particular, the voluntary guidance looks to protections mandated by the US Constitution or implemented under existing US laws. It describes protections that should be applied with respect to all automated systems that have the potential to meaningfully impact individuals’ or communities’ ability to exercise civil rights, privacy, and freedom of speech. It also ensures protections from discrimination and unlawful surveillance; and emphasizes access to education, housing, credit, employment, healthcare, and more.

In guidance related to protecting against algorithmic discrimination, the blueprint states that “for every instance where the deployed automated system leads to different treatment or impacts disfavoring the identified groups, the entity governing, implementing, or using the system should document the disparity and a justification for any continued use of the system.”282

About a year after publishing the Blueprint for an AI Bill of Rights, on October 30, 2023, the White House unveiled its Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence.283 The sweeping order calls on multiple US federal agencies, including the Department of Commerce, NIST, the Department of Defense, the Department of Health and Human Services (HHS), the Department of Homeland Security, and others, to devise new standards, procedures, testing and measurement benchmarks, and reporting requirements related to AI safety and security, responsible innovation and competition, privacy and civil rights, worker and consumer protection, and health-related AI impacts.

For example, it calls on HHS to develop an AI assurance policy to evaluate important aspects of the performance of AI-enabled healthcare tools and infrastructure needs for enabling pre-market assessment and post-market oversight of AI-enabled healthcare-technology algorithmic system performance using real-world data.

The order calls for the launch of a pilot program implementing a National AI Research Resource (NAIRR); the program plan should address infrastructure, governance mechanisms, and user interfaces for initial integration of distributed computational, data, model, and training resources to be made available to the research community. In addition to use of private industry technology infrastructure,284 plans for NAIRR also allow for private industry use of the resource.285 286

The order also calls on federal agencies to designate a chief artificial intelligence officer to hold primary responsibility for coordinating their agency’s use of AI, promoting AI innovation in their agency, and managing risks from their agency’s use of AI.

Americas: Northern America: United States

US-Based Multistakeholder/Corporate Programs

Open Loop

Overseen by prominent corporate AI developer Meta, Open Loop is an experimental, multinational, multistakeholder tech governance program.287 It involves private AI startups and representatives from academia, civil society, and governments. The program aims to “co-create and test new governance frameworks through policy prototyping programs, and to support the evaluation of existing legal frameworks through regulatory sandbox exercises.”288

Meta’s Open Loop team coordinated in 2022 and 2023 with AI startups, academia, and civil society and trade groups, as well as Uruguay’s Agency for Electronic Government, the Information and Knowledge Society, and the Inter-American Development Bank, to test operational guidance for implementing Privacy Enhancing Technologies to reduce privacy risks of AI and other types of technical systems.289

Open Loop also recently led a policy prototyping effort on AI Transparency and Explainability in partnership with Singapore’s Infocomm Media Development Authority and Personal Data Protection Commission. A July 2022 report about the initiative details a variety of algorithmic model types, the degree to which they can be interpreted for explainability purposes, and the limitations of their interpretability.290 The report also addresses the limits of counterfactual explanatory strategies and SHapley Additive exPlanation, noting “several drawbacks of SHAP”291

The Meta-led Open Loop program also worked with several organizations in Europe to explore and test “alternative policy frameworks and regulatory pathways” for gauging the impacts of automated decision systems.292 In part a response to EU legislative proposals calling for AI risk assessment, Open Loop’s impact assessment “playbook” includes a step-by-step risk assessment methodology and examples of risk “mitigation” measures.293 A detailed overview of the project is featured in Open Loop’s January 2021 report, AI Impact Assessment: A Policy Prototyping Experiment.

The report presents steps for quantifying the severity of risks associated with automated systems, and suggests a “proper metric” for measuring AI system accuracy that takes into account the tradeoffs “between recall (no false negatives) and precision (no false positives).”294 The report also highlights Facebook’s previously stated desire for “self-assessment of AI risk” 295 and recommends that regulation requirements vary “in accordance with the specific AI application in question and the level and extent of the risks assessed, alongside the calculus of the benefits that application brings.”296

Open Loop’s AI Impact Assessment report also presents a prototype for a law that would call for AI developers to consult supervisory authorities prior to deployment of automated decision-making systems if their assessments, possibly conducted by the AI developer itself, indicate a high risk.297

Partnership on AI

About ML Program

The Partnership on AI is a non-profit organization founded in 2016 by Amazon, DeepMind, Google, Microsoft, IBM, and Meta (at the time known as Facebook).298

As part of its About ML program, the organization evaluated 152 explainable AI, or XAI, tools intended to help explain why AI systems make decisions. The initial outcome of that effort, published in 2022, was a framework for documentation of XAI tools according to 22 tool dimensions such as tool type, intended users, technical compatibilities, and datasets used to build the tools.299 In addition, the analysis identified dimensions of usability, such as the scope and formats of explanations the tools produced.

More recently in 2023, PAI published case studies of pilot projects involving ML life cycle documentation under its About ML initiative.300

The group also created a framework in 2021 to help guide decisions regarding AI data sourcing and related services.301

Figure 7: Dimensions of Explainable AI Tools Usability

(Source: Partnership on AI)

Asia: Eastern Asia: Japan

Government of Japan: Governance Guidelines for Implementation of AI Principles

Japan has been very active in the area of AI tools and governance, with notable AI governance efforts becoming public as early as 2016. In 2019, Japan published its Social Principles of Human-centric AI, which played a role in shaping the OECD AI Principles.302 Version 1.1 of its detailed principles for implementing AI in society, Governance Guidelines for Implementation of AI Principles, published in 2022,303 address risk analysis in relation to social aspects of AI and gap analysis in system design, as well as management of AI systems. Unique to the Japanese approach are the segments regarding evaluation, verification, and re-analysis of conditions and risks—which are classic, advanced governance principles.

Appendix 1 of the Guidelines includes specific action targets and practical examples, as well as a section on how to implement agile governance in AI systems. This is a unique contribution to AI governance tools.

Government of Japan: Contract Guidelines on Utilization of AI and Data

Japan’s Ministry of Economy, Trade, and Industry in 2019 created non-binding guidelines for how risk mitigation—including consumer and social protections, transparency, and safety thresholds—should be included in contracts for the development or utilization of AI software.304

Monetary Authority of Singapore (MAS)

Veritas Initiative

Tool Type: Practical Guidance and Process Framework with Self-assessment Questions and Technical Code

The Monetary Authority of Singapore (MAS), along with the financial industry, co-created what is now known as the MAS 2018 Principles to Promote Fairness, Ethics, Accountability and Transparency (FEAT)305 in late 2018. Work by MAS to develop the FEAT principles as applicable to the financial sector is already in use regionally through the Asia Development Bank.

Figure 8: Financial AI and Data Analytics Model Management

(Source: Monetary Authority of Singapore)

To provide further concrete guidance regarding how to implement these principles in the financial sector, MAS worked with a consortium of financial institutions and technology companies. This consortium, called the Veritas consortium, was launched in November 2019 with the specific goal of cooperating to provide financial institutions verifiable ways of incorporating FEAT principles, including tools.306 The initiative aims to devise a framework for financial institutions to promote adoption of responsible AI and data analytics.

The Veritas initiative began work on Phase 2 of the effort in January 2021, specifically addressing use cases regarding credit risk scoring, customer marketing, predictive underwriting, and fraud detection. These use cases were evaluated in relation to methods used to promote the FEAT principles.307 Notably, the MAS implementation report, published in 2022,308 provides a discussion of observations of the implementation of fairness principles by financial institutions in selected use cases.

The ongoing Veritas effort includes technical code for executing a fairness and transparency diagnosis of specific use cases. It also includes a question-based and procedural method for assessing fairness risks, along with practical guidance regarding the implementation of fairness principles for use of AI and machine learning by financial institutions.

In addition, FEAT Fairness Principles Assessment Methodology309 guidance published in 2022 mentions IBM’s AI Fairness 360310 and Microsoft’s Fairlearn,311 an open-source set of fairness metrics, algorithms, and other resources originally created by Microsoft Research.312 The guidance also references use of the Four-Fifths Rule to measure disparate impact and lists several examples of tools that employ it. As noted in the Findings section of this report, AI fairness methods based on the rule have drawn sharp criticism in scholarly literature reviewed here in Part I.

An earlier FEAT Fairness Principles Assessment Methodology published in 2020 acknowledges that “all fairness measures capture different notions of fairness, each with their own limitations. Which of these notions of fairness are important for a particular system, and whether any are sufficient, is a context-dependent ethical question with no objective or universally accepted answer.”313

Asia: South-Eastern Asia: Singapore

Singapore Infocomm Media Development Authority

AI Verify

Tool Type: Practical Guidance and Technical Framework with Technical Software

Compared to other tools reviewed in this report, Singapore has taken what might be considered the most prescriptive approach to implementing AI principles. What stands out is Singapore’s development of a detailed technical testing framework and use of software designed for its program.

Singapore’s Infocomm Media Development Authority in 2022 launched the international pilot of AI Verify,314 an AI governance testing framework that includes software. In addition to process guidance, AI Verify includes open-source plugin tools addressing AI robustness, fairness, and explainability. The AI Verify program is intended for companies’ use to validate the performance of their AI systems through standardized self-testing.

The program also features a process checklist addressing data governance, explainability, fairness, human agency and oversight, inclusive growth and societal and environmental well-being, reproducibility, robustness, safety, security, and transparency.

AI Verify was developed based on internationally-recognized AI principles from OECD as well as principles from the European Union.315 AI Verify also integrates Singapore’s own Model AI Governance Framework, established by its Personal Data Protection Commission in 2019, and updated in 2020.316

Thus far, companies in the financial, healthcare, human resources, and technology sectors have tried the AI Verify toolkit.317

In June 2023, Singapore’s IMDA established the not-for-profit AI Verify Foundation318 as a forum where people from the international open-source community could contribute to ongoing development of AI Verify testing frameworks, codebase, standards, and best practices. AI Verify developer tools are available to the open-source software community.319

The foundation’s “premier” members tasked with setting strategic direction and a development plan for AI Verify include Aicadium, Google, IBM, IMDA, Microsoft, Red Hat, and Salesforce. The Foundation also has more than 60 general members.320

A 2022 AI Verify project pilot document323 stated that AI Verify included the Adversarial Robustness Toolbox and AI Fairness 360, both of which were originally developed by IBM and donated in 2020 to the Linux Foundation AI Foundation.324 It also included Fairlearn, an open-source set of metrics, algorithms, and other resources originally created by Microsoft Research.325

As noted in the Findings section of this report, use of SHAP for AI explainability, as well as research that formed the foundation of the AI Fairness 360 Disparate Impact Remover algorithm, have drawn sharp criticism in scholarly literature reviewed here in Part I.

According to AI Verify documentation, AI Verify “does not define ethical standards” and “does not guarantee that any AI system tested under this Framework will be free from risks or biases or is completely safe.”326

Asia: South-Eastern Asia: Singapore

Singapore Infocomm Media Development Authority

Generative AI (Gen AI) Evaluation Sandbox Evaluation Catalogue

Tool Type: Practical Guidance

Singapore’s Infocomm Media Development Authority is among the first government agencies to recommend specific approaches to evaluating generative AI systems. On October 31, 2023, along with the AI Verify Foundation, the authority introduced its Generative AI (Gen AI) Evaluation Sandbox,327 which features an Evaluation Catalogue including baseline methods and recommendations for Large Language Models (LLMs).328

Although the Evaluation Catalogue notes the limitations and early stage of development of evaluation methods for LLMs, it nonetheless recommends specific evaluations including approaches that produce scores. The recommended evaluation methods directly reflect the 11 AI ethics principles listed in the AI Verify Framework, which include explainability, reproducibility, robustness, fairness, data governance, human agency and oversight, and security. In some cases the principles are translated to concepts more applicable to LLMs, such as factuality, bias, and toxicity generation.

Specific LLM evaluation and benchmarking methods mentioned in the Evaluation Catalogue include TruthfulQA, a benchmark developed by researchers at OpenAI and the University of Oxford329 that measures whether a language model is truthful in generating answers to questions; Bias Benchmark for QA (BBQ), a dataset of question sets intended to highlight social biases against people belonging to protected classes in Natural Language Processing model outputs;330 and Perspective, which uses machine learning models to produce numerical scores assessing levels of toxicity, insults, threats, and sexual explicitness. Perspective was created by Google’s Jigsaw unit and Google’s Counter Abuse Technology team.331

Notably, the Evaluation Catalogue states that LLM evaluation techniques tend to be Western-centric, and should consider user demographics and cultural sensitivities. It also notes the risks of “imprecise or faulty benchmarks” that “could potentially lead to developers being blindsided to critical areas of deficiency.”332 In addition, it states that evaluation recommendations “should not be taken as an endorsement of the reliability and validity of the identified evaluation and testing approaches.”333

The document also presents a taxonomy of evaluations assessing the capabilities of LLMs, such as evaluations used for Natural Language Understanding, LLM reasoning, evaluations related to LLMs for specific domains such as finance and healthcare, and evaluations related to LLM bias, robustness, and data governance.

As part of its taxonomy, the Evaluation Catalogue references work on extreme risks of LLMs, featured in a May 2023 paper focused on risks of so-called “frontier” AI models;334 the paper was authored by researchers from private sector corporations building LLMs, including Anthropic, Google’s DeepMind, and OpenAI, as well as other organizations focused on researching the existential risks (or “x-risks”) of AI, including the Centre for Long-Term Resilience and the Alignment Research Center.

Singapore’s Catalogue suggests that a baseline set of evaluations should be used to define a minimal level of LLM safety and trustworthiness for LLMs before deployment. However, it also notes that only advanced, large-scale general purpose machine learning models need to undergo evaluations in its Extreme Risks category; here, the document acknowledges the influence of the Frontier Model Forum, a private sector group comprised of Anthropic, Google, Microsoft, and OpenAI.335

Several technology corporations joined the Sandbox, such as Amazon Web Services, Anthropic, Google, IBM, Microsoft, and IBM, as well as AI app developers and third-party AI testers including consulting firms Deloitte and EY.

Asia: Southern Asia: India

NITI Aayog

The Responsible AI #AIFORALL Approach Document for India Part 1 – Principles for Responsible AI

Tool Type: Practical Guidance with Self-assessment Questions

India’s NITI Aayog is a government body formed in 2015 to provide a platform to discuss intersectoral, interdepartmental, and federal issues to accelerate the implementation of India’s national development agenda.336

NITI Aayog in 2021 published its two-part Responsible AI #AIFORALL Approach papers, which identified principles for responsible design, development, and deployment of artificial intelligence in India. These papers also included enforcement mechanisms for the operationalization of India’s Responsible AI principles.

Part 1, published in February 2021, includes a Self-Assessment Guide for AI Usage.337 The self-assessment guide features a series of consideration-based questions organized according to various stages of the AI development life cycle: Problem Definition and Scoping, Data Collection, Bias in Data Labelling, Model Selection, Training, Evaluation, Deployment and Ongoing Monitoring.

Each consideration set features one or more general approaches to reducing negative impacts of an AI system. For instance, the description of a “mitigation strategy” for identifying a way to address errors in decisions by an AI system states, “If the potential degree of harm for a decision is expected to be high, have appropriate mechanisms in place so stakeholders can contest and humans can get involved in the decision making process.”338

Some proposed strategies are more specific. A strategy intended to address privacy in relation to data collection states, “Create and document a process to continually scan for and identify new sources of personal and/ or sensitive data.”339

The self-assessment process also provides guidance for monitoring AI systems after deployment. It suggests tracking system performance and changes over time, and ensuring that mechanisms are in place to allow third-party agencies to review system behavior. 340

In addition to briefly mentioning LIME and SHAP in relation to AI explainability, 341 NITI Aayog’s document also refers to specific methods for model transparency, including Google’s Model Card Toolkit, IBM’s Fact Sheet for AI governance, and Datasheets for Datasets.342

The second portion of the paper, published in August 2021, addresses policy approaches for establishing Responsible AI use in India.343

India: Tamil Nadu State

Tamil Nadu State Policy for Safe and Ethical AI

Tool Type: Practical Guidance with Self-assessment Questions and Scoring Output

India’s Tamil Nadu State Policy for Safe and Ethical AI, published by the Tamil Nadu Information Technology Department in 2020,347 provides a plan for adoption of AI-based systems for Tamil Nadu’s policymakers. In particular, the policy recommends use of a scoring system and features a framework for evaluation of AI-based systems addressing transparency, accountability and legal issues, misuse protection, data deficiencies, and fairness and equity.

The policy proposes use of the “DEEP-MAX Scorecard,” described as “a transparent point-based rating system for AI Systems.” Scoring is based on a set of parameters that form the DEEP-MAX acronym: Diversity, Equity and Fairness, Ethics, Privacy and Data Protection, Misuse Protection, Audit and Transparency, and Cross Geography and Society.

Figure 9: DEEP MAX Scorecard

(Source: Tamil Nadu Information Technology Department)

The “Cross” in Cross Geography is represented by the “X” in DEEP-MAX, and is also referred to in the document as Digital Divide and Data Deficit. This Cross Geography criteria addresses how well a system performs across geographies and societies. This is a concept that matters everywhere: particularly in India, where the world’s largest population represents several distinct cultures, languages, and customs.

Details of the process used to determine scores are not included in the policy; however, it does note that testing with “suitably designed test data sets” might be used.348 The policy also proposes use of a blockchain-based system for storing DEEP-MAX scores for all AI systems used in the public domain, to be designed by Tamil Nadu’s e-Governance Agency (TNeGA).349

The Tamil Nadu State Policy for Safe and Ethical AI is applicable to Tamil Nadu government authorities, government-controlled organizations and partnerships, and joint venture companies of the government.350 It states that implementation of the guidelines would be overseen by a Safe and Ethical AI monitoring committee headed by a chief secretary, with members consisting of Secretary and Senior Officers of select government departments along with AI and policy experts from academic and research institutions. 351

Asia: Western Asia: United Arab Emirates: Dubai

Digital Dubai

AI System Ethics Self-Assessment Tool

Tool Type: Practical Guidance with Self-assessment Questions and Scoring Output

The AI System Ethics Self-Assessment Tool352 from the government of Dubai353 is intended to assist organizations developing or using AI systems to evaluate the “ethics level” of those systems and reduce potential problems that run afoul of the country’s AI Ethics Guidelines.354

The tool, from Dubai’s Digital Authority, Digital Dubai, incorporates a series of questions intended to identify whether the AI system under evaluation is used to make insignificant, significant, or critical decisions.

The self-assessment relies on the assessor to answer “yes” or “no” to statements such as, “Mitigating measures have been pursued to ensure individuals in the same circumstances receive equal treatment.”

Depending on the assessor’s responses, the tool suggests ways to address potential problems. When the questionnaire is complete, the tool generates a set of scores intended to assess the performance levels of the system under evaluation in relation to fairness, accountability, and transparency.

In its current beta stage, Dubai’s AI System Ethics Self-Assessment Tool is for self-assessment purposes only and will not be audited, checked, or regulated.

Figure 10: AI System Ethics Self-Assessment Tool Report

(Source: Government of Dubai)

Europe: Northern Europe: United Kingdom of Great Britain and Northern Ireland

UK Information Commissioner’s Office

AI and Data Protection Risk Toolkit

Tool Type: Practical Guidance & Process Framework with Scoring Output

The AI and Data Protection Risk Toolkit from the UK Information Commissioner’s Office (ICO) is intended to help AI developers and others assessing AI systems to gauge and reduce the risks of those systems to individual rights and freedoms. It is also intended to complement data protection impact assessments legally required where data processing is likely to result in high risk to individuals.

Originally launched in 2021355 and updated in March 2023,356 the AI and Data Protection Risk Toolkit357 takes the form of a spreadsheet that organizes steps in the AI life cycle: from design, to data acquisition and preparation, to training and testing, followed by deployment and monitoring. Each life cycle category includes risk areas related to issues such as accountability, security, and purpose limitation, all directly referencing specific articles in the UK General Data Protection Regulation.358

For each risk area, the process flow detailed in the spreadsheet outlines a control mechanism. Examples of control mechanisms include “Conduct a data protection impact assessment (DPIA)” or “Document clear audit trails of how personal data is moved and stored from one location to another during the training and testing phase.” The suggested controls are accompanied by practical steps for implementation.

The toolkit is not mandatory for AI developers or users, and the ICO makes a point of noting that it “is not designed to be ‘one size fits all.’” It adds, “There may also be additional risks that apply to your context that are not included in this toolkit.”359

Europe: Northern Europe: United Kingdom of Great Britain and Northern Ireland

Centre for Data Ethics and Innovation

CDEI portfolio of AI assurance techniques

The UK’s Centre for Data Ethics and Innovation (CDEI), part of its Department for Science, Innovation, and Technology, unveiled in June 2023 the CDEI portfolio of AI assurance techniques.360 Intended to provide guidance to people involved in designing, developing, deploying, or procuring AI-enabled systems, the portfolio features descriptions and case studies highlighting various AI assessment methods and products addressing fairness, explainability, robustness, redress, and other AI considerations.

The collection of AI governance tools was developed in conjunction with TechUK,361a tech industry trade association. Some products and techniques sold by TechUK member companies are featured in the CDEI portfolio.362 The portfolio spotlights impact assessments, bias and compliance audits, and performance testing services.363

Some techniques featured in the CDEI portfolio mention use of specific quantifiable measures, including SHAP.364 As noted in the Findings section of this report, use of SHAP in AI measurement has drawn sharp criticism in scholarly literature reviewed here in Part I.

According to the CDEI’s description of its portfolio, “the inclusion of a case study in the portfolio does not represent a government endorsement of the technique or the organisation, rather we are aiming to demonstrate the range of possible options that currently exist.” The agency plans to publish subsequent editions of its portfolio in the future.

Oceania: Australia and New Zealand: Australia

Australia Commonwealth Ombudsman’s Office

Automated Decision-making Better Practice Guide

Tool Type: Practical Guidance with Self-assessment Questions

Australia’s Automated Decision-making Better Practice Guide365 was originally published by the Commonwealth Ombudsman’s Office in 2007, long before automation software and systems incorporated machine learning and AI capabilities.

In part due to recognition of AI’s impact, the guide was updated in 2019.366 The updated version considers big data analytics, AI, and machine learning to be features of automated systems.

The guide includes a lengthy checklist of considerations that should be addressed when government agencies implement or update use of an automated system for administrative decision-making.367 Checklist topics address administrative law, privacy, system design and governance, system maintenance and upgrades, quality assurance assessment, and audit trails for transparency and accountability.

Oceania: Australia and New Zealand: New Zealand

The Ministry for Social Development of New Zealand

Model Development Lifecycle and Privacy, Human Rights and Ethics framework (PHRaE)

Tool Type: Practical Guidance with Self-assessment Questions and Scoring Output

New Zealand’s Ministry of Social Development created the Model Development Lifecycle (MDL) as a practical guide in March 2022368 to help manage operational algorithms such as AI-powered services and automated decision systems. Its operational algorithm governance structure, a guide for which was published in 2021,369 includes three approval stages or “gates” designed to identify and reduce risk throughout the life cycle of an operational algorithm. According to that guide, titled Governance Guide, Model Development Lifecycle, in order to progress from a peer review phase to deployment of an algorithmic model, project managers and coordinators must receive approval for sign-off at each gate based on how and whether identified risks have been addressed.

The MDL process for managing operational algorithms features a risk classification matrix intended to determine the risk level of a project. The matrix involves a self-assessment to gauge the severity of impact should a risk occur, and likelihood of the risk occurring. While it does not produce a quantifiable score, the MDL process establishes the risk classification level of a project.

Once risks are identified and classified, controls may be put in place to reduce them, and further external technical, legal, or ethical reviews by a Technical Advisory Group may be recommended.

This recent work follows New Zealand’s Algorithm Assessment Report of 2018. For the report, New Zealand’s chief data steward and the government chief digital officer assessed existing algorithms and their uses across government agencies. The report provides a summary of self-reported information submitted by 14 government agencies about the algorithms that they use to deliver their functions.374

Figure 11: Model Development Lifecycle Operational Algorithm Governance Structure

(Source: New Zealand Ministry of Social Development)Appendix A: AI Governance Tool Types Lexicon

This report uses the term AI Governance Tools, defined here as:

AI Governance Tools – Socio-technical Tools for mapping, measuring, or managing AI systems and their risks in a manner that operationalizes or implements trustworthy AI.

This definition encompasses the wide array of formats and methods reviewed here in Part II. This lexicon of AI Governance Tool Types further distinguishes differences among the various types of these formats and methods.

• These AI governance tool types were developed on the basis of evidence gathered through the research conducted for this report. These tool types reflect the commonly found components of the AI governance tools identified and named in the Part II review of tools. For more insight into this process, please refer to the AI Governance Tools and Features Comparison Chart (See Appendix B). Readers will also find a tool type listed alongside selected relevant tool review entries in Part II of this report.
• In general, all of these tool types are designed to improve or measure AI systems, particularly in relation to AI principles including fairness, explainability, and robustness. When applying these types to specific AI governance tools, they may be combined to form hybrid types depending on the components of each tool in question.

AI Governance Tool Types:

• Practical Guidance – Includes general educational information, practical guidance, or other consideration factors

Self-assessment Questions – Includes assessment questions or detailed questionnaire

Procedural Framework – Includes process steps or suggested workflow for AI system assessments and/or improvements

Technical Framework – Includes technical methods or detailed technical process guidance or steps

Technical Code or Software – Includes technical methods such as use of specific code or software

Scoring or Classification Output – Includes criteria for determining a classification, or a mechanism for producing a quantifiable score or rating reflecting a particular aspect of an AI system

Catalog – A collection of multiple AI governance tools and types

Appendix B: AI Governance Tools and Features Comparison Chart

Our AI Governance Tools Comparison chart spotlights key features of select AI Governance Tools from national governments and multilateral organizations. The features indicated here directly map to the tool types we assign to each tool, and reflect our AI Governance Tool Lexicon featured in Appendix A.

		Practical guidance	Assessment questions	Process steps	Technical process guidance, code or software	Score or classification output	Mentions specific metrics, code or software
Australia 2019	Automated Decision-making Better Practice Guide TYPE: Practical Guidance with Self-assessment Questions	✓	✓
Canada 2019	Algorithmic Impact Assessment tool TYPE: Practical Guidance with Self-assessment Questions & Scoring Output	✓	✓			✓
Chile 2022	AI Procurement Directorate TYPE: Practical Guidance	✓					✓
Dubai 2019	AI System Ethics Self-Assessment Tool TYPE: Practical Guidance with Self-assessment Questions & Scoring Output	✓	✓			✓
Ghana 2023	FACETS Framework TYPE: Practical Guidance with Self-assessment Questions & Scoring Output	✓	✓			✓
India 2021	The Responsible AI Approach Document for India Part 1 TYPE: Practical Guidance with Self-assessment Questions	✓	✓				✓
Tamil Nadu, India 2020	Policy for Safe and Ethical AI TYPE: Practical Guidance with Self-assessment Questions & Scoring Output	✓				✓
Latin America & Caribbean/IDB 2019	fAIr LAC in a box TYPE: Catalog	✓	✓	✓	✓		✓
Global/OECD 2023	Catalogue of AI Tools and Metrics to Promote Trustworthy AI TYPE: Catalog	✓	✓	✓	✓	✓	✓
New Zealand 2020	Model Development Lifecycle TYPE: Practical Guidance with Self-assessment Questions & Scoring Output	✓	✓	✓		✓
Singapore 2022	AI Verify TYPE: Practical Guidance with Technical Framework & Software	✓	✓	✓	✓		✓
Singapore 2022	Veritas Initiative TYPE: Practical Guidance & Process Framework with Self-assessment Questions & Technical Code	✓	✓	✓	✓		✓
Singapore 2023	Generative AI Evaluation Catalogue TYPE: Practical Guidance	✓					✓
UK 2021	AI and Data Protection Risk Toolkit TYPE: Practical Guidance & Process Framework with Scoring Output	✓		✓		✓
US 2021	Artificial Intelligence: An Accountability Framework TYPE: Practical Guidance & Process Framework with Self-Assessment Questions	✓	✓	✓			✓
US 2022	Artificial Intelligence Governance Toolkit TYPE: Practical Guidance & Process Framework with Self-Assessment Questions	✓	✓	✓
US 2022	Blueprint for an AI Bill of Rights TYPE: Practical Guidance	✓
US 2023	Artificial Intelligence Risk Management Framework TYPE: Practical Guidance & Process Framework with Self-Assessment Questions	✓		✓			✓

Appendix C: Some AI Governance Tools Feature Off-label, Unsuitable, or Out-of-context Uses of Measurement Methods:

A detailed explanation of Finding 2.

As noted in the Findings section of this report, 7 of 18—or more than 38%—of select AI governance tools reviewed in detail in Part II either mention, recommend, or incorporate at least one of three measurement methods shown in scholarly literature to be off-label, unsuitable, or out-of-context when applied to measure AI systems.

This finding demonstrates the connections between Parts I and II of this report. Analysis gathered through scholarly literature reviewed in Part I informed some criteria used in reviews of AI governance tools surveyed in Part II.

In particular, the criteria used in reviews of AI governance tools drew from the literature presented in the two use cases featured in Part I. These use cases expose and analyze problems associated with off-label use of the US Four-Fifths or 80% Rule to measure disparate impact in AI, and the use of SHAP and LIME in AI explainability measures.

Here, the term “off-label” refers to applications of methods that fall outside of the scope of the original intended applications, such as with “off-label” use of prescription drugs in clinical settings.375

Explaining the review sample

More than 30 AI governance tools and AI governance tool-adjacent items were reviewed for Part II of this report. Of those, 18 fully satisfied the definition of AI governance tools introduced in this report:

AI Governance Tools – Socio-technical tools for mapping, measuring, or managing AI systems and their risks in a manner that operationalizes or implements trustworthy AI.

As featured in Appendix A, the AI Governance Tool Type Lexicon created for this report introduces tool types present in the 18 tools reviewed in depth: Practical Guidance, Self-assessment Questionnaires, Process Frameworks, Technical Frameworks, Technical Code and Software. Also see the AI Governance Tools Types and Features Chart in Appendix B for more detail.

Explaining the Calculations

• The review of the select 18 AI governance tools found that 7 — or more than 38% — of those select tools mention, recommend, or incorporate use of at least one of the three problematic measures reviewed in the Part I use cases: SHAP, LIME, and fairness tools that base disparate impact measures on the US Four-Fifths Rule (or mention specifically the US Four-Fifths Rule and its 80% threshold). Seven represents just over 38.8% of 18.

Discovering off-label fairness measures hidden in AI governance tools

• Off-label applications in AI governance tools are not always obvious. While it appears that only one of the 18 AI governance tools directly names the Four-Fifths Rule, several of them mention, recommend, or incorporate brand-name AI fairness measures that base disparate impact measures on the US Four-Fifths Rule. In particular, the Part II review found three such brand-name AI fairness measurement methods: Aequitas, AI Fairness 360, and BlackBoxAuditing.

Aequitas

• Aequitas376 is an AI fairness measure referenced in AI governance tools reviewed in Part II of this report. The open-source bias measure developed by the Center for Data Science and Public Policy at University of Chicago is based on 2018 research377 that suggests use of the Four-Fifths Rule’s 80% benchmark to measure disparity.

AI Fairness 360

AI Fairness 360, or AIF360, was originally developed by IBM and donated in 2020 to the Linux Foundation AI Foundation.378 AIF360 supports several “bias mitigation algorithms,” including a Prejudice Remover Regularizer and Adversarial Debiasing algorithm. At issue here, the documentation for its Disparate Impact Remover algorithm379 380 specifically cites 2015 research introducing a disparate impact measurement based on the Four-Fifths Rule’s 80% benchmark.381

BlackBoxAuditing

• BlackBoxAuditing is another AI fairness measure referenced in one of the 18 AI governance tools. The BlackBoxAuditing repository382 bases its disparate impact “repair process” on the aforementioned 2015 research that incorporates the Four-Fifths Rule’s 80% benchmark.383

Seven specific AI governance tools featuring off-label measures found in the Part II survey

A total of 7 of the select 18 tools reviewed in depth for Part II of this report mention, recommend, or incorporate the use of SHAP and/or LIME for AI explainability, use of the US Four-Fifths Rule’s 80% threshold, or specifically name or recommend AI Fairness 360, Aequitas or BlackBoxAuditing.

Figure 12: AI Governance Tools Including Off-Label Measures

Government or Organization	Name of AI Governance Tool	Off-Label Measurement Methods Mentioned
Chile (ChileCompra)	AI Procurement Directorate	LIME
Inter-American Development Bank	fAIr LAC in a box/Responsible Use of AI for Public Policy: Data Science Toolkit	Shapley Values
India (NITI Aayog)	The Responsible AI Approach Document for India Part 1	SHAP, LIME
Monetary Authority of Singapore	Veritas Initiative FEAT Fairness Principles Assessment Methodology	Aequitas, AI Fairness 360, SHAP, LIME
Singapore Infocomm Media Development Authority	AI Verify	AI Fairness 360, SHAP
US Government Accountability Office	Artificial Intelligence: An Accountability Framework for Federal Agencies and Other Entities in 2021	AI Fairness 360
Organization of Economic Cooperation and Development	Catalogue of Tools and Metrics for Trustworthy AI	15 entries in the catalog include off-label measures including Aequitas, AI Fairness 360, BlackBoxAuditing, and several entries using SHAP and/or LIME, including InterpretML

(Source: World Privacy Forum, Research: Kate Kaye, Pam Dixon. Image/Data Visualization: John Emerson.)

Chile’s Standard Bidding Terms for Data Science and AI Projects mentions that LIME could be used as an explanation tool.384

2. IDB FairLAC’s Responsible Use of AI for Public Policy Data Science Handbook mentions use of Shapley Values as a quantitative explainability method for deep neural networks, and includes them in a detailed workbook section.385

3. India’s Responsible AI #AIFORALL Approach Document for India Part 1 – Principles for Responsible AI report mentions LIME and SHAP in relation to AI explainability.386

4. Monetary Authority of Singapore’s FEAT Fairness Principles Assessment Methodology, part of its Veritas Initiative, mentions AI Fairness 360 and Aequitas as commonly-used open-source AI fairness tools, and references use of the “four-fifths rule” to measure disparate impact.387 A related, earlier Veritas document states that LIME or SHAP could be used for AI explainability, but cautions that “these and other explainability techniques may themselves introduce bias .”388

• 5. Singapore’s AI Verify software features a technical plugin that uses SHAP to explain how AI system features affect overall predictions.389 The 2022 AI Verify project pilot document stated its inclusion of SHAP and AI Fairness 360 in its tool package.390
• 6. The US Government Accountability Office’s report, Artificial Intelligence: An Accountability Framework for Federal Agencies and Other Entities in 2021, mentions AI Fairness 360 as an example of guidance that might be considered when defining responsible AI goals and objectives.391

7. The OECD’s Catalogue of Tools and Metrics for Trustworthy AI is counted as a single tool in the Finding tally of 7 select tools that mention the use of off-label AI measures; however, this tool catalog itself includes 15 entries featuring off-label measures. These include the following AI measurement methods:

Aequitas392

AI Fairness 360393

BlackBoxAuditing394

InterpretML395

Shapley Additive Explanation (SHAP)396

Shapley Variable Importance Cloud (ShapleyVIC)397

Beta Shapley398

Data Shapley399

SHAP400

Shapley Explanation Networks401

Fastshap402

Shapley403

Shapley Additive Global Importance (SAGE)404

Local Interpretable Model-agnostic Explanation (LIME)405

Lime406

Appendix D: OECD Catalog of Tools and Metrics Framework

Figure 13. Tools for trustworthy AI: A framework to compare implementation tools for trustworthy AI systems, Page 15.

Source: OECD, June 2021, https://www.oecd-ilibrary.org/docserver/008232ec-en.pdf .

---

Footnotes