AI Governance on the Ground: Chile’s Social Security and Medical Insurance Agency Grapples with Balancing New Responsible AI Criteria and Vendor Cost

WPF’s AI Governance on the Ground Series 

This series highlights and expands on topics and issues from WPF’s Risky Analysis report and survey of AI Governance Tools around the world.

Download AI Governance on the Ground: When Purchasing AI, Chile’s Social Security and Medical Insurance Agency Grapples with Balancing New Responsible AI Criteria and Vendor Cost (PDF, 7 pages )

When Purchasing AI, Chile’s Social Security and Medical Insurance Agency Grapples with Balancing New Responsible AI Criteria and Vendor Cost

By Kate Kaye, Deputy Director

1 November 2024

Around 80 people working at Chile’s social security and medical insurance agency hold the livelihoods and future health of thousands of people in their hands. These medical claims processors, some of whom are doctors and health specialists, are responsible for deciding whether or not to cover expenses such as employee wages during medical leave and occupational mental health related costs. They are under pressure from a heavy workload; around 200,000 claims crossed their desks last year. When payments are denied, those decisions can and often do have life-changing effects.

Now, like many government agencies these days, Chile’s Department of Social Security Superintendence (SUSESO) — a division of the country’s Ministry of Labor that manages its social and health system — has begun to use machine learning models to help its limited staff process an abundance of medical claims. The idea is to streamline claims processing and generally automate certain parts of the claims cycle.

Behind the scenes, one particular AI governance tool that automates certain aspects of procurement has played an important role. The tool, a mandatory bidding template and process for government AI purchases, was used to determine the vendors chosen by SUSESO to develop and assess its medical claim AI models.

The procurement tool — and its evaluation criteria related to vendor cost and competition, AI bias, transparency, and data protection — is also influencing another major decision. That is whether the humans working at SUSESO will decide to approve or deny a medical claim, or whether that life-changing decision will be totally automated by a machine learning model. Those questions are also exposing some emerging AI governance policy gaps.

As World Privacy Forum found in our December 2023 Risky Analysis report surveying and analyzing AI governance tools around the world, the minute decisions, measurements and methods embedded inside the tools used to govern AI systems directly affect whether policy implementations actually align with policy goals. SUSESO’s experience using the AI bidding template, and questions inside the agency regarding how to weigh traditional tech procurement criteria such as vendor cost along with newer responsible AI criteria like discriminatory impacts, give a glimpse of the AI governance challenges happening on the ground today. The tensions this particular Chilean government agency is dealing with may be a sign of what other organizations around the world could encounter as they put their own responsible AI policies into practice and navigate the policy implications of AI-facilitated decision making

Balancing Cost and Competition with Responsible AI Considerations

As Head of SUSESO’s Technology and Operations Department Digital Transformation, Innovation and Project Unit, Moya must balance technical factors and traditional government procurement concerns of cost and competition with legal and ethical considerations. The unit serves as a technical counterpart in Chile’s work with the Ethical Algorithms Project (Proyecto  Algoritmos Éticos) at GobLab UAI, the public innovation laboratory of Chile’s Universidad Adolfo Ibáñez’s School of Government. For this story, World Privacy Forum spoke in August and October with Moya, along with Daniel Leon, his colleague in SUSESO’s Digital Transformation, Innovation and Project Unit, who helped with some English translation.

Moya manages SUSESO tech projects including two ongoing AI-related projects intended to help speed up resolution of medical claims; SUSESO is a last-resort administrative backstop for appealing medical leave related claims. Moya said the majority of those claims are made by people of lower income who may need the medical leave wages to afford to pay other bills and feed themselves and their families, so the tech projects he manages can have serious impacts. Today, Moya said the agency has approximately 20,000 claims awaiting decisions.

Managing Two AI Projects

Moya shared his experiences overseeing management of two ongoing AI-related projects intended to help speed up claims resolution at SUSESO, both of which involved a bidding process and contracts with vendors established in 2023:

• A Medical Claims Machine Learning Model: One of those projects currently in the development phase will produce and implement a predictive machine learning model intended to optimize management and resolution of medical leave claims. Those claims can be made by any type of worker employed in private industry or government. SUSESO anticipates deploying the model, which uses gradient boosting, in the coming months once it is integrated into the agency’s larger claim system.
• A Mental Health Claims Machine Learning Model Audit: Another project involves auditing of an existing labor insurance mental health predictive machine learning model used by SUSESO to resolve occupational mental health claims involving things such as depression related to workplace harassment or job stress. The audit is intended to assess the model, which uses classification trees, according to criteria including bias and transparency. Moya said SUSESO receives a lot of occupational mental health claims it needs to process.

This is where ChileCompra, Chile’s purchasing and public procurement directorate or agency, comes in. ChileCompra established its Standard Bidding Terms for Data Science and AI Projects in December 2022 as part of the country’s broader Ethical, Responsible, and Transparent Algorithms Project. For about a year, use of that template for vendor bids on data science and AI projects was mandatory; it required statistical equity and risk analysis and documentation to assist in detecting bias in training data and models, and was integrated as part of ChileCompra’s broader procurement platform. World Privacy Forum featured the bidding terms template in our survey of AI governance tools included in our December 2023 Risky Analysis report. (Pages 64-65.)

During the time it was mandatory, the procurement terms and template were used by government agencies in Chile including SUSESO. However, the enactment of a new public procurement law has put that effort on hold. ChileCompra revoked the bidding terms and template along with other procurement templates in December 2023 to allow for adjustments to be made in relation to the new law. In the meantime, a related Procurement Directorate for AI and Data Science published in December 2023 offers similar guidance for all types of purchases, but is not mandatory.

The Limits of Traditional Procurement Criteria

When vendors submitted bids for the two SUSESO AI related projects, use of the 2022 Standard Bidding Terms for Data Science and AI Projects template was still a mandatory part of ChileCompra’s procurement process. That template gave Moya and his team valuable insights into the importance of AI governance evaluation and measurement criteria, said Moya. In many local and national governments around the world, procurement processes and rules are often designed to ensure cost-efficiency and prevent conflicts-of-interest or graft when choosing suppliers. Chile is no different. And according to Moya, despite including responsible AI-related criteria, the 2022 procurement terms and template process was designed to weigh price considerations and competition among vendors bidding for government projects more heavily than those other criteria. Documents associated with SUSESO’s procurement of the model to optimize medical claims management can be downloaded here (see ‘ver adjuntos’).

Moya lamented what he called ChileCompra’s “rigid” procurement template, which he worried placed too much emphasis on price and competition and not enough on ensuring vendor quality and experience working on similar complex AI projects. Moya said he would have preferred that the template placed more emphasis on a vendor’s capabilities and experience related to data protection and security, evaluation and reduction of model bias, and ensuring model transparency and explainability.

Technologies that enable algorithmic decision making create new policy implications — whether they are used to automate medical claims management, or other commonly found government uses such as taxation decisions, automating responses to constituent letters, or in pretrial risk assessments in policing. Researchers Deirdre Mulligan and Kenneth Bamberger have analyzed the limits of traditional government procurement approaches, as well as the policy and political choices embedded in machine learning systems acquired and used by governments. Their overarching conclusion was that traditional procurement methods need to be updated for the impacts of AI and digitalization.

Before the 2022 bidding template was removed for an update, it required bidders to undergo robust evaluation according to several responsible AI related criteria including model accuracy, transparency, explainability mechanisms, and bias and equity. The results of the analyses were documented in project deliverables. Universidad Adolfo Ibáñez’s GobLab UAI assisted SUSESO in performing these evaluations.

Specifically, GobLab provided guidance for implementing ethical parameters of SUSESO’s medical claims management model, according to Reinel Tabares Soto, alternate director of GobLab’s Proyecto Algoritmos Éticos. He told WPF in September that the GobLab group participated in the project’s weekly progress meetings involving discussions about technical and strategic decisions in relation to model development and choice of training variables, with a particular emphasis on transparency and fairness. Moya said both models discussed in this story were evaluated and scored using GobLab’s Algorithmic Transparency Sheet Tool and its Bias and Equity Measurement Tool; the models were also evaluated using metrics such as Statistical Parity Difference and Disparate Impact Ratio.

(Note: WPF will feature research sharing more details on SUSESO’s collaboration with GobLab in future work.)

Human Agency, Fairness, and an Emerging AI Governance Policy Gap

As all levels of governments around the world – from national to municipal and local level governments — struggle to keep up with workloads, sometimes amid budget cuts and staff reductions, many are using algorithmic scoring systems or more complex machine learning systems to automate some components of administrative processes and decisions.

However, there are chances that any algorithmic or machine learning model can make mistakes or produce unfair decisions, even if a bias or fairness assessment indicates low likelihood of discriminatory impacts. Use of allegedly faulty or discriminatory algorithmic systems to facilitate administrative decisions already have had devastating effects on people, and led to legal and political repercussions for governments. For instance, algorithmic scoring systems used for social welfare and tax related decisions by government agencies in France, Australia and The Netherlands have led to financial ruin for people, political scandal, lawsuits, resignations of government officials and additional harms.

Moya suggested those sorts of potential negative impacts on people should guide organizations toward using algorithmic models in conjunction with human decision makers, particularly if they involve decisions that affect people’s livelihoods, such as taxation-related decisions, or approval or denial of government assistance. In other words, Moya said these types of models or systems should be employed only in support of, or to inform, how a human — possibly someone with relevant subject-matter expertise — ultimately makes the decision. He pointed to key principles in The National Artificial Intelligence Policy of Chile, noting that while used to optimize costs and resources, data science and AI should be used in collaboration with human management, in ways that recognize human rights and ethical data considerations.

How AI assessments are interpreted matter. Moya suggested that fairness assessments indicating there is only a very minor likelihood or risk of inaccurate or biased decisions can be used to buoy the arguments of people in favor of an entirely automated decision process. He said he questions this approach, and worries that model performance or “success” measures might obscure the potential for serious negative impacts on some people. Instead, he argued “success” might be defined another way; in situations affecting people’s wellbeing and livelihoods, use of a well-designed and assessed model in support of speedier-yet-thoughtful human decisions can constitute success.

ChileCompra’s procurement template did not provide guidance on how algorithmic systems should be used in conjunction with humans, said Moya. Unresolved questions related to the interplay between humans and automated decision systems are creating meaningful obstacles to implementation of AI governance policy inside government agencies, in Chile and elsewhere. In particular, World Privacy Forum has spoken with people inside other government agencies who are using AI governance tools that do not adequately address key considerations related to interactions between humans and algorithmic systems. How and when should humans be involved in making decisions when automated or algorithmic systems are in use? How should human involvement in the decision-making process affect application of AI governance tools – such as algorithmic impact assessments?

The issue of human agency in relation to automated decision-making remains an open area of exploration as details of AI governance policy implementation are tested in real life situations.

Internal Buy-in for Responsible AI Criteria

A goal of ChileCompra’s short-lived mandatory bidding template was to incentivize companies: If they wanted to win government contracts, they had to build AI systems with responsible and trustworthy AI considerations in mind. Despite some misgivings about the procurement process and how it weighed responsible AI criteria, Moya said he thinks the template accomplished that goal, in part because it forced AI suppliers to conduct impact analyses and provide documentation related to responsible AI requirements. 

The addition of AI criteria and measures related to bias, transparency and data protection in the template also had a positive impact internally inside SUSESO, said Moya. The existence of those evaluation requirements helped convey the importance of those criteria to people in other units within the agency, such as those who are focused on finances or may not have emphasized or considered responsible AI criteria otherwise. 

And, Moya admitted that the bidding template’s mandatory requirement to assess AI systems even gave him a necessary push to dedicate the extra time and resources toward addressing ethical considerations seriously — something that is not easy when managing projects while up against time and budget pressures. 

World Privacy Forum’s December 2023 Report 

Risky Analysis: Assessing and Improving AI Governance Tools, An international review of AI Governance Tools and suggestions for pathways forward https://www.worldprivacyforum.org/2023/12/new-report-risky-analysis-assessing-and-improving-ai-governance-tools/

Chile’s entry in WPF’s AI Governance Tools Survey begins on pages 64-65 of the report.

Publication information: 

Author: Kate Kaye, Deputy Director WPF

Editing: Pam Dixon, Executive Director WPF

Original publication date: 1 November 2024

URL: https://www.worldprivacyforum.org/wp-content/uploads/2024/11/WPF_ChileCompra_Oct2024_fs.pdf 

References 

Ethical Algorithms Project (Proyecto Algoritmos Éticos), Universidad Adolfo Ibáñez’s School of Government, GobLab UAI, (September 2020), https://goblab.uai.cl/en/ethical-algorithms/ .

Standard Bidding Terms are now available for algorithm and artificial intelligence projects with ethical requirements (Ya se encuentra disponible Bases Tipo para licitar proyectos de algoritmos e inteligencia artificial con requisitos éticos, ChileCompra), Dirección ChileCompra, Government of Chile, (January 2023), https://www.chilecompra.cl/2023/01/ya-se-encuentra-disponible-bases-tipo-para-licitar-proyectos-de-algoritmos-e-inteligencia-artificial-con-requisitos-eticos/ .

The National Artificial Intelligence Policy of Chile, Ministerio de Ciencia, Tecnología, Conocimiento e Innovación, Government of Chile, (October 2021), https://minciencia.gob.cl/uploads/filer_public/bc/38/bc389daf-4514-4306-867c-760ae7686e2c/documento_politica_ia_digital_.pdf  .

Resolution revokes use of Standard Bidding Terms while adjustments are made to current regulations, Dirección ChileCompra, Government of Chile, (December 2023), https://www.chilecompra.cl/2023/12/resolucion-revoca-uso-de-bases-tipo-mientras-se-realicen-adecuaciones-con-normativa-vigente/ .

New ChileCompra Directive for the management of Artificial Intelligence or Data Science projects, Dirección ChileCompra, Government of Chile, (December 2023), https://www.chilecompra.cl/2023/12/nueva-directiva-de-chilecompra-entrega-recomendaciones-para-la-gestion-de-proyectos-de-inteligencia-artificial-o-ciencia-de-datos/ .

Superintendencia de Seguridad Social, Analysis and Development of the Medical Licenses Model (Bid ID: 1607-28-LE23), MercadoPublico, Dirección ChileCompra, Government of Chile, Anexo N°1 Declaración Jurada Para Contratar, (October 2023), (see ver adjuntos/Anexos para ofertar)
https://www.mercadopublico.cl/Procurement/Modules/RFB/DetailsAcquisition.aspx?qs=+CmsHS5Rn9jNsjNenY+rxA== .

Mulligan, Deirdre K. and Bamberger, Kenneth A., Procurement As Policy: Administrative Process for Machine Learning, Berkeley Technology Law Journal, Vol. 34, 2019, (October 2019),
https://ssrn.com/abstract=3464203 or http://dx.doi.org/10.2139/ssrn.3464203 .

Kate Kaye, Why old-school government procurement needs a rewrite for new AI tech, IAPP News, (February 2019), https://iapp.org/news/a/why-old-school-government-procurement-needs-a-rewrite-for-new-ai-tech .

Ethical Algorithm Tools (Herramientas de Algoritmos Éticos), GobLab UAI, Universidad Adolfo Ibáñez’s School of Government, https://algoritmospublicos.cl/herramientas .

High-Level Expert Group on AI, Ethics guidelines for trustworthy AI, European Commission, (April 2019), https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai .

Jon Henley, Dutch government faces collapse over child benefits scandal, The Guardian, (January 2021), https://www.theguardian.com/world/2021/jan/14/dutch-government-faces-collapse-over-child-benefits-scandal   .

Frances Mao, Robodebt: Illegal Australian welfare hunt drove people to despair, BBC News, July 2023, https://www.bbc.com/news/world-australia-66130105 .

The “SyRI” welfare fraud risk-scoring algorithm, Digital Freedom Fund, Case Studies,

https://digitalfreedomfund.org/the-syri-welfare-fraud-risk-scoring-algorithm/ .

Gabriel Geiger, How a Discriminatory Algorithm Wrongly Accused Thousands of Families of Fraud, Vice, (March 2021), https://www.vice.com/en/article/how-a-discriminatory-algorithm-wrongly-accused-thousands-of-families-of-fraud/.

Morgan Meaker, Algorithms Policed Welfare Systems For Years. Now They’re Under Fire for Bias, Wired, (October 2024), https://www.wired.com/story/algorithms-policed-welfare-systems-for-years-now-theyre-under-fire-for-bias/ .