AI Procurement Checklists: Revisiting Implementation in the Age of AI Governance

2404.14660

YC

0

Reddit

0

Published 4/24/2024 by Tom Zick, Mason Kortz, David Eaves, Finale Doshi-Velez

πŸ€–

Abstract

Public sector use of AI has been quietly on the rise for the past decade, but only recently have efforts to regulate it entered the cultural zeitgeist. While simple to articulate, promoting ethical and effective roll outs of AI systems in government is a notoriously elusive task. On the one hand there are hard-to-address pitfalls associated with AI-based tools, including concerns about bias towards marginalized communities, safety, and gameability. On the other, there is pressure not to make it too difficult to adopt AI, especially in the public sector which typically has fewer resources than the private sector$unicode{x2014}$conserving scarce government resources is often the draw of using AI-based tools in the first place. These tensions create a real risk that procedures built to ensure marginalized groups are not hurt by government use of AI will, in practice, be performative and ineffective. To inform the latest wave of regulatory efforts in the United States, we look to jurisdictions with mature regulations around government AI use. We report on lessons learned by officials in Brazil, Singapore and Canada, who have collectively implemented risk categories, disclosure requirements and assessments into the way they procure AI tools. In particular, we investigate two implemented checklists: the Canadian Directive on Automated Decision-Making (CDADM) and the World Economic Forum's AI Procurement in a Box (WEF). We detail three key pitfalls around expertise, risk frameworks and transparency, that can decrease the efficacy of regulations aimed at government AI use and suggest avenues for improvement.

Create account to get full access

or

If you already have an account, we'll log you in

Overview

  • Growing use of AI in the public sector over the past decade
  • Efforts to regulate government use of AI have recently gained attention
  • Promoting ethical and effective deployment of AI in government is a complex challenge
  • Tensions between mitigating risks and making AI adoption easy for resource-constrained public agencies

Plain English Explanation

The use of AI by government agencies has been quietly increasing for years, but it's only recently that there have been serious efforts to regulate this technology in the public sector. While it may seem straightforward to ensure the ethical and effective use of AI systems in government, it's actually a notoriously difficult task.

On one hand, there are significant concerns about the potential downsides of using AI-based tools, such as bias towards marginalized communities, safety issues, and the risk of the systems being gamed. On the other hand, there's pressure not to make it too difficult for government agencies to adopt AI, especially since the public sector often has fewer resources than private companies - the ability to save scarce government resources is often a big reason for using AI tools in the first place.

These competing tensions create a real risk that any regulations or procedures put in place to protect vulnerable groups from being harmed by government use of AI will end up being more performative than truly effective. To inform the latest wave of regulatory efforts in the US, the authors of this paper looked at the lessons learned by officials in other jurisdictions with more mature regulations around government AI use, like Brazil, Singapore, and Canada.

Technical Explanation

The paper investigates two specific checklists that have been implemented to govern AI procurement and use in the public sector: the Canadian Directive on Automated Decision-Making (CDADM) and the World Economic Forum's AI Procurement in a Box (WEF).

Through interviews with government officials, the researchers identified three key pitfalls that can undermine the effectiveness of regulations aimed at ensuring the responsible use of AI by public agencies:

  1. Expertise: Ensuring the right experts are involved in the design and implementation of AI governance frameworks is crucial but challenging, as government agencies often lack in-house technical AI expertise.

  2. Risk Frameworks: Developing appropriate risk assessment frameworks to categorize different AI use cases and determine the level of scrutiny required is complex, and risks being overly subjective or narrow in scope.

  3. Transparency: Striking the right balance between transparency requirements and legitimate concerns about protecting sensitive information or intellectual property is difficult, and can lead to regulations that are more about visible compliance than meaningful oversight.

The paper suggests several avenues for improvement, such as building AI audit standards and boards to provide ongoing guidance, and taking a more systematic approach to mapping the landscape of AI use in government to better inform regulatory efforts.

Critical Analysis

The paper provides a nuanced and well-researched look at the challenges of regulating government use of AI. It rightly highlights the inherent tensions between mitigating risks and making AI adoption feasible for cash-strapped public agencies. The three key pitfalls identified - around expertise, risk frameworks, and transparency - seem like valid and important concerns based on the evidence presented.

One potential limitation of the research is the relatively narrow focus on specific checklist frameworks implemented in a few jurisdictions. While these case studies provide valuable insights, the findings may not be fully generalizable to all regulatory approaches or government contexts.

Additionally, the paper could have delved deeper into potential solutions or best practices to address the identified pitfalls. While it suggests some avenues for improvement, a more comprehensive discussion of possible remedies would have strengthened the overall contribution.

Nevertheless, this paper makes a valuable contribution to the ongoing discourse around responsible AI governance in the public sector. By shedding light on the practical challenges faced by policymakers, it encourages readers to think critically about how to build more durable and effective frameworks for regulating the use of AI in government.

Conclusion

This paper provides a nuanced look at the complexities of regulating government use of AI. It highlights the inherent tensions between mitigating risks and making AI adoption feasible for resource-constrained public agencies, and identifies three key pitfalls - around expertise, risk frameworks, and transparency - that can undermine the effectiveness of regulatory efforts.

While focused on specific case studies, the paper's findings offer important lessons for policymakers and AI governance experts working to ensure the ethical and responsible deployment of AI in the public sector. By shedding light on these practical challenges, the research encourages critical thinking about how to build more durable and effective frameworks for governing the use of AI by government agencies.



This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!

Related Papers

Position Paper: Technical Research and Talent is Needed for Effective AI Governance

Position Paper: Technical Research and Talent is Needed for Effective AI Governance

Anka Reuel, Lisa Soder, Ben Bucknall, Trond Arne Undheim

YC

0

Reddit

0

In light of recent advancements in AI capabilities and the increasingly widespread integration of AI systems into society, governments worldwide are actively seeking to mitigate the potential harms and risks associated with these technologies through regulation and other governance tools. However, there exist significant gaps between governance aspirations and the current state of the technical tooling necessary for their realisation. In this position paper, we survey policy documents published by public-sector institutions in the EU, US, and China to highlight specific areas of disconnect between the technical requirements necessary for enacting proposed policy actions, and the current technical state of the art. Our analysis motivates a call for tighter integration of the AI/ML research community within AI governance in order to i) catalyse technical research aimed at bridging the gap between current and supposed technical underpinnings of regulatory action, as well as ii) increase the level of technical expertise within governing institutions so as to inform and guide effective governance of AI.

Read more

6/12/2024

Global AI Governance in Healthcare: A Cross-Jurisdictional Regulatory Analysis

Global AI Governance in Healthcare: A Cross-Jurisdictional Regulatory Analysis

Attrayee Chakraborty, Mandar Karhade

YC

0

Reddit

0

Artificial Intelligence (AI) is being adopted across the world and promises a new revolution in healthcare. While AI-enabled medical devices in North America dominate 42.3% of the global market, the use of AI-enabled medical devices in other countries is still a story waiting to be unfolded. We aim to delve deeper into global regulatory approaches towards AI use in healthcare, with a focus on how common themes are emerging globally. We compare these themes to the World Health Organization's (WHO) regulatory considerations and principles on ethical use of AI for healthcare applications. Our work seeks to take a global perspective on AI policy by analyzing 14 legal jurisdictions including countries representative of various regions in the world (North America, South America, South East Asia, Middle East, Africa, Australia, and the Asia-Pacific). Our eventual goal is to foster a global conversation on the ethical use of AI in healthcare and the regulations that will guide it. We propose solutions to promote international harmonization of AI regulations and examine the requirements for regulating generative AI, using China and Singapore as examples of countries with well-developed policies in this area.

Read more

6/14/2024

The Journey to Trustworthy AI- Part 1: Pursuit of Pragmatic Frameworks

The Journey to Trustworthy AI- Part 1: Pursuit of Pragmatic Frameworks

Mohamad M Nasr-Azadani, Jean-Luc Chatelain

YC

0

Reddit

0

This paper reviews Trustworthy Artificial Intelligence (TAI) and its various definitions. Considering the principles respected in any society, TAI is often characterized by a few attributes, some of which have led to confusion in regulatory or engineering contexts. We argue against using terms such as Responsible or Ethical AI as substitutes for TAI. And to help clarify any confusion, we suggest leaving them behind. Given the subjectivity and complexity inherent in TAI, developing a universal framework is deemed infeasible. Instead, we advocate for approaches centered on addressing key attributes and properties such as fairness, bias, risk, security, explainability, and reliability. We examine the ongoing regulatory landscape, with a focus on initiatives in the EU, China, and the USA. We recognize that differences in AI regulations based on geopolitical and geographical reasons pose an additional challenge for multinational companies. We identify risk as a core factor in AI regulation and TAI. For example, as outlined in the EU-AI Act, organizations must gauge the risk level of their AI products to act accordingly (or risk hefty fines). We compare modalities of TAI implementation and how multiple cross-functional teams are engaged in the overall process. Thus, a brute force approach for enacting TAI renders its efficiency and agility, moot. To address this, we introduce our framework Set-Formalize-Measure-Act (SFMA). Our solution highlights the importance of transforming TAI-aware metrics, drivers of TAI, stakeholders, and business/legal requirements into actual benchmarks or tests. Finally, over-regulation driven by panic of powerful AI models can, in fact, harm TAI too. Based on GitHub user-activity data, in 2023, AI open-source projects rose to top projects by contributor account. Enabling innovation in TAI hinges on the independent contributions of the open-source community.

Read more

4/9/2024

Guiding the Way: A Comprehensive Examination of AI Guidelines in Global Media

Guiding the Way: A Comprehensive Examination of AI Guidelines in Global Media

M. F. de-Lima-Santos, W. N. Yeung, T. Dodds

YC

0

Reddit

0

With the increasing adoption of artificial intelligence (AI) technologies in the news industry, media organizations have begun publishing guidelines that aim to promote the responsible, ethical, and unbiased implementation of AI-based technologies. These guidelines are expected to serve journalists and media workers by establishing best practices and a framework that helps them navigate ever-evolving AI tools. Drawing on institutional theory and digital inequality concepts, this study analyzes 37 AI guidelines for media purposes in 17 countries. Our analysis reveals key thematic areas, such as transparency, accountability, fairness, privacy, and the preservation of journalistic values. Results highlight shared principles and best practices that emerge from these guidelines, including the importance of human oversight, explainability of AI systems, disclosure of automated content, and protection of user data. However, the geographical distribution of these guidelines, highlighting the dominance of Western nations, particularly North America and Europe, can further ongoing concerns about power asymmetries in AI adoption and consequently isomorphism outside these regions. Our results may serve as a resource for news organizations, policymakers, and stakeholders looking to navigate the complex AI development toward creating a more inclusive and equitable digital future for the media industry worldwide.

Read more

5/9/2024