Advancing AI Incidents Classification: Leveraging LLMs with Strategic Prompting

EasyChair Preprint 15820

Abstract

This study examines the efficacy of large language models (LLMs), particularly GPT-4, in classifying AI incident reports documented in the AI Incidents Database (AIID), with the goal of enhancing our understanding and management of AI-related harm. The data of incident reports are all on news events that detail specific incidents related to AI technology that have resulted in harmful effect on our society. We explore the use of different prompting techniques on GPT-4 and assess the classification results of those incidents by subjective and objective evaluations. This work lays the groundwork for a comprehensive, automated classification framework for AI incident reporting, balancing LLM capabilities with the intricacies inherent in human judgment.

Keyphrases: LLM Classification, Prompt Engineering, Responsible AI., large language models

@booklet{EasyChair:15820,
  author    = {Yian Chen and Lana Do and Liheng Yi and Ricardo Baeza-Yates and John A. Guerra-Gomez},
  title     = {Advancing AI Incidents Classification: Leveraging LLMs with Strategic Prompting},
  howpublished = {EasyChair Preprint 15820},
  year      = {EasyChair, 2025}}