Exploration of essential practices around use of Foundation Models in real-world settings  

WAI Labs, Women in AI 

The state-of-the-art of Large Language Models(LLMs) or Foundation Models(FMs), their varied and changing capabilities and increasing multimodality have led to discussion around their use and promise to be in applied domains. The expectation of AI to deliver reliable outcomes in domains such as healthcare, climate and weather, and robotics[1, 2] on tasks suitable for FMs can be used and could be beneficial for automation remains in the promising stage. To bridge the gap towards real-world implementation, this workshop focuses on critical perspectives grounded in technological work on the issues of verification, robustness, data composition, reliability etc., around the use of these models in such real-world settings. 

The workshop offers a space for discussion, sharing of practices, and the steps needed to ensure the inclusion of stakeholders - who are impacted by the use of these models and transparency in operations - if the effects of using these models in complex environments are as intended.

While research and discussions around improving models and capabilities are important, the methodologies required for using models in real-world settings and assessing their consequences is an underexplored research perspective. 

The goal of the workshop is to create a forum for discussion of these research directions that are important for the adoption and use of FMs in a reliable manner in real-world settings, with also a critical approach to current practices 

List of Topics 

We welcome extended abstracts that cover the following, but are not necessarily limited to:

• Description of tasks for which FMs are used in real-world settings
• Description of complex environments that FMs are to be used in 
• Methodologies for assessment of the impact of using FMs within specific operation scenarios such as healthcare, robotics, transportation, climate etc.
• Description or establishment of datasets, as it relates to the scenario of operation 
• Articulation of biases, be it in the data or output from models - in the context of use
• Negative results for some methodologies in the use of FMs 
• Templates for impact assessment of deployment of AI on stakeholders
• Standards for describing operation scenarios for FMs, from any domain

Targeted Audience and Submissions:

We target researchers from various domains and fields contributing to the above mentioned main themes and topics. Researchers with different backgrounds and varying levels of experience are explicitly invited to participate in the discussion at the workshop.

We invite submissions of novel work, even in the form of initial ideas, methodology design (great example [3]), and summaries of past work that are relevant for the topic or opinion papers. We further allow submissions of short paragraphs/abstracts in which participants are invited to explain their intentions concerning future uses of Foundation Models in real-world settings. 

Program Structure 

• Welcome and Introduction -  Women in AI
• Talks from accepted submissions
• Panel Discussion and Networking 

References

[1]Chen, Shengchao, Guodong Long, Jing Jiang, Dikai Liu, and Chengqi Zhang. 2023. “Foundation Models for Weather and Climate Data Understanding: A Comprehensive Survey.” arXiv. https://doi.org/10.48550/arXiv.2312.03014.

[2] Moor, Michael, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M. Krumholz, Jure Leskovec, Eric J. Topol, and Pranav Rajpurkar. 2023. “Foundation Models for Generalist Medical Artificial Intelligence.”Nature 616 (7956): 259–65. https://doi.org/10.1038/s41586-023-05881-4.

[3]Dembrower, Karin, et al. "Artificial intelligence for breast cancer detection in screening mammography in Sweden: a prospective, population-based, paired-reader, non-inferiority study." The Lancet Digital Health 5.10 (2023): e703-e711.

Submission Guidelines

Submission Details

• Title
• 1000 - 2500  words -  position papers or preliminary work 
• 3-5 Keywords
• Submission link  - coming soon

Committees

Organizing committee

• Vinutha Magal Shreenath
• Frincy Clement
• Jauwairia Nasir
• Auxane Boch

Program committee

• Vinutha Magal Shreenath
• Jauwairia Nasir
• Auxane Boch

Contact

All questions about submissions should be emailed to vinutha@womeninai.co