BibTeX does not have the right entry for preprints. This is a hack for producing the correct reference:
@booklet{EasyChair:8260,
author = {Aiden Boyd and Shibani and Will Callaghan},
title = {Learning to Self-Modify Rewards with Bi-Level Gradients},
howpublished = {EasyChair Preprint 8260},
year = {EasyChair, 2023}}