[Archimedes NLP Group Invited Talk]Prompt Stability Scoring for Text Annotation with Large Language Models

Title: "Prompt Stability Scoring for Text Annotation with Large Language Models"

Speaker: Christopher Barrie, NYU - Assistant Professor, Department of Sociology, New York University

Abstract: Researchers are increasingly using language models (LMs) for text annotation. These approaches rely only on a prompt telling the model to return a given output according to a set of instructions. The reproducibility of LM outputs may nonetheless be vulnerable to small changes in the prompt design. This calls into question the replicability of classification routines. To tackle this problem, researchers have typically tested a variety of semantically similar prompts to determine what we call "prompt stability." These approaches remain ad-hoc and task specific. In this article, we propose a general framework for diagnosing prompt stability by adapting traditional approaches to intra- and inter-coder reliability scoring. We call the resulting metric the Prompt Stability Score (PSS) and provide a Python package PromptStability for its estimation. Using six different datasets and twelve outcomes, we classify >150k rows of data to: a) diagnose when prompt stability is low; and b) demonstrate the functionality of the package. We conclude by providing best practice recommendations for applied researchers.

Speaker Bio:
Christopher Barrie is Assistant Professor of Sociology at NYU. He is also Core Faculty at CSMaP and Research Fellow at the Department of Sociology, University of Oxford. 

Physical presence only for NLP theme members due to capacity restrictions. Check for additional ways to participate/dial in at the bottom of the announcement.

You can join virtually via Zoom Meetings - https://zoom.us/j/95703943754?pwd=BRZKJr7b6njXaBaD9YkIyYL8I3bTmK.1

• Meeting ID: 957 0394 3754
• Passcode: 2K3YDx

Dial-in information:
One tap mobile
+302111984488,,95703943754#,,,,*857682# Greece
+302311180599,,95703943754#,,,,*857682# Greece

Dial by your location
• +30 211 198 4488 Greece
• +30 231 118 0599 Greece

Meeting ID: 957 0394 3754
Passcode: 857682

For ways to receive news about the NLP Group and its meetings, as well as to get check the latest information about the meetings of Archimedes NLP Theme and AUEB NLP Group, check http://nlp.cs.aueb.gr/news.html. To subscribe to the mailing list of AUEB NLP Group, send a message with subject "subscribe nodigest" to This email address is being protected from spambots. You need JavaScript enabled to view it.. If you have an AUEB account and want to receive announcements about AUEB NLP Group Meetings through MS Teams, subscribe to "AUEB NLP Group meetings" group on MS Teams (code: 01j65ny). Team members can also send text messages (chat) to other team members.

If you are an AI researcher or practitioner, please consider becoming a member of the Hellenic Artificial Intelligence Society (EETN, http://www.eetn.gr/en/).