archimedes-Artificial Intelligence, Data Science, Algorithms-greece

 
Artificial Intelligence
 
Data Science
 
Algorithms

[Archimedes NLP Group Invited Talk]Prompt Stability Scoring for Text Annotation with Large Language Models

Dates
2025-01-13 17:30 - 19:00
Venue
Artemidos 1 - Meeting Room
Title: "Prompt Stability Scoring for Text Annotation with Large Language Models"

Speaker: Christopher Barrie, NYU - Assistant Professor, Department of Sociology, New York University

Abstract: Researchers are increasingly using language models (LMs) for text annotation. These approaches rely only on a prompt telling the model to return a given output according to a set of instructions. The reproducibility of LM outputs may nonetheless be vulnerable to small changes in the prompt design. This calls into question the replicability of classification routines. To tackle this problem, researchers have typically tested a variety of semantically similar prompts to determine what we call "prompt stability." These approaches remain ad-hoc and task specific. In this article, we propose a general framework for diagnosing prompt stability by adapting traditional approaches to intra- and inter-coder reliability scoring. We call the resulting metric the Prompt Stability Score (PSS) and provide a Python package PromptStability for its estimation. Using six different datasets and twelve outcomes, we classify >150k rows of data to: a) diagnose when prompt stability is low; and b) demonstrate the functionality of the package. We conclude by providing best practice recommendations for applied researchers.

Speaker Bio:
Christopher Barrie is Assistant Professor of Sociology at NYU. He is also Core Faculty at CSMaP and Research Fellow at the Department of Sociology, University of Oxford. 


Physical presence only for NLP theme members due to capacity restrictions. Check for additional ways to participate/dial in at the bottom of the announcement.

You can join virtually via Zoom Meetings - https://zoom.us/j/95703943754?pwd=BRZKJr7b6njXaBaD9YkIyYL8I3bTmK.1
• Meeting ID: 957 0394 3754
• Passcode: 2K3YDx

Dial-in information:
One tap mobile
+302111984488,,95703943754#,,,,*857682# Greece
+302311180599,,95703943754#,,,,*857682# Greece

Dial by your location
• +30 211 198 4488 Greece
• +30 231 118 0599 Greece

Meeting ID: 957 0394 3754
Passcode: 857682


For ways to receive news about the NLP Group and its meetings, as well as to get check the latest information about the meetings of Archimedes NLP Theme and AUEB NLP Group, check http://nlp.cs.aueb.gr/news.html. To subscribe to the mailing list of AUEB NLP Group, send a message with subject "subscribe nodigest" to This email address is being protected from spambots. You need JavaScript enabled to view it.. If you have an AUEB account and want to receive announcements about AUEB NLP Group Meetings through MS Teams, subscribe to "AUEB NLP Group meetings" group on MS Teams (code: 01j65ny). Team members can also send text messages (chat) to other team members.

If you are an AI researcher or practitioner, please consider becoming a member of the Hellenic Artificial Intelligence Society (EETN, http://www.eetn.gr/en/).


 
 

Vision

To position Greece as a leading player in AI and Data Science

image
image

Mission

To build an AI Excellence Hub in Greece where the international research community can connect, groundbreaking ideas can thrive, and the next generation of scientists emerges, shaping a brighter future for Greece and the world

 

Welcome to ARCHIMEDES, a vibrant research hub connecting the global AI and Data Science research community fostering groundbreaking research in Greece and beyond. Its dedicated core team, comprising lead researchers, affiliated researchers, Post-Docs, PhDs and interns, is committed to advancing basic and applied research in Artificial Intelligence and its supporting disciplines, including Algorithms, Statistics, Learning Theory, and Game Theory organized around 8 core research areas. By collaborating with Greek and Foreign Universities and Research Institutes, ARCHIMEDES disseminates its research findings fostering knowledge exchange and providing enriching opportunities for students. Leveraging AI to address real-world challenges, ARCHIMEDES promotes innovation within the Greek ecosystem and extends its societal impact. Established in January 2022, as a research unit of the Athena Research Center with support from the Committee Greece 2021, ARCHIMEDES is funded for its first four years by the EU Recovery and Resilience Facility (RRF).

 
 

NEWS

 
Archimedes Talk by Alexandra Meliou on

Archimedes Talk by Alexandra Meliou on "Data Analysis and Manipulation through a Constrained Optimization Lens"

On Tuesday 4 November, 2025, from 1:00 pm to 2:00 pm, at the Archimedes Amphitheatre (1 Artemidos Street, 15125, Marousi, Archimedes, Athena Research Center, Greece), Professor Alexandra Meliou, Professor at the College of Information and Computer Sciences at the University of Massachusetts Amherst, USA, and a visiting researcher at the Archimedes Research Center, Athena Research Center, Greece, will deliver an Archimedes talk on "Data Analysis and Manipulation through a Constrained Optimization Lens."

Exploring the Frontiers of AI and Biomechanics at ICIST 2025

Exploring the Frontiers of AI and Biomechanics at ICIST 2025

Archimedes, Athena Research Center, Greece is happy to see its work being featured at the 15th International Conference on Information Society and Technology (ICIST 2025), at Kaunas University of Technology, Lithuania, through an invited keynote speech by Dr. Serafeim Moustakidis, titled: “Bringing Biomechanics into the AI Era: A New Frontier for Gait and Joint Health.”

Archimedes Talk by Manos Athanassoulis on

Archimedes Talk by Manos Athanassoulis on "Harnessing the Untapped Benefits of Near-Sortedness for Data Systems"

On Tuesday 21 October, 2025, from 1:00 pm to 2:00 pm, at the Archimedes Amphitheatre (1 Artemidos Street, 15125, Marousi, Archimedes, Athena Research Center, Greece), Professor Manos Athanassoulis, Associate Professor of Computer Science at Boston University (BU), USA, Director and Founder of the BU Data-intensive Systems and Computing Laboratory, BU, USA, and Co-Director of the BU Massive Data Algorithms and Systems Group, BU, USA, will deliver an Archimedes talk on "Harnessing the Untapped Benefits of Near-Sortedness for Data Systems."

GameSec 2025 just Around the Corner!

GameSec 2025 just Around the Corner!

The 16th Conference on Game Theory and AI for Security (GameSec25), an event strongly supported by Archimedes, Athena Research Center, Greece will take place at the National Technical University of Athens, Greece from October 13-15 2025. "With the rapid advancement of artificial intelligence, game theory, and security technologies, the resilience and trustworthiness of modern systems is more critical than ever. The 2025 Conference on Game Theory and AI for Security focuses on leveraging strategic decision-making, adversarial reasoning, and computational intelligence to address security challenges in complex and dynamic environments.

11 Papers Accepted at NeurIPS 2025!

11 Papers Accepted at NeurIPS 2025!

We are happy to announce that 11 papers from Archimedes, Athena Research Center, Greece, have been accepted at the Thirty-Ninth Annual Conference on Neural Information Processing Systems (NeurIPS 2025)!NeurIPS is a major annual event in the artificial intelligence and machine learning community and this year it will take place from December 2 to December 7 at the San Diego Convention Center, California, USA.

 
 

The project “ARCHIMEDES Unit: Research in Artificial Intelligence, Data Science and Algorithms” with code OPS 5154714 is implemented by the National Recovery and Resilience Plan “Greece 2.0” and is funded by the European Union – NextGenerationEU.

greece2.0 eu_arch_logo_en

 

Stay connected! Subscribe to our mailing list by emailing sympa@lists.athenarc.gr
with the subject "subscribe archimedes-news Firstname LastName"
(replace with your details)