archimedes-Artificial Intelligence, Data Science, Algorithms-greece

 
Artificial Intelligence
 
Data Science
 
Algorithms

[Archimedes NLP Group Invited Talk]Prompt Stability Scoring for Text Annotation with Large Language Models

Dates
2025-01-13 17:30 - 19:00
Venue
Artemidos 1 - Meeting Room
Title: "Prompt Stability Scoring for Text Annotation with Large Language Models"

Speaker: Christopher Barrie, NYU - Assistant Professor, Department of Sociology, New York University

Abstract: Researchers are increasingly using language models (LMs) for text annotation. These approaches rely only on a prompt telling the model to return a given output according to a set of instructions. The reproducibility of LM outputs may nonetheless be vulnerable to small changes in the prompt design. This calls into question the replicability of classification routines. To tackle this problem, researchers have typically tested a variety of semantically similar prompts to determine what we call "prompt stability." These approaches remain ad-hoc and task specific. In this article, we propose a general framework for diagnosing prompt stability by adapting traditional approaches to intra- and inter-coder reliability scoring. We call the resulting metric the Prompt Stability Score (PSS) and provide a Python package PromptStability for its estimation. Using six different datasets and twelve outcomes, we classify >150k rows of data to: a) diagnose when prompt stability is low; and b) demonstrate the functionality of the package. We conclude by providing best practice recommendations for applied researchers.

Speaker Bio:
Christopher Barrie is Assistant Professor of Sociology at NYU. He is also Core Faculty at CSMaP and Research Fellow at the Department of Sociology, University of Oxford. 


Physical presence only for NLP theme members due to capacity restrictions. Check for additional ways to participate/dial in at the bottom of the announcement.

You can join virtually via Zoom Meetings - https://zoom.us/j/95703943754?pwd=BRZKJr7b6njXaBaD9YkIyYL8I3bTmK.1
• Meeting ID: 957 0394 3754
• Passcode: 2K3YDx

Dial-in information:
One tap mobile
+302111984488,,95703943754#,,,,*857682# Greece
+302311180599,,95703943754#,,,,*857682# Greece

Dial by your location
• +30 211 198 4488 Greece
• +30 231 118 0599 Greece

Meeting ID: 957 0394 3754
Passcode: 857682


For ways to receive news about the NLP Group and its meetings, as well as to get check the latest information about the meetings of Archimedes NLP Theme and AUEB NLP Group, check http://nlp.cs.aueb.gr/news.html. To subscribe to the mailing list of AUEB NLP Group, send a message with subject "subscribe nodigest" to This email address is being protected from spambots. You need JavaScript enabled to view it.. If you have an AUEB account and want to receive announcements about AUEB NLP Group Meetings through MS Teams, subscribe to "AUEB NLP Group meetings" group on MS Teams (code: 01j65ny). Team members can also send text messages (chat) to other team members.

If you are an AI researcher or practitioner, please consider becoming a member of the Hellenic Artificial Intelligence Society (EETN, http://www.eetn.gr/en/).


 
 

Vision

To position Greece as a leading player in AI and Data Science

image
image

Mission

To build an AI Excellence Hub in Greece where the international research community can connect, groundbreaking ideas can thrive, and the next generation of scientists emerges, shaping a brighter future for Greece and the world

 

Welcome to ARCHIMEDES, a vibrant research hub connecting the global AI and Data Science research community fostering groundbreaking research in Greece and beyond. Its dedicated core team, comprising lead researchers, affiliated researchers, Post-Docs, PhDs and interns, is committed to advancing basic and applied research in Artificial Intelligence and its supporting disciplines, including Algorithms, Statistics, Learning Theory, and Game Theory organized around 8 core research areas. By collaborating with Greek and Foreign Universities and Research Institutes, ARCHIMEDES disseminates its research findings fostering knowledge exchange and providing enriching opportunities for students. Leveraging AI to address real-world challenges, ARCHIMEDES promotes innovation within the Greek ecosystem and extends its societal impact. Established in January 2022, as a research unit of the Athena Research Center with support from the Committee Greece 2021, ARCHIMEDES is funded for its first four years by the EU Recovery and Resilience Facility (RRF).

 
 

NEWS

 
Archimedes Flagship Project in Cardiology and AI is Featured in the News!

Archimedes Flagship Project in Cardiology and AI is Featured in the News!

Archimedes Research Unit of the Athena Research Center, Greece, is featured in a recent article in Dnews. This article is about an Archimedes flagship project in cardiology and AI that aims to use "two-dimensional echocardiographic data to develop deep learning tools and improve the treatment of heart problems."

Best Paper Award at FAIEMA 202

Best Paper Award at FAIEMA 202

Vasileios Moustakas, PhD student at the School of Electrical and Computer Engineering - NTUA and Academic Fellow at Archimedes, Athena Research Center, Greece, Konstantinos Cheliotis and Anna Mylona, both MEng students at the School of Electrical and Computer Engineering - NTUA and interns at Archimedes, Athena Research Center, Vassilis Alimisis, Postdoctoral Researcher at Archimedes, Athena Research Center, and Paul Sotiriadis, Lead Researcher at Archimedes, Athena Research Center, and a Professor at the School of Electrical and Computer Engineering - NTUA, received the Best Paper Award (PhD Symposium) at the

Nature Communications Publication on Advanced AI in Biological Research by Giorgos Papanastasiou

Nature Communications Publication on Advanced AI in Biological Research by Giorgos Papanastasiou

Giorgos Papanastasiou, Lead Researcher at the Archimedes Research Unit of the Athena Research Center, Greece,and Faculty Research Fellow at Edinburgh Imaging, at the University of Edinburgh, the Queen’s Medical Research Institute, Edinburgh, UK, has co-published a Nature Communications paper on "Clinical implications of bone marrow adiposity identified by phenome-wide association and Mendelian randomization in the UK Biobank."Prof. Papanastasiou mentions that "this project is a strong testament to the power of augmenting biological research with advanced AI and data science methods."

 
 

The project “ARCHIMEDES Unit: Research in Artificial Intelligence, Data Science and Algorithms” with code OPS 5154714 is implemented by the National Recovery and Resilience Plan “Greece 2.0” and is funded by the European Union – NextGenerationEU.

greece2.0 eu_arch_logo_en

 

Stay connected! Subscribe to our mailing list by emailing sympa@lists.athenarc.gr
with the subject "subscribe archimedes-news Firstname LastName"
(replace with your details)