Inquiring the 'oracle'. An empirical study on how artificial intelligence literacy and prompt engineering influence the use of LLMs and GAI in higher education

Abstract

The release of ChatGPT by OpenAl on November 2022 marked a significant paradigm shift in the landscape of human-artificial intelligence interaction (HAII); for it signified the accessibility of generative artificial intelligence (GAI) and, in particular, large language models (LLMs) outside the technological élite. As of August 2024, ChatGPT was reported to have more than 200 million weekly active users, engaging with it to accomplish a wide variety of purposes, including professional or academic activities. To many, ChatGPT represents a polymathic entity, an oracle harnessing the extensive spectrum of human knowledge. Despite the enthusiasm with which this chatbot has been greeted, there is a pressing need to identify the numerous technical, architectural, ethical, and legal constraints associated with its development and usage within academic environments. This study aims to investigate whether possessing adequate literacy on both GAI, with its advantages and limitations, and fundamental prompt engineering techniques, can provide quantitatively and qualitatively improved interactions for university students, together with enhanced awareness of what LLMs can and can not achieve. In this regard, students from various humanities faculties at Ca’ Foscari University of Venice were selected to participate in a literacy workshop and empirical tests in order to contribute towards the development of an effective framework for AI literacy and utilisation within higher education.