The latest development within HMD-based Augmented Reality and Context-Aware Systems points towards a potential future where people could carry nudging AI+AR glasses (see figure) that almost invisibly help them perform both professional and everyday tasks. Apart from ethical and privacy challenges (do we really want our perception to be invisibly manipulated in this way?) a number of advanced system functionalities would need to be engineered. Core mechanisms include manipulation of how the human individual perceives the surrounding environment (subtle visual attention guidance) driven by a story generation process which on the fly, using available real-world and digital objects, gradually lead users of the device towards a better understanding of a phenomena and/or simply conclusion of the task at hand.   The aim of this project is to develop a first version of a story generation component based on Large Language Model (LLM) technologies, used in for instance Open AI’s Chat GPT. The story generation is intended to be performed both a) in order for the system itself to make sense of the current situation, and b) in order to help the user of the system better make sense of the situation through subtle attention guidance (nudging). More concretely:

1. Can LLM take in the signals from the environment and from cognitive loading (like pupil dilation information and EEG) can create a rich story about what is going on?
2. Based on the above, can LLM suggest what to do in order to nudge action XYZ?

In the developing field of AI-enhanced Augmented Reality (AR) and Virtual Reality (VR), the concept of “nudging”—subtle guidance to influence human cognition and behavior—presents both promise and challenge. Can these methods help us understand and manage complex issues, improving lives or helping us deal with societal problems? Do we really want our perception to be invisibly manipulated in this way? To properly investigate such issues, a range of advanced functionalities must be developed. These include systems that alter how individuals perceive their environment through subtle guidance, and using techniques to generate and guide developing narratives to help users gain a better understanding of complex subjects in real-time. This project aims to investigate these dynamics in the context of immersive VR-based climate data visualization. The objective is to develop a VR component that leverages AI-driven nudging to assist users in navigating and interpreting complex information. A central focus is the creation of narratives within the data; as users navigate the visualization, the system will direct their attention to key elements, forming narratives that enhance understanding. By exploring these issues, the project aims to contribute to ongoing research in AI-enhanced nudging techniques for educational and decision-making applications in AR/VR environments.