OL23 LLM-based story generation for AI+AR glasses based on environmental and biometric sensing

Aim/goal/research question

The latest development within HMD-based Augmented Reality and Context-Aware Systems points towards a potential future where people could carry nudging AI+AR glasses (see figure) that almost invisibly help them perform both professional and everyday tasks. Apart from ethical and privacy challenges (do we really want our perception to be invisibly manipulated in this way?) a number of advanced system functionalities would need to be engineered. Core mechanisms include manipulation of how the human individual perceives the surrounding environment (subtle visual attention guidance) driven by a story generation process which on the fly, using available real-world and digital objects, gradually lead users of the device towards a better understanding of a phenomena and/or simply conclusion of the task at hand. The aim of this project is to develop a first version of a story generation component based on Large Language Model (LLM) technologies, used in for instance Open AI’s Chat GPT. The story generation is intended to be performed both a) in order for the system itself to make sense of the current situation, and b) in order to help the user of the system better make sense of the situation through subtle attention guidance (nudging). More concretely:

Can LLM take in the signals from the environment and from cognitive loading (like pupil dilation information and EEG) can create a rich story about what is going on?
Based on the above, can LLM suggest what to do in order to nudge action XYZ?

Method

Iterative prototyping of an LLM-based software system able to take input both from other components of the envisioned AI+AR nudging glasses from one or several sources (depending on the interests of the students) such as a) the mixed reality modelling component developed in a separate project, b) eye movements of the user, c) EEG signals.

Recommended past experience/interest

Machine Learning, Python, C++/C#.

Related Work

Pederson, T., Janlert, L-E., & Surie, D. (2011). Setting the Stage for Mobile Mixed-Reality Computing – A Situative Space Model based on Human Perception. IEEE Pervasive Computing Magazine vol. 10, no. 4, pp. 73-83, Oct. 2011. DOI: 10.1109/MPRV.2010.51
Jalaliniya, S. and Mardanbegi, D. (2016) EyeGrip: Detecting Targets in a Series of Uni-directional Moving Objects Using Optokinetic Nystagmus Eye Movements. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). Association for Computing Machinery, New York, NY, USA, 5801–5811. https://doi.org/10.1145/2858036.2858584

Other comments

The project is part of the bigger Nudging AI+AR glasses project.

Resource limitations

Depending on the interest and focus of the students, existing AR/VR/eye-tracking hardware will be used or new hardware (e.g. EEG brainwave sensors) will be purchased. Som of the equipment can be borrowed over the time of the project while other needs to worked on inside Open Lab (room J337).

Contact

amit.kumar-mishra@hv.se, thomas.pederson@hv.se