Large Language Models as A Paradigm Shift in Next-Generation Virtual Reality Interaction: A Comprehensive Investigation

Abstract

The convergence of Large Language Models (LLMs) and Virtual Reality (VR) represents an emerging frontier in Human-computer interaction (HCI). Contemporary LLMs, such as GPT-4 and Gemini 1.5, demonstrate advanced multimodal reasoning capabilities, while VR has evolved from purely visual immersion to semantically rich, generative environments enabled by neural scene encoding and real-time holographic generation. However, current VR systems predominantly rely on preconfigured interactions and static interfaces, failing to harness the adaptive, generative, and dynamic potential of LLMs which is a significant research gap. Prior studies have focused on isolated aspects, such as graphical fidelity or voice-based navigation, without exploring the integration of LLMs as the central intelligence for real-time dialogue, agent behavior, and procedural content generation. Moreover, literature lacks a systematic analysis of the architectural frameworks, opportunities, and challenges inherent in deep LLM-VR integration. This review addresses these gaps through three primary objectives: (1) synthesizing recent advancements in multimodal LLMs and generative VR environments to assess their technical compatibility; (2) analyzing key application areas of LLMs in VR, including agent modeling, user experience adaptation, and procedural content generation, with detailed evaluation of methodologies, innovations, and empirical outcomes; and (3) proposing a structured framework to guide the development of next-generation intelligent VR systems. The findings affirm that LLM integration constitutes not merely an enhancement but a fundamental paradigm shift—enabling dynamic, context-aware, and highly immersive virtual experiences. This review provides a roadmap for researchers and practitioners aiming to leverage LLM-VR synergy across domains such as education, healthcare, and entertainment.