XR IMMERSIVE ROOM FOR CLIMATE CHANGE EDUCATION
Multisensory Interactive Storytelling System
ABOUT THE PRODUCTION
An interactive XR immersive environment designed to deliver climate change education through multisensory storytelling. Developed within the Kemendikti Saintek Climate Change Research Consortium 2025 as an S.Tr.Kom. capstone project at Politeknik Elektronika Negeri Surabaya in collaboration with the Surabaya City Environmental Agency (DLH Kota Surabaya). Features a 3-wall immersive projection booth, MediaPipe gesture tracking, Kinect V2 wall-touch interaction, bilingual voice commands with NLP, real-time WAQI pollution data, and Arduino-controlled 4D physical effects (wind, mist, smoke, lightning). Resulted in a registered Indonesian Simple Patent (S00202601377) and an international IEEE research publication at IES 2026 ("A Multimodal Interactive Immersive Room for Climate Change Education").
TECHNOLOGY & RIG
App No: S00202601377
First Author & Presenter
DLH Surabaya Collaboration
Applied Multimedia Engineering
KEY SYSTEM BENCHMARKS & EVALUATION METRICS
Comprehensive quantitative evaluation across Jakob Nielsen 10 Usability Heuristics.
Continuous 30-minute stress test with 0 memory leaks on NVIDIA RTX 5070 Ti.
Projector output delay benchmarked using 240 FPS high-speed camera frame tracking.
3 × 3 × 2.5 meter room (18:5 aspect ratio) powered by 3x Epson short-throw laser projectors.
01 // EXECUTIVE SUMMARY & CLIMATE CONTEXT
XR Immersive Room for Climate Change Education
Academic Thesis: "Development of XR Immersive Environment for Interactive Storytelling on Climate Change Experience"
The XR Immersive Room for Climate Change Education is an interactive Extended Reality installation developed within the 2025 Kemendikti Saintek Climate Change Research Consortium. It translates complex climate science into an engaging, multi-sensory storytelling experience by combining three-wall panoramic projection, touchless gesture tracking, bilingual natural language voice commands, live environmental open-data telemetry (WAQI API), and synchronized 4D physical actuators (simulated wind, rain mist, pollution smoke, and lightning flashes).
Sritrusta Sukaridhoto, Ph.D.
Expert Validator: Agustinus Hendra (NIP. 19870822015011001). Part of the Kemendikti Saintek Climate Change Research Consortium 2025.
02 // SPECIFIC TECHNICAL CONTRIBUTIONS & ROLES
- ▹Architected the end-to-end multi-process pipeline linking input sensors, central logic, and physical actuators.
- ▹Engineered interactive storytelling scenarios in Unity 3D with dynamic story branches and waste sorting minigames.
- ▹Bridged Unity and TouchDesigner with Resolume Arena via Spout Inter-Process Communication (IPC) for zero-latency triple-wall slicing.
- ▹Implemented dual-tier gesture recognition combining Google MediaPipe (RGB camera) and Kinect V2 for virtual wall-touch interactions.
- ▹Engineered bilingual voice commands integrating offline Vosk (English) and cloud Google Speech-to-Text (Indonesian) with rule-based NLP intent parsing.
- ▹Programmed an Arduino Uno microcontroller managing 4-channel relays for real-time wind, mist, smoke, and lighting effects.
- ▹Integrated live World Air Quality Index (WAQI) and OpenWeather REST APIs into the projection canvas.
- ▹Formulated a Coordinate Remapping Algorithm in Unity, reducing Kinect V2 tilted perspective Euclidean error from 10.1 cm to <1.1 cm.
- ▹Devised a Pre-Triggering Strategy with an 88-frame offset in Unity to compensate for 366.7 ms of mechanical actuator inertia.
- ▹Conducted cross-modality latency profiling via 240 FPS high-speed camera frame-by-frame analysis in Adobe Premiere Pro.
- ▹Executed a rigorous 30-minute system stress test (GPU 94%, CPU 25%, 380+ FPS, zero memory leaks).
- ▹Facilitated expert validation with Surabaya City Environmental Agency (DLH) and conducted usability testing with end users.
03 // TECHNICAL SYSTEM & ENGINEERING DEEP-DIVE
End-to-End Multimodal Dataflow & Inter-Process Communication (IPC)
The system architecture separates compute loads across modular processes: MediaPipe in TouchDesigner extracts hand landmarks and transmits gesture events via Open Sound Control (OSC) to Unity 3D. Kinect V2 monitors point-cloud depth for virtual wall touches corrected through a custom Coordinate Remapping Algorithm. High-resolution textures are routed to Resolume Arena via zero-copy GPU shared memory using Spout IPC for triple-wall projection slicing without frame drops (>380 FPS).
Sensor & Audio Input
- • Logitech C270 RGB: 30 FPS MediaPipe Hand Landmark tracking.
- • Kinect V2 Depth: Tilted 280 cm ceiling mount, virtual wall touch.
- • Studio Mic Array: Dual bilingual speech streams (Vosk + Google STT).
- • Live Open API: World Air Quality Index (WAQI) & OpenWeather data.
Unity & TouchDesigner
- • Unity 3D: Central storytelling logic, scene triggers, minigames.
- • TouchDesigner: Generative particle simulations & gesture OSC bridge.
- • Spout IPC: Zero-copy GPU shared memory video texture routing.
- • Pre-Trigger Engine: 88-frame actuator offset delay compensation.
Triple-Wall & 4D Actuators
- • Resolume Arena: Triple-wall geometry mapping & slicing.
- • 3x Epson Laser Short-Throw: 4000 Lumens, 18:5 aspect ratio.
- • Arduino Serial Relays: Blower, Mist Pump, Smoke Machine, Flash LED.
- • Edifier MR4: Dedicated multi-channel spatial soundscapes.
04 // DISTINGUISHED VISITS, EXHIBITIONS & EVALUATIONS
Nanyang Technological University (NTU) Singapore
Direct system review and testing by international academic delegates from NTU Singapore, evaluating global standards in interactive Extended Reality public education.
Ministry of Higher Education, Science, and Technology RI
Official inspection by Prof. Dr. Eng. Yudi Darma, S.Si., M.Si. to evaluate commercialization prospects and potential adoption across public science centers.
Directorate General of Intellectual Property (DJKI)
On-site inspection and verification of technical novelty claims by official DJKI patent examiners inside the operational immersive room.
Dinas Lingkungan Hidup (DLH) Kota Surabaya
Rigorous review by Expert Validator Agustinus Hendra (DLH Surabaya) to verify municipal waste data and validate ecological messaging accuracy.
Politeknik Elektronika Negeri Surabaya (PENS)
Public exhibition at PENS auditorium before faculty examiners, university leaders, and multimedia engineering students.
05 // VISUAL DOCUMENTATION REPOSITORY
public/images/projects/xr-immersive-room/
Booth construction photos, MediaPipe hand landmark tracking, Kinect V2 wall-touch minigames, Arduino physical actuators, and high-profile visits from NTU Singapore, Kemendikti Saintek, and DJKI will render here.