Hand Gesture Recognition for Smartphone-Based Augmented Reality Applications

Eric Cesar E. Vidal Jr.
Department of Information Systems and Computer Science

Virtual, Augmented and Mixed Reality (VAMR) can potentially improve educational applications by increasing immersion and user interactivity. In line with the effort to make VAMR more accessible to the public, a hand gesture recognition (HGR) system is developed for iOS devices to improve user interaction on mobile handheld AR systems, bringing these systems’ functionality on par with expensive AR systems with built-in HGR such as Microsoft HoloLens. Usability testing shows that hand- held HGR using our system is comparable in usability to an equally configured Microsoft HoloLens system.

As a proof-of-concept, this HGR system is utilized in a cross-platform basketball simulator, with similar functionality across both handheld and head-mounted VAMR configurations. This basketball simulator uses natural hand gestures for throwing and shooting the ball and for allowing effortless movement within the virtual court on all supported platforms. Options for simplified gesture execution are also provided to allow disabled persons to play basketball alongside non-disabled peers. This system demonstrates the possibility of using a cost-effective mix of hand-held and head-mounted devices for space-limited VAMR applications, e.g., usage in interactive kiosks within museums, or facilitating online virtual instruction during a pandemic.

Publications/Relevant Links

  • Vidal ECJE, Rodrigo MMT (2020) Hand Gesture Recognition for Smartphone-Based Augmented Reality Applications. In: Virtual, Augmented and Mixed Reality. Design and Interaction. Springer International Publishing, pp 346–366
  • Vidal ECJE, Rodrigo MMT (2021) VAMR Basketball on Head-Mounted and Hand-Held Devices with Hand-Gesture-Based Interactions. To be published in CCIS (Communications in Computer and Information Science), Springer International Publishing.