Mobile Augmented Reality

CAREER: Enhancing Energy Efficiency in Mobile Augmented Reality Apps

NSF Award Page: Link

PROJECT ABSTRACT — Mobile Augmented Reality (MAR) apps leverage the camera of mobile devices to augment the real user environment with a set of virtual objects. Users can thus achieve a high level of engagement by interacting with the augmented environment through the mobile user interface. Typical MAR apps require executing several compute-intensive tasks to render virtual objects and, using Artificial Intelligence (AI), analyze the environment. However, due to their relatively small form factor, mobile devices only have limited computing resources and energy available for the MAR app execution, which may lead to short battery life and poor user experience. To ensure the future success of MAR, it is thus important not only to conduct novel research on how to enable high energy efficiency in MAR apps but also to educate the future workforce in the best practices for energy efficient mobile software design. To achieve these goals, this project will propose an edge-assisted and energy-efficient framework for MAR apps. Specifically, the project designs runtime algorithms that (1), on each mobile device, control user-perceived performance and energy consumption by manipulating virtual object quality, AI-model complexity, and AI-task allocation on the heterogeneous computing resources, (2), on the supporting edge servers, ensure scalability by grouping similar virtual object decimation requests for minimized cross-user performance cost under limited edge computing and storage resources, and (3) will be implemented and tested on a physical testbed.

By providing a framework that makes MAR apps more energy-efficient, this project will help speed up the adoption of augmented reality in many different key sectors of modern society, such as education and workforce training, and will unlock the full potential of new technologies such as augmented remote meetings and augmented experiences in public places. The project will also result in the creation of new graduate-level courses to educate the future workforce on energy-efficient mobile resource management, new research opportunities for undergraduate students, and new summer camps for K12 students. The course material will be made available to other universities interested to introduce similar courses, projects, and summer camps. The results of this project will be disseminated through publications in top-tier conferences and journals. Similarly, all the source code of the proposed algorithms will be made available as open source.

Publications and Artifacts Related to This Award

PAPER ABSTRACT — Mobile Augmented Reality (MAR) apps may cause short battery life due to high-quality virtual objects rendered in the augmented environment. State-of-the-art solutions propose to balance energy consumption and user-experience using a static set of decimated object versions within the app. However, they do not consider that each object has unique characteristics, which highly influence how the user-perceived quality changes according to user-object distance and triangle count. As a result, they may lead to limited energy savings, a high storage overhead, and a high burden on the MAR app developer. In this paper, we propose eAR, an edge-assisted autonomous and energy-efficient framework for MAR apps designed to solve the limitations of state-of-the-art solutions. eAR features an offline software running on an edge server that leverages Image Quality Assessment (IQA) to model user-perceived quality for each virtual object in terms of triangle count and user-object distance. In addition, eAR features a runtime lightweight optimization algorithm that dynamically decides the most energy-efficient virtual object triangle count to request from the edge server based on (i) the per-object models of user-perceived quality, (ii) energy consumption models for mobile GPU and network interface, and (iii) a user path prediction system that estimates near-future user-object distances. eAR is completely autonomous and can be easily integrated into most MAR apps as an open-source library. Our results show that eAR can help reduce energy consumption by up to 16.5% while reducing storage overhead by almost 60% compared to existing schemes, with minimal MAR app developer effort and minimal impact on user-perceived quality.

eAR Artifact: Github Repository

Graduate Research Training Funded by this Award

Undergraduate Research Training in Mobile Augmented Reality

  • Saloni Gupta
    Honors Thesis Title: “Prediction of User Position in Augmented Reality Apps”
    Thesis: Thesis PDF
  • Affan Atif
    Honors Thesis Title: “Modeling User-Perceived Performance and GPU Utilization in MAR Apps”
    Thesis: Thesis PDF
  • Joseph Bommarito
    Honors Thesis Title: “AI Application Component Development for the Study of Tradeoff between User-Perceived Performance and AI Model Inference Throughput in Mobile Augmented Reality Apps”
    Thesis: Thesis PDF
  • Syed Safwaan
    Honors Thesis Title: “Framework Development for Energy-Efficient Mobile Augmented Reality Applications and Analysis of AI/AR Task Inference Resource Utilization”
    Thesis: Thesis PDF
  • Kanta Islam
    Research Experience for Undergraduates Site (Summer Academy in Sustainable Manufacturing): “Studuying Animated Virtual Object Energy Consumption/Performance Trade-off in Mobile Augmented Reality”

Invited Talks

  • University of Michigan – Dearborn, December 9th, 2022: Flyer