Major Breakthrough for AR Glasses! A Team From Shanghai Jiao Tong University Has Proposed A Mask-balancing Scheme, Overcoming The Challenge of Light Transmission in Occlusion-capable AR Displays.

In 2026, a collaborative team comprising researchers from Shanghai Jiao Tong University, Graz University of Technology (Austria), Nara Institute of Science and Technology (Japan), and Zhejiang University of Science and Technology published a landmark study titled "Mask Balancing: Perception-Driven Dynamic Visibility Enhancement for Occlusion-Capable Optical See-Through Head-Mounted Displays" in the prestigious journal IEEE Transactions on Visualization and Computer Graphics. The study introduces an innovative "Mask Balancing" solution that addresses a critical industry challenge—excessively low real-world light transmittance in occlusion-capable optical see-through AR head-mounted displays (OC-OSTHMDs)—by simultaneously optimizing polarization optical paths and human visual perception, thereby clearing a key obstacle to the commercialization of consumer- and industrial-grade AR glasses.

I. Core Industry Pain Point: Pixel-level Occlusion Causes Irreversible Luminance Loss

Optical see-through AR head-mounted displays (OC-OSTHMDs) capable of mutual occlusion between virtual and real elements are essential for rendering virtual objects with realistic depth and seamless integration into the physical environment. They utilize spatial light modulators (SLMs) to achieve pixel-level image occlusion, thereby resolving the "floating phantom" issue associated with virtual images in traditional semi-transparent AR systems. However, existing optical architectures suffer from an unavoidable drawback regarding light transmission:

1. To achieve the occlusion effect, light from the real-world scene must undergo polarization, resulting in an immediate 50% loss in brightness;

2. Signal attenuation caused by the stacking of multiple components—such as SLMs, Polarizing Beam Splitters (PBS), and optical combiners—limits theoretical maximum light transmission to under 20%, with actual prototypes often achieving less than 7%;

3. Multiple optical path reflections and spectral losses from polarization components further reduce ambient brightness; in dim indoor environments, the real-world view becomes excessively dark, hindering user observation and interaction while posing safety risks.

Previous solutions have focused primarily on hardware iterations—such as increasing SLM transmittance, switching to reflective LCoS/DMD technologies, or exploring photochromic materials. However, optical components face physical limits on light transmission, and photochromic approaches suffer from drawbacks like slow response times and insufficient pixel-level occlusion precision, failing to fundamentally resolve the conflict between virtual and real-world brightness levels. Balancing the need for realistic occlusion depth for virtual objects with the requirement for high visibility of the real world has become the critical bottleneck for the mass commercialization of next-generation AR devices.

II. Technical Innovation: Dual-Polarization Optical Path Fusion + Perception-Driven Dynamic Modulation

Moving beyond the approach of merely improving individual hardware components, the team from Shanghai Jiao Tong University proposed a "mask-balancing method." This method combines low-cost polarization optical path modifications with a quantitative model of human visual perception to dynamically adjust the light transmission ratio between the real-world scene and the virtual imagery. The entire solution requires only the addition of a rotating linear polarizer (Balancing LP) to an existing OC-OSTHMD (Optical See-Through Head-Mounted Display), offering high compatibility and low modification costs.

1. Dual-Polarization Optical Fusion Architecture

Natural light is split by a polarizing beam splitter into two independent polarization paths, each serving a specific function:

s-polarization occlusion path (masking vision): Passes through the original AR optical system and utilizes an SLM to achieve pixel-level occlusion of the real world by virtual content, providing a sense of spatial depth, albeit with significant light loss;

p-polarization direct-transmission path (native real-world vision): Bypasses all occlusion-related optical components, fully preserving the ambient scene's original brightness without additional light attenuation.

The system employs a rotating module to adjust the angle φ between the linear polarizer and the polarizing beam splitter, precisely controlling the transmission ratios of the real-world light and the virtual image light. Transmission rates are governed by the following formulas: direct real-world transmission \(T_{pass} = \cos^2φ\); and transmission for the occluded real-world view and virtual image \(T_{mask}/T_{image} = \sin^2φ \cdot T_{HMD}/T_{display}\). Adjusting this angle allows for real-time switching of visual weighting: in low-light environments, the proportion of the direct-transmission path is increased to brighten the real-world view; in high-light environments, the direct-transmission ratio is reduced to enhance the contrast of virtual objects.

2. Perception-Driven Real-Time Modulation System

The entire dynamic balancing logic operates in real-time based on a dual-sensor approach—combining eye tracking with scene visual analysis—and aligns with the principles of human visual perception rather than relying solely on calculations based on optical hardware specifications:

Real-world visibility assessment: A scene camera captures the real-world view, and the image is decomposed into seven band-pass levels using a Peli contrast pyramid. The system selects the Root Mean Square (RMS) contrast of the 4.7 cpd (cycles per degree) mid-frequency band as the evaluation metric. By integrating the Barten Contrast Sensitivity Function (Barten CSF) with pupil diameter data from the eye-tracking camera, the system calculates—in real-time—the minimum contrast threshold required for the human eye to perceive the real-world scene. If visibility falls below this standard, the system automatically adjusts the polarization angle to increase the brightness of the direct optical path.

Virtual object visibility assessment: Power Spectral Density (PSD) analysis is performed on virtual models to distinguish between high-frequency textured models (e.g., maple trees, Christmas trees) and low-frequency minimalist models (e.g., rabbits, teapots), with Weber contrast used to quantify the clarity of the virtual imagery. The team calibrated perceptual thresholds through two sets of human-subject experiments: the critical Weber contrast for recognizing virtual textures was determined to be 0.537, while the critical threshold for recognizing lighting and shadow effects was 0.443. When the contrast of the virtual image falls below these thresholds, the system automatically increases the weighting of the occlusion optical path to enhance the perceived texture and definition of the virtual objects.

The system employs a Kalman filter to smooth fluctuating pupil data captured by the eye-tracking camera, eliminating equipment noise and ensuring smooth, stable adjustment during transitions between different lighting conditions. The algorithm suite operates using dual-thread parallelism, with scene image analysis running independently of the main rendering thread; this results in low computational overhead, making it compatible with mainstream AR real-time rendering engines.

III. Human Perception Experiments: Quantifying Visual Thresholds and Validating the Proposed Scheme

To establish a precise baseline for perceptual balance, the team conducted three sets of standardized user experiments—all approved by the Human Research Ethics Committee of Shanghai Jiao Tong University—recruiting a total of 36 participants with experience in AR or computer graphics.

1. Virtual texture perception experiment (12 participants): Textures of large, medium, and small sizes were used to simulate virtual objects with varying spatial frequencies, covering four levels of indoor ambient illuminance (29–626 cd/m²). Results showed that high-frequency, fine textures required higher contrast for visual recognition, whereas recognition thresholds for medium- and low-frequency models differed minimally. Factors such as gender, AR usage experience, and pupil size had no significant impact on perception thresholds, indicating highly consistent perceptual patterns across the participant group.

2. Virtual lighting and shadow perception experiment (24 participants): Paired virtual models (e.g., teapot, robot, Christmas tree)—one with lighting/shadow reflections and one without—were compared. Participants adjusted the polarization angle until the two models appeared visually identical. The experiment demonstrated that lighting and shadow effects inherently contain high-frequency details, allowing them to be easily distinguished even on low-frequency geometric models; spatial frequency showed no statistically significant effect on perception thresholds for lighting and shadows.

3. Comprehensive comparative experiment in multi-lighting scenarios (12 participants): A tabletop prototype was constructed to simulate four real-world lighting conditions: indoor lights-off (8 cd/m²), indoor lights-on (135 cd/m²), outdoor shaded (287 cd/m²), and outdoor bright light (414 cd/m²). A blind comparison was conducted among three conditions: the proposed solution (MB), traditional unoccluded AR (AR), and static occlusion headsets (OC).

· Low-light indoor environments: Traditional AR held the lead (47%), followed closely by this solution (44%), while traditional occlusion devices trailed at only 8%;

· Bright indoor environments: This solution led with a 42% preference rate;

· Outdoor bright light / shaded scenarios: This solution achieved preference rates of 58% and 53% respectively, significantly outperforming traditional solutions;

· Overall assessment: 50% of participants preferred the mask-based balancing system, 36% chose traditional occlusion headsets, and only 14% opted for standard see-through AR.

Real-world footage clearly demonstrates the differences: with traditional occlusion-based AR, the real-world view appears severely washed out in all environments; with standard non-occlusion AR, virtual content virtually disappears in bright outdoor light; in contrast, mask-balancing technology adaptively adjusts brightness—maintaining a clear view of the real world indoors while fully preserving the textures, lighting, and shadows of virtual objects in bright outdoor conditions—thereby ensuring an optimal visual experience in both scenarios.

IV. Application Value and Industry-Wide Paradigm Shift

1. Diverse Implementation Scenarios

This technological breakthrough extends beyond consumer-grade AR glasses, offering immense practical value in professional industrial settings:

· Industrial O&M: Overlaying blueprints and annotations during equipment maintenance while maintaining a clear view of the physical mechanical structure;

· Medical assistance: Intraoperative image overlay and rehabilitation training, combining the real-world view of the human body with digital annotations;

· Applications such as education and training, in-vehicle navigation, remote collaboration, and architectural design, significantly enhancing AR device usability in complex lighting environments.

2. A New Approach to AR Display Design

This research proposes a paradigm shift in the industry: while AR optical design previously focused primarily on optimizing physical parameters such as hardware light transmittance and resolution, this study demonstrates that human visual perception is a critical design dimension that cannot be overlooked. Previous research by the team confirmed a significant discrepancy between actual human visual perception and measurements taken by optical instruments, indicating that optimizing hardware alone cannot achieve the best visual experience. A perception-driven control scheme can overcome the physical limitations of optical hardware; it simultaneously enhances the quality of both virtual and real-world imagery—without compromising device size or cost—while ensuring user visual comfort and operational safety.

V. Current Limitations and Future Research Directions

There is still room for optimization in the current desktop prototype, and the team has planned the following directions for future iterations:

Occlusion accuracy optimization: Real-world brightening can cause slight light leakage at occlusion boundaries; future improvements will mitigate this by integrating high-brightness micro-OLEDs, color compensation algorithms, and task-oriented balancing strategies (prioritizing virtual occlusion accuracy in high-precision modeled scenes).

Smooth brightness adjustment: Current polarization rotation speeds are high, leading to abrupt brightness changes that can cause visual discomfort; future development will focus on algorithms for seamless, gradual brightness transitions.

Binocular adaptation prototype: The current device is monocular; the next step involves building a binocular mask balancing system and incorporating a binocular visual contrast perception model.

Automatic object spectral classification: Enables real-time, automatic recognition of high- and low-frequency features of virtual objects, eliminating the need for manual configuration of perception thresholds.

The mask-balancing technology proposed by the team at Shanghai Jiao Tong University resolves the long-standing brightness conflict that has hindered the widespread adoption of optical see-through AR. By combining hardware improvements for dual-polarization optical paths with a perception-driven dynamic balancing algorithm, the solution is easy to implement and compatible with mainstream AR optical architectures. Validated through extensive human-subject experiments and diverse lighting scenarios, it simultaneously enhances the visibility of the real-world environment and the realistic texture of virtual objects across all conditions—from dim indoor settings to bright outdoor environments. By deeply integrating visual perception science with near-eye display engineering, this technology paves a new optimization path for next-generation AR glasses, offering a viable solution for the all-day, real-world deployment of AR devices in consumer, industrial, and medical sectors, thereby advancing optical see-through AR from laboratory prototypes to practical products.

Source: CINNO