Foveated What?

Foveated What?
​

Ultra-high-resolution displays are difficult to make, but necessary for VR headsets to avoid what is called the ‘screen-door effect’ that comes from being able to see the spaces between pixels.  As VR displays are extremely close to the user’s eyes, screen pixel density (pixels/inch) becomes a major factor in the quality of the image, and the desire for ultra-high-resolution displays.  The problem is that the more pixels you try to squeeze into a small space, the smaller and closer together they have to be.  RGB OLED displays are produced by placing three (red, green, and blue) sub-pixels together to form a pixel, with each color being deposited through a metal mask, essentially a screen with very small holes.  While the mask material is particularly rigid, the more holes you cut in a sheet, the more flexible the mask becomes, and if the mask is even the slightest bit warped at any point, the display will not function correctly while making the mask thicker will cause ‘shadows’ that will misplace pixels and make the display unusable. 
 
These issues and more limit the pixel density of current VR displays and even a small amount of ‘screen-door’ can contribute to rapid fatigue for users, so display manufacturers continue to push the limits of display technology to move VR ahead.  As display technology moves forward, techniques for rendering images become more sophisticated, and higher display refresh rates that reduce motion blur, are now up to 120 times/second.  However the computing power needed to refresh the screen more frequently will drain the battery faster than a slower refresh rate, and that can be problematic for VR headset users.  Adding to the problem is that higher resolution displays inherently require more processing (more pixels) power for shading, artifact detection and removal, and a host of other functions, all of which require increased computing power.
.

Screen Door Effect - Source: Byteside.com

Some VR device manufacturers have come up with a solution taken from the human eye.  In the human eye, the retina, the portion of the eye that contains light sensitive receptors, ~7m cones and ~75m to 150m rods.  The cones respond to bright light and resolve color information, while the rods respond to low light with less color accuracy, which is why you see less color definition in low-light situations.  The rods and cones are not spread across the retina evenly, with a small (0.5mm) area, known as the fovea, covered only with cones that are packed tightly together, making it the point in the eye with the highest visual acuity.  As the eye focuses on an object, it uses that specialized region to provide the best possible image information to the brain, while the periphery is less detailed.  

Fovea focus rendering - Source: Rebuildyourvision.com

VR designers have taken this cue from the human eye and used it to maximize the systems computing power on rendering exactly what you are focused on, while reducing the rendering quality for the edges of the image, and reducing the overall power requirements of the system  In theory this should work exactly as the human eye does, which rations the ‘brain power’ needed for things on the edges of your vision, but the human eye can do one thing that is absolutely necessary to make this concept work, and that is movement.  The human eye optical system moves eye focusing structures, constantly refocusing whatever you are looking at on that most sensitive portion of the retina, the fovea, but VR displays are static, which makes it impossible for the system to reduce processing in areas where you are not focused, as it does not know where you are looking at any given time.

Figure 3 Cones Concentrated in the Fovea – Source: Webvision.med.utah.edu Figure 4 - Human Retina Detail - Source: Ortuno-Lizaran and Cuenca (2018)

​Not to be stymied, VR headset engineers came up with the idea of eye-tracking, by which using cameras , sensors, and infrared light sources to capture eye motion by reflecting a non-visible light source on the eye’s cornea and lenses, or in some cases the blood vessels of the eye.  This location information is fed to the system which then is able to lower the computing power needed to process the image outside of the user’s actual gaze, and as the user’s focus changes the system responds by shifting the ‘processing focus’.  Neural networks and Ai algorithms are also being developed that will ‘learn’ from the user’s movements and help the system to predict where the user’s gaze might move next to improve the system response time. 
While only a few VR systems use the concept of ‘foveated rendering’ combined with eye-tracking to improve VR performance, eye-tracking is becoming more the norm in the latest crop of VR headsets, which paves the way for the use of foveated rendering as a more common feature over the next few years.  Approximately 30.8% of VR headsets that have been or are scheduled for release this year will have eye-tracking and 62.5% of those already incorporate foveated rendering systems.  As eye-tracking becomes more common we would expect the technology to be adopted by most, if not all, mid to upper tier VR headsets, improving battery life or allowing for additional processing that will image quality.  As ultra-high-resolution displays push toward higher resolutions, such rendering systems will become even more important to VR designers who have to contend with the balance between battery power and the weight of a VR headset,