average guy or gal in the street might not know much about this 5-day event, or even care.  In most instances there is little reason for the average Apple user to pay much attention to what is discussed at the conference, as they will likely not see the actual changes to their devices until much later in the year if they notice them at all.  But this year could be different in that not only will Apple present developers with a new version of iOS (iOS 17), and new versions of iPad OS, MacOS, Watch OS, and TV OS, as they typically do, but the company is expected to announce the long-awaited XR headset that has been rumored for years.
Apple’s XR development program has been around in its current form since 2017, but Apple has been making acquisitions relating to AR/VR since 2014 (see below) and allows developers access to a number of tools and resources that they can use to create AR and VR applications for iOS and iPad OS and have access to the Apple Store.  But Apple itself has not championed a physical XR headset, disappointing developers and fans a number of times in the past.  The most recent ‘rumored’ device, the Apple Reality Pro, would be Apple’s entry into the XR hardware space, and would be a driver for the industry, that in the long run, would likely have even more impact than Meta’s (FB) Quest series of VR headsets, as Apple’s
hardware following, with over 2 billion active devices, to Meta’s ~20m Quest headsets sold.  Of course, Meta has over 2b active Facebook users, along with Instagram, Messenger, etc., but hardware is Apple’s thing.
If Apple does decide to make an announcement concerning an XR device, it will be to stimulate developers to build or modify apps to operate under what will likely be a new sub-OS specifically designed for Apple’s XR hardware, seemingly called xrOS.  This code will work under iOS and will allow the device to communicate and process applications that reside on a secondary device, such as an iPhone or iPad.  Making the headset a ‘non-standalone’ device, reduces the bulk, weight, and computing power on the headset itself, making it more comfortable to wear, a concept not lost to Apple developers who are likely following many Apple design mantras that are consumer oriented over technology oriented, while tethering also means at least one other Apple device must be purchased or owned for the headset to be operated.
Expectations are that the headset battery, which in some headsets is in the headset itself, will likely be external, perhaps a belt or pocket clip-on, with a cable to the headset, while the communication between the headset and the paired device could be a cable or wireless, but the Apple XR device is also expected to not sport controllers, which would be a departure from the norm.  It has been suggested that Apple will use gestures to control the headset, with a large number of cameras scanning body movements, eye movements, area mapping, and even facial expressions for information that the xrOS can use to keep the user’s field of view correct.
The displays are expected to be something close to 4K micro-OLED displays, with on-board processing through Apple’s own SoC, either the M2 or a specially designed processor, with a focus on power efficiency given the battery-driven nature of the device, but while developers will be interested in the internals, consumers will be more concerned with how it looks, how it feels, and can it do things other AR/VR headsets cannot, and the idea that the device can operate as both a VR and AR device is not new.  Other VR headsets allow a black mask to be removed from the headset, allowing it to operate as an AR device through cameras, so the proof will be in the applications themselves and how easily the headset operates. 
​As we have noted recently eye-tracking is becoming  more popular as it facilitates foveated rendering, and movement tracking are common, so it will be incumbent on Apple to devise a gesture system that is intuitive for consumers, but on an overall basis, for Apple to have a successful product, especially on that is expected to cost between $2,500 and $3,000, it must be able to provide functionality, as while there will be an initial rush from ‘I always buy the newest Apple product’ people, it better provide more than ‘coolness’, and much of that will come from applications.  While the iPhone is a well-designed device, as are most Apple products, it also serves as a communication hub for its users, and while we do not expect an Apple XR headset to become as ubiquitous as an iPhone in the near future, it has to be able to do more than play games or allow you to buy things in the Metaverse.  We keep our expectations low but our hopes high.

ederico Capasso, a former Bell Labs (NOK) applied physicist who was one of the inventors of the quantum cascade laser, and now a Senior Research Fellow at Harvard, came up with the idea of using metasurfaces for flat optics in 2016 and has been leading research in this new field since, with over 70 US patents and over 500 peer-reviewed journals (not all on metasurfaces).  The idea of metasurfaces, or sub-wavelength thickness structures that are placed in a horizontal manner, is similar to that of more typical optical lenses that focus light using refraction, while metasurface optics use small nanostructures to scatter light and by capturing and changing a number of optical characteristics, including phase, polarization, and spectrum, can be used for a number of optical processes that are typically done with glass or similar medium.  The shape and patterning of these nanoobjects determines their optical characteristics, opening an almost infinite range of possible shapes, sizes, and materials for these lenses, which are typically in the range of 100nm thick, or 1/100,000 of the equivalent optical lens.

What makes metalenses important is that they are produced using photolithography or ALD (Atomic Layer Deposition), both of which are used in the manufacturer of semiconductors and a few years ago MIT scientists developed a computational technique that determined the optimal makeup and arrangement of metalens elements, enabling designers to meet specific optical parameters, reducing prototype and rework timelines.  As, in theory, the process by which the metrastructures scatter and re-emit the source light is !95% efficient, such lenses are thought to have an advantage over conventional glass lenses in terms of cost, size, weight, and performance.  Further a single refractive lens (glass) can project an undistorted image onto a curved surface, but in order to project on a flat surface, additional lenses are needed to eliminate the curvature distortion.  As much in the image sensor world is flat, only a single metalens is needed for a flat (2D) surface, such as in a smartphone, eliminating the need for the lens ‘bumps’ seen on most smartphones.
Companies like Imagia (pvt), NIL Technologies (pvt), and Metalenz (pvt) are developing applications that try to simplify the optical stacks used in a number of consumer electronics’ applications.  STMicro (STM) is using a metalens in its 6/22 release of a ToF (Time-of-flight) module, using Metalenz technology that is produced on STM’s silicon lines.   The device uses two metalenses that substitute for the more typical optical lenses and has a power consumption of 31.3% less than that of the optical version in continuous mode.

​While metalenses sound game-changing, and they are to a degree, they have two flaws.  The first is that they are small.  Achromatic lenses, meaning that the color remains the same through all parts of the lens, is relatively easy to do with glass, essentially making the lens thicker as it gets larger.  Metalenses are achromatic when small (few hundred microns), but have color aberrations as they get larger, which limits them to very small applications.  There have been positive lab results using a large number of such lenses to stich together a wide-angle image, but with no commercial application yet.  The efficiency of metalenses is the 2nd flaw, and while such is over 80%, meaning 80% of the light that enters the metalens is ’converted’, Fresnel lenses (plastic on glass) and pure glass lenses are higher, and as noted below, this can be an important factor in some applications.
One area of particular interest is VR, where complex optics add considerable weight and bulk to headsets.  Flat metalenses can be less than 1mm thick and are rectangular, which matches the format of most digital image sensors.  With pixel pitch below 10um, such lenses allow pixel or even sub-pixel beam steering and can replace optics that require a number of lenses or steering mechanisms.  Since VR displays are small, typically 1” or less, the size issue is less of a factor, but efficiency is more so, as the clarity of the image in VR is key, but considerable research continues and as better materials and patterns are found, there is considerable hope that higher efficiencies can become available, with a recent lab process having 94% efficiency, albeit not in a commercial setting.  The use of DUV (Deep Ultraviolet) patterning has led to higher efficiencies but work still needs to be done to move those results from the lab to commercialization.
All in, if metalens research is able to conquer a few limitations, they open a whole new world for AR/VR and a host of other applications that rely on physical optics.  Glass manufacturers and optical component companies do not have to worry about their business yet, as metalenses are still primarily a highly specialized field that has potential but no guarantee that it can be competitive with 3-dimensional optics, but every ounce of weight that can be removed from a VR headset will make them more compatible with the general public and metalenses are trying to fill that bill.

Ultra-high-resolution displays are difficult to make, but necessary for VR headsets to avoid what is called the ‘screen-door effect’ that comes from being able to see the spaces between pixels.  As VR displays are extremely close to the user’s eyes, screen pixel density (pixels/inch) becomes a major factor in the quality of the image, and the desire for ultra-high-resolution displays.  The problem is that the more pixels you try to squeeze into a small space, the smaller and closer together they have to be.  RGB OLED displays are produced by placing three (red, green, and blue) sub-pixels together to form a pixel, with each color being deposited through a metal mask, essentially a screen with very small holes.  While the mask material is particularly rigid, the more holes you cut in a sheet, the more flexible the mask becomes, and if the mask is even the slightest bit warped at any point, the display will not function correctly while making the mask thicker will cause ‘shadows’ that will misplace pixels and make the display unusable. 
 
These issues and more limit the pixel density of current VR displays and even a small amount of ‘screen-door’ can contribute to rapid fatigue for users, so display manufacturers continue to push the limits of display technology to move VR ahead.  As display technology moves forward, techniques for rendering images become more sophisticated, and higher display refresh rates that reduce motion blur, are now up to 120 times/second.  However the computing power needed to refresh the screen more frequently will drain the battery faster than a slower refresh rate, and that can be problematic for VR headset users.  Adding to the problem is that higher resolution displays inherently require more processing (more pixels) power for shading, artifact detection and removal, and a host of other functions, all of which require increased computing power.
.

Some VR device manufacturers have come up with a solution taken from the human eye.  In the human eye, the retina, the portion of the eye that contains light sensitive receptors, ~7m cones and ~75m to 150m rods.  The cones respond to bright light and resolve color information, while the rods respond to low light with less color accuracy, which is why you see less color definition in low-light situations.  The rods and cones are not spread across the retina evenly, with a small (0.5mm) area, known as the fovea, covered only with cones that are packed tightly together, making it the point in the eye with the highest visual acuity.  As the eye focuses on an object, it uses that specialized region to provide the best possible image information to the brain, while the periphery is less detailed.  

VR designers have taken this cue from the human eye and used it to maximize the systems computing power on rendering exactly what you are focused on, while reducing the rendering quality for the edges of the image, and reducing the overall power requirements of the system  In theory this should work exactly as the human eye does, which rations the ‘brain power’ needed for things on the edges of your vision, but the human eye can do one thing that is absolutely necessary to make this concept work, and that is movement.  The human eye optical system moves eye focusing structures, constantly refocusing whatever you are looking at on that most sensitive portion of the retina, the fovea, but VR displays are static, which makes it impossible for the system to reduce processing in areas where you are not focused, as it does not know where you are looking at any given time.

​Not to be stymied, VR headset engineers came up with the idea of eye-tracking, by which using cameras , sensors, and infrared light sources to capture eye motion by reflecting a non-visible light source on the eye’s cornea and lenses, or in some cases the blood vessels of the eye.  This location information is fed to the system which then is able to lower the computing power needed to process the image outside of the user’s actual gaze, and as the user’s focus changes the system responds by shifting the ‘processing focus’.  Neural networks and Ai algorithms are also being developed that will ‘learn’ from the user’s movements and help the system to predict where the user’s gaze might move next to improve the system response time. 
While only a few VR systems use the concept of ‘foveated rendering’ combined with eye-tracking to improve VR performance, eye-tracking is becoming more the norm in the latest crop of VR headsets, which paves the way for the use of foveated rendering as a more common feature over the next few years.  Approximately 30.8% of VR headsets that have been or are scheduled for release this year will have eye-tracking and 62.5% of those already incorporate foveated rendering systems.  As eye-tracking becomes more common we would expect the technology to be adopted by most, if not all, mid to upper tier VR headsets, improving battery life or allowing for additional processing that will image quality.  As ultra-high-resolution displays push toward higher resolutions, such rendering systems will become even more important to VR designers who have to contend with the balance between battery power and the weight of a VR headset,
 

Way back in June of 2019 we noted that a small California-based company known as Mojo Vision (pvt) was in the process of developing a contact lens with a Micro-LED display that was to be 1/50th of an inch across, with a pixel density of 14,000 ppi (pixels/inch) that would allow users to see images and data superimposed on the user’s vision, without the use of AR glasses.  The company, which had raised $108m at that juncture, had the backing of Motorola (MSI), Google, LG Electronics (066570.KS) and Hewlett Packard (HPE), was in stealth mode at the time and we had little contact until last year when it showed a prototype of its AR contact lens, which included a power management chip, a communication module, motion sensor, accelerometer, gyroscope, magnetometer, and micro-processor, along with the Micro-LED display in a breathable plastic shell..

​ 
Unfortunately the target market, the visually impaired, proved to small to support the necessary product development, so the company shifted focus toward the development of ultra-high-resolution displays based on Micro-LEDs and recently indicated that it had developed the technology far enough to reach a pixel density of 28,000 ppi, using sub-micron sized blue Micro-LEDs and quantum dots for color shifting.  As 14,000 ppi already would be the highest pixel density display available, the push to 28,000 ppi would be a significant leap ahead.  Mojo Vision has also developed the necessary backplane and bonding technology and advertises the ability to manufacture 300mm GaN on Silicon at high volumes.
Along with the shift in focus comes a 75% workforce reduction, as the contact lens project required further funding that became scarce as the company tried to miniaturize the battery and processor to include them in the device, as the prototypes used external power and processing, so Mojo Vision is added to the list of those that have tried to develop such a device, including Google’s parent Alphabet () that abandon a contact lens development project that could also monitor glucose levels in a user’s tears.  Hopefully Mojo Vision can capitalize on their Micro-LED technology, which seems closer to actual availability than the contact lens concept and would certainly be a step forward for Micro-LED AR displays if it can be produced commercially at a reasonable cost.

Yesterday we spent some time looking at the characteristics of AR glasses, particularly focused on the progression of the technology behind those devices.  While the technology is certainly important and will help to drive the industry forward, we noted that over the last year or two, the AR space has been seeing new interest from larger consumer electronics companies, some of whom are new to the space but have significant clout in other CE product categories.
In the very early days of AR, the industry was dominated by large CE and technology companies as indicated below, however as there was relatively little consumer traction, by 2017 all of the AR players were small and/or private companies.  This continued until 2019 when both Microsoft (MSFT) and Google (GOOG) released updated versions of previous models, now more oriented toward the business community.  This drove a new crop of small AR developers to enter the space in 2020, which saw a large expansion in the number of models released, although all were from small AR companies.  By 2021 the space had attracted a number of Chinese CE companies, as well as small players, and last year major Chinese CE players offered their first forays into the AR space.
While detailing all of the companies in the AR space would be an arduous task, we show a quick summary of those companies having announced AR products that are expected to be released this year.  Some are well-known, while others are just passing through start-up mode.  We expect more such announcements this year, and potentially an MR device from Apple (AAPL), but even at this early point in the year, there are a 13 brands scheduled to release AR product this year, only two short of the total number of brands releasing Ar product for the entire 2022 year, so we expect the 2023 total will be higher than last year as the space develops better hardware and continues to look for practical applications.

​Here are the companies expected to release AR product this year:
Digilens (pvt)
Owned by SBG Labs (pvt) – Sunnyvale, CA – Purchased at foreclosure sale.  Leader in holographic Optical waveguides.  Investors – Dolby, Samsung, GLW, OLED
 
TCL (000100.CH) – $24.11B Sales - 2021
Broad-based Chinese CE company.  Appliances, TV, Smartphone, etc.  Owns  – Chinastar (pvt)
            
Oppo – (pvt)
Owned by BBK Electronics (pvt) who also owns Vivo (pvt), OnePlus (pvt), and iQOO (pvt).  Major Chinese smartphone brand.  Also produces other CE products. ~17,000 employees
 
Ximmerse (pvt)
Guangdong, China – Investors include Lenovo (), & Qualcomm (QCOM)
“Rhino” series of mixed reality devices.  Specializes in position and tracking.
 
Dream Glass (pvt)–
Fremont, CA.  Small VC backed AR Developer
 
Lenovo (992.HK)
Large Hong Kong based Chinese CE manufacturer best known for laptops. – $60.74B Sales – 2021
 
Engo (pvt)
Owned by MicroOled (pvt) – Producer of Micro-OLED displays and optical modules based in Grenoble, France.  Also promotes Activelook™ smart sport glasses.
 
Guangli (pvt)
Small Hangzhou, China based producer of AR glasses specifically designed for swimmers
 
Brilliant (pvt)
Owned by Brilliant Labs (pvt), a small NFP based in New Brunswick, Canada.  Product is an open platform monocle that can be used for educational purposes.
 
Nimo (pvt)
Milpitas, CA based (India really) company building AR glasses that are a substitute for a laptop or computer.
 
Campfire (pvt)
San Mateo, CA start-up that is focused on a collaborative AR platform that is currently oriented toward CAD and is built around an easy to use AR device that also converts to VR.
 
Rokid (pvt)
Hangzhou, China based AR developer that has released Ar devices each year since 2020 with a focus on sources that have a video output.  Investors include Bilibili (9626.HK), a major Chinese streaming service
 

While we are on the subject, we thought it helpful to share a few facts concerning the AR/VR space.  In 2022 15 VR headset models were released from 10 brands, down slightly from 2021 when 17 models were released, again from 10 brands, with 2022 seeing 2 new brands releasing product and two that released in 2021, skipping 2022.  In the AR space, in 2022 17 models were released from 15 brands, up from 13 models in 2021 from 12 brands, with 8 new brands entering the AR space last year.  As smaller (and sometimes larger) companies tend to announce new models with release dates in the future, we enter 2023 with 22 announced VR models, 5 of which were announced in 2023, 11 last year, and 6 in 2021. 
While carrying an announced but unreleased model since 2021 might seem foolish, the Sony (SNE) Playstation VR2 headset was announced in October 2021 and will ship (hopefully) in late February, 730 days after its announcement, so while some announcements might fade into non-existence, we carry them until we know they are officially withdrawn.  The AR space enters 2023 with 14 announced but unreleased models from 13 brands, with two announced this year, 10 remaining from 2022, and 2 left over from 2021. 
One factor for AR/VR headsets that we monitor is whether the devices are stand-alone or tethered,, with standalone meaning they do not have to be attached (either by wire or wirelessly) to a separate device, such as a smartphone or PC.  There are both positives and negative to each mode, with the tethered devices requiring less local (on headset) computing capacity and power, usually equating to a lighter headset, at the cost of being less mobile.  Of the VR devices announced but unreleased this year 63.6% are standalone, while last year that spiked to 80.0%, with 2021 coming in at 64.7%,averaging 69.4% over the current and past two years.  Given the computing and power intensive imaging necessary for VR headsets, we would expect standalone VR headset share to remain between 65% and 75% this year.
In theory AR headsets should be less computing and power intensive and should therefore show a higher rate of standalone vs. tethered devices, however that is not the case as the current and past two year standalone average for AR headsets is 56.5%, lower than that of power hungry VR devices.  That said, the sequential change goes from 45.4% standalone in 2021 to 52.9% in 2022, and 71.4% this year, which establishes a significant trend toward standalone AR.  While we expect some of that trend can be accounted for as we noted above, it is important for consumer AR glasses to be ‘good-looking’, which implies thin, light, and unobtrusive, which we expect has pushed AR headset designers to trade off extended battery life in lieu of a more normal looking device that allows the user full freedom of movement, leading us to expect the trend for AR headsets to be standalone to continue.
Headset price is an obvious factor that has considerable influence on both AR and VR headset sales, but the price range for VR and AR headsets can be so varied that the inclusion of a single high-priced headset can skew the average drastically.  That said, with that understanding, the average price of VR headsets continues to decline, dropping from $3,083 in 2021 to $2,249 in 2022, and $1,374 for those expected to be released this year, although we note that there are a considerable number of potential price variables for some VR headsets, with prices ranging from $300 to $38,500.  AR headsets are generally less expensive that VR headsets, with less display and optical hardware and a more simplistic frame,, but again there are big variations in prices, with a range between $99 and $3,300, including those we are able to price that are to be released this year.  The average price for AR headsets has remained between $965 and $1,002 in 2021 and 2022, inclusive of a relatively small sample of those for release this year.
Even with a large number of variable for both AR and VR headsets there are few consistencies.  Every VR headset for which we have chipset data, produced since 2019 has used a Qualcomm (QCOM) chipset, and while the models have changed over time from the Snapdragon 835/845 to the XR1 and currently the XR2, Qualcomm has established itself as the basis for VR processing.  AR is a bit less narrow, with an occasional Intel (INTC) or AMD (AMD) chipset showing up, but again, Qualcomm is the de facto leader in the AR space.
Displays, especially in VR headsets continue to evolve, with all display types (LCD, OLED, Mini-LED, Micro-OLED, and QLED) all represented in this year’s potential devices, a change from the almost universal LCD displays used in VR in previous years.  This year’s AR headset displays are also quite varied as to display types, but given the lesser display requirements of AR, AR displays, all display types have been common in those headsets for a number of years, including LCoS, which is not used in VR.  All in, we track over 35 variables for AR and VR headsets in our database, and while it can be difficult to get data from some brands, there are many discernable trends that become visible.  More to come…

​Rumors and speculation continue to swirl around Apple’s (AAPL) plans for its entry into the world of AR/VR, with a multitude of hardware and options ranging from a relatively expensive (neighborhood of $3,000) and/or a low-priced alternative to compete with mass market VR headsets such as the Oculus (FB) Quest 2 or the Pico (pvt) Neo 3.  Apple’s entry into the AR/VR market will be a watershed event for the industry, but we see a trend that will blur the lines between AR and VR over the next year or two.  Currently there is a distinctive difference between headset designs for AR and VR, with AR headsets trending toward almost normal looking eyeglasses and VR headsets still more massive and obtrusive.  The technology behind AR was developed to combine the user’s view of the outside world, and an artificial image that can be superimposed on a normal visual image.  VR systems do not use external visual images, and in fact, require a darkened, enclosed space inside the headset to be fully effective.
This dichotomy is obvious in most cases but we have noticed that both VR and AR headset developers are more extensively looking over their shoulders and wondering if they each might be missing out on something.  In the vase of AR, we have seen a few systems that include technology that can darken the image an AR user sees, emphasizing the artificial overlay and reducing the impact of the pass-through image.  As the display overlay systems become more sophisticated in AR headsets, it seems AR developers are trying to gain at least some access to VR, albeit in a rudimentary way, as the AR market develops broader applications.
The same seems true for VR, where a number of systems either allow the user to remove the light blocking cover typically seen on VR headsets, which allows the system to function like an AR device, at least in a limited capacity and other systems use pass-through cameras to mix the outside world with VR.  While it might seem that giving a VR headset the ability to be used for AR is overkill, we expect the VR case is more likely used to allow a VR user the ability to intermittently ‘see’ someone or something in the user’s location, without removing the headset, but we expect the trend toward such mixed use devices will continue, making data collection a bit more difficult as AR and VR devices move toward each other’s home ground.
We believe both AR and VR headset developers will continue to improve this ‘mixed reality’ feature set and over the next two years, while there will still be dedicated AR and VR headsets, a greater number of new releases will gravitate toward becoming mixed reality devices, in order not to miss an unexpected growth path of application classification, and while Apple’s entry into the space seems to be inevitable, we expect Apple will be careful not to limit their market specifically to AR or VR.  Perhaps that might not be the case with the company’s first AR/VR iteration, but we believe Apple is smart enough not to pick a side and hope that consumers agree.

While rumors that Apple (AAPL) has postponed its AR device project in favor of an XR device to be released this year or next, there has been considerable activity concerning AR devices before and during CES 2023.  While we still favor AR over VR in terms of practical applications, the extended reality universe has been gradually attracting larger CE companies, and with those companies comes the need for high volume products.  Gaming is the most obvious application for VR, particularly as enthusiasm for the Metaverse wanes a bit, and as a ~$200b market, the gaming software market makes the hardware that feeds such a large space quite attractive to CE manufacturers looking to become a new player or expand their share of CE hardware.
That said, while VR is the focus for most hardware manufacturers, the more recent trend has been toward creating a mixed environment that allows VR headsets to also be used for AR, and while this is really a convenience for VR users, it adds a bit more credibility to the AR space, which saw 17 models from 15 brands released in 2022.  In that group were 8 brands that were new to the AR space, although some have VR offerings, and of those eight, three are likely recognizable to our readers, Oppo (pvt), Xiaomi (1818.HK), and Huawei (pvt), all well-known Chinese CE companies with representation in a number of CE segments.  Further, announcements of upcoming AR devices were made by 13 brands, including TCL (000100.CH) and Lenovo (992.HK), also major Chinese CE companies, indicating the shift from small, private brands to large Chinese high-volume manufacturers, although we do note that the earliest supporters of AR hardware were Google (GOOG), Microsoft (MSFT), and Meta (FB) years ago.
We believe it is important for investors to understand a bit about both the AR and VR space, both from the perspective of being able to understand trends in the industry and to better understand how new products will affect the space and the private and public companies involved.  SCMR LLC has developed a comprehensive database of both AR and VR products (headsets) which breaks down the hardware into relatively simple categories, but before we detail some of that data, we believe it would be helpful for investors to understand the basics concerning AR headsets.
The idea of Augmented Reality, in a limited form has been around for many years but much of the early work on developing a system that could provide such visual information to a user was developed by and for the military, either as a ‘simulator’ or a HUD (Heads-up Display), with much of the research being done at various universities.  Early pioneers such as Evans & Southerland (pvt) and Boeing (BA) were limited by the need for computing power and graphic capabilities, but in 2013 Google opened the concept of AR to the consumer world with the release of Google Glass to a limited group of ‘Explorers’ (You could ‘explore’ for $1,500 & ~8,000 did), and roughly a year later to the general public, and followed with ‘Enterprise’ and ‘Edition 2’ models in 2017 and 2019, although consumer oriented production was ended in 2015.  The device was able to project data to the user’s eye using either a touch system on the side of the device or by voice commands, and could take pictures and short videos and was able to project requested directions, messaging, Google search results, or connect to your smartphone.
Unfortunately, the cost was high and while the headset was remarkably sophisticated for its time, the concept of walking down a street with such glasses while viewing data was a bit ahead of its time, and privacy concerns about not knowing when the Google Glass user was recording, left many concerned over privacy.  No matter what the problems were with Google Glass at the time, you have to give Google considerable credit for trying to bring AR to the consumer market, and despite Google’s problems with ‘glass’, Microsoft released the Hololens in 2016, a Windows AR device that was oriented toward business applications (~$3,000), and Oculus (now Meta) released the ‘2’, a less expensive (~$1,500) but bulkier AR headset in the same year.  From that point forward AR device releases came primarily from small, VC or private investor based firms, and while both Microsoft and Google released updated models in 2019, the number of devices released each year was minimal.  That changed in 2020 with over 10 models released, 2021 with 13, and last year with 17 new models.

What makes AR different from VR is that AR is not an immersive experience and allows the user to see and interact with the real world while seeing projected data or images overlaid, essentially a ‘pass-through’ device.  Simplistic uses, such as directions, similar to automobile HUDs, give the user the ability to follow directional indicators that move with the user without holding a smartphone, and more sophisticated uses, such as language translation are basic to AR (Watch the video here to see the translation function in action) and have real world uses that over time, will become ubiquitous with over 7,000 languages spoken across the globe and a global population of over 430m considered deaf (ASL can be translated). 
Marketing is an even more obvious use, allowing a shopper to look at an object, such as through a store window, and see information about that item instantly, sort of having a personal assistant available 24/7, with sophisticated retailers opening the door to consumers ordering items as they wander through a store, without the need for ‘advocates’ to help them make a physical sale.  More commonly seen uses for AR are in the business world, where manuals, data, and other information about complex systems can be projected directly on such items, giving workers the ability to see how repairs are made, or procedures for maintenance, and during meetings, users can see data and images while also seeing facial expressions and objects around them, rather than avatars in an unreal VR environment.

Of course, AR devices vary considerably, both in feature sets and price, and with 14 AR headset models already announced but unreleased and the potential for Apple to enter the market at some point, it is important to understand the potential differences and a bit about how AR devices work.  Among the most important items to understand is whether the device is a standalone one, or one that needs to be tethered to another device, in many cases a smartphone.  This is important as non-tethered headsets would need the ability to communicate with the outside world directly, while tethered devices could use the communications hardware of a smartphone or similar device.  The hardware needed for direct communication would add weight, bulk, cost, and power usage to the device, but would free the user from some mobility constraints.
There are a number of display technologies used in AR headsets.  Some display the AR image on a transparent display, some use non-transparent displays combined with camera images, and some project the AR image directly on the user’s eye, but the quality of the display, the cameras, and or the optics are important factors in determining the quality of the device and what its uses might be.   The display itself is important and a variety of display technologies are used in current AR headsets (OLED, Micro-LED, Micro-OLED, and LCD), but combining camera images, and AR images requires optical systems that take up room and add weight, which makes the development of consumer AR not only based on the quality of the display, but on the optical technology. 
Along with the display and optical hardware, the systems needs to be able to understand the user’s position in space, so it can move the overlayed image as the user’s gaze moves.  This is done with tracking cameras, accelerometers, and other sensors, and in some cases eye tracking.  All of this and built-in speakers, a microphone, memory & storage capacity, at least some CPU capacity, and a battery, must fit in a lightweight and thin frame (averages ~70 grams or 2.5 oz. as opposed to VR which averages ~17.5 oz.) that looks (hopefully) almost like normal glasses.
As with VR, the resolution of the displays themselves is important, especially as they are so close to the user’s eye, with typical AR devices sporting full HD (1920 x 1080) displays, but as AR displays are used in normal or even bright light situations, display brightness is also a factor that must be considered when evaluating AR headsets. While we do not trust advertised ‘peak brightness’ metrics used by many brands, brightness levels (peak) have been increasing over time, with Micro-OLED (OLED on Silicon) and Micro-LED displays offering the brightest current headsets. 
There is one headset characteristic that is important to both AR and VR, and that is FOV or field of view.  In VR headsets an FOV of 90° or more is essential, as the brain is more easily fatigued when it receives a more narrow visual field, and while AR devices are essentially visual pass-through devices, the larger the FOV in AR devices the more natural the overlay seems and a full view requires less head movement but at a higher cost.  Therefore, the cost of AR headsets can vary considerably based on hardware, with 2022 released devices ranging from $349 to $3,300 (~$500 average), and as larger CE companies move into the space, the ability to lower overall product costs is improved.
While we believe VR is still a technology looking for an application, we see so many use cases for AR that we wonder why AR headset brands seem less inclined to promote the technology to the general public than VR brands.  We do note that some AR applications require software developed specifically for that use case, such as AR used in a surgical suite or use in vehicle repair and maintenance, but when looking at such a basic use as language translation, much of the software has been written and is in the public domain or easily licensed.  A reasonable looking AR headset that sells for $150, would be able to solve communication issues for millions of individuals, families, and organizations, and while we are not quite there yet, we believe such a target is not that far off based on the progress made over the last few years.

​Key:
Top Left – ThirdEye Razor
Top Right – Vuzix Blade 2
Bottom Left – Magic Leap 2
Bottom Right – Huawei Vision Glass
 

​With over 3,200 exhibitors at the 2023 Consumer Electronics Show, of which over 1,000 are start-ups, getting product releases out can be a difficult task, especially when competing with the massive PR machines that are part of major CE companies.  This leaves companies to spend an inordinate amount of time formulating a single headline that is fashioned to catch the attention of the media, who then, hopefully, pass it on to consumers[1].  When searching for a word to describe many of the headlines that emanate from CES, the word ‘bombastic’ comes to mind (Oxford definition - “high-sounding but with little meaning; inflated”).  “Words like ‘revolutionary’, ‘game-changing’, and ‘unbelievable’ pop up in many releases, but when one digs a bit further into what was actually released or announced, things tend to be a bit less than those adjectives might imply.
One such headline from a company we have mentioned recently was “Magic Leap 2 Receives 60601 Certification for Use in The Operating Room”.  Magic Leap (pvt) is a well-funded AR unicorn that recently became the property of the Saudi government, when it took a controlling stake in the company through a $450m purchase made by the country’s sovereign wealth fund.  Magic Leap, as we have previously noted has been through some highs and lows since its founding in 2010, including the stepping down of its founder in 2020 when the company faced financial trouble.  Magic Leap refocused itself from a consumer orientation to a commercial one and last September released the Magic Leap 2 AR headset, the follow-up to the Magic Leap 1 released in 2018, an improved version that has been touted for use by both repairmen and surgeons, albeit the $3,300 price.  
 
We have no issue with the Magic Leap 2 device itself, however the release seemed to indicate that there was some governmental approval for the device’s use in operating rooms, opening up the world of AR to surgical suites across the globe, however if one looks at what 60601 certification actually is, the headline is a bit misleading.  IEC 60601 is a series of technical standards for the safety and effectiveness of medical electrical equipment.  It is the de facto requirement for bringing new medical devices to market in many countries, but is quite specific as to what it covers which is the operation of a device’s power supply.  With a number of certification levels as noted below, the certification is an absolute necessity for any device used in a healthcare facility or even a home environment and measures the potential for a number of potential electromagnetic hazards that could emanate from such devices.  In some cases those emanations might cause problems with other equipment in the vicinity, similar to the way in which smartphones must be certified for excessive RF radiation before they are made available to the public.
 
·       No body contact – such as with X-Ray or MRI machines, or even hospital beds and lighting
·       Physical Contact – such as blood pressure monitors, ultrasound tools, or even thermometers
·       Cardiac Contact – such as defibrillators or dialysis machines
 
In no way does this certification mean that the Magic Leap 2 AR headset has been certified by a government organization like the FDA or other drug or medically-related organization and the option to use the ML 2 headset would be in the hands of the user.  We are certainly not criticizing the use of AR in a surgery suite, just the opposite, but rather than tout a certification that sounds like a tacit road to AR or VR assisted surgery, it might have been more beneficial to produce a headline that highlighted the company’s partnership with SentiAR (pvt), a company that has developed the software to visualize cardiac catheter lab procedures, bring them from typical 2D visualization on monitor screens, to 3D visualization using AR devices such as the Magic Leap 2 (https://youtu.be/rsS2D7PrkEo) or EchoPixel (pvt) working toward similar cardiac 3D imaging solutions.
 
The FDA is still a bit iffy about software  used in or around medical devices, so there are a number of classifications that can be bandied about in headlines for companies associated with AR in a healthcare setting, and while there is no overarching FDA policy toward medical software, when software is used to analyze medical device data it is considered an accessory to the device, however when the software is part of the device’s functions, and could pose a risk, the FDA oversight can become more stringent.  While in some such cases, such as the use of a small or low resolution screen in a radiological exam, could lead to a poor or misleading diagnosis, the FDA seems to recognize the importance of managing those risks through regulation, but the field of AR or VR in practical medicine is relatively new and will likely take years for the FDA to develop regulations that are specific to their use in medical settings.  If the use of devices that indicate blood glucose levels or display ECG information are any indication, it will be even more difficult for the FDA to classify how it will regulate AR or VR software that can create 3D visual images that can be viewed before or during procedures.
 
Headlines do not make products, or at least not in the medical AR/VR space.  Devices and software that make complex medical procedures safer, simpler, and more successful can bring XR into the public eye far more significantly than eye-catching headlines, just as robotically assisted surgery has reduced the risks related to on-pump CABG procedures.  Being able to visualize actual patient organs in 3 dimensions should make procedures faster and therefore less risky, so if companies like Magic Leap want to popularize AR devices in the medical space, they should focus on how such devices can be used to save lives.  If they are as effective as the company (and others) say they are at improving surgical outcomes, and the media can make that information available to the public, it would go a long way toward improving the regulatory processes needed to maintain a safe use environment.  If the public thought there was a real benefit to using AR/VR in healthcare and puts some pressure on regulatory agencies to codify better regulations, things will move more quickly for AR companies than confusing headlines.


[1] We note that CES is not open to the general public.  One has to prove they are a part of the media or are ‘in’ the CE industry in some way, either as a producer or consumer (retail or otherwise) of CE products.

We are big fans of the use of AR in applications outside of retail, and while much of the AR/VR world has been negatively painted by Meta’s (FB) haphazard spending on developing the Metaverse and the issues surrounding cryptocurrency, we note that AR is alive and well, and not waiting for the metaverse or for gamers to sing its praises.  As with all other consumer technology, the XR space is application based, with successful products filling a need, either consumer related or business related, and while VR tends to be consumer (gaming) related, AR is more oriented toward business.  There are of course, business applications for VR, where you can sit at a virtual board room table and speak with other VR enabled avatars that could be in other parts of the world, but as a practical business application, VR still has a way to go.
AR however is a less complex and more accessible technology that is able to provide information to the user without requiring isolation from the real world, and will become imbedded in a number of business segments, particularly the service industry.  By service here we mean ‘servicing’, or the repair or maintenance of equipment.  Anyone business owner or consumer who has had to call in a repair person for even the most minor issue, is aware that there is no end to the complications and costs involved. 
Home appliances are a perfect example, as when a repair is needed, almost regardless of the item, a flat fee is charged just to make the visit, with an additional hourly charge as soon as the repair person walks through the front door.  Then, there is the location of the model information, which is hopefully easily accessible, and the inevitable question of, “Do you realize how old this unit is?”, which sets the tone for expensive parts that could be in short supply.  Next comes the phone call to the office, asking about the particular potential problems that might be at cause, even before a screw is turned, with the result being, “…It could be the motor (or whatever) which is going to be a problem as they don’t make it anymore…”.  If the repair person is younger than 25, there is another call to the office to talk to the boss who explains the possible issues of said pre-digital appliance, and then more parts removed. 
When the issue is discovered there is another call to the office, this time to speak with Loretta, the keeper of the parts catalogs, to find a replacement part number, and another call to Bill’s Appliance Parts Emporium to order the replacement part, followed by the inevitable, “looks like they have to order it, so we should have it within 5 to 7 business days…”, at which point you are left with a bill for the time spent, a promise as to when the part will come in, the hope that they will call when it does, and that the same repair person will be available to install the replacement part.  When the part come in, if you are lucky the same repair person is available, although if not there will be another series of phone calls to get the situation explained and some tips on how to install the part.  Eventually your appliance gets fixed (7 to 10 days?) and you geta bill for the ‘arrival fee’, time spent, and the marked-up parts, which, in the case of a refrigerator, can cost as little as $40 and as much as $1,000 ($350 average) according to Home Advisor.

What if that appliance is not an appliance but an escalator that is the only alternative to stairs for those that are disabled in a commercial building?  Just think of that home refrigerator times 100 in terms of complexity, repair difficulty, and cost, while keeping in mind that the number of repair persons who have the specialized knowledge to fix an elevator, escalator or moving walkway is diminishing as younger workers gravitate toward more digitally oriented service jobs.  Here’s where AR comes in.  Fujitec (6406.JP), Japan’s and the world’s largest manufacturer and servicer of elevators and escalators, just signed a deal with Vuzix (VUZI) under which Fujitec will adopt the Vuzix M400 smart glasses series to support its elevator engineers (installation and repair persons), along with functioning as a training tool.
Given that the company’s products are extremely complex and therefore require years of experience to install and repair, Fujitec has been testing the AR glasses for the last few years and has now decided to deploy the glasses across the company’s service engineers.  The glasses can be used while wearing helmets, regular glasses, or gloves, and have an extended battery life.  By using the glasses, experienced engineers can guide those providing installation or repair support without travel, reducing repair cost and the need for more extensive training.  The glasses themselves are technically adaptive frames that can be used with or without eyeglasses, weigh under 3 oz. and can be mounted in a hardhat if necessary.  Based on an 8 core Qualcomm (QCOM) XR1 processor and 64GB of internal flash (6GB RAM), the glasses have 3 DOF head tracking, 3 axis gyro, accelerometer, and compass, along with and an OLED display[1] with a brightness of over 2,000 nits and 24 bit color. 
The hot swappable battery pack (2 to 12 hours) can be worn as part of the headset or a 3rd party pack can be used, while the noise cancelling microphone and integrated speakers serve as adjuncts to the 12.8 MP (4k 30 FPS) camera  with image stabilization and barcode scanner.  The glasses are wireless using Bluetooth, Wi-Fi a/b/g/n/ac and 5G connectivity, all of which means that pretty much anything you see will be seen by others at a remote location as you see it, and detailed guides and information can be superimposed over the item under repair to make diagnosis and repair a simpler task. 
The M-400 series units retail for $1,800 with a variety of accessories, although we expect Fujitec will be working under a more discounted pricing system given the company’s employee base of ~10,000.  There is no information as to the timing of the roll-out and there will likely be a relatively long learning curve for those who have not participated in the trials, but such a deal is a feather in the cap for the AR world and is a good example of applications where AR has very practical applications.  We use this as an example only and are not indicating that it ushers in the ‘era of AR’ but serves as a point along the evolutionary path of AR and its potential applications in the business world, beyond any consumer applications.  While it might seem an expensive tool currently, we expect it would pay for itself quickly in the kind of applications Fujitec would be using the device.


[1] Equivalent to a 5” mobile device at 17”

​Note:  We have no connection with or receive any compensation from companies mentioned herein.  All product specifications and applications are from reference sources and/or product literature.