Harvard Business Review covers Virtual Worlds

HBR published a short piece last month titled "Getting Real About Virtual Worlds" in which Paul Hemp talks about the business benefits of using 3D environments. Coverage by the Harvard Business Review, given its stature and high-level distribution, is a good sign for an industry.

To me, the article underscores once more the notion that compelling, useful content is already out there. Unlike many non-impressive virtual reality offerings in the 90's, CPU and GPU power is good enough to create a very compelling experience...if you have a good-enough display system.

Using these compelling 3D worlds on a desktop monitor - usually in 2D - is like watching a new High-Def DVD release on a small black&white television. Try it on a high-resolution HMD with wide FOV and good peripheral vision, and you'll never want to go back.

The HMD and the Washing Machine

Should see-through head-mounted displays be also used for completely immersive (non see-through) applications? Technically, it's certainly possible. Just put a dark cover on the front glass, and you have something similar to being completely immersive. However, having the extra weight, the not-quite-as-bright image and a bulkier product is certainly a downside for dual-use HMDs.

This reminds me of the washing machine / dryer combination. What a great idea on paper - no need to move laundry from one to the other. In practice, there are so many drawbacks to this combined approach that most everyone still has separate washers and dryers.

Not to showcase my age too much, but as a teenager I owned an Apple II computer. Not an Apple IIc. Not an Apple IIe. A real Apple II (serial number had 4 digits) and at that time, it would connect to our standard TV through a small conversion box. But, even though it made sense at the time, computers migrated very quickly to separating the TV from the computer monitor. Same concept, different time.

So... I still think see-through HMDs and fully-immersive HMDs should be kept separate for now.

The Mind-Reading Virtual Binoculars

Northrop Grumman Corporation (NYSE:NOC) announced that it was awarded a contract to develop electronic binoculars that use brain activity to detect threats. The press release is here. My company is proud to partner with Northrop on this project.

The project seeks to present a panoramic day/night optical system to a user and monitor brain waves - yes, brain waves - to detect targets of interest. A panoramic view and high-resolution image was deemed very important for this project, which is why Sensics panoramic displays were chosen.

However, the drive to present users with diverse visual stimuli and measure brain activity is certainly not unique to the defense markets. We are working with several academic institutions to achieve much of the same goals within a research context.

Presenting users with narrow field of view displays could be considered not to put the human brain to full use as a super-fast object/pattern recognition computer. This is especially true given that difference parts of the visual field are used for different purposes: central vision is used for reading and perception of fine detail. Peripheral vision is used in humans and also in animals for threat detection, sensing movement and presence of peripheral objects. Try walking around with toilet paper rolls on your eyes (see this demo on the Washington Post) and you'll understand how critical peripheral vision is.

See-through HMDs have lots of ground to cover

See-through HMDs (sometimes referred to as 'personal displays') are sometimes portrayed as combining eyeglasses with a video ipod. In a perfect world, they look like cool fashion accessories, are so light that you can wear them anytime, yet overlay what you see through them with computer-generated images. Perhaps it's a movie that you can watch on a plane. Perhaps it's driving directions or text messages from your friends.

However, to get there, many problems remain to be solved:
- Brightness: can these devices be used in a sunny day?
- Weight: can you really walk around with them all day?
- Power consumption: how long does the battery last and where is the battery placed? If the battery is part of the HMD, it adds weight. If it's on a belt-clip, it adds inconvenience and perhaps a cable.
- User interface: how can you control what information is displayed, or when it is shown.
- Dual use: if the goal is to watch a movie, it's often better to block out external imagery. But how? Do you add a cover to the front? Do you make the glass part so dark that it's effectively like sunglasses?
- Placement of information: if the information is placed in the central vision, it might intefere with reading tasks. If it's placed in the peripheral vision, it is difficult to read. If it's both, you need a very wide display element.
- Stereo vision? At some level, a monocular display (one eye) might be sufficient for textual information. However, if you're looking to augment a scene, the feeling of stereo may be very important. Stereo, however, nearly doubles the weight, cost, power consumption and cabling requirements of the display electronics.
- Sound - can you or should you integrate the display with an earpiece
- Cost, for obvious reasons

I'm curious to see what Apple (who recently published some patents on this) and Sony (recently showing a cool display at SID 2008) have up their sleeves for this product category.

The Alienware curved monitor and a curved HMD

Earlier this year, Alienware (now part of Dell), demonstrated a high-resolution, panoramic monitor. The 'Curved monitor' blends together four DLP projectors to create a wide display that is over 3 ft wide and has 2880x900 pixel resolution. You can read the Gizmodo review of this monitor, and see some videos at here

The process of tiling together multiple displays to achieve a curved, panoramic, high-resolution product is exactly what my company does to generate a curved, panoramic, high-resolution head-mounted display. The multiple displays are positioned so that they approximate a sphere around the eye, for a 'surround video' experience. Resolution is similar or higher than the curved monitor.

The decision whether to use a curved panoramic HMD or a curved panoramic monitor
is a bit like comparing a car to a plane: there are uses where one is better than the other, with some area of overlap. If you need many people to see the image at once, a monitor is better. If you are not inclined to wear anything on the head, the monitor is better. However, if you want stereo images, if you need portability, or if you want 360 degree immersion (image changes when you move your head), the HMD is your choice.

Some of the things I'm curious regarding the curved monitor:
- Does it require special video output modes, or can it accept stadnard display resolutions?
- How many video inputs does it require?
- How good is the tiling/stitching between the displays? In the Gizmodo video you can clearly see the seams between the projectors. Is there a way to achieve a seamless image?
- From a manufacturing standpoint, how does Alienware perfectly align the DLP projectors to achieve pixel-by-pixel matching? Does this alignment change over time?
- How good is the projector-to-projector color matching?

It's great to see tiled displays hit high-end consumer applications. I'm looking forward to trying a curved monitor one day soon and comparing it - with the same application - to a curved HMD

The Display Inside The Display

When users talk about their recent experience with head-mounted displays, they often refer to image quality. "The colors were great", "the image was very bright" and so forth. Indeed, image quality (lack of smear, brightness, contrast, color gamut) scores very high on HMD requirement surveys, right next to high resolution and wide field of view.

It turns out that the micro-display technology used inside the HMD can critically impact the user experience, often more than other design decisions for a particular HMD.

Many HMDs today using LCOS (liquid crystal on silicon), which has similarities to LCD technology, with the exception the LCD is transmissive - that is, the light source is behind the display, where LCOS is reflective - which means the light source needs to project the front of the display. LCOS displays are a common choice because they come in higher resolution than comparable single-chip OLED (Organic LED) display, and are available from multiple vendors. Unfortunately, LCOS-based designs have some disadvantages:
- They require a light source which reduces the display contrast (the light is on even when a black picture is displayed) as well as substantially increases power consumption.
- They often have limited temperature range, or require somewhat exotic solutions like local LCOS heaters to warm up the display before initial use.
- They typically exhibit motion blur and smear, just like many of us experienced with LCD monitors. This is particularly relevant in HMDs since images inside an HMD constantly change to track user head movements.
- Depending on the specific design, users may experience color flashes or other artifacts during head movement.

OLEDs are also used in several HMDs. OLEDs are self-emitting, meaning that they do not require independent lighting. OLEDs, such as those available from eMagin, offer a larger color gamut, higher brightness, higher contrast and lower power consumption. OLEDs are also faster than LCOS and thus are free of LCOS artifacts. However, OLEDs typically come in lower resolutions that LCOS displays and are available from fewer vendors. To overcome the lower resolution of individual displays, vendors such as my company, use a tiled approach to combine multiple micro-displays into a larger, higher-resolution image.

So... look inside! Next time you look at an HMD, inquire what display technology is inside the display.

The 2008 HMD Market Study Survey

Sensics released today the 2nd annual HMD market survey. In this survey, conducted in February of 2008, we re-examine the same questions asked last year in the 2007 HMD survey, and then go on to compare and contrast the market needs.

The 2007 survey was titled "Are existing HMDs good enough?". The answer was a resounding "no!", and it helped quantify what we all intuitively know - that HMDs with narrow field of view ("tunnel vision") and low resolution may be sufficient for very early adopters, but are certainly not good enough for widespread use.

The 2008 survey is titled "Are we there yet?". The short answer is also "no", though perhaps without the exclamation point. Requirements have not changed much, and vendors are certainly making progress in working towards what users are asking for - wider field of view, higher resolution, in a light-weight, easy-to-use product. Panoramic field of view products such as those that we make are maturing, and other companies have innovative products (though sometimes curious-looking, such as the "wedding cake HMD") that try to solve these problems. However, the "holy grail of HMDs": panoramic field of view, high def, low cost, easy to use - has not been conquered yet. Look for additional fascinating developments in 2008.

I'll be posting additional insight from these survey from time to time. Stay tuned.

Optical see-through vs. Video see-through

If you're interested in augmented reality, you've probably thought about optical see-through and video see-through.

Is one better than the other? That's like asking whether a car is better than an plane. It depends on what you need to do: get groceries? the car is unmatched. Get from New York to San Francisco? a plane is very often your choice.

Video-see through systems present video feeds from cameras inside head-mounted devices. You can see an example here. This can be useful when you need to experience something remotely: a robot which you send to fix a leak inside a chemical plant; a vacation destination that you're thinking about. This is also useful when using an image enhancement system: a thermal imagery, night-vision devices, etc.

Optical see-through systems combine computer-generated imagery with "through the glasses" image of the real world, usually through a slanted semi-transparent mirror. If you are in a mission-critical application and you're concerned what happens should your power fail, an optical see-through solution will allow you to see something in that extreme situation. If you are concerned about the utmost image quality, portable cameras and fully-immersive head-mounted display can't match the "direct view" experience.

One aspect of video see-through systems is that it's much easier to match the video latency with the computer graphics latency. Latency (delay) is inherent to immersive imaging systems: motion trackers are not instantaneous; computer graphic generation is not immediate and even when refreshing images at 60, 70, even 120 Hz, there is a lag from sensing to imaging. When computer graphics need to be overlaid on the image from the actual world, there is a difference between video see-through and optical see-through. Optical see through offers no latency, which sounds great, except that there is an inherent mismatch or lack of synchronization between what you see through the glasses and the graphics. If you're showing a virtual sofa inside a real living room, this mismatch can be distracting. In contrast, using video see-through allows you to synchronize the delay so that your video and graphics are always in sync.

With optical and video see-through, no solution is perfect for all problems.