Variable Reality II (2015)
INTRODUCTION
Variable Reality is a novel Augmented Reality (AR) based interactive system focused on bringing intuitive on-the-go reading experiences to the users by enabling always-available access to and presentation of any digital book contents in a complete spatial form of a book. Variable Reality introduces two types of books: the Augmented Book and the Virtual Book.
The Augmented Book incorporates computer vision based corner detection and tracking systems to detect the corners of a physical book, information of which is used to overlay paper book pages with digital pages using a Head-Mounted Display (HMD) device. As a result, any real book can instantly turn into a different book while still retaining the physical properties of the book, conforming to the legacy of the paper book metaphor. The Virtual Book is an entirely virtual 3-D book that extends the accessibility and mobility of reading materials. When in absence of a physical book, the user can retrieve this virtual book on the palm of a hand –a well-established pose of how one holds a book when reading. The weightless and portable nature of the Virtual Book makes it possible for users to comfortably hold larger physical books such as unabridged dictionaries or the World Book Encyclopedia.
In both augmented and virtual book scenarios, natural hand gestures inherently associated with the reading activity are defined and used in various Variable Reality applications.
CONCEPT
The basic concept of Variable Reality is essentially based on the idea of enabling users to read a digital book anywhere while maintaining the physical quality of a real book. The Variable Reality system uses a lightweight and compact head- mounted display (HMD) device that is see-through. Two types of books designed for the system are the Augmented Book and the Virtual Book.
Variable Reality Augmented Book
In the Augmented Book scenario, the user can take any book from the real environment and turn it into a digital book, which can be seen through the HMD (as shown in the image above). The book becomes an augmentable surface for overlaying paper pages with digital information without the need of a visual marker for tracking. As a result, it enables the user to quickly switch between reading a paper book and reading a digital book with a single physical book. The title of a book is detected using a feature detector and the page number can be detected with the help of the OCR technology. This enables the user to store and retrieve digital data on a specific physical page. The Augmented Book also integrates natural hand gestures associated with reading activity in order to provide an intuitive reading experience. A fingertip is predominantly used as a mouse cursor but is used as a pen tip for annotations.
Variable Reality Virtual Book
Ideally, the Virtual Book makes it possible for users to hold a 3-D virtual book in the hand and read it anywhere. It is designed to bring extended mobility and accessibility to the digital library designed for on-the-go reading experience. The user does not need anything other than the hand to bring up a virtual book on the palm. Similar to the Augmented Book, it integrates the hand gesture recognition system, which enables users to efficiently manipulate the book.
Variable Reality Virtual Bookshelf
Variable Reality further incorporates a three-dimensional digital bookshelf that closely resembles the look of a real bookshelf. Using a simple gesture, the user can display it, zoom into a portion of it and pick out a book to read. This book can be in the form of either the Augmented Book or the Virtual Book.
INTERACTION TECHNIQUES
(a) Index Fingertip —the user’s index fingertip is used for interaction just like a mouse cursor on screens
(b) Quick Swipe —a quick opening and closing of the hand motion
(c) Stay Click — staying (or hovering) at a place for 1.8 seconds using the index fingertip
(d) Open-Close Hand —a quick motion of moving the hand from right to left while keeping the pointing gesture
(a) Open/Close Hand (Face Up) —a quick opening and closing the hand with the palm looking up
(b) Right Hand Swipe —a swift, right-to-left hand movement using the right hand
(c) Right Hand Twist —a quick rotating motion of the right hand around the middle finger using the wrist
(d) Right Hand Pinch —a hand gesture similar to how one would pinch someone
(e) Two-Hand Swipe Open/Close — a quick two hand movement separating apart from one another
(f) Finger Fold —folding one of the five fingers
(g) One-Hand Swipe —left hand movement from left to right with the palm facing up
(h) One-Hand Quick Bounce —a sudden down and up hand movement
(i) Quick Open-Close (Face Down) —a quick closing and reopening of the left hand
APPLICATIONS
Based on the stable book- and hand-tracking algorithms along with the recognition of various gestures designed for user interaction, I built several Variable Reality applications for both the Augmented Book and the Virtual Book. These applications import convenient navigation techniques available on digital books and yet maintain the physicality of a paper book. The goal of each application is to create an intuitive and effective reading experience for the users. Incorporating natural hand gestures associated with reading activity adds to the positive user experience in reading.