Human-Computer Interaction is a concept that has been with us for decades. Because of technological limitations, it has perhaps more been about the computer than the human, and people needing to learn new methods of interacting. However, this is starting to change. Gestural technologies allow for more widely and easily-understood methods of interacting with computer systems, with the increasing ownership of smartphones becoming a catalyst for a more relaxed relationship between computer and user. Swiping, pinch/pull and flicking to turn pages are now commonplace, although one assumes that millions of dollars have been invested to develop such levels of simplicity.
Last year's launch of the Kinect from Microsoft, allowed interaction to move even further away from the physical. As Kinect can "read" human interaction from a camera, low-cost development of gestural interfaces to be enacted anywhere, now becomes possible. Although Microsoft has positioned Kinect to be the start of a new phase of computer gaming, there are, of course, many more ways in which gestural technology can be offered.
Code Computerlove has developed a prototype, which positions the Kinect in a retail environment. The screen is positioned in the shop window, with the Kinect camera facing outwards towards an area around the shop. Visitors can then gesture to interact with the shop-mounted system, and enjoy an immersive experience without having to be in the shop.
The idea for the project came from the agency's Jono Casley, who spotted an estate agency with a window-mounted digital display. The display rotated through many properties, to which it would have taken a long time for a bystander to go through them all. Such an environment, according to Casley, was ripe for improvement.
Casley's first idea was to develop a touchscreen interface. However, this was discounted on the grounds of safety: it could be easily smashed, either deliberately or through an over-exuberant use of the interface. Replacing the system would be costly, and so when the opportunity to develop with Kinect came up, it was the natural choice in terms of developing something which was safe, cost-effective, and enjoyable, while clearly extending the environment offered by the shop itself.
The project uses a Primesense camera, with the project utilising OpenNI middleware. However, since the project was developed, Microsoft has announced that the Kinect SDK is to be launched for non-commercial use. The opening up of such possibilities is something that interests and excites Casley. "[Gestural technology] is more natural, and is a better way to simulate what you're doing digitally. Developers are getting together and creating these open source platforms, and you can see agencies using these technologies to promote their clients and their work."
The availability of OpenNI and the Kinect SDK will allow for an explosion of new experiments across a wide variety of disciplines, although the clear focus for this project was to develop a gestural interface for a retail environment. Casley sees gestural technology becoming more prevalent in retail: through touchscreen gestures, headtracking, and many other permutations. However, as Casley notes, the remit of this project is clear: "We were looking at a good reason to market products, or just to inform the users of more information than they currently get."
Although gestural interfaces allow for a much more natural approach, Casley and his colleagues had to think very hard about how to make their project easy to learn, understand and use, for someone approaching the system for the first time. "In terms of the user interface. It needs to be extremely simple in terms of people walking in off the street. They have no training, and don't have time. We're talking about a short turnaround of people using this, and they need to intuitively know how to operate it.
"I've seen a lot of examples which are very complex, and you need a few minutes just to understand them. You need to be very visual in terms of representing what's happening on screen. We have used buttons where when you hover over them, they pop out a little arrow which you can follow, which is a swipe gesture to activate the call to action. We have tested that with sample users, and as soon as they found out that they had to swipe - there was a little tutorial which said 'over' and 'slide' - they found it very easy to operate. Without that prompt, they would not have understood."
Casley flows with ideas for where to take this technology, and its use clearly extends to in-store environments as well as those around the shopfront. "I see it everywhere - not just through windows, but general interactive pieces without the use of a branded computer. If you go into any shop which offers customised products, you're generally presented with a keyboard and mouse. The way in which people interact with these technologies will become ubiquitous."
One of Casley's more thoughtful ideas is based on a shopping mall. With colour recognition, it would be hypothetically possible for a customer to have themselves "scanned" by the gestural camera, and for the system to offer ideas for new clothes. After the customer has made a purchase in that store, they could then be scanned at a later store for matching accessories. Although this makes for a seamless experience and gives the user what they want based on a set of defined choices and contexts, there are clearly issues in terms of privacy and data sharing which need to be addressed. However, from a commercial perspective, it opens up possibilities for shopping centre owners to effectively deploy retargeting ad networks across their estate, and to mine and harvest that data in order to deliver a greater degree of customer intelligence to brands and shop owners.
There is also an opportunity here for social networks to connect to gestural interfaces. Casley observes that within the store, a customer could input data to help to customise a product of their choice – something which is increasingly commonplace in retail environments such as sport. However, it will be increasingly possible for this data to be saved by the user in mass networks such as Facebook, and to take that data somewhere else if necessary – inviting opportunities for e-commerce as well as in-store.
Casley points out that whether it's an in-store offering or a rich, interactive window-mounted interface, the content and context has to be thought through. "Anywhere that's got a window could have an interacvtive window display, but you've got to think about having it available towards closing times and in the evenings – for example, showing customers additional information after the store has closed. If it's an abstract visualisation, it could be there all the time if the shop can spare the window real estate."
The sky's the limit in terms of gestural technology, and early hacks such as this could be the start of something big. In five years' time, if you are buying some snacks from a self-service late night store by swiping through its screen-mounted product catalogue, it will be people such as Casley and his colleagues that have helped you to get what you want.
Jono Casley is a Flash developer at Code Computerlove.