Fun With Augmented Reality

street tests from Najork on Vimeo.

Direct Brain-Computer Interface Will Require a New Language of Interaction

[Cross-posted from Signtific.]

When Apple Computer recently released the 3.0 version of its iPhone OS one of the most anticipated new features was Cut & Paste. This simple task has been a staple of computing since GUIs were part of the OS, so why did it take Apple until it’s 3rd OS version to implement the feature for the iPhone?

As Apple tells it, there was incredible deliberation over how best to design the user experience. This is, after all, the first and only fully multi-touch mobile computing device. Apple has been meticulously developing and patenting the gestural language through which users interact with the device. Every scroll and pinch, zoom and drag is a consciously designed gesture adding to Apple’s growing lexicon of multi-touch interface. Implementing Cut & Paste was a substantial challenge to create the most accessible gestural commands within the narrow real-estate of the mobile screen.

Now, consider interacting with the same content types available on an iPhone or anywhere in the cloud, but remove the device interface and replace it with a HUD or direct brain interface. If the content is readily visible, either as an eyeglass orverlay or directly registered in the visual cortex, how do we give a UI element focus? How do you make a selection? How do you scroll and zoom? How do you invoke, execute, and dismiss programs? Can you speak internally to type text? How might a back-channel voice be distinguished from someone standing behind you? How do you manage focus changes between the digital content and the visual content of the real-world when both are superimposed in some state?

The fields of Human Computer Interaction and User Interface & Experience Design address these challenges for interacting with digital content and processes, but what new interaction modalities may be developed to better interface humans and computers? As we internalize computation and interaction, the disciplines of HCI & BCI will begin to interpenetrate in ways that may radically alter the conventions of the Information Age.

Brain-Computer Interface

In my present tenure as a Visiting Researcher at the Institute for the Future I’ve been posting a lot of Signals pertinent to Brain-Computer Interface over at the Signtific open source research site. My Signals are listed under the tag “ProgrammableEverything”.

Check ‘em out if you’re interested in the fascinating & accelerating field of BCI. Also feel free to add your own Signals you see in the world or are engaging in your professional research.

Cheers!

Reflections on Thailand

[Extensive photo album here.]

Japan crossed with Mexico. Hack, mash, and lash everything together. Very hot and thick, humid and prone to short heavy rains. Bangkok is larger than expected, with a higher skyline. Slum-like in many ways but comfortable. Dirty, aged, grafitti’d, tagged, polluted, smelly, hungry, buggy, feral. Friendly, smiley, reverent, strong, spiritualized, watery, creative, delicious, surviving with tenacity. Temples & tenements, luxury hotels and megamalls. Insane traffic and transport. Little regard for lanes or right of way. Swarms of motorbikes, vespas. Cheap and dangerous tuk tuk 3-wheelers. Families piled onto scooters, kids asleep, baggage strapped on. Traffic flow like a logjam, shifting metal slabs moving within inches of each other, victory goes to the bold in a cloud of exhaust. The mighty Chao Phraya cutting its way through Bangkok and out to the coast, it’s headlands in the foothills below Burma. These are river people, with traffic on the waterways as busy and chaotic as the streets. The river is deep, a 1/4 mile wide, running green & tan, dirty and littered with commercial & vegetal detritus. After the rains clumps of fallen jungle float on its surface, carried down from farms and foothill tributaries. Black & yellow birds land on leafy branches half submerged to dine on nuts and berries. Water taxis from hotel to Sky Train. Fantastic monorail, the SRT, its cement track a modern work of civil engineering adding to the Tokyo vibe of downtown Bangkok.

Tangled mess of black utility cable slashing horizontal lines across most everything, tied in to huge transformers, burnt metal grills pumping amperage for the teeming metropolis of 6 million. The twisted infrastructure grows organically like a banyan, stretching out axonal to connect and communicate. Most buildings are old haggard tenements, their facades stained with a dark grey wash like grease and ash drawn out of the thick air. Structures that seem abandoned, uninhabitable, are strung with drying laundry drawn perpendicular to the necessarily ubiquitous swamp coolers lining the sides of each floor. Broken concrete fields under freeway overpasses offer football grounds lined by graffiti mural walls under chainlink divisions.

Downtown, luxury malls with Louis Vuitton and Burberry fronted by large altars of golden Buddha’s and Ganesha’s, black marble elephants flecked with gold, yellow floral garlands and incense offered by shoppers to their immaterial gods. A sign at Wat Phrao Keo in broken Thaiglish sagely, if not inadvertently, warns visitors to “Beware of your valuable possessions”. Technology, commerce, wealth, and western aesthetics have moved in with the economic development afforded here as in every other large city by the realities of globalized communication and trade.

Down crowded alleyways lined with merchant stalls and open air ad hoc kitchens, thick with pedestrians, cars, tuk tuks, and manic motorcyclists weaving through the narrow channels, over rooftop patios caged against some unseen menace, rise countless golden and white and glittery temple spires. Buddhist Wats take residence everywhere, themselves seemingly hacked into the dense fabric of the city, rising like aspirational fruiting bodies of ancient mycelial webs. Wat Arum, Wat Pho, Wat Phra Keo & the Grand Palace, and innumerable others. Religion & myth is woven throughout the populace. Every building has it’s own adjacent spirit house offering residence to the disincarnate lest they move into your own home. City walls are tacked with incense holders between stores. Banyans breaking through the sidewalks are wrapped with rainbow sashes honoring their freakish holy treeness. Every taxi has a statue on the dash or mala hanging from the rearview or Buddhist stencil on the headboard or any combination of the aforementioned. A 3-day Buddhist holiday shut down all government and banking.

The current Thai king is the longest reigning monarch of the modern age, holding office since 1950. Thailand was the only East-Asian country to resist British colonialism, sparing its autonomy by ceding a few bits of territory along the Burmese & Malay borders. Indeed there are long running conflicts with the Burmese, and Buddhist Thailand is in the midst of an insurgency along the Malaysian border from an advancing Islamic populace. The cabinet of the prime minister and the military have provided ongoing political theater as each vie back and forth for the seat of power. Most transfers of power, even in the case of multiple coups, have been bloodless. The Thai people themselves seem to have little interest in these power games, preferring a life of pragmatic spirituality while maintaining a deep abiding love and respect for the king. The two possibly mortal social offenses in Thai society are speaking ill of the Buddha and speaking ill of the king.

Farming is honored. Rubber trees and palms cover most southern land, providing two of the country’s largest exports. The Thai peninsula includes all the most breathtaking exotic tropical beach locations you could imagine, including the stunning Railay Bay – famed for the movie The Beach. Beautiful light blue waters, ridiculously warm and salty, stretched for ages across the gulf. Koh Samui running on Full Moon inertia, tourist trinkets, and scattered luxury resorts sheltered from the hustle. Low inland jungles bring minimal shade to island shanties in seemingly impossible poverty. Yet they survive & persist and move through generations like the rest of us. Koh Phangan also still milking their internationally notorious Full Moon Rave scene, adding a Half Moon party to underwrite the Euro draw. Even away from the main strips the beach scenes has a fun accidental Burning Man vibe, a shoreline esplanade of shanty bars and sound systems. Expats all over the place. Seems easy to get lost for months, years, decades in some seaside shack eating fruit and fish in a poor man’s paradise. Impossible walls of insects whip up into sudden frenzy, a cacophonous wail of screamapillars, giant cicadas that still don’t seem anywhere near big enough to make such a pitch. Monkey troops swing across canopies carpeting tall rock slabs jutting from the water. A rock climber’s joy, sheer faces hung with dripping stalactites and pocked with rope tie-ins. These tall rocks are scattered by the hundreds – thousands? – across the Gulf of Thailand and Andaman Sea. A boater’s paradise. You could spend months exploring thin beaches stretched around the edges of countless small jungle rock islands.

In the South, each night was attended by thunderstorm, often over sea or above the island peaks. Big black charcoal canvas lumbering across, flicker flashed with lightning bursts every few moments, often too distant to hear the thunderclap, then a sudden ear-shattering rend of ozone right above. When the heavy rains hit they come quickly and with ferocity. Never seen rain like it. So thick that it occluded line of sight to 20 meters or so, hiding everything beyond in watery showers. From the steep island peaks water rushes down in sudden rivers cutting through beach sands, pushing tan clouds out into the bay, a shimmering clear layer of fresh water forcing the saline back out over the ocean’s surface. Giant raindrops agitate the bugs forcing them to take flight in peppery swarms. Small opportunistic swift-like birds take to the skies darting and arching, turning and diving to pluck the insects mid-air in some ancient deeply programmed ballet of the food chain. Life goes on. It must. When rains come often and fiercely you can’t just drop your business. This was especially so in Bangkok whose streets are lined with tirelessly deployed open markets bare to the sky save for a small canvas over each. In 20 or 30 minutes the rain will likely pass so there’s no point in worrying much about the interlude.

While the deep south is struggling with a mounting Islamic insurgency, and the peninsula is attending the construction of more new mosques, the buddhist majority continues to permeate life with the spirit of their patron, accompanied by a host of Garuda and Nagas and a menagerie of mythic beasties syncretized from India and China. If Thai Buddhist Bangkok is feral and lashed and relentlessly modded in ghetto slapdash, the Bangkok Chinatown is 10x more so compressed into tighter alleyways, with more people and motorcylces (Vespas apparently seek Chinatown to live out their golden years), hung with impossibly more spaghetti cables, and festooned with walls of neon Mandarin signage casting a little too much light onto freakish displays of animal carcass and presumably inedible seafood and giant transparent sacks of fried pork product and stall after stall of fashionable Versace & Loius Vuitton knock-offs. Imagine threading your way down a dark, narrow alley lined with flea market stalls and no-health-code/no-insurance open air cart kitchens, filled with people pressing in all directions through dense heat and smell and rot, then send a motorcycle down the alley every few moments to do battle with cross-traffic carts and tuk tuks. Now imagine the alley is a whole network labyrinth covering multiple blocks between several-story tenaments streaked with black soot and stain and hung with drying clothes and black cables. This is why we western pansies stay in the nice hotel with A/C and a pool.

The final capper to the trip was in Bangkok the night before our departure. After the evening rains subsided, my partner and I went down to the pool for a night swim, around 9pm. Refreshing and fun we frolicked and generally soaked up the remaining moments of our stay. Then, in the poolside darkness moving low between the lounge chairs, I saw a large reptilian form lumbering along. “Dude, there’s a fricken alligator coming towards the pool!” I exclaimed excitedly. As it marched into the light we realized it was actually a monitor lizard – Varanus salvator, to be precise – about 5-6ft long with a fattened belly like it just ate a dog or possibly a small European child. “If that thing gets in the water, we get out immediately” I said with some urgency. I knew it could swim and see underwater much better than we could. No reason to tangle with a 6ft thunderlizard in a foreign country with questionable health care. Sure enough the beast slipped into the pool and sidled along the swim-up bar. We hopped out, laughing nervously, and I approached the lizard from a careful distance. Grabbing the pool attendant I motioned towards the monster. “That’s bad”, he said in a way that suggested that, bad as it may be, it wasn’t unusual. And so he casually splashed the creature with water nudging it along until it climbed out of the pool slowly, begrudgingly, made it’s way back into the riverside brush. It was easily the biggest lizard I’d ever seen in the wild.

The final day we were denied pool access during a particularly solid rain. When it’s always 90+ degrees & 90+% humidity, swimming in the rain is quite nice. But no, we were not allowed. “Why?” I protested. “Lightning” retorted the attendant. Fair enough, I thought. Then, in a casual but cautionary aside, the attendant reflected, “We had an accident last year”. This is the Bangkok Riverside Marriott, a fancy if not dated family hotel. Apparently buried somewhere deep in the boilerplate legalese fine print of our hotel contract is the clause, “Marriott Properties takes no liability in the event of any hotel guest or visitor getting suddenly struck by lightning and then slowly eaten by ferocious monitor lizards”.

Every Enterprise Needs Its Own Twitter

To the title of this post, I was on a tech support call with AT&T/Apple this afternoon. Facing surprising SMS overage charges on my mobile account, I wanted to know if Twitter apps like Tweetie might be using the SMS channel to conduct their transactions. I figured not but the Tweetie site gave me no info.

So I asked the support person, who knew about Twitter (better than AT&T) but not about whether Twitter iPhone apps use SMS. She put me on hold to ask a support specialist. About 15 minutes later she came back with a reasonable response (“they probably don’t use SMS”) and a suggestion to talk to the app vendor.

All this had me thinking about the huge inefficiency at play with the responder trying to locate the specialist, get their attention and time, probably juggling multiple phone lines, to then give me one person’s measured response. I imagined a not-too-distant future where the call responder typed my query into a local Twitter-clone running on the tech support network inside AT&T/Apple. This query would immediately push out to subscribers – some required by their manager to subscribe to all tech support feeds, and others who just want to see the problems customers are encountering. I then imagined that some of these subscribers would run search filters on the messaging stream in order to be alerted to those queries they were most interested in tracking.

So somewhere in the bowels of tech support there’s a guy who is a Twitter power-user, or an engineer who wrote the SMS api’s, or a community developer that helps 3rd parties like Tweetie write iPhone apps… and they get pinged every time a tag of interest comes across the network. They see “iPhone Twitter SMS” and respond with the info. With enough of these transactions the call responder will have an archive of tech support tweets they can search through to see if someone has already responded. Of course, this archive provides another layer of analytic data that can be mined to get more info about the problem areas most often reported to tech support.

This concept isn’t particularly new. Businesses have been trying to do this with IM for years. The difference is that IM only gets you access to one person at a time and you have to think to contact them specifically. Then you get caught up in a conversation when you really just need an answer. The Twitter broadcast model quickly gets your query out to a pool of possible responders. Even more importantly, by subscribing to the posts of others throughout the business (eg sales, dev managers, support, evangelists, brand managers, etc…) employees extend their sensors out to include many more valuable inputs. Once they get beyond a certain size, most businesses inevitably Balkanize into distinct units that gradually build up walls and grow insular. Enterprise Twitter (for lack of a better term) would help dissolve these boundaries. This is especially critical in an age of convergence where even the most diverse businesses are feeling the need to integrate and build interoperability across their portfolios.

Again, enterprise Twitter isn’t a new concept but it’s one that so far seems to have escaped either the demand side of the equation – businesses (I’m constantly amazed by the deep endemic failures of communication within companies), who desperately need better & more efficient forms of internal communication; and the supply side, which remains unable to provide any sort of internal enterprise-grade broadcast messaging solutions.

[Updated: Check out Mike Gotta's note on Enterprise Version's of Twitter.]

E-Tech 2009 Twitter Round-up

Here’s a selection of my tweets from the O’Reilly Emerging Technology Conference this past week. These are the ones I think grab the juicy nuggets from the speaker’s presentations. [In temporal order with the earliest (ie Monday eve) listed first.]

Tim O’Reilly: “We have greatness but have wasted it on so much. ”
We have an unprecedented opportunity to build a digital commonwealth. #etech
Work on something that matters to you more than money. This is a robust strategy. #etech
Niall Kennedy: Energy Star rating for web apps? Thinking of clouds & programming like tuning a car for better gas mileage. #etech
Cloud computing: no reasonable expectation of privacy when data is not in your hands. Not protected by 4th amendment. #etech
Alex Steffen: Problems with water supply are based in part on our lack of beavers. #etech
Social media for human rights. http://hub.witness.org #etech
Gavin Starks – Your Energy Identity & Why You Should Care. see http://amee.com #etech
Maureen Mclugh – Consider that technology may be evolving in ways that are not particularly interested in us. #etech
Becker, Muller: We have under-estimated the costs and over-estimated the value of our economy. #etech
Becker, Muller: We assume economic trade must be the primary framing of value in our lives. Why? #etech
Design Patterns for PostConsumerism: Free; Repair Culture; Reputation Scaled; Loanership Society; Virtual Production. #etech
NYT: emerging platforms, text reflow, multitouch, flexy displays, smart content, sms story updates, sensors, GPS localized content. #etech
Jeremy Faludi: Buildings & transport have the largest impact on climate change. Biggest bang for the buck in re-design. #etech
Jeremy Faludi – Biggest contributor to species extinction & habitat loss is encroachment & byproducts from agriculture. #etech
Jeremy Faludi – Best strategies to vastly reduce overpopulation: access to birth control & family planning, empowerment of women. #etech
Tom Raftery: Grid 1.0 can’t manage excess power from renewables. Solution: electric cars as distributed storage. #etech
Considering the impact of pluging AMEE (@agentGav) data in ERP systems for feedback to biz about supply chain impacts. BI meets NRG ID.
Mike Mathieu: Data becoming more important than code. Civic data is plentiful and largely untapped. Make civic apps! #etech
Mike Mathieu: Take 10 minutes today and pick your crisis. Figure out how to create software to help. #etech
What is #SantaCruz doing to make civic data available to service builders? We want to help SC be healthier & more productive.
Mark Fraunfelder: “I haven’t heard of anybody having great success with automatic chicken doors.” #etech [re-emerging technology]
Realities of energy efficiency: 1gallon of gasoline = ~1000hrs of human labor. #etech
Kevin Lynch: Adobe is saving over $1M annually just by managing energy. #etech
Designing backwards: Think about the destiny of the item before thinking about he initial use. (via Brian Dougherty) #etech
RealTimeCity: physical & digital space merges, people incorporate intelligent systems, cities react in accord w/needs of pub welfare. #etech
Oh my we’re being LIDAR’d while Zoe Keating plays live cello n loops. ZOMG!!!
zoe keating & live lidar is blowing my mind at #etech 1.3M points per sec!
Julian Bleeker cites David A. Kirby: “Diegetic prototypes have a major rhetorical advantage over true prototypes” #etech
Julian Bleeker: Stories matter when designing the future, eg. Minority Report. #etech
Julian Bleeker: “Think of Philip K. Dick as a System Administrator. #etech
Rebecca MacKinnon: Which side are we helping, River Crabs or Grass Mud Horses? #etech
Kati London: How can we use games to game The System and how can they be used to solve civic problems? #etech
Nathan Wolfe: Trying to fight pandemics only at the viral human level ignores deep socioeconomic causes of animal-human transmission. #etech
Nathan Wolfe, re: viral jump from animal to human populations: “What happens in central Africa doesn’t stay in central Africa.”
Nathan Wolfe: need to work with % of population w/ hi freq of direct contact with animals for early detection of viral transmission.
Nathan Wolfe: Vast majority of biosphere is microscopic, mostly bacterial & viral. Humans: very small piece of life on Earth. #etech

Another Rant: On the Cloud, Augmented Reality, & the Networked World

[This is a reply I left recently to a Global Futures question about the near-future of the web. It goes a little off-topic at the end but such is the risk of systems analysis. Everything's connected.]

Within 10-15 years mobile devices will constantly interact with the world around us, analyzing objects, faces, signage, locations, and anything else their sensors can engage. Camera viewfinders will identify visual sources using algorithms to match them up with cloud data repositories. Bluetooth and GPS will interact on sub-channels silently exchanging relationships with embedded sensors across devices and objects. A user’s mobile device will become their IP address hosting much of their profile information and mediating relationships across social nets, commercial transactions, security clearances, and the array of increasingly smart objects and devices.

Cloud access and screen presence will be nearly ubiquitous further blurring the line between desktop, laptop, server, mobile devices, and the objects in our world. It will all be screens interfacing between data, objects, and humans. Amidst the overwhelming data/content glut we will outsource mathematical chores to cloud agents dedicated to scraping data and filtering the bits that are pertinent to our personalized affinities and needs. These data streams will be highly dynamic and cloud agents will send them to rich media layers that will render the results in comprehensible and meaningful displays.

The human sensorium and its interaction with reality will be highly augmented through mobile devices that layer rich information over the world around us. The digital world will move heavily into the natural analog world as the boundaries between the two further erode. This will be readily apparent in the increasing amount of communication we will receive from appliances, vehicles, storefronts, other people, animals, and even plants all wired to the cloud. Meanwhile, cloud agents will sort through vast amounts of human behavioral information creating smart profiles and socioeconomic and environmental systems models with incredible complexity and increasing predictive ability. The cloud itself will be made more intelligible to agents by the standardization of semantic web protocols implemented into most new sites and services. Agents will concatenate to tie services together into meta-functions, just as human collectives will be much more common as we move into increasingly multicellular functional bodies.

The sense of self and our philosophical paradigms will be iterating and revising on an almost weekly basis as we spread out across the cloud and innumerable virtual spaces connected through instantaneous communication. Virtual worlds themselves will be increasingly common but will break out of the walled-garden models of the present, allowing comm channels and video streams to move freely between them and the social web. World of Warcraft will have live video feeds from in-world out to device displays. Mobile GPS will report a user’s real-world location as well as their virtual location, mashing both into Google Maps and the SketchUp-enabled virtual map of the planet.

All of this abstraction will press back on the world and create even greater value for real face-to-face interactions. Familial bonds will be more and more cherished and local communities will take greater and greater control of their lives away from unreliable global supply chains and profit-driven corporate bodies. Most families will engage in some form of gardening to supplement their food supply. The state itself will be hollowed out through over-extended conflicts and insurgencies coupled with ongoing failures to manage domestic civic instabilities. Power outages and water failures will be common in large cities. This will of course further invigorate alternative energy technologies and shift civic responsibilities to local communities. US manufacturing will have partially shifted towards alternative energy capture and storage but much of the real successes will be in small progressive towns rallying around local resources, small-scale fab, and pre-existing economic successes.

All in all, the future will be a rich collage. Totally new and much the same as it has been.

Cisco, video, digital hardware… and Adobe?

When Redmonk’s James Governor opined that Cisco might make a play for Adobe Systems, bells went off in my head. It suddenly made a lot of sense and made me realize I should really be paying more attention to Cisco.

From James:

Cisco competing with Apple? Who would have thunk it? To really make its ambitions count I believe Cisco will make a play for Adobe, filling out a video internet value chain from low to high production to the web.

Adobe is arguably the predominant enabler of web video, with much of the web firmly invested in Flash and all of it’s platform components & accessories. Cisco knows how to make hardware and has not been at all shy about their goals in the consumer digital market, pursuing rich internet-enabled media on set top boxes and in TVs. Recall not too long ago the legal battle with Apple over the term “iPhone”. And at CES in January Cisco is expected to introduce a new line of consumer media products.

Cisco Systems, the dominant provider of the digital pipes that run the Internet, is making a big play in digital entertainment. At the Consumer Electronics Show in January in Las Vegas, it plans to introduce a new line of products, including a digital stereo system that is meant to move music wirelessly around a house.

That is the first small move in a long-term strategy to take on Apple, Sony and the other giants of consumer electronics. Cisco is working on other gadgets that will let people watch Internet video on their televisions more easily. And its biggest bet is that people will want to use a version of its corporate videoconferencing system called Telepresence to chat with their friends over their high-definition televisions.

While this article notes the looming battle with Apple & Sony it should be considered that the set-top and video market is clearly of interest to Adobe, as well as the obvious similarities between Telepresence & Adobe Connect for video conferencing solutions. Cisco & Adobe are both invested in the Open Screen project but the relationship between the two will surely get closer whether or not some sort of acquisition is in play.

The strategy for Cisco, of course, is to encourage more high-bandwidth content running through all those Cisco routers that will need to be upgraded to keep up with the throughput. There’s no greater bandwidth hog than video and its just exploding with the boom in cheap consumer video hardware and turnkey hobbyist solutions. Adobe would be wise to pursue the authoring side of this hobbyist video boom, and focus on an aggressively marketed, cross-platform Premiere Express solution, as well as developing an ecology for video capture and publish from mobile devices. Meanwhile, their set-top Flash initiative will continue to intrigue Cisco and if the two are not already talking to make sure Cisco is using Flash everywhere possible, then somebody at Adobe needs to get busy and make it happen.

It remains to be seen whether Cisco might make an acquisition play or Adobe but it seems likely that the future of the two companies will be tightly coupled.

Enabling Rich Content Rendering for Dynamic Ambient Displays

Now more than ever, screens are competing for our attention in daily life. The distinctions between desktop PC, laptop PC, mobile, etc… are being replaced by the simple abundance and omnipresence of digital screens conveying information and mediating interaction. As LCD newsfeeds, airport terminal displays, set-top video box menus, billboards, advertisements, multi-touch screens, and augmented mobile devices, data and content are everywhere. All of it requires a lightweight and dynamic graphic layer in which to render content. Dynamic render plugins like Flash and Silverlight are very well engineered to address the requirements of these displays but it should be understood that much of the technology represents a shift away from the 2D ad/interactive paradigm primarily addressed by the Flash runtime.

Mobile augmented reality solutions require heads-up-display, alpha-channel rendering of text and graphics. Dynamic data visualization requires strong integration with back-end databases as well as messaging protocols like JSON, SOAP, and SMS. To enable rendering large datastreams in plugin runtimes like Flash & Silverlight, companies should aggressively pursue runtime adoption across mobile devices and smart phones, while defining prototypes for active camera overlays. They should optimize rendering and expand into all rich-content displays (set-top, automotive, kiosks, smart objects, and embedded systems built on the Android/Linux platform). They should optimize for dynamic rendering of large data streams, like the Nasdaq AIR application. The runtimes should be increasingly exposed to SEO and analytics retrieval so that secondary services and agents can easily be built on user workflows and data collection. And they should not neglect the 3D gaming market, but should consider how it can play a role in immersive worlds as a dynamic data layer (eg billboard adverts that can be remotely updated).

Competitive landscape: Adobe Flash (dominant market share, full turn-key solution), Microsoft (Silverlight remains the strongest competitor to Flash though it continues to play catch-up with our tech), Java (JavaFX just released but has little traction and is too late to the party), W3C (HTML5 has perhaps the largest share of hearts but is also the slowest to move), Google (might be cutting around Flash & Silverlight by pushing its interests into HTML5, Mozilla, and through Android & Chrome).

Related: Augmented reality solutions will require semantic architecture and image recognition
algorithms (identification, recognition, relation).

MMOGs and CCTVs: 3D Games For Spectators

Killer feature: Immersive gaming companies like Rockstar, Blizzard, ID, Epic, NCsoft, Valve et al should implement a way for game server admins to tap live in-scene camera feeds that can be bussed out to HTML. Liberty City should have public CCTV’s that viewers can watch from their browser or mobile client. Like a virtual Adam’s Block people could tune in to an alleyway or freeway underpass and let the action unfold in the streets below. World builders should enable first/third-person game cameras to send their viewport to Flash embeds.

Imagine tuning into a South Korean Quake deathmatch tournament as a spectator and being able to view the world through the game eyes of the champion or to switch across various fixed camera feeds in corridors, over walkways, and above central arenas to witness the gameplay from alternate angles. It might seem odd in sports-addled America and the UK but in Asia hundreds of millions are online and they love 3D gaming. The gameworld becomes a performance space. Second Life could finally entertain more than just the local cliques by broadcasting the actions of art collectives and protest groups out to the world.

It doesn’t take much effort to see how this multiplies the available media advertising real-estate considerably. Fixed camera views become hot advert property rented to savvy marketers who know they can reach both the local gamers and the viewing masses. Attention property goes meta. This is a two-way street as external feeds begin to pipe into the game worlds. The walls really start coming down when you can take a cellphone call in-game and then wave to the camera for your friends to see. They take a screenshot and then send it to the clan web site and the GTA Flickr feed that gets displayed on the side of a building in downtown Liberty City. Sponsored by Verizon of course.

Blizzard claims something like 10 million regular users of World of Warcraft. Their WoW wikia page is the largest collection of data on anything in the world. You think they might be interested in being able to view their clan members remotely and communicate with them at any time? Or tune into video feeds showing the night elves in Darnassus, or watch the pass at Chillwind’s Point? (Truth: I googled these.) And if Blizzard wins the fight to keep the gameworld free from advertising, marketers can accrete around the popular viewing channels and their Flash embeds.

These worlds are dark clouds, opaque from all but those that pay the playing fee. Yet there’s so much entertainment and interest and real human nature playing out in these worlds. People will watch. Agents will sift through the user data feeds streaming out along with the A/V feeds. And think of the up-sell as viewers convert to players. Someday I’m certain some of the reality tv channels on my mobile device will be looking into such immersive worlds and showing me where in my friends are and who they’re battling and what type of car they just jacked for a joyride through Liberty City. (I *really* want the GTA CCTV’s!)

G-Speak MIT Gestural Interface –> Minority Report

General Guidelines for Cloud Tech

These are my brief (and very rough) notes from 5 minutes ago summarizing some guidelines I feel are critical for application & service development:

The cloud is everywhere.
Applications grab eyes.
Mobile/desktop/cloud – Don’t draw partitions.
Seek integrations across platforms.
Scale services by UI. Eg editing photos on a mobile is not appropriate but capturing images and uploading them to a workspace is.
Build communities.
Provide ubiquitous workspaces.
Communicate, Collaborate, Create, Share

2016 Metaverse Roadmap

I’m heartened to find the Metaverse Roadmap, sponsored by the Accelerating Studies Foundation. While I’ve been moaning about the shortcomings of immersive 3D technologies, they’ve been defining the template for progress. Much of their thoughts align with my own, painting an exciting future of convergence across modalities, devices, and workflows.

The emergence of a robust Metaverse will shape the development of many technological realms that presently appear non-Internet-related. In manufacturing, 3D environments offer ideal design spaces for rapid-prototyping and customized and decentralized production. In logistics and transportation, spatially-aware tags and real-time world modeling will bring new efficiencies, insights, and markets. In artificial intelligence, virtual worlds offer low-risk, transparent platforms for the development and testing of autonomous machine behaviors, many of which may be also used in the physical world. These are just a sampling of coming developments based on early stage Metaverse technologies.

In sum, for the best view of the changes ahead, we suggest thinking of the Metaverse not as virtual space but as the junction or nexus of our physical and virtual worlds.

Second Life Avatar Controlled By Thoughts of Paraplegic

I have a lot of issues with Second Life – mostly because I’m frustrated by their potential and their seeming inability to act on it – but it’s nevertheless an interesting sandbox to explore the greater frontiers of virtual immersion and social ontology. To this end, Japanese researchers have wired up a Second Life avatar to respond to the thoughts of a paraplegic.

…he wore headgear with three electrodes monitoring brain waves related to his hands and legs. Even though he cannot move his legs, he imagined that his character was walking.

He was then able to have a conversation with the other character using an attached microphone, said the researchers at Japan’s Keio University.

…”In the near future, they would be able to stroll through Second Life shopping malls with their brain waves… and click to make a purchase,” Ushiba said.

Convergence and Continuity Across Virtual Worlds

In games, immersive worlds, forums, social networks, and in blogs we inhabit multiple selves. In most cases, theses virtual spaces are walled islands with little relation between them. Increasingly it’s becoming apparent that continuity is necessary to resolve these fractured selves and to open up the channels of communication between the diversity of online containers. This can be seen in the new wave of web 2.0 aggregators like FriendFeed and Plaxo that aim to collate our myriad profiles, friends and content streams into a single portal. Now, Technology Review reports that several companies are working to enable avatars to move between virtual worlds.

More and more, such affordances will move into virtual spaces. 2D content streams and communication pipelines will feed into and across immersive worlds. A WoW player should be able to call up a HUD console in the game and locate their friends across all of the virtual worlds they’re currently in. They should then be able to communicate with them through IM or VoIP and subsequently transport to join them in another world. GTA4 has announced a feature to allow users to call each other in-world using the game cell phone. Shouldn’t this extend across game worlds and out into real-world mobiles? API’s could evolve to mine user communications (Twitter in WoW?) and chart locations on world maps. In the age of digital society, findability is key.

The vast amounts of personal profiling we’re building up around ourselves in MySpace, Facebook, blogs, and other forums should be accessible through our avatars and from all places we inhabit, virtually and in reality. It should be present in our devices and our profiles. As avatars, it should follow us like a digital skin (secured and opt-in, of course) layered in transaction-appropriate trust profiles that fly-out on mouseover. My avatars should contain more information than just polygons and scripted motions. Social transactions are information exchanges. My LinkedIn profile should be accesible to anyone in 2D and 3D if I so desire.

The realness of immersive worlds should leverage the fundamental reality of our digital profiles and interests. If these platforms are going to become truly compelling, they must work to integrate the API’s, content streams, and communication channels of the web2.0 revolution. We’re in the midst of a completely unprecedented historical shift as all of our cultural and intellectual content is going digital, made manifest in searchable, findable, and persistent datalogs. The profiles we create around our virtual selves are growing larger and larger, and they are being recorded and left open for many eyes to see. Imagine the political candidates running 10 or 15 years from now. So much of their lives will be a matter of public record easily searchable and graphed out to show affiliations, donations, histories and contradictions. So much of who they are will live online like a shadow. SO much of who we all are.

Virtual worlds are poised to engage directly in this shift and draw culture and identity into their domain. Instead of closed platforms, worlds like Second Life must open up and grow to become contiguous spaces whose character arises from the types of people that choose to gather there by affiliation, interest, and intention. MMORPG’s like WoW will continue to offer highly crafted narratives, specialized social groups and hierarchies, and bleeding edge rendering tech but will acknowledge the tremendous personal content within each player distributed across their digital and analog lives.

Of course, if virtual platforms become more open, their business models will inevitably shift towards advertising. Space is space, whether 2D, 3D, or 4D, and eyes are eyes especially when they gather in great enough concentration. As in the real world, the exchange of goods and services will always be of great value in any domain, so the shift towards continuity will be a shift towards reality. Virtual worlds have the unique proposition of creating fantasy within the world of life. So the shift towards reality in the context of a realized fantasy brings both closer together. It is part of the alchemical formula of bringing spirit into matter. It is the power of gods to create in an unlimited universe. It is the movement of the ghost in the machine as our real selves grow more and more to include virtual, digital, non-local aspects of identity and presence. Who am I but the sum of my transactions with the world? These words I’m writing and posting on the global billboard become preserved bits of my self. Your interactions with them extend my identity into the virtual world. All my words are facets of my expanding digital identity. My self-reflection extends from my body, my deeds, my actions towards others around me, to include the ideas and statements I leave online, the avatars I inhabit, and the webs of disembodied people I associate with. In 100 years I may roll up in bits under some social anthropologist’s data-mining PhD nudging their graphs this way or that with my Tweets and posts.

Aggregation of social data serves a very practical role of making it easier for us to manage an increasingly vast amount of data, but it also serves a larger role of helping us defragment our sense of self as it fractures out across so many new digital domains rising and falling daily. If we’re to walk like new gods through worlds both real and virtual, shouldn’t we do so with as much wholeness as possible? In a world that’s made it so challenging to have a fully integrated psyche it’s really imperative that we lay down a strong foundation of holism and continuity as we move into the unfettered vastness of the digital noosphere. As strong cohesive selves we can better wear the masks of avatars and wield the power of virtual gods.