Category: interface

Direct Brain-Computer Interface Will Require a New Language of Interaction

[Cross-posted from Signtific.]

When Apple Computer recently released the 3.0 version of its iPhone OS one of the most anticipated new features was Cut & Paste. This simple task has been a staple of computing since GUIs were part of the OS, so why did it take Apple until it’s 3rd OS version to implement the feature for the iPhone?

As Apple tells it, there was incredible deliberation over how best to design the user experience. This is, after all, the first and only fully multi-touch mobile computing device. Apple has been meticulously developing and patenting the gestural language through which users interact with the device. Every scroll and pinch, zoom and drag is a consciously designed gesture adding to Apple’s growing lexicon of multi-touch interface. Implementing Cut & Paste was a substantial challenge to create the most accessible gestural commands within the narrow real-estate of the mobile screen.

Now, consider interacting with the same content types available on an iPhone or anywhere in the cloud, but remove the device interface and replace it with a HUD or direct brain interface. If the content is readily visible, either as an eyeglass orverlay or directly registered in the visual cortex, how do we give a UI element focus? How do you make a selection? How do you scroll and zoom? How do you invoke, execute, and dismiss programs? Can you speak internally to type text? How might a back-channel voice be distinguished from someone standing behind you? How do you manage focus changes between the digital content and the visual content of the real-world when both are superimposed in some state?

The fields of Human Computer Interaction and User Interface & Experience Design address these challenges for interacting with digital content and processes, but what new interaction modalities may be developed to better interface humans and computers? As we internalize computation and interaction, the disciplines of HCI & BCI will begin to interpenetrate in ways that may radically alter the conventions of the Information Age.

Brain-Computer Interface

In my present tenure as a Visiting Researcher at the Institute for the Future I’ve been posting a lot of Signals pertinent to Brain-Computer Interface over at the Signtific open source research site. My Signals are listed under the tag “ProgrammableEverything”.

Check ‘em out if you’re interested in the fascinating & accelerating field of BCI. Also feel free to add your own Signals you see in the world or are engaging in your professional research.


Reflections on Thailand

[Extensive photo album here.]

Japan crossed with Mexico. Hack, mash, and lash everything together. Very hot and thick, humid and prone to short heavy rains. Bangkok is larger than expected, with a higher skyline. Slum-like in many ways but comfortable. Dirty, aged, grafitti’d, tagged, polluted, smelly, hungry, buggy, feral. Friendly, smiley, reverent, strong, spiritualized, watery, creative, delicious, surviving with tenacity. Temples & tenements, luxury hotels and megamalls. Insane traffic and transport. Little regard for lanes or right of way. Swarms of motorbikes, vespas. Cheap and dangerous tuk tuk 3-wheelers. Families piled onto scooters, kids asleep, baggage strapped on. Traffic flow like a logjam, shifting metal slabs moving within inches of each other, victory goes to the bold in a cloud of exhaust. The mighty Chao Phraya cutting its way through Bangkok and out to the coast, it’s headlands in the foothills below Burma. These are river people, with traffic on the waterways as busy and chaotic as the streets. The river is deep, a 1/4 mile wide, running green & tan, dirty and littered with commercial & vegetal detritus. After the rains clumps of fallen jungle float on its surface, carried down from farms and foothill tributaries. Black & yellow birds land on leafy branches half submerged to dine on nuts and berries. Water taxis from hotel to Sky Train. Fantastic monorail, the SRT, its cement track a modern work of civil engineering adding to the Tokyo vibe of downtown Bangkok.

Tangled mess of black utility cable slashing horizontal lines across most everything, tied in to huge transformers, burnt metal grills pumping amperage for the teeming metropolis of 6 million. The twisted infrastructure grows organically like a banyan, stretching out axonal to connect and communicate. Most buildings are old haggard tenements, their facades stained with a dark grey wash like grease and ash drawn out of the thick air. Structures that seem abandoned, uninhabitable, are strung with drying laundry drawn perpendicular to the necessarily ubiquitous swamp coolers lining the sides of each floor. Broken concrete fields under freeway overpasses offer football grounds lined by graffiti mural walls under chainlink divisions.

Downtown, luxury malls with Louis Vuitton and Burberry fronted by large altars of golden Buddha’s and Ganesha’s, black marble elephants flecked with gold, yellow floral garlands and incense offered by shoppers to their immaterial gods. A sign at Wat Phrao Keo in broken Thaiglish sagely, if not inadvertently, warns visitors to “Beware of your valuable possessions”. Technology, commerce, wealth, and western aesthetics have moved in with the economic development afforded here as in every other large city by the realities of globalized communication and trade.

Down crowded alleyways lined with merchant stalls and open air ad hoc kitchens, thick with pedestrians, cars, tuk tuks, and manic motorcyclists weaving through the narrow channels, over rooftop patios caged against some unseen menace, rise countless golden and white and glittery temple spires. Buddhist Wats take residence everywhere, themselves seemingly hacked into the dense fabric of the city, rising like aspirational fruiting bodies of ancient mycelial webs. Wat Arum, Wat Pho, Wat Phra Keo & the Grand Palace, and innumerable others. Religion & myth is woven throughout the populace. Every building has it’s own adjacent spirit house offering residence to the disincarnate lest they move into your own home. City walls are tacked with incense holders between stores. Banyans breaking through the sidewalks are wrapped with rainbow sashes honoring their freakish holy treeness. Every taxi has a statue on the dash or mala hanging from the rearview or Buddhist stencil on the headboard or any combination of the aforementioned. A 3-day Buddhist holiday shut down all government and banking.

The current Thai king is the longest reigning monarch of the modern age, holding office since 1950. Thailand was the only East-Asian country to resist British colonialism, sparing its autonomy by ceding a few bits of territory along the Burmese & Malay borders. Indeed there are long running conflicts with the Burmese, and Buddhist Thailand is in the midst of an insurgency along the Malaysian border from an advancing Islamic populace. The cabinet of the prime minister and the military have provided ongoing political theater as each vie back and forth for the seat of power. Most transfers of power, even in the case of multiple coups, have been bloodless. The Thai people themselves seem to have little interest in these power games, preferring a life of pragmatic spirituality while maintaining a deep abiding love and respect for the king. The two possibly mortal social offenses in Thai society are speaking ill of the Buddha and speaking ill of the king.

Farming is honored. Rubber trees and palms cover most southern land, providing two of the country’s largest exports. The Thai peninsula includes all the most breathtaking exotic tropical beach locations you could imagine, including the stunning Railay Bay – famed for the movie The Beach. Beautiful light blue waters, ridiculously warm and salty, stretched for ages across the gulf. Koh Samui running on Full Moon inertia, tourist trinkets, and scattered luxury resorts sheltered from the hustle. Low inland jungles bring minimal shade to island shanties in seemingly impossible poverty. Yet they survive & persist and move through generations like the rest of us. Koh Phangan also still milking their internationally notorious Full Moon Rave scene, adding a Half Moon party to underwrite the Euro draw. Even away from the main strips the beach scenes has a fun accidental Burning Man vibe, a shoreline esplanade of shanty bars and sound systems. Expats all over the place. Seems easy to get lost for months, years, decades in some seaside shack eating fruit and fish in a poor man’s paradise. Impossible walls of insects whip up into sudden frenzy, a cacophonous wail of screamapillars, giant cicadas that still don’t seem anywhere near big enough to make such a pitch. Monkey troops swing across canopies carpeting tall rock slabs jutting from the water. A rock climber’s joy, sheer faces hung with dripping stalactites and pocked with rope tie-ins. These tall rocks are scattered by the hundreds – thousands? – across the Gulf of Thailand and Andaman Sea. A boater’s paradise. You could spend months exploring thin beaches stretched around the edges of countless small jungle rock islands.

In the South, each night was attended by thunderstorm, often over sea or above the island peaks. Big black charcoal canvas lumbering across, flicker flashed with lightning bursts every few moments, often too distant to hear the thunderclap, then a sudden ear-shattering rend of ozone right above. When the heavy rains hit they come quickly and with ferocity. Never seen rain like it. So thick that it occluded line of sight to 20 meters or so, hiding everything beyond in watery showers. From the steep island peaks water rushes down in sudden rivers cutting through beach sands, pushing tan clouds out into the bay, a shimmering clear layer of fresh water forcing the saline back out over the ocean’s surface. Giant raindrops agitate the bugs forcing them to take flight in peppery swarms. Small opportunistic swift-like birds take to the skies darting and arching, turning and diving to pluck the insects mid-air in some ancient deeply programmed ballet of the food chain. Life goes on. It must. When rains come often and fiercely you can’t just drop your business. This was especially so in Bangkok whose streets are lined with tirelessly deployed open markets bare to the sky save for a small canvas over each. In 20 or 30 minutes the rain will likely pass so there’s no point in worrying much about the interlude.

While the deep south is struggling with a mounting Islamic insurgency, and the peninsula is attending the construction of more new mosques, the buddhist majority continues to permeate life with the spirit of their patron, accompanied by a host of Garuda and Nagas and a menagerie of mythic beasties syncretized from India and China. If Thai Buddhist Bangkok is feral and lashed and relentlessly modded in ghetto slapdash, the Bangkok Chinatown is 10x more so compressed into tighter alleyways, with more people and motorcylces (Vespas apparently seek Chinatown to live out their golden years), hung with impossibly more spaghetti cables, and festooned with walls of neon Mandarin signage casting a little too much light onto freakish displays of animal carcass and presumably inedible seafood and giant transparent sacks of fried pork product and stall after stall of fashionable Versace & Loius Vuitton knock-offs. Imagine threading your way down a dark, narrow alley lined with flea market stalls and no-health-code/no-insurance open air cart kitchens, filled with people pressing in all directions through dense heat and smell and rot, then send a motorcycle down the alley every few moments to do battle with cross-traffic carts and tuk tuks. Now imagine the alley is a whole network labyrinth covering multiple blocks between several-story tenaments streaked with black soot and stain and hung with drying clothes and black cables. This is why we western pansies stay in the nice hotel with A/C and a pool.

The final capper to the trip was in Bangkok the night before our departure. After the evening rains subsided, my partner and I went down to the pool for a night swim, around 9pm. Refreshing and fun we frolicked and generally soaked up the remaining moments of our stay. Then, in the poolside darkness moving low between the lounge chairs, I saw a large reptilian form lumbering along. “Dude, there’s a fricken alligator coming towards the pool!” I exclaimed excitedly. As it marched into the light we realized it was actually a monitor lizard – Varanus salvator, to be precise – about 5-6ft long with a fattened belly like it just ate a dog or possibly a small European child. “If that thing gets in the water, we get out immediately” I said with some urgency. I knew it could swim and see underwater much better than we could. No reason to tangle with a 6ft thunderlizard in a foreign country with questionable health care. Sure enough the beast slipped into the pool and sidled along the swim-up bar. We hopped out, laughing nervously, and I approached the lizard from a careful distance. Grabbing the pool attendant I motioned towards the monster. “That’s bad”, he said in a way that suggested that, bad as it may be, it wasn’t unusual. And so he casually splashed the creature with water nudging it along until it climbed out of the pool slowly, begrudgingly, made it’s way back into the riverside brush. It was easily the biggest lizard I’d ever seen in the wild.

The final day we were denied pool access during a particularly solid rain. When it’s always 90+ degrees & 90+% humidity, swimming in the rain is quite nice. But no, we were not allowed. “Why?” I protested. “Lightning” retorted the attendant. Fair enough, I thought. Then, in a casual but cautionary aside, the attendant reflected, “We had an accident last year”. This is the Bangkok Riverside Marriott, a fancy if not dated family hotel. Apparently buried somewhere deep in the boilerplate legalese fine print of our hotel contract is the clause, “Marriott Properties takes no liability in the event of any hotel guest or visitor getting suddenly struck by lightning and then slowly eaten by ferocious monitor lizards”.

Every Enterprise Needs Its Own Twitter

To the title of this post, I was on a tech support call with AT&T/Apple this afternoon. Facing surprising SMS overage charges on my mobile account, I wanted to know if Twitter apps like Tweetie might be using the SMS channel to conduct their transactions. I figured not but the Tweetie site gave me no info.

So I asked the support person, who knew about Twitter (better than AT&T) but not about whether Twitter iPhone apps use SMS. She put me on hold to ask a support specialist. About 15 minutes later she came back with a reasonable response (“they probably don’t use SMS”) and a suggestion to talk to the app vendor.

All this had me thinking about the huge inefficiency at play with the responder trying to locate the specialist, get their attention and time, probably juggling multiple phone lines, to then give me one person’s measured response. I imagined a not-too-distant future where the call responder typed my query into a local Twitter-clone running on the tech support network inside AT&T/Apple. This query would immediately push out to subscribers – some required by their manager to subscribe to all tech support feeds, and others who just want to see the problems customers are encountering. I then imagined that some of these subscribers would run search filters on the messaging stream in order to be alerted to those queries they were most interested in tracking.

So somewhere in the bowels of tech support there’s a guy who is a Twitter power-user, or an engineer who wrote the SMS api’s, or a community developer that helps 3rd parties like Tweetie write iPhone apps… and they get pinged every time a tag of interest comes across the network. They see “iPhone Twitter SMS” and respond with the info. With enough of these transactions the call responder will have an archive of tech support tweets they can search through to see if someone has already responded. Of course, this archive provides another layer of analytic data that can be mined to get more info about the problem areas most often reported to tech support.

This concept isn’t particularly new. Businesses have been trying to do this with IM for years. The difference is that IM only gets you access to one person at a time and you have to think to contact them specifically. Then you get caught up in a conversation when you really just need an answer. The Twitter broadcast model quickly gets your query out to a pool of possible responders. Even more importantly, by subscribing to the posts of others throughout the business (eg sales, dev managers, support, evangelists, brand managers, etc…) employees extend their sensors out to include many more valuable inputs. Once they get beyond a certain size, most businesses inevitably Balkanize into distinct units that gradually build up walls and grow insular. Enterprise Twitter (for lack of a better term) would help dissolve these boundaries. This is especially critical in an age of convergence where even the most diverse businesses are feeling the need to integrate and build interoperability across their portfolios.

Again, enterprise Twitter isn’t a new concept but it’s one that so far seems to have escaped either the demand side of the equation – businesses (I’m constantly amazed by the deep endemic failures of communication within companies), who desperately need better & more efficient forms of internal communication; and the supply side, which remains unable to provide any sort of internal enterprise-grade broadcast messaging solutions.

[Updated: Check out Mike Gotta’s note on Enterprise Version’s of Twitter.]

E-Tech 2009 Twitter Round-up

Here’s a selection of my tweets from the O’Reilly Emerging Technology Conference this past week. These are the ones I think grab the juicy nuggets from the speaker’s presentations. [In temporal order with the earliest (ie Monday eve) listed first.]

Tim O’Reilly: “We have greatness but have wasted it on so much. ”
We have an unprecedented opportunity to build a digital commonwealth. #etech
Work on something that matters to you more than money. This is a robust strategy. #etech
Niall Kennedy: Energy Star rating for web apps? Thinking of clouds & programming like tuning a car for better gas mileage. #etech
Cloud computing: no reasonable expectation of privacy when data is not in your hands. Not protected by 4th amendment. #etech
Alex Steffen: Problems with water supply are based in part on our lack of beavers. #etech
Social media for human rights. #etech
Gavin Starks – Your Energy Identity & Why You Should Care. see #etech
Maureen Mclugh – Consider that technology may be evolving in ways that are not particularly interested in us. #etech
Becker, Muller: We have under-estimated the costs and over-estimated the value of our economy. #etech
Becker, Muller: We assume economic trade must be the primary framing of value in our lives. Why? #etech
Design Patterns for PostConsumerism: Free; Repair Culture; Reputation Scaled; Loanership Society; Virtual Production. #etech
NYT: emerging platforms, text reflow, multitouch, flexy displays, smart content, sms story updates, sensors, GPS localized content. #etech
Jeremy Faludi: Buildings & transport have the largest impact on climate change. Biggest bang for the buck in re-design. #etech
Jeremy Faludi – Biggest contributor to species extinction & habitat loss is encroachment & byproducts from agriculture. #etech
Jeremy Faludi – Best strategies to vastly reduce overpopulation: access to birth control & family planning, empowerment of women. #etech
Tom Raftery: Grid 1.0 can’t manage excess power from renewables. Solution: electric cars as distributed storage. #etech
Considering the impact of pluging AMEE (@agentGav) data in ERP systems for feedback to biz about supply chain impacts. BI meets NRG ID.
Mike Mathieu: Data becoming more important than code. Civic data is plentiful and largely untapped. Make civic apps! #etech
Mike Mathieu: Take 10 minutes today and pick your crisis. Figure out how to create software to help. #etech
What is #SantaCruz doing to make civic data available to service builders? We want to help SC be healthier & more productive.
Mark Fraunfelder: “I haven’t heard of anybody having great success with automatic chicken doors.” #etech [re-emerging technology]
Realities of energy efficiency: 1gallon of gasoline = ~1000hrs of human labor. #etech
Kevin Lynch: Adobe is saving over $1M annually just by managing energy. #etech
Designing backwards: Think about the destiny of the item before thinking about he initial use. (via Brian Dougherty) #etech
RealTimeCity: physical & digital space merges, people incorporate intelligent systems, cities react in accord w/needs of pub welfare. #etech
Oh my we’re being LIDAR’d while Zoe Keating plays live cello n loops. ZOMG!!!
zoe keating & live lidar is blowing my mind at #etech 1.3M points per sec!
Julian Bleeker cites David A. Kirby: “Diegetic prototypes have a major rhetorical advantage over true prototypes” #etech
Julian Bleeker: Stories matter when designing the future, eg. Minority Report. #etech
Julian Bleeker: “Think of Philip K. Dick as a System Administrator. #etech
Rebecca MacKinnon: Which side are we helping, River Crabs or Grass Mud Horses? #etech
Kati London: How can we use games to game The System and how can they be used to solve civic problems? #etech
Nathan Wolfe: Trying to fight pandemics only at the viral human level ignores deep socioeconomic causes of animal-human transmission. #etech
Nathan Wolfe, re: viral jump from animal to human populations: “What happens in central Africa doesn’t stay in central Africa.”
Nathan Wolfe: need to work with % of population w/ hi freq of direct contact with animals for early detection of viral transmission.
Nathan Wolfe: Vast majority of biosphere is microscopic, mostly bacterial & viral. Humans: very small piece of life on Earth. #etech

Another Rant: On the Cloud, Augmented Reality, & the Networked World

[This is a reply I left recently to a Global Futures question about the near-future of the web. It goes a little off-topic at the end but such is the risk of systems analysis. Everything’s connected.]

Within 10-15 years mobile devices will constantly interact with the world around us, analyzing objects, faces, signage, locations, and anything else their sensors can engage. Camera viewfinders will identify visual sources using algorithms to match them up with cloud data repositories. Bluetooth and GPS will interact on sub-channels silently exchanging relationships with embedded sensors across devices and objects. A user’s mobile device will become their IP address hosting much of their profile information and mediating relationships across social nets, commercial transactions, security clearances, and the array of increasingly smart objects and devices.

Cloud access and screen presence will be nearly ubiquitous further blurring the line between desktop, laptop, server, mobile devices, and the objects in our world. It will all be screens interfacing between data, objects, and humans. Amidst the overwhelming data/content glut we will outsource mathematical chores to cloud agents dedicated to scraping data and filtering the bits that are pertinent to our personalized affinities and needs. These data streams will be highly dynamic and cloud agents will send them to rich media layers that will render the results in comprehensible and meaningful displays.

The human sensorium and its interaction with reality will be highly augmented through mobile devices that layer rich information over the world around us. The digital world will move heavily into the natural analog world as the boundaries between the two further erode. This will be readily apparent in the increasing amount of communication we will receive from appliances, vehicles, storefronts, other people, animals, and even plants all wired to the cloud. Meanwhile, cloud agents will sort through vast amounts of human behavioral information creating smart profiles and socioeconomic and environmental systems models with incredible complexity and increasing predictive ability. The cloud itself will be made more intelligible to agents by the standardization of semantic web protocols implemented into most new sites and services. Agents will concatenate to tie services together into meta-functions, just as human collectives will be much more common as we move into increasingly multicellular functional bodies.

The sense of self and our philosophical paradigms will be iterating and revising on an almost weekly basis as we spread out across the cloud and innumerable virtual spaces connected through instantaneous communication. Virtual worlds themselves will be increasingly common but will break out of the walled-garden models of the present, allowing comm channels and video streams to move freely between them and the social web. World of Warcraft will have live video feeds from in-world out to device displays. Mobile GPS will report a user’s real-world location as well as their virtual location, mashing both into Google Maps and the SketchUp-enabled virtual map of the planet.

All of this abstraction will press back on the world and create even greater value for real face-to-face interactions. Familial bonds will be more and more cherished and local communities will take greater and greater control of their lives away from unreliable global supply chains and profit-driven corporate bodies. Most families will engage in some form of gardening to supplement their food supply. The state itself will be hollowed out through over-extended conflicts and insurgencies coupled with ongoing failures to manage domestic civic instabilities. Power outages and water failures will be common in large cities. This will of course further invigorate alternative energy technologies and shift civic responsibilities to local communities. US manufacturing will have partially shifted towards alternative energy capture and storage but much of the real successes will be in small progressive towns rallying around local resources, small-scale fab, and pre-existing economic successes.

All in all, the future will be a rich collage. Totally new and much the same as it has been.

Cisco, video, digital hardware… and Adobe?

When Redmonk’s James Governor opined that Cisco might make a play for Adobe Systems, bells went off in my head. It suddenly made a lot of sense and made me realize I should really be paying more attention to Cisco.

From James:

Cisco competing with Apple? Who would have thunk it? To really make its ambitions count I believe Cisco will make a play for Adobe, filling out a video internet value chain from low to high production to the web.

Adobe is arguably the predominant enabler of web video, with much of the web firmly invested in Flash and all of it’s platform components & accessories. Cisco knows how to make hardware and has not been at all shy about their goals in the consumer digital market, pursuing rich internet-enabled media on set top boxes and in TVs. Recall not too long ago the legal battle with Apple over the term “iPhone”. And at CES in January Cisco is expected to introduce a new line of consumer media products.

Cisco Systems, the dominant provider of the digital pipes that run the Internet, is making a big play in digital entertainment. At the Consumer Electronics Show in January in Las Vegas, it plans to introduce a new line of products, including a digital stereo system that is meant to move music wirelessly around a house.

That is the first small move in a long-term strategy to take on Apple, Sony and the other giants of consumer electronics. Cisco is working on other gadgets that will let people watch Internet video on their televisions more easily. And its biggest bet is that people will want to use a version of its corporate videoconferencing system called Telepresence to chat with their friends over their high-definition televisions.

While this article notes the looming battle with Apple & Sony it should be considered that the set-top and video market is clearly of interest to Adobe, as well as the obvious similarities between Telepresence & Adobe Connect for video conferencing solutions. Cisco & Adobe are both invested in the Open Screen project but the relationship between the two will surely get closer whether or not some sort of acquisition is in play.

The strategy for Cisco, of course, is to encourage more high-bandwidth content running through all those Cisco routers that will need to be upgraded to keep up with the throughput. There’s no greater bandwidth hog than video and its just exploding with the boom in cheap consumer video hardware and turnkey hobbyist solutions. Adobe would be wise to pursue the authoring side of this hobbyist video boom, and focus on an aggressively marketed, cross-platform Premiere Express solution, as well as developing an ecology for video capture and publish from mobile devices. Meanwhile, their set-top Flash initiative will continue to intrigue Cisco and if the two are not already talking to make sure Cisco is using Flash everywhere possible, then somebody at Adobe needs to get busy and make it happen.

It remains to be seen whether Cisco might make an acquisition play or Adobe but it seems likely that the future of the two companies will be tightly coupled.

Enabling Rich Content Rendering for Dynamic Ambient Displays

Now more than ever, screens are competing for our attention in daily life. The distinctions between desktop PC, laptop PC, mobile, etc… are being replaced by the simple abundance and omnipresence of digital screens conveying information and mediating interaction. As LCD newsfeeds, airport terminal displays, set-top video box menus, billboards, advertisements, multi-touch screens, and augmented mobile devices, data and content are everywhere. All of it requires a lightweight and dynamic graphic layer in which to render content. Dynamic render plugins like Flash and Silverlight are very well engineered to address the requirements of these displays but it should be understood that much of the technology represents a shift away from the 2D ad/interactive paradigm primarily addressed by the Flash runtime.

Mobile augmented reality solutions require heads-up-display, alpha-channel rendering of text and graphics. Dynamic data visualization requires strong integration with back-end databases as well as messaging protocols like JSON, SOAP, and SMS. To enable rendering large datastreams in plugin runtimes like Flash & Silverlight, companies should aggressively pursue runtime adoption across mobile devices and smart phones, while defining prototypes for active camera overlays. They should optimize rendering and expand into all rich-content displays (set-top, automotive, kiosks, smart objects, and embedded systems built on the Android/Linux platform). They should optimize for dynamic rendering of large data streams, like the Nasdaq AIR application. The runtimes should be increasingly exposed to SEO and analytics retrieval so that secondary services and agents can easily be built on user workflows and data collection. And they should not neglect the 3D gaming market, but should consider how it can play a role in immersive worlds as a dynamic data layer (eg billboard adverts that can be remotely updated).

Competitive landscape: Adobe Flash (dominant market share, full turn-key solution), Microsoft (Silverlight remains the strongest competitor to Flash though it continues to play catch-up with our tech), Java (JavaFX just released but has little traction and is too late to the party), W3C (HTML5 has perhaps the largest share of hearts but is also the slowest to move), Google (might be cutting around Flash & Silverlight by pushing its interests into HTML5, Mozilla, and through Android & Chrome).

Related: Augmented reality solutions will require semantic architecture and image recognition
algorithms (identification, recognition, relation).

MMOGs and CCTVs: 3D Games For Spectators

Killer feature: Immersive gaming companies like Rockstar, Blizzard, ID, Epic, NCsoft, Valve et al should implement a way for game server admins to tap live in-scene camera feeds that can be bussed out to HTML. Liberty City should have public CCTV’s that viewers can watch from their browser or mobile client. Like a virtual Adam’s Block people could tune in to an alleyway or freeway underpass and let the action unfold in the streets below. World builders should enable first/third-person game cameras to send their viewport to Flash embeds.

Imagine tuning into a South Korean Quake deathmatch tournament as a spectator and being able to view the world through the game eyes of the champion or to switch across various fixed camera feeds in corridors, over walkways, and above central arenas to witness the gameplay from alternate angles. It might seem odd in sports-addled America and the UK but in Asia hundreds of millions are online and they love 3D gaming. The gameworld becomes a performance space. Second Life could finally entertain more than just the local cliques by broadcasting the actions of art collectives and protest groups out to the world.

It doesn’t take much effort to see how this multiplies the available media advertising real-estate considerably. Fixed camera views become hot advert property rented to savvy marketers who know they can reach both the local gamers and the viewing masses. Attention property goes meta. This is a two-way street as external feeds begin to pipe into the game worlds. The walls really start coming down when you can take a cellphone call in-game and then wave to the camera for your friends to see. They take a screenshot and then send it to the clan web site and the GTA Flickr feed that gets displayed on the side of a building in downtown Liberty City. Sponsored by Verizon of course.

Blizzard claims something like 10 million regular users of World of Warcraft. Their WoW wikia page is the largest collection of data on anything in the world. You think they might be interested in being able to view their clan members remotely and communicate with them at any time? Or tune into video feeds showing the night elves in Darnassus, or watch the pass at Chillwind’s Point? (Truth: I googled these.) And if Blizzard wins the fight to keep the gameworld free from advertising, marketers can accrete around the popular viewing channels and their Flash embeds.

These worlds are dark clouds, opaque from all but those that pay the playing fee. Yet there’s so much entertainment and interest and real human nature playing out in these worlds. People will watch. Agents will sift through the user data feeds streaming out along with the A/V feeds. And think of the up-sell as viewers convert to players. Someday I’m certain some of the reality tv channels on my mobile device will be looking into such immersive worlds and showing me where in my friends are and who they’re battling and what type of car they just jacked for a joyride through Liberty City. (I *really* want the GTA CCTV’s!)

General Guidelines for Cloud Tech

These are my brief (and very rough) notes from 5 minutes ago summarizing some guidelines I feel are critical for application & service development:

The cloud is everywhere.
Applications grab eyes.
Mobile/desktop/cloud – Don’t draw partitions.
Seek integrations across platforms.
Scale services by UI. Eg editing photos on a mobile is not appropriate but capturing images and uploading them to a workspace is.
Build communities.
Provide ubiquitous workspaces.
Communicate, Collaborate, Create, Share

2016 Metaverse Roadmap

I’m heartened to find the Metaverse Roadmap, sponsored by the Accelerating Studies Foundation. While I’ve been moaning about the shortcomings of immersive 3D technologies, they’ve been defining the template for progress. Much of their thoughts align with my own, painting an exciting future of convergence across modalities, devices, and workflows.

The emergence of a robust Metaverse will shape the development of many technological realms that presently appear non-Internet-related. In manufacturing, 3D environments offer ideal design spaces for rapid-prototyping and customized and decentralized production. In logistics and transportation, spatially-aware tags and real-time world modeling will bring new efficiencies, insights, and markets. In artificial intelligence, virtual worlds offer low-risk, transparent platforms for the development and testing of autonomous machine behaviors, many of which may be also used in the physical world. These are just a sampling of coming developments based on early stage Metaverse technologies.

In sum, for the best view of the changes ahead, we suggest thinking of the Metaverse not as virtual space but as the junction or nexus of our physical and virtual worlds.

Second Life Avatar Controlled By Thoughts of Paraplegic

I have a lot of issues with Second Life – mostly because I’m frustrated by their potential and their seeming inability to act on it – but it’s nevertheless an interesting sandbox to explore the greater frontiers of virtual immersion and social ontology. To this end, Japanese researchers have wired up a Second Life avatar to respond to the thoughts of a paraplegic.

…he wore headgear with three electrodes monitoring brain waves related to his hands and legs. Even though he cannot move his legs, he imagined that his character was walking.

He was then able to have a conversation with the other character using an attached microphone, said the researchers at Japan’s Keio University.

…”In the near future, they would be able to stroll through Second Life shopping malls with their brain waves… and click to make a purchase,” Ushiba said.

Convergence and Continuity Across Virtual Worlds

In games, immersive worlds, forums, social networks, and in blogs we inhabit multiple selves. In most cases, theses virtual spaces are walled islands with little relation between them. Increasingly it’s becoming apparent that continuity is necessary to resolve these fractured selves and to open up the channels of communication between the diversity of online containers. This can be seen in the new wave of web 2.0 aggregators like FriendFeed and Plaxo that aim to collate our myriad profiles, friends and content streams into a single portal. Now, Technology Review reports that several companies are working to enable avatars to move between virtual worlds.

More and more, such affordances will move into virtual spaces. 2D content streams and communication pipelines will feed into and across immersive worlds. A WoW player should be able to call up a HUD console in the game and locate their friends across all of the virtual worlds they’re currently in. They should then be able to communicate with them through IM or VoIP and subsequently transport to join them in another world. GTA4 has announced a feature to allow users to call each other in-world using the game cell phone. Shouldn’t this extend across game worlds and out into real-world mobiles? API’s could evolve to mine user communications (Twitter in WoW?) and chart locations on world maps. In the age of digital society, findability is key.

The vast amounts of personal profiling we’re building up around ourselves in MySpace, Facebook, blogs, and other forums should be accessible through our avatars and from all places we inhabit, virtually and in reality. It should be present in our devices and our profiles. As avatars, it should follow us like a digital skin (secured and opt-in, of course) layered in transaction-appropriate trust profiles that fly-out on mouseover. My avatars should contain more information than just polygons and scripted motions. Social transactions are information exchanges. My LinkedIn profile should be accesible to anyone in 2D and 3D if I so desire.

The realness of immersive worlds should leverage the fundamental reality of our digital profiles and interests. If these platforms are going to become truly compelling, they must work to integrate the API’s, content streams, and communication channels of the web2.0 revolution. We’re in the midst of a completely unprecedented historical shift as all of our cultural and intellectual content is going digital, made manifest in searchable, findable, and persistent datalogs. The profiles we create around our virtual selves are growing larger and larger, and they are being recorded and left open for many eyes to see. Imagine the political candidates running 10 or 15 years from now. So much of their lives will be a matter of public record easily searchable and graphed out to show affiliations, donations, histories and contradictions. So much of who they are will live online like a shadow. SO much of who we all are.

Virtual worlds are poised to engage directly in this shift and draw culture and identity into their domain. Instead of closed platforms, worlds like Second Life must open up and grow to become contiguous spaces whose character arises from the types of people that choose to gather there by affiliation, interest, and intention. MMORPG’s like WoW will continue to offer highly crafted narratives, specialized social groups and hierarchies, and bleeding edge rendering tech but will acknowledge the tremendous personal content within each player distributed across their digital and analog lives.

Of course, if virtual platforms become more open, their business models will inevitably shift towards advertising. Space is space, whether 2D, 3D, or 4D, and eyes are eyes especially when they gather in great enough concentration. As in the real world, the exchange of goods and services will always be of great value in any domain, so the shift towards continuity will be a shift towards reality. Virtual worlds have the unique proposition of creating fantasy within the world of life. So the shift towards reality in the context of a realized fantasy brings both closer together. It is part of the alchemical formula of bringing spirit into matter. It is the power of gods to create in an unlimited universe. It is the movement of the ghost in the machine as our real selves grow more and more to include virtual, digital, non-local aspects of identity and presence. Who am I but the sum of my transactions with the world? These words I’m writing and posting on the global billboard become preserved bits of my self. Your interactions with them extend my identity into the virtual world. All my words are facets of my expanding digital identity. My self-reflection extends from my body, my deeds, my actions towards others around me, to include the ideas and statements I leave online, the avatars I inhabit, and the webs of disembodied people I associate with. In 100 years I may roll up in bits under some social anthropologist’s data-mining PhD nudging their graphs this way or that with my Tweets and posts.

Aggregation of social data serves a very practical role of making it easier for us to manage an increasingly vast amount of data, but it also serves a larger role of helping us defragment our sense of self as it fractures out across so many new digital domains rising and falling daily. If we’re to walk like new gods through worlds both real and virtual, shouldn’t we do so with as much wholeness as possible? In a world that’s made it so challenging to have a fully integrated psyche it’s really imperative that we lay down a strong foundation of holism and continuity as we move into the unfettered vastness of the digital noosphere. As strong cohesive selves we can better wear the masks of avatars and wield the power of virtual gods.

A Little Virtual Spice Please

To briefly elaborate on an earlier post about Second Life… And specifically, ways in which I believe a modern 3d immersive world can leverage the new wave of cloud tech and create a truly compelling experience:

I want downtown billboards streaming Twitter feeds, rich dataviz, global network traffic, weather patterns, Flickr streams, and cycling media channels. I want to Dj from Traktor directly into a virtual club. I want interactive music and video remix tools that include the world as a substrate. I want to endow my avatar with metadata callouts, grouped in trust profiles, that display my affinities, affiliations, tag cloud, LinkedIn profile, sms number, twitter id, and credit accounts as appropriate to those I meet. I want to be free to re-purpose 3D assets from 3DSM, Maya, and Sketchup into my worldspace. I want a beautiful living homeworld that gathers the populace and inspires users and developers to create their own content elsewhere on distributed servers. I want to join friends on a virtual hilltop and watch the clouds drift past, watch the sun set, and the moons rise. I want to get lost in emergent behaviors, intelligent agents, and the beauty of physical dynamics. I want to easily find friends across multiple servers, across social nets, and out into mobile, gsm, and phone networks. I want an open-standard, opt-in, cloakable virtual ID that can be searched for and found across all dominant gaming and immersive networked worldspaces – and then when I find my friend I want to be able to join them wherever they are. I want peer-to-peer drop-boxes and back-channels that can address files to dominant industry and open-source applications, then back to in-world interfaces. I want an in-world, heads-up fly-out phone/sms/notepad/web-browser overlay that’s data synched to my mobile phone. I want to stumble into sinuous plotlines that sweep me away to distant parts of the virtual world. And yes, I want an SDK that allows EA to stick the Tony Hawk trick and physics model into a nice binary that can be purchased and installed into my client so I can skate around the place. And yes, I will try to grind your avatar if you have any linear edges sticking out.

I’m totally dreaming, I know. But dreams are what the future is built upon.

Parting Notes on ETech

This was a great conference and the most consistent collection of speakers and topics I’ve ever experienced. Very fun and inspiring. Lots of hip 30-somethings trying to dream up tomorrow and make it real. It was a a very balanced, yet cutting-edge talk aimed at an eager (and surprisingly mixed-gender)crowd. I noticed that most folks were using Mac laptops – this part of the edge seems to prefer Apple – and it was fascinating to watch many who were blogging the talks while pulling up references dropped by the speakers, tweeting out to Twitter, and snapping/downloading/posting photos in real-time. As speakers dropped references I was pulling them up on my laptop and dropping links into my blog notes.

In the lobby a team was showing off a data viz video mapping real-time communications connecting NYC to the rest of the world. Andrea noticed that a surprising number were with an Italian city called Perugia. Maybe next year they could map the live feed of all web traffic from ETech. Imagine the bitstreams rising off such a gathering of digiterati.

Maybe it was just the Sudafed coursing through our virus-ridden veins (thank you Portland) but ETech was a total intellectual turn-on, from ambient objects, Asian mobile media, green policy and sustainability, hardware hacking & drone building, Austrian post-Situationists, neuroengineering, and the digital salvation of Democracy itself.

I hope I can go back next year!

Synthetic Neurobiology: Towards Engineering Brain Circuits for Health and Human Augmentation (Ed Boyden) – ETech08

What are the mechanisms of normal and pathological human function? And how can we improve them? Bridging the gap of human behavior, social behavior, and molecular function. Can we augment emotion & cognitive functions? Why do we feel and act the way we do? HOw can we cure intractable disorders?

Controlling neural circuits. INvent new tools and understand how to use them. What is the abstraction layer we need to deal with? Need to understand entire neural circuits from input to output.

Augmentation: towards a cognitive augmentation tool box. Transcranial Magentic Stimulation. Modifying magnetic fields across the cortex, activating brain circuits. Approved to treat depression. OpenStim open source brain stim – wearable device, cheap, light. How to target deep noninvasive stim?

Targeted brain stim can precisely alter cognitive processing. Ex: increase/decrease risk-taking; trust/judgement; memory. Can we make a creativity prosthetic?

Cognitive behavioral therapy: cancel out a negative thought by behavioral/psychological strategies. Better than pharm. Easy to learn, difficult to excercise in the midst of depression. POss to use hypnotherapy to treat procedural anxiety. State of suggestibility, attention , and distance from feelingss.

Customized adaptive treatment engine: generate scripts to match anxiety and pain-release strategies to individuals. Can be used before and during surgical/invasive procedures to reduce anxiety.

Neuroeconomics: scan the brain during work tasks to identify areas of activity, derive consequences of brain lesions and other problems. Why does someone do what they do, buy what they buy, invest or not invest? Intervene to temporarily perturb specific low-level function in order to study it’s role in higher-level emergent behavior. [scary! how to better market to the lizard brain]

Brain disorders are a huge problem. Problems are sever, robbing happiness, self, identity. Great untapped need inspiring many business opportunities for treatment. 20th century the era of pharmaceuticals. Developments are slowing down, lots of side effects, nonspecific.

Optical neural control: precisely sculpt activity in specific regions and cell types. Thousands of cell types. Most severe disorders are loss of specific cell types. Can tune molecular promoters to express chemical modulator, using viral vectors (gene therapy). Adeno-associated virus seems very safe so far. Modify specific cells to respond to light. Can drive neural activity in modified cells using colored light. Can turn on with blue light, off with yellow. Looking at hardware light arrays (ex: modulate epilepsy neural circuit, Parkinsons). Shuffling neural code to offset epileptic spike trains.

Convergence Mobil in Tokyo

From a post at Electroplankton about high convergence functionality in Japanese mobile phones.

Claude is a 27 y.o. Japanese male… (His) typical day starts with him checking his email on his phone. He gets all his daily tasks and calendaring events this way. He then syncs it with his computer. He pays for the subway by placing the phone on a kiosk granting him access past the gates. The commute is spent watching TV on his phone by rotating the screen. A small antenna extends up and catches the wireless digital TV signals (something we will never have here in America). About 45 minutes later, he’s in Tokyo and heads to a vending machine to buy fresh fruit and water. He places the phone up against a pad. The vending machine reads his bank information which is tied into his phone. He then places his thumb on the phone’s tiny thumbprint reader to verify his identity. As he makes his way to the office, he waves the phone near the door handle to unlock it. During a 10 minute break, he’s flips thru a magazine and sees something he wants to buy. The item has a tiny stamp size barcode pictogram next to it. He scans the pictogram with his phone. A receipt and shipping confirmation hits his email minutes later. As the day ends, he syncs with his work computer and goes grocery shopping paying for items with his phone. Before heading home, he heads to a bar his friend has invited him too. He uses the phone to give him step-by-step directions. The day is finally over and his phone’s battery is nearing the end of its life. He plugs it in and goes about the rest of the evening relaxing before bed.

Ribbit Wires Phones To Apps, Web Pages

Very cool tech from Ribbit uses Flash/Flex to emulate telephony in web pages and software. This enables direct 2-way communication between widgets and mobiles. The lines between software and webware continue to blur, as are the distinctions between mobile and desktop.

Ribbit software hooks up standard phone services to the Web. Users simply forward their mobile phone numbers to Ribbit, which delivers the calls back to personalized Amphibian Web pages that offer a series of unified communications features.

…As a demonstration of the power of Ribbit, one independent developer using new Adobe AIR software has built a full-featured version of Apple’s iPhone that works on Web pages.