Outliers & Complexity

Or, The Risk of Extrapolating Linear Trends Against Non-Linear Systems.

A common habit in forecasting, particularly in energy futures & economic growth, is to take roughly linear trends and extend them over the next few decades. The notion is that there is inertia in what has already happened that will make the future look markedly similar, or at least there will likely be a more-or-less linear movement along an existing path. For example, many forecasts suggest that energy consumption will increase by 50% towards the year 2035. This is based on data over the past 30 years that is then extrapolated forward along expectations, so you get graphs that look like this one from the EIA’s 2009 Annual Energy Outlook Early Release Overview:

The graph shows mostly linear growth in energy consumption. The assumptions here are that, given previous growth rates, and given a rough set of expectations about future growth, energy consumption will steadily grow across all sectors. Yet you’ll notice a few bumps & dips for transportation & industrial in the later months of 2008 and early 2009. These suggest outlier events. Outliers are the unexpected events, the Black Swans that come out of nowhere and blow expectations out of the water. In this case, economic activity got a big boost by the inflated gains of the securities market, then took a dive after all the hidden risks came to the surface. The following graph from the same EIA report highlights the 2008 economic black swan:

Here we see the market prices for the primary energy sources. This graph really shows the instability churned up by the securities outlier. As the ultimate determinant of just about all economic activity (nothing happens without energy) we can see energy prices climbing at the same time demand was ramping up (compare to the last graph of consumption). Then heading into the crash energy prices plummet as fears mount, workforces are downsized, factories go dark, and productivity retracts in the face of economic doom. In spite of expectations the market collapse came as a surprise. Yet, forecasts still commit global energy consumption to a future of roughly 50% growth in demand (see those post-2010 consumption lines in the first graph?). In spite of obvious turbulence in past performance the forecasts assume typical, linear economic growth out to 2035.

While such linear approximations offer hope of anticipating and, hence, preparing for the future, to some degree they represent a logical fallacy of projecting linear trends onto complex, non-linear systems. Living systems like weather patterns, anthills, and global economics are approximately non-deterministic. That is, they’re so complex and have so many feedback mechanisms that they’re mostly unpredictable (weather predictions are still only more-or-less valid for about 5 days out). Much of this complexity arises from the turbulence generated by feedback loops and interconnections across every scale of the system. The power laws underlying dynamic systems take small values and iterate them over time into very large values. This is the mechanism underlying the oft-mentioned Butterfly Effect and one of the drivers for outlier events. Imagine a dust devil spinning up on an otherwise calm desert floor.

Nature seeks homeostasis – a dynamic equilibrium around a point of stability. The counterpoint to runaway feedback loops and suddenly emergent outliers are the damping effects of control elements. In climate, the tendency for hot & cold to equalize will usually mitigate a storm and return clear skies. The dust devil gives up it’s angular momentum to shifting pressure & temperature gradients. Looking at our current affairs we see that total economic collapse has (so far) been averted through aggressive attempts to dampen the turbulence by injecting massive amounts of state capital into the financial system. These interventions & market regulations are control structures put in place to govern for relative economic homeostasis. When they work and things are relatively quiet, they keep those trend projections nice & linear.

Linear projections help us continue to get things done based on fairly reliable expectations. But avoiding the next economic catastrophe requires a deep study of the many threads & amplifiers that drive black swan events. Outliers occupy the thin edge of statistical possibility yet almost always have tremendous consequences. They are, by nature, entropic & disruptive, shifting the territory and demanding new adaptations. To return to the global energy domain, what outliers might be slowly iterating to challenge the forecasts of 50% growth in demand? What catastrophic black swans might be lurking off the radar? What scientific breakthroughs and game-changing innovations might be weaving together towards a complete re-orientation of power requirements, transport, or industrial fuel?

The mobile phone is a great example of a high-impact outlier with a small physical footprint that achieved global ubiquity within 10 years, shredding the linear projections of numerous industries. The pace & breadth of it’s adoption suggests that interventionary technologies can rather quickly have major impacts, challenging heavily invested and entrenched businesses. Imagine an energy outlier with a similar device profile that enabled people to generate & store enough power to run a small home or drive an electric car 100 miles.

Studying a system for outliers and looking for the signals & trends that might lead to the next Black Swan, as well as examining the conditions that have led to previous outlier events, can inform forecasts that are much more attuned to resiliency and adaptation.

10 comments

  1. Pingback: The Risk of Extrapolating Linear Trends Against Non-Linear Systems « •||•• Edge of Tomorrow ••|||•••
  2. Renée A.D.

    “While such linear approximations offer hope of anticipating and, hence, preparing for the future, to some degree they represent a logical fallacy of projecting linear trends onto complex, non-linear systems”

    Absolutely! Great post, thanks for pointing things out. People are increasingly relying on models that are oversimplified. Bad news.

  3. chris arkenberg

    Thanks, Renee! I’d argue we’ve been relying on oversimplified models for millenia but it seems that we’re starting to get better at addressing (and modeling) more complex systems. And we don’t really have a choice given the urgency with which our systems appear to be straining and buckling while we face up to the consequences of our myopia. We built industrialism on newton & descartes with little awareness of non-local effects. But, for better or worse, systems talk back as we see now with climate, economy, food systems, and governance all starting to shake up a bit. Gradually we’ll instrument these systems more and more and feed the data through superclusters that weave it all into multiscalar simulations. For example, check out the work over at ESRI to link up earth system models.
    http://www.esrl.noaa.gov/news/quarterly/winter2009/noaa_environmental_software_infrastructure_and_interoperability.html

  4. John Merryman

    Chris,

    I think you can put Einstein at the top of that list of those endorsing a myopic linearity, with the idea of four dimensional spacetime.

    Time as a dimension extending from the past into the future is really just the narrative construct. The reality is a changing energy configuration which creates events and replaces them. For example, the earth doesn’t travel the fourth dimension from yesterday to tomorrow. Tomorrow becomes yesterday because the earth rotates. Time is a consequence of motion, not the basis for it.

    Now yes, you can use the speed of light to say it takes a certain amount of time to cross a certain distance and the faster you go, the slower the rate of change and this relation is mathematically precise, but the same argument could be applied to the relationship between temperature and volume. Given that a certain amount of energy in a certain volume will be a certain temperature and if the volume is changed, the temperature will change proportionally, so it could be argued that temperature is another parameter of volume.

    The issue of the direction of time is fundamental to our conception of reality. When we view it as the series of events, it is the individual moving relative to their context, but if we view it as the events coalescing out of future probabilities and receding into past circumstance, then the individual is integral to the events.

    Now obviously our entire concept of human history and narrative, at least the western version, is based on the linear concept of time, in which the present is a point moving from past events to future ones, much like ones reads a book, but then for much of human history, we viewed the earth as the center of the universe and the sun revolving around that and this model is the basis of mechanical clocks, which evolved from sun dials and so the hands represent the motion of the sun, but it is actually the opposite. The earth rotates west to east, while the sun is stable, just as the hands of the clock represent the present and it is actually the units of time going future to past/counterclockwise.

    Then we get to the idea of space being represented by three dimensions, which are in reality just the coordinate system of the centerpoint and are no more the basis of space than longitude, latitude and altitude are the basis of ones location on earth, but are simply an extremely reductionistic model of it.

    Not a popular argument in physics discussions, but I find it makes sense to people for whom spacetime is not the heart of the Holy Canon.

  5. chris arkenberg

    Thanks, John. I like what you said here:

    “…if we view it as the events coalescing out of future probabilities and receding into past circumstance, then the individual is integral to the events.”

    I’m wondering if complex systems are non-reversible… and if this, as you suggest, in some way challenges the arrow of time (or does it reinforce the forward continuum?).

  6. John Merryman

    Chris,

    I’m almost thinking of time as more of a scalar, emergent effect, where the past/present is constantly peeling away from the present/future in much more of a non-linear, cumulative effect, with different sectors moving at different speeds. The linear concept of time is focused, reduced to one clock, but reality consists of many clocks, operating at different speeds. A good example of our tendency to reduce the many to the one is that months are based on the cycles of the moon and have no particular alignment with years, but civilization has converted them into sections of the year, that have only a passing relationship to lunar cycles.

    This is just how our minds function. We take enormous amounts of information in and reduce it to digestible, serial snapshots, called thoughts. This freeze faming of reality creates the impression that reality is a series of digitized instants, but the mental alternative would be total white noise of information overload. (Of course, some of the narratives constructed are fantastical, bizarre and delusional, but that’s just my perception of what others must be thinking.)
    We, as mobile organisms, move in a subjective linear direction and our brains are navigation organs functioned with preserving our singular selves and larger communal bodies. What if we were not mobile? Would we even need the focal perception of a brain?How do plants interrelate with their environment? Where we relate mechanically, they relate chemically. The thermal function controls the temporal function. We are the perception of cause and effect. They are cause and effect. We make distinctions. They are connections. Both a tree and a lightning bolt are a connection between the sky and the earth.

    So in this sense, you might say I see the arrow of time as a construct, those flashes of thought as we move about. Reversing it, time travel, etc. isn’t a physical reality because time is not some meta-dimension along which events exist and if we were to find a way to fold it, could travel it. Rather it is the physical reality transforming from one event to the next, so that neither past or future exist, as the energy to manifest them is currently manifesting the present. The reality isn’t the present traveling from past to future, but what exists creating and replacing those events which go future to past.

    Actually there is a lot of precedent in eastern thought, which is more contextual and it is said they view the past as in front, since it is visible and the future as behind, since it is not. Also we are only seeing information of past events, whether it’s across the room, or across the universe. They also are not monotheistic, which is an idealization of the individual, but tend towards ancestor worship, which is a reverence for the past.

    This is an essay I wrote last summer, trying to tie divergent thoughts into a larger whole:

    http://www.exterminatingangel.com/index.php?option=com_content&task=view&id=612&Itemid=479

  7. chris arkenberg

    Fantastic response, John. Really. Thanks a bunch for unpacking your thoughts here. Definitely made it much more clear to me what you’re proposing. I especially like these passages:

    “The linear concept of time is focused, reduced to one clock, but reality consists of many clocks, operating at different speeds.”

    “Time… is the physical reality transforming from one event to the next, so that neither past or future exist…”

    “How do plants interrelate with their environment? Where we relate mechanically, they relate chemically…We are the perception of cause and effect. They are cause and effect. We make distinctions. They are connections.”

    “The reality isn’t the present traveling from past to future, but what exists creating and replacing those events which go future to past.”

    Though, for our sake, I’d say it’s both the present traveling into the past and the future unfolding from the present.
    BTW, would you be interested in letting me feature this comment as a post here? With full attributions, of course.

  8. John Merryman

    Chris,

    It does go both ways. Relativity, you might say. That’s how I originally intuited it, but in unwrapping it further, I see one as the physical reality and one as the perception. Much like we perceive the sun moving across the sky from east to west, but the underlaying physical reality is of the earth rotating west to east. Although relativistically they are moving about the same center of gravity, it just happens to be located near the center of the sun.
    I don’t know why that link didn’t work for you. I clicked on it and it loaded. Damn pipes. I tried their archives, but it didn’t come up that way. I’ll try posting it email.

  9. John Merryman

    for some reason the pop up doesn’t have a send button and when i find where it is, says sorry, wrong captcha, but there was no captcha. So I sent it regular email, but I suspect it might end up in your trash, if it goes through, so might look for it there.

  10. John Merryman

    and yes, no problem with posting it, but be forewarned, it’s not a popular observation among physicists.
    Here is recent example;

    http://www.fqxi.org/community/forum/topic/618

    The discussion about time starts Feb 25, 3:51.

    It takes a bit of unpacking. Tom is a good example of someone who has put a great deal of effort into mastering the box and isn’t about to think outside it.

Post a comment

You may use the following HTML:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>