Clay Shirky, previously featured here for his book Here Comes Everybody, has provoked a lot of interest through another proposition. Adapted from a conference talk related to Web 2.0, Shirky knit together a surprising combination of elements identified in the title:

Gin, Television, and Social Surplus

His first contention is that television sitcoms served the same sociological midwifery role during the American post-WWII ‘leisure age’ as gin during the Industrial Revolution. First it might help to understand just what gin had to do with anything. (more…)

Albums added in April: 10,769

I hope you got your Rolling Stones fix last month, since they’ve now been stripped out of the catalog for “events outside of our control” according to eMusic. This reversal is part of a larger issue affecting eMusic’s catalogue, which invisibly loses some portion of its releases even as it publicly reveals all of new additions. Just among the artists I note below, at least one is a repost of an album by Future Sound of London that had previously been offered and then removed at some point, plus all of Ladytron’s albums apart from the new “Ghosts” single have likewise vanished. The unfortunate lesson is that you can’t assume that your “Save for Later” selections will actually still be there when you have the downloads to spare.

(more…)

Apropos for the previous post (on Darwinian adaptation among malware), the article itself attracted one of those keyword-matching comments from an apparent spamblog (somewhat different from straightforward splogs). I had not previously heard of these before operating this blog in other than stealth mode, so here’s how I infer they function just by observation:

1. A new post is scanned, either via its feed or one of the aggregator services like Technorati, looking for certain keywords.
2. A corresponding post is created on the spamblog with a generic blurb like “[author] had an interesting post about [keyword]” and a short 1- or 2-line excerpt centering on the keyword match.
3. A comment is submitted to the originating blog, linking back to the spamblog.
4. The spamblog post is then able to attract traffic either through clickthroughs from the comments thread, or from increased PageRank from Google since their blog gradually increases its network of keyword-linked sites.

The ultimate purpose is still simply to gain visitors which in turn trigger ad revenue through a combination of Google text ads, banner ads, and other pay-to-host content. The spamblog itself is often a default template, e.g. the Kubrick WordPress theme, consisting only of these short linked posts. For blogs that either don’t moderate comments or who don’t scrutinize excerpting sites individually, growth is mostly automatic. The adaptation is that they propagate links without the prior telltale markers of comment spam like overt sales messages included in the actual comment text.

So far I’ve seen these keyword comments triggered by an unusual set of terms: ‘elevator operator,’ ‘turquoise jewelry,’ a ’sequel to 5 People You Meet in Heaven,’ ‘Apple,’ ‘zebrafish,’ and ‘plumbing license.’ As an exercise for the reader, I leave it to you to guess which original posts generated each of those matches (hint: keywords don’t have to be sequential). I’m also curious whether having listed those now all together, I will get a repeat entry of all prior spam comment attempts.

This brings to mind what I am sure has already been codified into the equivalent of Sturgeon’s Law, which would go something like: “Any sufficiently popular mainstream communications system will generate spam” or perhaps the more prescriptive, “A communication system can be considered mainstream once it attracts spam.” Spam is generally considered to have originated with electronic systems like e-mail and Usenet forums, but extending the definition backwards, one could potentially designate parallels like telemarketing and robocalls for telephones and junk mail for postal service as examples. Did telegraph operators ever suffer from unsolicited commercial Morse Code transmissions? Certainly spam has gained tremendous genetic diversity in jumping to every emerging communication form—chat spam (first IRC then IM), forum spam (first newsgroups then web), mobile phone spam via SMS, online games, search engines (aka spamdexing), blog spam, and even video-sharing sites like YouTube. Twitter? Check.

Part of the original blame can be placed on the idealism of academic groups like the IETF who established standards for communication protocols like SMTP and NNTP without incorporating more robust authentication and authorization to deter spoofing and other common tactics. Except, of course, that those standards were created long before the very notion of a commercial Internet had been considered, and the online community was small enough to police itself by etiquette alone. Certainly we could assert that newer protocols should learn the lessons of the past and instill greater protection against potential abuses, right? Except, instead, the rapid evolution of spam in response to antispam efforts has created ’superbugs’ and an extensive evolutionary toolbox of techniques that can thwart most any systemic precaution. Just like our immune system and pharmacology have developed to deal with ever more sophisticated organic threats, inspiring ever stronger virii and bacteria, so the race continues between platform developers and those who would distribute spam over them. It is effectively now almost impossible to create a communications system that is actually usable, capable of reaching mainstream acceptance, and totally immune to spam-like behavior. Instead, like the common cold, we now aim instead to reach a détente where we can take steps to prevent infection and minimize symptoms, but no longer envision a ‘cure for spam.’

A fascinating microcosm of technical Darwinism is the ceaseless escalation of sophistication between malware authors and anti-virus vendors. Formerly solo practitioners acting out of bravado or malcontent, malware developers are increasingly dedicated professionals bankrolled by organized crime syndicates or even governments in areas like Eastern Europe and Asia. With the huge financial incentive in identity fraud, online theft, and electronic blackmail, black hat hackers aim to exploit the twin vectors of technical vulnerability and human laziness. Anti-virus firms meanwhile have developed a huge market base by playing a largely defensive game against new attack types, constantly scouring the underground community for new examples of attack vectors and building massive databases of ’signatures’ or ‘fingerprints’ of specific variants. Yet just like organic mutation, each new form of defense is the inspiration for a variety of alternatives that seek to bypass the Maginot lines of AV software.

Botnet agent plays lost sheep to avoid detection | The Register

The Register reports on one such development in the ongoing mutation of the strain ‘Kraken’ (AKA Bobax) bot. Earlier botnets were susceptible to interruption by attacking not the zombie clients but, in military parlance, going after their C&C (command and control) elements that issue the bot its orders—attack a certain host via distributed denial-of-service (DDOS), for example. These were often IRC servers that allowed pseudonymous communication via protected messages. Yet IRC is, with the advent of IM and Web-based chat, now itself something of an anomaly that an AV program could view as suspicious.

The Kraken adaptation adopts dynamic DNS through generated domain names, passing encrypted commands through HTTP further obfuscated with bogus headers to fool SPI-capable firewalls. In this fashion, the bot homes in on the current location of its control server without having any hardcoded lists that can be used to target them for shutdown. The Australian firm PC Tools that analyzed the new code compared the process to the way a lost sheep tries to locate its shepherd (hence the article’s title). Similarly, new variations of P2P software have attempted to replace the fragile centralized tracker with distributed databases and multi-hop obfuscation through efforts like TOR and I2P.

Kraken also employs a random word generator to vary its infection host filenames, which spread through IM networks like MSN Messenger. This is a tactic previously adopted by spammers, who in turn were responding to naive Bayesian filtering built into antispam engines like SpamAssassin.

A parallel story at The Register notes a new kind of SQL injection attacks that targets DATE and NUMBER fields. Previously SQL injections exploited unchecked parameters or syntactic tricks to pass SQL code, often limited to text-based fields like VARCHAR. The recent nihaorr* mass attack on older ASP-based sites, for example, used a combination of techniques: the injection was appended as a POST in place of a standard GET query, overloading the request with a 4000-character hex string set within a CAST function. Decoding the hex to text revealed a procedural cursor that trawled the sysobjects DB for any char-based columns, to which it then proceeded to append rogue JS code via UPDATEs. Since most remedies for SQL injection have centered around validating text-based input, this variation bypasses such defenses by manipulating date and number data routines in Oracle’s PL/SQL.

Not content to be left on the defensive, particularly with suspicions that certain governments could be building up their own stockpiles of zombie PCs to act as a botnet in the effect of a (sigh) cyberwar, the US Air Force Cyber Command (AFCYBER) has recently published their consideration of mustering a military botnet. Ars Technica reviews the salient points of Col. Charles Williamson’s proposal, including the need for offensive capability (essentially to attack the attacker) in light of the indefensibility of our present infrastructure, and the potential political fallout were we to, say, pingflood France due to a DDOS mounted from a botnet there controlled by a rogue group elsewhere. The implications for this are somewhat provocative—even if AFCYBER were to build a managed botnet out of decommissioned military PCs, would some other branch like the NSA or CIA also receive a secret mandate to develop ‘offsite assets’ by infecting civilian PCs in other countries? Will some portion of future conflicts consist of shadowy agents provocateur wielding heavily anonymized zombie PC armies trying to provoke retaliation against the enemy’s allies by launching DDOS attacks from within their civilian networks?

In yet another Daring Fireball-inspired tract, Charlie Wood asks why Apple appears to escape the "innovator’s dilemma" presented in Clayton Christensen’s eponymous work. The idea is a follow-on from Christensen’s earlier depiction of ‘disruptive technology-cum-innovations’ and how they evolve within a market (similar to Kuhn’s structure of scientific revolutions). Once the disruptive paradigm has established a beachhead, its progenitor often overdoes its development and in turn loses out to second-tier players that leapfrog it by adopting the innovation at a ‘good enough’ level, undercutting the market leader. Yet Apple, at least in its current incarnation, seems to avoid that pitfall.

Moonwatcher: Why Doesn’t Apple Face The Innovator’s Dilemma?

Wood argues that this is in due to Apple distinguishing itself by design, which appeals to taste and is harder to usurp than a typical feature matrix. This certainly helps explain why, for example, the iPod has utterly eclipsed any imitators (which add features at the expense of usability) and why the iPhone was able to dominate mindshare so quickly in an established smartphone segment (which has always buried functionality behind clunky interfaces).

However, another straightforward business answer is that Apple acts to undercut itself rather than leaving that to a competitor. For example, when the iPod Mini was the bestselling flash-based music player, they discontinued it and introduced the Nano, which reconsidered the Mini in both design and features rather than just making minor changes. With the iPhone less than a year old, already intense speculation mounts about a likely successor with enhancements like 3G wireless, effectively hamstringing competition which may already have 3G-capable handsets. To re-iterate: even the rumor of a future iPhone feature is somehow perceived (at least in the breathless press coverage) as superior to other brands already in the field.

Perhaps even more apt than Wood’s own rationalization is the observation made in a comment by Martin Pilkington that:

"the problem with most companies once they become larger is

a) they become more bureaucratic
b) everyone starts to protect their own territory
c) marketing takes over or they ignore marketing"

This you may recognize as a business-specific case of systemantics, where the business effectively ends up at war with itself in unconscious internecine competition for resources. He also adds the specific point—which has also been espoused by Steve Jobs in interviews about the Apple design philosophy—that they do not add features to products merely to reach feature parity in reaction to competition, or through typical focus group artificiality, but through something more akin to user cases. That is, they imagine how people want to use a device, and then build a feature to make that possible in as intuitive a way as possible. As far back as 1998 (when their resurgent success was much less assured), Jobs told BusinessWeek that:

"It’s really hard to design products by focus groups. A lot of times, people don’t know what they want until you show it to them."

Is there a better shorthand for product innovation than ‘thinking of what people want before they knew they wanted it’? 

Wrapping up today’s trifecta of psychological judo, Brad Bird discusses lessons on encouraging innovation he applied at Pixar, as well as what institutional enablers the company offers:

Pixar’s Brad Bird on Fostering Innovation

GigaOM extracts his interview with the McKinsey Quarterly into 9 lessons. While all make for interesting insights into creative teambuilding, perhaps the most universal is morale as multiplier:

“If you have low morale, for every $1 you spend, you get about 25 cents of value. If you have high morale, for every $1 you spend, you get about $3 of value.”

Although occasionally given mention, it’s still uncommon to hear morale recognized so starkly as a driver of value. Compare, for example, the divergent way in which Wal-Mart and CostCo treat their employees. ‘Cost’ as measured strictly on the balance sheet does not factor in lost productivity due to malaise or innate rebellion resulting from poor employee morale.

The availability of interdisciplinary learning via Pixar University also offers an antidote to the two previous articles’ recognition of skill calcification. One interesting aside about the company offering Krav Maga as a class alongside storytelling and improvisation is that, in contrast to most other fighting styles, KM is built much more around principles instead of techniques. Students are trained for real-world contingencies, and great emphasis is placed on conditioning the student to react instictively against an attack and escape versus get caught in a traditional ‘battle’ as found in other styles. Thus, Krav Maga could be seen as much as psychological adaptation as physical defense.

Finally, the influence of Steve Jobs is evident in the overall layout of the campus, such as a central atrium to maximize crossover contact between functional teams as they visit the cafeteria or even the bathrooms. One other example of cross-disciplinary inspiration comes from today’s Fortune story “Apple and Eve” about the role of chief Apple designer Jonathan Ive in affirming the character of Eve from the upcoming Pixar film Wall-E. What caught my attention more than even the premise of a character based on Apple projected into the 28th century is the limits placed on Ive’s involvement:

“Apple is so proprietary and so secretive that he couldn’t even really allude to where the future of technology was going,” says Stanton. “The most he could do is nod his head to the things we said we wanted to do.”

Whether this reticence was at Ive’s own initiative or reinforced by the looming ire of Jobs and lawyers et al., it reminds me of how straitjacketed corporate culture can become—’corporate’ here referring to almost any size company whose investments of intellectual property and shareholder value demand these precautions of silence and measured response. Even the tiny startup Epiphyte in Neal Stephenson’s Cryptonomicon has to employ elaborate security to protect their corporate interests.

Another variation of the ‘cognitive trap,’ David Weiss explores the inverse relation between confidence and knowledge—again through the lens of software development, sort of the zebrafish of organization psychology.

David Weiss: Metacognitive Miscalibration

He goes on to characterize several cases of the miscalibration of confidence and thinking. “Wicked Problems” could be considered as similar to those physics problems you first struggled to complete in high school, which helpfully neglected messy factors like air resistance at the expense of effective accuracy. As you add in all of the variables required by the actual underlying complexity, the problem eventually collapses. The “Desire to Learn” dovetails with the previously linked Raganwald predicament, where a sense of sufficient knowledge forestalls efforts to deepen understanding. “Personal Pride” evokes the admonition common in entrepreneurship and venture capital to “fail quickly” and not let fear of failure paralyze you.

Finally, the “Well Intended Deception” describes a situation more specific to software programming where levels of abstraction can hide deeper problems—sometimes an emphasis on simplicity through inheriting framework code results in an offset in opacity. The upfront ease of using pre-made tools and resources can be undone by the lack of transparency into what’s really going on when you need to dig into the details. This is a balancing act, as it’s often more efficient to build on proven platforms and add just distinguishing features as custom effort. Thus a new product like Pixelmator, even with only one designer and one developer, can usurp some of the mighty Adobe Photoshop’s turf by building on established open-source and Apple-provided APIs. The parallel in videogame development is whether to build one’s own engine, a la Id Software, or license a middleware product like the Unreal Engine. The former gains full optimization and customization options at the cost of major additional engineering effort; the latter can focus on just a specific game’s logic rather than the underlying plumbing, but remain fundamentally limited by the licensed engine’s capabilities.

This ‘buy or build’ decision ripples through most manufacturing processes. Despite its long history of ‘Not Invented Here‘ myopia, Apple has lately shown a great deal of maturity in this area, with the move to Intel processors freeing up engineering to focus on new products like the iPhone and Apple TV; yet their recent purchase of P.A. Semi also shows that they intend to maintain a toehold at the lower levels of chip design.

One of the small joys of introspection is identifying those cognitive traps that restrict our growth. This ego-spelunking process has featured prominently in Western philosophy within the role of the skeptic (e.g. Descartes); in Eastern disciplines such as Buddhism; and in various self-help tomes that provide a mash-up of both (e.g. Dan Millman’s Way of the Peaceful Warrior). In laying out the idiosyncratic ‘metaphysics of Quality’ in his Zen and the Art of Motorcycle Maintenance, Robert Pirsig proposes a related concept called the ‘gumption trap’ which siphons off your enthusiasm when you encounter “affective, cognitive and psychomotor blocks” in performing a task.

Reginald Braithwaite discusses what might be characterized as one of these self-limiting habits:

Why we are the biggest obstacles to our own growth

This begins by riffing on an observation made by Daring Fireball’s John Gruber in analyzing the import of the mainstream success of Apple among youth, which has now filtered down from the obvious case to financial analysis. Braithwaite carries the idea in a more professional direction, namely that what you know often interferes with your acceptance of something you don’t. Building up a degree of expertise in any discipline means you have potentially more to lose - in comfort and initial efficiency at least - when switching to a novel alternative. Braithwaite’s examples are specific to software programming (Lisp vs Factor, regular expressions in Ruby) but the principle can be applied more broadly. The effects can be seen in resistance to new technologies or methodologies, resulting in foot-dragging up through overt sabotage reminescent of Luddites.

The solution at the individual level is to learn how to stretch, just as you do to extend physical reach. This can mean challenging long-accepted notions on technique that may no longer be the pinnacle solutions they once were, particularly when crossing disciplines—keeping with the software design idea, , for example, when switching from procedural to object-oriented programming. At the group level, extra attention should be paid to transitional aids and training to help lower the resistance borne out of knee-jerk anxiety triggered by a perceived threat to one’s identity.

[EB: Luddite]