Atmospheric Rivers

Here comes the rain!

If you find that image a bit hard to view, all you need to know is that my house is at the right end of that long yellow line.

Two Years for Randall Munroe

Here's one of the most touching and beautiful comics you could see: Two Years.

Congratulations and best wishes to Randall and partner!

WDW vs DL

It's one of those eternal debates: DisneyLand? Or Walt Disney World?

Myself, I come down strongly on the DisneyLand side, but that betrays my heritage, growing up in on the edge of Orange County in the 1970's.

Anyway, don't listen to me: go read FoxxFur's superb photo-essay on the subject: The Awkward Transitions of Disneyland!

As you may have noticed by now, rails, fences, planters, and flowerboxes do an unusual amount of heavy lifting of the transition-spots of Disneyland. This may because these crowd control barriers generally are felt more than carefully studied: we interact with them and they set a tone without really being noticed in specific detail. Disneyland has a huge and astonishing variety of simply beautiful railings.

As a child, I recall spending an absurd amount of time worrying about questions such as: is the Swiss Family Robinson Treehouse in Adventureland? in Frontierland? or in New Orleans Square? As FoxxFur observes, this is no trivial question:

Nearby, these rails, part of New Orleans Square but also part of the Riverbelle Terrace visual experience - ie, belonging to both Frontierland and New Orleans Square but being actually inside Adventureland - abruptly terminate at the base of a tree

It's a wonderful essay, with lots to chew on. Don't miss the section where she discusses how careful foliage (non-)trimming can control the amount of light that reaches certain areas, governing the overall mood and atmosphere and sub-consciously giving you clues that you're leaving one "land" and entering the next.

On being a developer

Matt Sencenbaugh's terrific essay is getting some well-deserved attention: On Being a Junior Developer.

But the title is misleading: this isn't just a great regimen for a junior developer; it's a superb list of goals for a master developer as well. Here's Sencenbaugh's list:

1. Read other people’s code
2. Plan things out
3. Have an opinion
4. Ask questions
5. Explore new technologies
6. Embrace unit testing
7. Refactor

The essay is tightly-written, clear, and filled with choice nuggets such as:

Ask questions about peculiar things, there is often wisdom and learning to be had.

I wouldn't remove a thing; all his goals are admirable and wise.

My only addition would be:

1. Communicate. Developers find it easy to work with computers, challenging to work with other developers, and nearly-impossible to communicate with non-developers. You must constantly strive to improve your communication skills and overcome your natural desire to spend all day inside your cube, emitting nothing but code. Practice writing, speaking, and listening constantly.

Sencenbaugh, of course, clearly has the communication thing down pat!

The fourth bore

As the justifiably famous Caldecott Tunnel nears completion of its latest expansion, take this nice tour of the geology of the project from Oakland Geology's Andrew Alden: Rocks of the Caldecott Tunnel

It’s pretty crummy rock, and the tunneling was quite slow there. Every couple of meters, the tunnelers drilled a fan of holes over the roof of the upcoming dig segment, each one reinforced and grouted. That reinforcement allowed the tunnel roof to stay up long enough to spray it with shotcrete, which in turn held up the roof until the real concrete tunnel lining could be emplaced. The tunnel’s final shaft is round, as strong as an eggshell in its resistance to the earth’s weight. (To see what I mean, try gripping a raw egg firmly in one hand, fingers on all sides, and attempt to crush it. You can’t do it.)

Tis the season...

We have now officially entered the Catalog Season, that annual season of mail-order catalogs arriving in your mailbox.

As you sit down with your mulled wine or egg-nog for a satisfying catalog read, let Drew Magary be your guid: The Hater’s Guide To The Williams-Sonoma Catalog

While certain retailers like Hammacher Schlemmer are almost intentionally ludicrous ("Buy this personal hovercraft for $80,000!"), there's no wink to be found in a Williams-Sonoma catalog. The people at W-S aren't the least bit self-conscious about getting you to pay $35 for mailed gravy. So I thought I would go through this holiday season's catalog, which has spent a solid week atop my shitter, and point out some of the more ridiculous items. Because there are people out there who buy this shit. The question is ... who? And why? Let's try to figure that out now.

Warning: NSFW (language).

But extremely funny.

Zlatan's goal

The New York Times considers the question: Ibrahimovic’s Goal: Best Ever?

The reaction was immediate. Steven Gerrard, the captain of the English team, after the game called it the best goal he had witnessed. Twitter rattled with wonderment. Office computer screens toggled toward YouTube replays. All were astonished.

So where does this one rank among the best?

Great videos!

The Maradona goal is justifiably worthy of its title.

I love the video of Neymar, the young Brazilian. It's just remarkable how much faster he is than anyone else on the field.

Thanksgiving edition of stuff I'm reading

Here's some things I found interesting; you might find them interesting too!

• Want a petroglyph of your own? Apparently, some people want them so badly they'll go and steal them: Petroglyph thefts near Bishop stun federal authorities, Paiutes

  The theft required extraordinary effort: Ladders, electric generators and power saws had to be driven into the remote and arid high desert site near Bishop. Thieves gouged holes in the rock and sheared off slabs that were up to 15 feet above ground and 2 feet high and wide.

• Lots of fascinating stuff to read here: Winners Named in 2012 AAAS Kavli Science Journalism Award Competition . Among the stories:

  Nijhuis donned a protective suit and went underground to observe both bats and biologists as she reported on white-nose syndrome, a fast-moving fungal disease that has killed more than a million cave-dwelling bats in the northeastern United States and is threatening to spread across the continent. The judges noted the scope of the Nijhuis story, which provided an in-depth look at an issue that has been emerging since 2007 when the disease was first discovered in bats behaving oddly in upstate New York.

  And:

  On the Olympic Peninsula in the state of Washington, the largest dam-removal project in North America is underway. At a cost of $325 million, two dams that have blocked salmon runs on the Elwha River for more than a century are being removed in a grand experiment in ecological restoration that is posing challenges for engineers and scientists alike. State, federal and tribal scientists are gathering baseline data on what the river basin is like today and what it could become as 800 acres drowned by the dam reservoirs are seeded with hundreds of thousands of native plants.

• In this week's discussion of MOOCs, Jon Bruner makes an interesting observation about one reason this might appeal to certain faculty members: Will online learning destroy America’s colleges?

  Top-tier schools that survive the spread of MOOCs could find themselves subject to new costs and transformations by the creation of a star system for faculty, in which popular teachers will have an international audience.

• Great story in Wired about the overlaps between computational linguistics, machine translation, and cryptography: They Cracked This 250-Year-Old Code, and Found a Secret Society Inside

  Knight was part of an extremely small group of machine-translation researchers who treated foreign languages like ciphers—as if Russian, for example, were just a series of cryptological symbols representing English words. In code-breaking, he explained, the central job is to figure out the set of rules for turning the cipher’s text into plain words: which letters should be swapped, when to turn a phrase on its head, when to ignore a word altogether. Establishing that type of rule set, or “key,” is the main goal of machine translators too. Except that the key for translating Russian into English is far more complex. Words have multiple meanings, depending on context. Grammar varies widely from language to language. And there are billions of possible word combinations.

• Should counter-hacking be legal? The law firm of Steptoe and Johnson hosts a fascinating debate: The Hackback Debate

  Legalizing self-help would also encourage foul play designed to harness the new privileges. One possibility is the bankshot attack: If I want a computer to be attacked, I can route attacks through that one computer towards a series of victims, and then wait for the victims to attack back at that computer because they believe the computer is the source of the attack.

• Why did Cisco spend over a billion dollars to buy Meraki? The Networking Nerd says it's all about dynamic network management

  Their single management platform allows them to manage switches, firewalls, and wireless in one single application. You can see all the critical information that your switches are pumping out and program them accordingly. The demo I saw at WFD2 was isolating a hungry user downloading too much data with a combination of user identification and pushing an ACL down to that user limiting their bandwidth for certain kinds of traffic without totally locking that person out of the network. That’s the kind of thing that Cisco is looking for.

• The great folks over at Netflix have pushed the mature, but still widely-used, log4j framework to the extreme: Announcing Blitz4j - a scalable logging framework

  Blitz4j overrides key parts of the log4j architecture to remove the locks and replace them with concurrent data structures.Blitz4j puts the emphasis more on application performance and stability rather than accuracy in logging. This means Blitz4j leans more towards the asynchronous model of logging and tries to make the logging useful by retaining the time-order of logging messages.

  As a guy who spends huge amounts of my waking days poring through logs, doing forensic analysis and post-mortem debugging, I get nervous about people who are willing to trade away accuracy in logging. But as the Netflix team point out, logging is pointless if it can't keep up with the load:

  It had worked fine for us, until the point where there was a real need to log lots of data. When our traffic increased and when the need for per-instance logging went up, log4j's frailties started to get exposed.

• Lastly, a spot of delightful news: Perforce Cuts the Ribbon on Alameda's Little Ice Rink

  Supporting the local community – whether that’s Alameda or other causes close to employee’s hearts – has always been part of the fabric of life at Perforce. The Foundation was established in 1998, just a few years after Perforce was started. The Foundation’s focus is on “community” and The Little Ice Rink is an awesome way to bring families and friends together during the holidays.

  Thank you Perforce!

The OAK watch incident

Here's a bit of a follow-up to last week's "weird watch arrest" at the airport by my house: McGann Tells His Story of Unusual Watch Arrest at Oakland Airport

McGann also explained that he had been travelling with his homemade watches for the past six months and even got a go ahead from the TSA in Los Angeles the first time he travelled with them.

"It is not a surprise that I got stopped by TSA...They are precarious looking watches. It's just they said it was alright if I put it in the bin," McGann said. "I knew I might be stopped sometime and that I might have to explain the watch as I did the first time I went to TSA months ago."

Check out the pictures of his other watches!

The "too large shoes" issue seems to have been resolved, as well: Updated: No Charges Filed Against McGann for Weird Watch

Police also said that at the time of his arrest, McGann was wearing boots that were two sizes too big and were stuffed with layers of homemade insoles. According to Nelson, the insoles created cavities in the shoes where items could be hidden.

Horngrad denied that the shoes were being used by McGann for any nefarious purposes.

"The shoes were, first of all, Uggs," said Horngrad. "Mr. McGann is short and the shoes were a little big and modified to accommodate lifts ... The shoes were meant to accomplish only that."

The Oakland airport is now safe for art again.

HP Autonomy

Holy cow!

HP Issues Statement Regarding Autonomy Impairment Charge

This appears to have been a willful effort on behalf of certain former Autonomy employees to inflate the underlying financial metrics of the company in order to mislead investors and potential buyers. These misrepresentations and lack of disclosure severely impacted HP management’s ability to fairly value Autonomy at the time of the deal.

HP has referred this matter to the US Securities and Exchange Commission’s Enforcement Division and the UK’s Serious Fraud Office for civil and criminal investigation. In addition, HP is preparing to seek redress against various parties in the appropriate civil courts to recoup what it can for its shareholders. The company intends to aggressively pursue this matter in the months to come.

I was shocked when HP paid $12 billion for a software company I'd never heard of. But then, there are lots of software companies I've never heard of, and 30-person companies get sold for nearly a billion dollars nowadays, so what do I know?

I guess I don't understand the due diligence process very well, but in the cases I've been (somewhat) involved in, it's hard to see how the acquiree can hide much of anything. The acquiring company generally gets access to everything, for as long as they want. So did HP (a) rush the process? (b) not assign a very good team of examiners? (c) actually have the information at the time but disregard it? or (d) find themselves prevented from getting the information because Autonomy operated in a different country, under different rules?

It sounds like they think they can get some of their money back, but I think this is mostly about ethics in business.

This is a very interesting development.

The bubble is back!

I guess I had somewhat foolishly thought that the crazed gold-rush days of the early Internet were behind us, and people were no longer acting quite so desparately.

But I guess I was wrong; things seem to be just as crazy as ever, if not more so.

• The New York Times has been running a series called "The iEconomy"; the most recent article in the series is As Boom Lures App Creators, Tough Part Is Making a Living. The article describes, in detail, two different experiences of those who write apps for the iPhone: one made a million dollars, while the others were not so fortunate:

  They sold one of their cars, gave some possessions to relatives and sold others in a yard sale, rented out their six-bedroom house and stayed with family for a while.

  This for a job which is

  freelance work that came with nothing in the way of a regular income, health insurance or retirement plan.

  After nearly two years, the result is that their

  quest cost them more than $200,000 in lost income and savings. So far this year, their eight apps have earned $4,964.

  As the Times observes, this is a common outcome, though the lure of the mother-lode is strong.

  Streaming Color Studios, a game developer, did a survey of game makers late last year. The 252 respondents, while not a scientifically valid sample and restricted to one segment of the app market, indicated what many people had suspected: the app world is an ecology weighted heavily toward a few winners.

  A quarter of the respondents said they had made less than $200 in lifetime revenue from Apple. A quarter had made more than $30,000, and 4 percent had made over $1 million.

  A few apps have made it extremely big, including Instagram, the photo-sharing app that was bought by Facebook in April for $1 billion. When app developers dream, they dream of triumphs like that.

• And iPhone app development isn't the only part of the Internet attracting desparate souls. The online magazine The Verge reports on the strange world of Bitcoin mining:

  O’Shea has 24 computers running constantly in a shed behind his house, making Bitcoins. "My setup is kind of... ghetto. I have bugs crawling around on my rigs and there’s dust and pollen and cigar smoke," he told The Verge recently by phone. "I'm out here now, I don’t know if you can hear them in the background. Can you hear the hum?"

  Again, the odds seem wildly stacked against striking it rich:

  O’Shea’s backyard operation brings in about $3,000 a month, he estimates, although the take is always changing because the price of Bitcoin is extremely volatile. He’s spent more than $60,000 on equipment, and his electricity costs run between $2,200 and $2,400 a month. He’s defrayed his cost significantly but has yet to break even.

I suppose it's somewhat in the nature of computers that people will charge off on unlikely quests such as these, and the computer industry is full of people who are working on their next great side project. After all, with computers, it seems almost as though you can't possibly fail

We didn’t need offices or fax machines or secretaries to get going. We could rent all our computing needs for next to nothing until customers with cash in hand started using our services and taxing our servers. This meant basically “no money down!” and no need to go hat in hand begging banks or venture capitalists for money.

There's nothing wrong with trying, and I wish all these folks well. I must say, though, this guy's approach seems much more sensible, and much more realistic, to me.

Stuff I'm reading about

This week was a bit fractured for us. It started with Hannah breaking her thumb, and ended with the company potluck, and had a little bit of everything else sprinkled in.

Along with it all, I'm plugging away on the beginnings of a large new project at work, and still trying to keep up with duh Netz. So:

• 30 years ago, when I was just starting out programming, I was an avid reader of Peter Neumann's Risks Digest. Somewhere along the way I lost track of it, but it turns out that both the Digest and Dr Neumann himself are still going strong. There's a nifty writeup of him in the New York Times: PROFILES IN SCIENCE PETER G. NEUMANN: Killing the Computer to Save It. I hope I am still an active Computer Scientist when I am 80 years old!
• A super paper by a team including Professor Dan Boneh of Stanford: The most dangerous code in the world: validating SSL certificates in non-browser software.

  We demonstrate that SSL certificate validation is completely broken in many security-critical applications and libraries. [...] Any SSL connection from any of these programs is insecure against a man-in-the-middle attack. The root causes of these vulnerabilities are badly designed APIs of SSL implementations (such as JSSE, OpenSSL, and GnuTLS) and data-transport libraries (such as cURL) which present developers with a confusing array of settings and options. We analyze perils and pitfalls of SSL certificate validation in software based on these APIs and present our recommendations.

• Facebook engineering has "crossed the Google threshold": they are now at the point where they re-write every piece of software to their own exact specifications, build their own hardware, etc. Only a handful of organizations are this sophisticated, and Facebook are now clearly one of them: Under the Hood: Scheduling MapReduce jobs more efficiently with Corona. Note that, if I'm understanding this correctly, high-end data centers are now at the point where they devote entire computers to just a single task.

  Corona introduces a cluster manager whose only purpose is to track the nodes in the cluster and the amount of free resources. A dedicated job tracker is created for each job, and can run either in the same process as the client (for small jobs) or as a separate process in the cluster (for large jobs).

  Once there was multi-processing, then there was multi-threading, but at a certain scale things go inverted and every thread of every process gets its own dedicated computer.
• I'm not sure how this Niantic Project idea is going to turn out, but my son is on-board and playing, so more as it develops...
• Speaking of games, how did I manage to pick up every expansion of Carcassonne but this one? As we're mostly a 2-person gaming group right now (though my granddaughter is coming along fast), I'll be picking this up shortly.
• At the Counterparties blog, Ben Walsh tries to help us understand the drivers of tuition inflation. We were doing the math recently, and we're just crossing the $100K line for my youngest's college education (at a public university!), with still nearly a year to go. The overall costs tripled in just the 10 years between our oldest child and our youngest. Thankfully, we're one of those lucky families who've been able to afford it; how long will that last?
• I absolutely adored Wolf Hall, and am so looking forward to Bringing Up The Bodies, but I really need to get through several other (long) books first...
• The sub-heading on this Slate story seems to hit a bit close to home: The Secret History of the Aeron Chair: It wasn’t originally designed for office warriors. It was intended for the elderly.. I certainly can't call myself "elderly", but I'll tell you this: until I took my current job, I had a long procession of left-over office chairs purchased from various auction houses and office supply stores. But I've had my Herman Miller chair for 30 months now and it is fantastic!
• It's only been two weeks, but I'm enjoying my subscription to Weekend Sherpa's newsletter. Of course, this weekend is a big wet winter storm, but hopefully we'll sneak out for at least a little bit in-between the raindrops...

Perforce Thanksgiving Potluck

Today was my third Perforce Thanksgiving Potluck.

The Perforce Thanksgiving Potluck is my favorite company event of the year. It is not only the best potluck I've ever attended, it might be the best potluck on the planet. The entire company participates, attends, and spends time together. We set up outside in the parking lot; we all sit at one long "table" (many tables, joined end-to-end).

It's hard to describe, but wonderful. A great end to a rather stressful (but ultimately successful) week.

Gates Foundation gets involved in MOOCs

There's big news in the world of MOOCs today: Establishment Opens Door for MOOCs

The Bill & Melinda Gates Foundation is funding that effort as part of $3 million in new, wide-reaching MOOC-related grants, including research projects to be led by ACE, the Association of Public and Land-grant Universities (APLU) and Ithaka S+R, a research group that will team up with the University System of Maryland to test and study the use of massive open online courses across the system.

It seems that all the news about MOOCS involves their movement toward some sort of accreditation, even though the MOOCs themselves continue to protest (too much?) that they don't want any part of that:

They have repeatedly said the company has no desire to become an accredited, credential-issuing institution, arguing that it will be an extension of higher education, rather than a direct competitor. Ng and Koller have also shown little interest in pushing a pathway to college credits for Coursera’s offerings, at least until now.

While the Gates Foundation involvement will bring both visibility and massive resources, Joshua Gans wonders if this whole effort isn't heading in the wrong direction: Accreditation and MOOCs: How about we just don’t do it.

The alternative is that they just wanted to learn. That means that when lectures are prepared, the professors involved don’t have to worry about what will be on a test and what is testable. They can just teach. And there is a difference. You can go on digressions, add potential sources of confusion and discomfort all without the potential come back of “is this going to be on the final.” Now the professors on CourseRA haven’t yet broken from their learned shackles of teaching what can be examined but the ones on Udemy seem to move on that path. That is just my assessment, you can all decide for yourselves. But my point is that MOOCs offer that potential. Think too of the non-academic contributions of Vi Hart and CP Grey among others. This is pure knowledge people and it is what it looks like when there is no test involved. Think also about the radical teaching styles in Codeacademy, Treehouse, and LearnStreet; not to mention the Khan Academy which is in a league of its own.

For my own part, I'm not interested in credit, I'm not looking for a degree, and I'm not likely to pay additional money for a certificate or other document from the MOOCs. I guess I'm that weird guy that just wants to learn.

Is Gans right? Am I the target audience?

Meanwhile, Alex Tabarrok focuses his attention on the fundamentals of the new approaches: Why Online Education Works. Tabarrok is strongly optimistic about the potential:

Technology is rapidly changing how much interaction can occur online. The future is lectures plus intelligent, on the fly assessment. The GRE, for example, is a computer-adaptive test—when you answer questions correctly you get a harder question; when you answer incorrectly you get an easier question. The adaptive nature of the test makes it possible to zero in more quickly on true ability. The future of online education is adaptive assessment, not for testing, but for learning. Incorrect answers are not random but betray specific assumptions and patterns of thought. Analysis of answers, therefore, can be used to guide students to exactly that lecture that needs to be reviewed and understood to achieve mastery of the material. Computer-adaptive testing will thus become computer-adaptive learning.

I think we're at the point now where the technology is operational, the educators are starting to become comfortable with how to use it, the existence of an audience is thoroughly established, and the institutions are at least willing to debate how things should proceed.

One signal event that I'm looking for is: when will the various offerings (Coursera, Udacity, edX, etc.) start to include more advanced material? So far, it's mostly been "Intro to this, Intro to that".

That is, what I've seen, at least in the fields with which I'm familiar, is introductory (college-level) material, survey material, and summary material. Relatively few of the classes that I've seen have gone beyond that 1st-year or 2nd-year undergraduate level of offering.

For example, in the Computer Science arena, Tim Roughgarden's classes at Coursera are superb.

But when will we see a Coursera-type delivery of something like MIT's graduate school class in algorithms: 6.851: Advanced Data Structures? (Note that MIT already makes the lectures for 6.851 available online, and they're superb; thanks MIT! But 6.851 may be something of an exception; 6.854 only appears to make the instructor's notes available).

Would giving something like 6.851 the "Coursera treatment" be a valuable next step? Or is it indeed better to develop the infrastructure for accreditation, credit transfer, and certification?

In the meantime, I'll just continue being a happy student :)

Yes, it's true, I just can't get enough...

I've signed up for Algorithms: Design and Analysis, Part 2, and got this enthusiastic auto-response:

Thank you for signing up for Algorithms: Design and Analysis, Part 2! Part 2 of Algorithms: Design and Analysis will be action-packed with great topics: the greedy algorithm design paradigm, with applications to computing good network backbones and good codes for data compression; the tricky yet widely applicable dynamic programming algorithm design paradigm, with applications to routing in the Internet and sequencing genome fragments; NP-completeness and the famous “P vs. NP” problem and what they mean for the algorithm designer; and strategies for dealing with hard (i.e., NP-complete) problems , including the design and analysis of heuristics. We're shooting for a launch date of December 2nd, 2012.

Data security in the cloud

There's lots to chew on in this strongly-worded editorial on the website NextGov.com: Op-ed: Encryption, not restriction, is the key to safe cloud computing.

The specter of non-U.S. citizens having physical control over and access to U.S. data understandably gives the government pause. The same is true of almost every other country in the world.

As a result, many federal, state and local governments and agencies are starting to require that their data remain within geographic control.

Taking this school of thought further, the U.S. government is engaged in an opaque rule-making process that is poised to create a requirement that federal data be stored at a U.S. location and handled only by U.S. citizens.

As hinted at by the title of their editorial, the authors suggest that this is the wrong direction to proceed.

There is an easier solution -- encryption at rest. A system of encryption where the customer controls the encryption keys solves many of the security problems that have bedeviled public clouds for the government. It would eliminate the need to insist on U.S.-only location for government cloud data centers and support personnel. All that is required is to implement an architecture that enables customers to apply encryption to data at rest before that data is transitioned to the cloud and for their customers to be the sole holders of their own encryption keys.

But as the article notes, the main objection to this proposal is not technological, but financial:

Why do some in the industry resist this solution?

In part it is because encryption with customer controlled keys is inconsistent with portions of their business model. This architecture limits a cloud provider’s ability to data mine or otherwise exploit the users’ data. If a provider does not have access to the keys, they lose access to the data for their own use. While a cloud provider may agree to keep the data confidential (i.e., they won’t show it to anyone else) that promise does not prevent their own use of the data to improve search results or deliver ads. Of course, this kind of access to the data has huge value to some cloud providers and they believe that data access in exchange for providing below-cost cloud services is a fair trade.

Remember: if you don't know what the product is, the product is you.

The fastest boat in the world?!

It's about as non-traditional as you can imagine, but the Vestas Sailrocket is a fascinating vessel.

This week, it's racing at the Luderitz Speed Challenge in Walvis Bay, Namibia.

The always-fun Kimball Livingston picks up the story from here: Sixty-one Knots (!). Check out the videos!

And it's not just the Sailrocket that's having a great week: Surfer Today reports that the windsurfers are flying along, too: Anders Bringdal breaks the 50 knots speed windsurfing frontier.

Zoom!

AWS 10/22 outage

I've been looking through Amazon's information about the October 22 AWS outage: Summary of the October 22,2012 AWS Service Event in the US-East Region.

As you might expect, given the incredible sophistication of AWS, the outages are becoming more sophisticated as well; this outage was no exception, as it involved a cascade of events:

• A hardware failure caused an internal monitoring server to do down
• The DNS information describing the replacement for that server was not successfully propagated to all the internal DNS servers (Amazon runs their own DNS implementation)
• This meant that other servers continued to try to contact the no-longer-operational server
• A memory leak in the error-handling for failure-to-contact-internal-monitoring-server caused memory pressure in the production servers
• which then caused those servers to run out of swap space and become non-responsive

I can sympathize with the AWS developers: memory leaks in error-handling code are easy things to overlook, and it is quite challenging to write thorough-enough test suites to be able to detect memory leaks in error-handling code:

• First you have to provoke those errors, which can be quite challenging
• Then you have to have a test harness which can observe memory leaks
• And then you have to provoke the leak a sufficient number of times that the harness is able to detect it

As Jeff Darcy observes, these sorts of testing (dealing with distributed failures, and testing to ensure that your error-handling is up to snuff), are the sorts of things for which there have been some good tools developed:

Another possibility would be from the Recovery Oriented Computing project: periodically reboot apparently healthy subsystems to eliminate precisely the kind of accumulated degradation that something like a memory leak would cause. A related idea is Netflix’s Chaos Monkey: reboot components periodically to make sure the recovery paths get exercised.

It's also interesting to observe that some of the human actions that the AWS operations team took while trying to deal with the problem caused problems of their own:

We use throttling to protect our services from being overwhelmed by internal and external callers that intentionally or unintentionally put excess load on our services. A simple example of the kind of issue throttling protects against is a runaway application that naively retries a request as fast as possible when it fails to get a positive result. Our systems are scaled to handle these sorts of client errors, but during a large operational event, it is not uncommon for many users to inadvertently increase load on the system. So, while we always have a base level of throttling in place, the team enabled a more aggressive throttling policy during this event to try to assure that the system remained stable during the period where customers and the system were trying to recover. Unfortunately, the throttling policy that was put in place was too aggressive.

Higher-level software systems also encountered problems in their attempts to handle and recover from the lower-level problems, such as this behavior in Amazon Relational Database Service:

The second group of Multi-AZ instances did not failover automatically because the master database instances were disconnected from their standby for a brief time interval immediately before these master database instances’ volumes became stuck. Normally these events are simultaneous. Between the period of time the masters were disconnected from their standbys and the point where volumes became stuck, the masters continued to process transactions without being able to replicate to their standbys. When these masters subsequently became stuck, the system blocked automatic failover to the out-of-date standbys.

Given the amount of complexity in the overall system, it's remarkable that Amazon were able to analyze all of these events, and deal with them, in only eight hours. It wasn't so long ago that outages such as these were measured in days or weeks.

Still, eight hours is a long time for AWS.

As the team at Netflix describe, businesses that build atop AWS need to consider these issues, and ensure that they have their own processes and tools to handle such situations: Post-mortem of October 22,2012 AWS degradation:

We’ve developed a few patterns for improving the availability of our service. Past outages and a mindset for designing in resiliency at the start have taught us a few best-practices about building high availability systems.

The Netflix analysis is well worth reading; it contains lots of information about the tools and techniques that Netflix use for these purposes, many of which they've open sourced to the world.

Thanks once again to the AWS team for providing such detailed information; each time I read these reports, I learn more, and think about more ways that I can improve my own testing and development.

The old guy can still bring it

I love the fact that Anatoly Karpov won the Anatoly Karpov chess tournament!

As The New York Times notes in Former World Champion Wins Namesake Event, Karpov has had an astonishingly long career:

Karpov, 61, of Russia, became world champion in 1975, succeeding Bobby Fischer, who was feuding with the World Chess Federation and refused to defend the title. Karpov quickly proved that he was a worthy champion, and he successfully defended the title in 1978 and 1981 in matches against Viktor Korchnoi, a Russian who defected from the Soviet Union in the 1970s. It was a period in which Karpov dominated the chess world.

Karpov became world champion when I was 14 years old. Although Bobby Fischer was of course my idol when I was just learning chess, I remember being fascinated by Karpov's cool and crisp precision. Fischer's games were ravaging annihilation, with attacks, sacrifices, pieces en prise everywhere; Karpov's were clinical devastation: simple, accurate, flawless.

Let's hear it for the old guy!

Vibe Managers

I enjoyed reading this story about one of the Bay Area's hottest software companies: How Heroku’s ‘Vibe Managers’ Could Transform Salesforce’s Culture

Heroku was rapidly approaching legendary status in the Bay Area software startup scene when they were acquired by Salesforce in 2010.

You could hardly imagine two companies with more opposite cultures: Heroku was a Ruby programming shop, small and agile and focused on small users and their needs; Salesforce was already becoming an enterprise titan, selling and delivering to the largest organizations on the planet and about as buttoned-down and sober as Bay Area software companies get.

You might expect that the corporate titan would drain the soul from the little quirky operation, but so far, at least, things are, according to the article, at least somewhat the other way around:

Left alone to maintain its own unique working environment with weekly yoga, a company dashboard with a ping-pong champion leaderboard, and blinking red lights that blind everyone when the site goes down, Heroku has flourished under Salesforce. But more impressive perhaps is that now Heroku is the company exerting influence on its much larger parent.

At least part of Heroku's influence is credited to their Vibe Managers, a most unusual job description:

These aren’t just people who order supplies and help you fill out benefits forms (though they will help you with either task), they are there to help install a hipster-approved coffee station near your desk, or roll a kombucha kegerator into the office if that is what the staff requires. Vibe Manager Sharon Schmidt’s duties have included everything from organizing team bar outings to finding an old TV at the local thrift store to go with the almost-vintage Nintendo system in the office.

Although we don't have Vibe Managers, my day job has a very similar feel. We have beautiful office space, with art on the walls, comfortable and well-provided break areas, game rooms, company-provided massage, a wonderful grotto-like lunch area, and more.

Company culture isn't just something that gets written up in Harvard Business Review; it's something that turns man-months and resources into a productive and coherent team. I'm pleased to see Wired celebrating Heroku's vivid and inspirational culture.

Election 2012

I mentioned to a few close friends that I was completely uninspired, unmoved, and de-motivated by this election, and they reacted with surprise, for they felt it was exciting and important.

For me, it seems like the last five years of political discourse have been numbing and mundane.

It seems like every public discussion, every political debate, is petty, and is about nothing more than an argument over money: who should have more money, who should have less, whose taxes should be higher or lower.

This started all the way down at the local level, as the election for the school district in my town was about nothing more than the city's parcel tax revenues, and whether they had been used to give the administrators an outsized raise.

And it continued at the state level, where the major propositions on the ballot were about whose taxes should be raised or lowered.

And it persisted all the way to the national election, where the debates were almost solely about debt, deficits, taxes, and spending.

I don't understand why there aren't people debating issues that seem truly important to me: how do we reduce conflict over religion, and over race; how do we end these seemingly endless wars; how do we make sure that people around the world have enough to eat, have clean water to drink, have a safe place to sleep and raise their families; how do we solve the problems of drug wars, of gangs, of ghettos and youth violence; how do we move away from policies that put millions of people in horrible dangerous prisons for trivial reasons, turning minor miscreants into hardened criminals; how do we stop polluting the planet; how do we arrange to bring new scientific discoveries to the people of the world?

All this bickering about money and jobs and taxes just seems so mundane and selfish and ordinary, when there are so many more important things that people ought to be discussing.

So that's why this year's elections just left me cold and drained and listless about politics.

I don't mean to bring everybody down, just wanted to write on my blog a bit about it.

It's not just a game ...

... it's an opportunity to reflect on what it is that causes us to form emotional attachments to each other: Analyzing XCOM: Enemy Unknown’s soldier progression with producer Garth DeAngelis

My X-Com soldiers earn their “abilities” while I grant such perks to my troops in XCOM. That contrast in agency of character growth makes all the difference between a real sense of attachment and playing with action figures. So I asked DeAngelis how the skill system develops a feeling of affection toward your troops.

“Because your soldiers experience combat before they level up,” he said, “… it’s easy to create a story where soldiers use the experience they just went through to help them define their skills. Maybe that Heavy was just in a fight where nobody was making their shots — it’s easy to choose Holo-Targeting [editor's note: a skill that increases aim] at that point and say, ‘Alessandro “Crater” Guzman is not going to go through that again!’”

Oh, and by the way: I've finished the first base mission, and have almost built the holodeck beacon.

Meanwhile, if you can stomach strong language, I'd have to say that Ben Croshaw's review in The Escapist pretty much hits the nail on the head: Zero Punctuation reviews XCOM: Enemy Unknown..

Consider the source

Today, the Wired website is running an article by Manny Schecter, the Chief Patent Counsel at IBM: With All Due Respect: The Patent System’s Not Broken.

It's an odd article; I encourage you to read it for yourself. It's full of strange arguments such as:

The success of the U.S. software industry correlates with its use of software patents to protect its innovations. If patent litigation caused by the U.S. patent system stifled innovation, U.S. software companies would not be the most successful in the world.

I suspect that Wired has, queued up, an article by the Chief Earth Scientist at Exxon: With All Due Respect: The Use of Hydrocarbon Fuel's Not Harming the Earth's Environment; that article will probably include arguments such as: If the use of hydrocarbon fuels were harming the Earth's environment, U.S. energy companies would not be the most successful in the world.

Meanwhile, Apple has, indeed, successfully patented the rectangle.

Don't see me!

When she was young, and she had done something she was embarrassed by or felt guilty about, my daughter would sometimes hold up her hand to block her eyes and say, loudly,

Don't see me!

The idea, of course, was that by blocking her eyes, she felt that she had blocked us from seeing her.

My wife and I always found that behavior interesting, and just recently I noticed that some others had found it interesting, too: Why do children hide by covering their eyes?

Now things get a little complicated. In both studies so far, when the children thought they were invisible by virtue of their eyes being covered, they nonetheless agreed that their head and their body were visible. They seemed to be making a distinction between their "self" that was hidden, and their body, which was still visible.

SSL Certificate Validation

I quite enjoyed this recent paper: The Most Dangerous Code in the World: Validating SSL Certificates in Non-Browser Software

We present an in-depth study of SSL connection authentication in non-browser software, focusing on how diverse applications and libraries on Linux, Windows, Android, and iOS validate SSL server certificates. We use both white- and black- box techniques to discover vulnerabilities in validation logic. Our main conclusion is that SSL certificate validation is completely broken in many critical software applications and libraries. When presented with self-signed and third-party certificates—including a certificate issued by a legitimate authority to a domain called AllYourSSLAreBelongTo.us —they establish SSL connections and send their secrets to a man-in-the-middle attacker.

Security is interesting; there are so many different ways to get it wrong!

Am I slowly learning something?

Perhaps there's hope for me yet.

I'm about halfway through my Introduction to Computer Networking class, and although I'm still a newb dunce, I'm at least learning enough so that when I read a story like this, most of the article actually makes sense!

Going back to validate, things started to make sense. I also noticed a little hint in tcpdump verbose mode (tcp cksum bad) which was missed before. A Montreal machine receiving this packet discarded it at the kernel level after realizing it’s corrupt, never passing it to the userland ssh daemon. London then re-transmitted it, going through the same corruption, getting the same silent treatment. From ssh and sshd’s perspective, the connection was at a stalemate.

In some ways, what's been best about the networking course is that I'm starting to understand what an incredible achievement it is that the Internet works at all, not to mention how incredibly well it actually works.

With luck, given the current syllabus, by Thanksgiving I will have learned enough to understand this story.

the internet is a collection of networks called autonomous systems. Autonomous systems communicate with each other through the Border Gateway Protocol, or BGP, which is a system for exchanging information about routes from one location in the network to another. If you want to access Google, your ISP needs to have a route from your computer to Google’s servers.

Patience grasshopper. Hold the course, watch the videos, it will slowly continue to make sense.

Future Perfect: a very short review

About a decade ago, I read a fairly clever book by Steven Johnson: Emergence: The Connected Lives of Ants, Brains, Cities, and Software. The book explored the idea that complex behaviors can arise out of the apparently disconnected and unrelated behaviors of a group of smaller independent objects. The book was rigorous enough to be informative but also descriptive enough to be entertaining.

Ten years later, Johnson's latest book is Future Perfect: The Case For Progress In A Networked Age.

Future Perfect attempts to reveal ways that we can improve our political institutions and our public organizations, using the techniques that have arisen from the highly-connected Internet and the so-called "social media" approaches.

Johnson's book bounces all over the place, making some points very well, while in other cases demonstrating what I found to be at best tenuous support for his proposed ideas.

In a strong early chapter, Johnson discusses how the Internet, through tools such as Kickstarter, could literally revolutionalize the way that artists get funded and find their audience:

Art that probes the boundaries of accepted ideas or taste rarely attracts enough of an audience to sustain itself financially. We have the phrase "starving artist" for a reason. And yet society as a whole benefits greatly from the network edges of experimental writing and music and theater and seventy-one-minute music videos. Subcultures expand the possibility space of our experience and our understanding; yesterday's underground is tomorrow's mainstream.

Johnson then explains why the Kickstarter approach is so revolutionary:

The donors decide which projects deserve support. There are no experts, no leaders, no bureaucrats -- only peers. New creative ideas don't need to win over an elite group of powerful individuals huddled in a conference room, and they don't need to win over a mass audience. All they need is an informal cluster of supporters, each contributing a relatively small amount of money.

Although Johnson didn't make this connection, I was strongly reminded, while reading this chapter, of micro-investment web sites such as Kiva, which seem very similar both in their approach, and in their potential.

Later in the book I thought Johnson succeeded again, in his discussion about the decline of mass media. Johnson makes a point that I think is often overlooked: Internet-based sources of news and current events often bring a locality of focus, offering something that the mass media was never capable of delivering:

Yet every week in my neighborhood there would be easily twenty stories that I would be interested in reading: a mugging three blocks from my house; a new deli opening; a house sale; the baseball team at my kid's school winning a big game. The New York Times can't cover those things in a print paper not because of some journalistic failing on their part, but rather because the economics are all wrong: there are only a few thousand people potentially interested in those news events, in a city of 8 million people.

...

We've never thought of it as a failing of the newspaper that its metro section didn't report on a deli closing, because it wasn't even conceivable that a big centralized paper could cover an event with such a small radius of interest. But peer networks can. They can find a way around the pothole paradox.

Not all of Johnson's book hits pay dirt: his notions of "liquid democracy" seem full of hand-waving and elision to me, and I think that he confers a decision of success on efforts that seem to me to be still nascent and most likely benefiting greatly from faddishness and the 'cool' factor.

However, he has some nifty ideas, he writes well, and, most importantly, he is quite talented at taking current events and ideas and extrapolating and projecting them just slightly into the future, providing a certainly-possible view of what might be.

Best of all, I think, he is a looker-on-the-bright-side. With all the doom and gloom (much of it well-deserved, I admit), it is refreshing and stimulating to find somebody like Johnson who looks at occurrences such as US Airways Flight 1549, Porto Alegre's participatory budgeting, and Girl Walk // All Day, and concludes that the future, if not as perfect as his title makes it out to be, is at least hopeful and worth fighting for and celebrating.

Looking for a good way to pass a long plain trip, or a rainy weekend afternoon? Give Future Perfect a try, and let me know what you think!

Pulpit Rock

Perhaps if there were more music videos like this, there would be more chess players in the world!

Another quick collection of Sandy links

A few Sandy-recovery stories that intrigued me over the last few days:

• The Coast Guard has called off its search for Captain Walbridge, and has moved into an analysis stage: Coast Guard scrutiny on the Bounty's sinking begins, investigation expected to last months
• A plan has been developed and put into action to secure and dismantle the damaged construction crane: Dangling Crane to Be Secured in 36-Hour Weekend Operation

  Over the next three or four weeks, workers will build another crane alongside the damaged one and use it to remove the parts that are broken, said the mayor

  Dave Winer's blog has some good pictures here.

• Large parts of New Jersey will practice gas rationing, at least for a few days: Christie Orders Odd-Even Rationing System For Filling Up Gas Tanks

  Residents with license plates ending in an odd number can make gas purchases on odd-numbered days of the month Residents with plates ending in an even number will be able to buy gas on even-numbered days, the governor said.

  Specialized plates or those not displaying a number will be considered odd numbered plates, a release from the governor’s office stated.

  This being New York, there's an extensive black market, on Craigslist: Energy emergency: Sandy profiteers sell gas, generators at predatory prices on post-apocalypse Craigslist

  to get a taste of how bad it is: search for "gasoline," "gas," or "generator" on NY Craigslist right now

• A somewhat-under-reported story is the impact of the closure of the stupendously-enormous Port of New York and New Jersey: Cargo industry suffers huge losses in wake of storm

  Commercial vessels aren't yet allowed to wait for the terminals to reopen. "If you don't have an approved berth, you can't come in and sit for a week and wait for something to open," Ward says.

  Ocean shippers are rerouting ships to other East Coast ports, such as Philadelphia. For example, Evergreen Marine, a shipping company based in Hong Kong, told customers Friday that it would steer vessels to Baltimore or Norfolk until the New York port resumes normal operation.

  The Wall Street Journal, behind a paywall, says that the biggest problem is not the waterways or the physical docks, but the fact that the lack of power means they can't run the container cranes and lifts.

• Lastly, to end on a somewhat lighter note, this amusing story about why there happened to be eight generators already present at the New York Army National Guard's Lexington Avenue armory: How Victoria’s Secret Saved the National Guard During Hurricane Sandy

  As they had done for the last three years running, the lingerie company was holding its annual television event at the Regiment’s historic armory, located at 25th street and Lexington Avenue in Manhattan. For the show, the producers had hauled in eight massive 500 kilowatt generators. Of course, the producers said, we’d be happy to help. Hours later, the lights flashed back on.

Techy Sandy Stories

A kind of quick grab-bag of some interesting tech-related Sandy stories I noticed:

• Hurricane Sandy’s Lesser-Known Victims: Lab Rats

  The collection of carefully bred rodents was considered one of the largest and most valuable of its kind in the country. The animals lived in colonies in the cellar of the Smilow Research Center, on 1st Avenue near 30th Street.

  New York University medical and research staff worked furiously to protect their human patients — and others threatened by the storm — in all three of its facilities in Kips Bay. Though most of the animals at the center were unharmed, the center staff could not rescue the animals in one of the facilities, despite hours of work amid the flooding that started at the institute on Monday night.

• Here’s How Army Engineers Are ‘Unwatering’ NYC’s Tunnels

  The Corps is “looking at bringing in” two types of pumps, a “high-head submersible” and a centrifugal one, Pogue says. The high-head pump goes below the surface, extracting water down from the top, while the pump itself may be submerged as far down as 100 feet. The centrifugal pump is more familiar, using a hose “similar to a straw,” as Pogue put it, to suck the water out. The plan is to pump the water back out to sea.

• Why Salt Water in the Subway Is So Extremely Dangerous

  This type of rail system is safe to use in nearly any type of environment except being submerged in salt water. When two different types of metal (or metal with two different components) are placed in water, they become a battery: the metal that is more reactive corrodes first, losing electrons and forming positive ions, which then go into water, while the less reactive metal becomes a cathode, absorbing those ions. This process happens much more vigorously when the water is electrically conductive, and salt water contains enough sodium and chloride ions to be 40 times more conductive than fresh water.

• We Are Deploying Mobile Wi-Fi Hotspot and Recharging Stations in Lower Manhattan

  To help New Yorkers in hard-hit areas like Lower Manhattan and Staten Island who are without power, we are deploying multiple vehicles with mobile charging stations and free WiFi access points. Local residents are welcome to charge their consumer devices such as smartphones and laptops and access a 4G WiFi connection.

• This Is What A Starbucks In Manhattan Looks Like Right Now

  Because huge parts of the subway system are still down and much of lower Manhattan is without power, lots of people in my neighborhood are "working from home" today.

  For many of them, that actually means working from Starbucks, which has power, wi-fi, and an abundance of caffeine on tap.

• Even a superstorm is no excuse for journalists not to check Twitter trolling

  Here's the thing: while what Tripathi did was stupid, inappropriate, ill-timed and loathsome, the reaction to it was entirely out of scale to the actual offense. The truth is, Tripathi had a relatively small niche on Twitter. His influence would have been limited had not journalists on Twitter been desperate for information to share, regardless of provenance.