A few interesting articles

At least for the time being, the base-board installation upstairs is complete, so it's back to surfing the Internet...

• The Weird Stuff Warehouse is where old tech goes to retire

  Tucked neatly between Yahoo! headquarters and Lockheed Martin is a row of unmarked warehouses. To the common passersby, it's nothing more than an office park surrounded by perfectly coiffed lawns. But to those who are in on the secret, there's a place full of technology treasures waiting to be unearthed. It's called the Weird Stuff Warehouse and for more than 27 years it's been providing the Bay Area with a surplus of old and new technology.

• Making of White One

  A 4k intro is a executable file of at most 4 kilobytes (4096 bytes) that generates video and audio. That is, it puts something moving on your screen and something audible on your speakers. The finished product runs for a few minutes, has some coherent theme and design and ideally, sound and visual effects complement each other. On top of that, it's a technical challenge: It's impossible to store 3D models, textures or sound samples in 4 kilobytes, so you have to generate these things at runtime if you need them.

• Revenge, ego and the corruption of Wikipedia

  Taken all together, the edits strongly suggest a focused attempt to diminish Hannah’s legacy. But why? Who was Qworty and what axe did he have to grind with Hannah?

• With Ubiquity, Sears is Turning Shuttered Stores into Data Centers

  The first Ubiquity project will be a Sears store on the south side of Chicago, nestled alongside the Chicago Skyway. The 127,000 square foot store is closing at the end of June, and will be retrofitted as a multi-tenant data center. Farney says he already has a commitment for the first tenant at the site on East 79th Street, which has 5 megawatts of existing power capacity and the potential to expand.

• Finding peace in post-disaster Haiti

  In fact, Grand Rue melees were the exception, not the rule. Just as in New York after Sandy, responders and many journalists looking back on the postquake moment would highlight the lack of unrest in their after-action reports, often crediting their own presence, such as the UN adviser who told the Los Angeles Times: “There has been no rioting over food, and we avoided people dying of hunger or thirst. This is no small accomplishment.” As Auf der Heide has written, “Even when looting is not actually observed, that fact is often attributed to the extraordinary security measures that have been taken rather than the fact that such behavior is inherently uncommon.”

• ABC: Always Be Coding. I like:

  2. Be passionate. If you don’t care, then nobody else will. Be passionate about something. It might be programming, but what about it? Do you enjoy building compilers in your spare time? Do you build and fly RC helicopters? It doesn’t really matter because if you’re passionate about it, then you can make it interesting.

• Statistical Formulas For Programmers

  Most of these formulas can be found in Wikipedia, but others are buried in journal articles or in professors' web pages. They are all classical (not Bayesian), and to motivate them I have added concise commentary. I've also added links and references, so that even if you're unfamiliar with the underlying concepts, you can go out and learn more. Wearing a red cape is optional.

Buildings, rising and falling

Like almost every other engineer on the planet, I woke up this morning fascinated by the bridge collapse in Washington State.

It's interesting that it seems like the incident was at least partly a human factors issue: Skagit River bridge collapse: Looking for a temporary fix to get traffic moving

The bridge has a maximum clearance of about 17 feet – higher than the truck’s load in this case – but the clearance curves down to 14 feet 5 inches along the sides, where the collision occurred. The tractor-trailer was hauling drilling equipment southbound.

The article makes much of the idea of putting a "temporary" bridge in place while a replacement is studied. Well, all structures are temporary, of course; the question is just how long do we expect they will last?

A friend, traveling to Boston, remembered that we spent our first years together there, and sent us a picture of the beautiful Zakim Bunker Hill Bridge. It is indeed beautiful, though we've never seen it; when we lived in Boston we used to drive over the Charlestown High Bridge, which we called the "Fitzgerald Bridge", and which surely must be in the running for Ugliest Bridge Ever.

Getting back to the Skagit River Bridge collapse, any engineering failure is important and needs to be studied. Even if the issues here are pretty simple, in other cases they are more complex. Another Christian Science Monitor article points out that studying the decay of structures is a constant process: Collapse of I-5 bridge in Washington State: no fatalities, many questions

The bridge was not classified as structurally deficient, but a Federal Highway Administration database listed it as being "functionally obsolete" – a category meaning that the design is outdated, such as having narrow shoulders and low clearance underneath.

The bridge is also classified as "fracture critical" by the National Bridge Inventory. That means the bridge is designed so that a failure in any one part of the bridge can collapse the entire span. There are some 18,000 fracture-critical bridges nationwide, of which 8,000 are also "structurally deficient," according to a 2012 review of Federal Highway Administration records by Bloomberg News.

Some failures are far more complex and need much more analysis. In the case of the horrible disaster in Bangladesh a month ago, the investigation will probably take many months or even years: Bangladesh factory collapse probe uncovers abuses

The man in charge of the investigation, Mainuddin Khandker, told BBC Bangla on Wednesday that "extremely poor" construction materials were used in the building and said the report identified five causes of the collapse.

"A portion of the building was also constructed on land which had been a body of water before and was filled with rubbish," he told the Associated Press news agency.

The 400-page report was submitted to the government on Wednesday.

Closer to home, our own bridge is still under construction, yet already we are studying its decay: Lawmakers seek probe as doubts are raised on Bay Bridge opening

"This issue does not affect the safety, strength or the lifespan of the Skyway," Caltrans said in a written statement Monday. "... Corrosion of steel tendons inside the bridge was extremely limited and was addressed."

But the construction record shows that Caltrans managers did not address frequent warnings from the agency's bridge inspectors until it was too late to head off substantial corrosion. Thomas Devine, a UC Berkeley engineering professor and a leading metallurgist, disagreed with Caltrans claims that its examination of the issue proved that the corrosion was insignificant.

Numerous scientific and methodological errors made the Caltrans findings "meaningless" and "essentially useless," Devine told The Bee.

Like other lawmakers, Cannella expressed frustration about the slow trickle of information coming out of Caltrans. "It's frustrating to me, as a lawmaker, to keep getting my information from The Sacramento Bee," Cannella said.

Sometimes a structure collapses, but the story has a happy ending: The remarkable twist to the tale of Oklahoma's tornado-hit hospital

Surveying the shredded shell of the medical centre, the two-floor structure torn apart as if a bomb had struck, it is hard to believe that there were no casualties inside.

Yet the accounts of those who rode it out inside make the story of survival all the more extraordinary. They describe people lifted off their feet and carried tumbling down hallways by screaming gusts; the thump of cars smashing into the building; shards of doorways, picture frames and tiles turned into projectiles and hurtling past them; the air choked with dust; and that all-encompassing noise -- a jet engine meeting a freight train.

That nobody suffered more than cuts and bruises is down to a combination of detailed planning, calm heads, quick thinking, selfless bravery and plain luck.

Meanwhile, new buildings continue to be built, even when it seems like there's no need: Sky High and Going Up Fast: Luxury Towers Take New York

Ultraluxury housing and construction is booming across Manhattan, which is now beginning to rival London in popularity with the world’s wealthy. The number of condominium buildings in the borough with apartments selling for more than $15 million has risen to 49, up from 33 in 2009, according to CityRealty.

...

As with many of these buildings, only about a quarter of the units will be occupied at any one time.

Software, of course, doesn't decay (although disk drives and flash memory do wear out, a phenomenon that we call "bit rot", though that's a gross over-simplification). But software does undergo constant change, which can introduce new problems even as we fix the old ones. So we software engineers love to study how other fields of engineering deal with failures.

And, of course, modern architecture and engineering design wouldn't be possible without software, since nearly everything is simulated and analyzed ahead of time on computers. Even software that was once considered to be just a toy is now used for serious architectural study: SimCity: An Interview with Stone Librande

Manaugh: Now that the game is out in the world, and because of the central, online hosting of all the games being played right now, I have to imagine that you are building up an incredible archive of all the decisions that different players have made and all the different kind of cities that people have built. I’m curious as to what you might be able to make or do with that kind of information. Are you mining it to see what kinds of mistakes people routinely make, or what sorts of urban forms are most popular? If so, is the audience for that information only in-house, for developing future versions of SimCity, or could you imagine sharing it with urban planners or real-life Mayors to offer an insight into popular urbanism?

Librande: It’s an interesting question. It’s hard to answer easily, though, because there are so many different ways players can play the game. The game was designed to cover as many different play patterns as we could think of, because our goal was to try to entertain as many of the different player demographics as we could.

Well, enough of all that. It's a beautiful day and time's a-wasting. We're off to go enjoy our weekend.

Stuff I'm reading on a May afternoon

Looking forward to that long weekend? Wishing you had something to read? Try some of these...

• What the State Birds Should Be

  Everyone knows that state birds are a big joke. There are a million cardinals, a scattering of robins, and just a general lack of thought put into the whole thing.

  States should have to put more thought into their state bird than I put into picking my socks in the morning.

• Networks, Crowds, and Markets: Reasoning About a Highly Connected World

  Networks, Crowds, and Markets combines different scientific perspectives in its approach to understanding networks and behavior. Drawing on ideas from economics, sociology, computing and information science, and applied mathematics, it describes the emerging field of study that is growing at the interface of all these areas, addressing fundamental questions about how the social, economic, and technological worlds are connected.

  The book is based on an inter-disciplinary course entitled Networks that we teach at Cornell.

• Bayesian Programming and Learning for Multi-Player Video Games: Application to RTS AI

  we review the current solutions to problems raised by multi-player game AI, by outlining the types of computational and cognitive complexities in the main gameplay types. From here, we sum up the transversal categories of problems, introducing how Bayesian modeling can deal with all of them. We then explain how to build a Bayesian program from domain knowledge and observations through a toy role-playing game example. In the second part of the thesis, we detail our application of this approach to RTS AI, and the models that we built up. For reactive behavior (micro-management), we present a real-time multi-agent decentralized controller inspired from sensory motor fusion. We then show how to perform strategic and tactical adaptation to a dynamic opponent through opponent modeling and machine learning (both supervised and unsupervised) from highly skilled players’ traces.

• Where in the World is Satoshi Nakamoto?

  Then, in April 2011, Satoshi said he had "moved on to other things." Satoshi has not been heard from since then. But the quest to find Satoshi continues. This is that story.

• When We Held Kings: The oral history of the 2003 World Series of Poker

  Poker went from a game understood by few and played in smoky backrooms to a television staple. In this 10th-anniversary oral history, more than 30 people who were part of the event explain what happened and what it meant for the poker business.

• A Virtual Weimar: Hyperinflation in a Video Game World

  A culmination of a series of unanticipated circumstances — and, finally, a most unfortunate programming bug — has over the last few weeks produced a new and unforeseen dimension of hellishness within Diablo 3: hyperinflation.

• Distributed Systems Reading Group reading list

  (note: we probably won’t read anything already covered here: http://pdos.csail.mit.edu/6.824/schedule.html)

• 6.824: Distributed Systems: Spring 2013

  It will present abstractions and implementation techniques for engineering distributed systems. Major topics include fault tolerance, replication, and consistency. Much of the class consists of studying and discussing case studies of distributed systems.

• 6.828: Operating System Engineering

  You will study, in detail, virtual memory, kernel and user mode, system calls, threads, context switches, interrupts, interprocess communication, coordination of concurrent activities, and the interface between software and hardware. Most importantly, you will study the interactions between these concepts, and how to manage the complexity introduced by the interactions.

• Learning NVP, Part 1: High-Level Architecture

  This blog post kicks off a new series of posts describing my journey to become more knowledgeable about the Nicira Network Virtualization Platform (NVP). NVP is, in my opinion, an awesome platform, but there hasn’t been a great deal of information shared about the product, how it works, how you configure it, etc.

• The Oldest Algorithmic Patent?

  While doing some research on cryptographic history, I stumbled on what may be one of the oldest algorithmic patents. That is, what is patented is an algorithm, rather than a physical, biological, or electronic device. It's US patent 1,356,546, filed on December 4, 1918, and issued on October 26, 1920.

Enjoy your Memorial Day weekend.

The intriguing theory of Expert Beginners

I've been enjoying Erik Dietrich's series of articles on the Daedtech blog about a phenomenon he calls the "Expert Beginner".

Dietrich's observation is that, while learning a new skill (such as computer programming, say), you might accidentally think that you had become an expert when in fact you were actually doing it all wrong. You just hadn't noticed that you were doing it wrong, and nobody was around to point that out to you.

For whatever reason, software appears to be a field that is particularly prone to this sort of Expert Beginner situation, perhaps because we software types are often learning things on our own, rather than having the opportunity to be taught from an established body of knowledge.

Not only is the Expert Beginner full of bad habits, but even worse: should he at some point encounter somebody who has greater expertise, and can help educate the Expert Beginner toward improvement, this assistance is likely to be met by resistance, as the Expert Beginner will have to start by un-learning all of his bad habits, which he is naturally reluctant to do (since they have served him well so far).

Dietrich is an entertaining writer and I recommend the essays:

• How Developers Stop Learning: Rise of the Expert Beginner

  The Expert Beginner has nowhere to go because progression requires an understanding that he has a lot of work to do, and that is not a readily available conclusion. You’ll notice that the Expert Beginner is positioned slightly above Advanced Beginner but not on the level of Competence. This is because he is not competent enough to grasp the big picture and recognize the irony of his situation, but he is slightly more competent than the Advanced Beginner due mainly to, well, extensive practice being a Beginner.

• How Software Groups Rot: Legacy of the Expert Beginner

  Perhaps it’s a lack of automated testing. Giant methods/classes. Lots of copy and paste coding. Use of outdated or poor tooling. Process. It can be any number of things, but the common thread is that you have a person or people in positions of authority that have the culturally lethal combination of not knowing much, not knowing what they don’t know, and assuming that, due to their own expertise, anything they don’t know isn’t worth knowing. This is a toxic professional culture in that it will force talented or ambitious people either to leave or to conform to mediocrity.

• How Stagnation is Justified: Language of the Expert Beginner

  The Expert Beginner believes that he and his ‘fellow’ Expert have a simple difference of opinion among ‘peers.’ While it may be true that one Expert speaks at conferences about source control best practices and the other one runs the IT for Bill’s Bait Shop and has never used source control, either opinion is just as valid.

• Up or Not: Ambition of the Expert Beginner

  An Expert Beginner’s entire career is built on a foundation of cognitive dissonance. Specifically, they believe that they are experts while outside observers (or empirical evidence) demonstrate that they are not. So an Expert Beginner is sentenced to a life of believing himself to be an expert while all evidence points to the contrary, punctuated by frequent and extremely unwelcome intrusions of that reality.

• Self-Correcting Organizations: Fall of the Expert Beginner

  Following the career arc of Expert Beginners is really quite sad. In the early stages, one feels annoyed and a little indignant at advancement by luck instead of competence. As things progress, real damage is caused by poor implementation and wrong-headed approaches, resulting for a lot of people in stress, frustration, failure, and at times even lost jobs and failed ventures. And, in the end, the fate of the one that caused these things is probably poetically just, but hard to find happiness in. A person ill-suited for a role assumed it, caused problems, and then suffered personal hardship. It’s not a great story.

To my mind, what this boils down to, for a person who's concerned with continuing to develop their own skills indefinitely, is: in order to improve my skills, I need to find somebody who can coach me.

You can always get better; you can always learn more. It's just a matter of finding somebody who can help you improve, and putting in the effort to learn from that person.

Just don't get stuck being an Expert Beginner.

Some benchmarking principles

Here's a nice slide deck on general benchmarking principles that I came across recently: Tokutek benchmarking principles

For hard-core perf types, there's not much new here, but I thought it was worth passing along in case some found it interesting.

Among my favorite observationss from the slide deck:

• benchmark frequently, to catch performance regressions soon after they occur
• keep a benchmark history, for analysis of data over time
• use graphs and plots to help with data interpretation
• share benchmarking data widely ("developers love data")

At various companies, at various times, I've worked with benchmarking teams, and it's nice to see the slow-but-steady "systematization of knowledge" going on here, since some of these are hard lessons and it's worth your time not to re-learn them yourself.

HotOS: Hot or not?

I didn't go to the biennial HotOS conference, which was held last week in New Mexico.

I am, however, extremely grateful to USENIX and to the sponsors and participants for making the technical session materials freely available to all.

There's a lot of dig through in the conference, and I'll share my thoughts on some of the work once I've had more of a chance to read it.

Meanwhile, Matt Welsh has stirred up a fair bit of discussion with his post-conference blog post: What I wish systems researchers would work on.

Of the 27 papers presented at the workshop, only about 2 or 3 would qualify as bold, unconventional, or truly novel research directions. The rest were basically extended abstracts of conference submissions that are either already in preparation or will be submitted in the next year or so. This is a perennial problem for HotOS, and when I chaired it in 2011 we had the same problem. So I can't fault the program committee on this one -- they have to work with the submissions they get, and often the "best" and most polished submissions represent the most mature (and hence less speculative) work.

Welsh has been involved with the conference for years; moreover, given his well-known move from member of the faculty of Harvard's Computer Science department to lead researcher at Google, he has a fascinating background in both the academic and industrial research arenas.

So his thoughts are worth considering.

And yet, I must say that I find Welsh's criticisms to ring hollow.

It doesn't seem right to critique a topic by whether it is brand new, or whether it "has been a recurring theme" that involves "problems that are 25 years old." In some ways, I think it is a mark of Computer Science's maturity that we've stopped completely changing our entire perspective every 18 months, and are instead returning to deep problems of enduring interest.

For example, the observation by an IBM research team that the last two decades have seen dramatic progress in network I/O performance but a dramatic lack of progress in storage I/O performance seems like a great topic for the operating systems community to be discussing. Why, just last week I was digging into continued research in the network I/O world, while the disk storage world appears to be still stuck in bloated, layered, horrifically complex implementations from gigantic enterprise companies that ladle on the complications while ratcheting up the price: it's not uncommon for an enterprise SAN device to cost 7 or 8 digits, as much as 1000 times the cost of the servers that are trying to process that imprisoned data.

And when Welsh outlines the areas he considers important, he notes that:

A typical Google-scale system involves many interacting jobs running very different software packages each with their own different mechanisms for runtime configuration: whether they be command-line flags, some kind of special-purpose configuration file (often in a totally custom ASCII format of some kind), or a fancy dynamically updated key-value store.

and also that:

the modes of interaction are vastly more complex and subtle than simply reasoning about state transitions and messages, in the abstract way that distributed systems researchers tend to cast things.

I couldn't agree more with Welsh's points, which makes me wonder how he reacted to the session from the RAMCloud team: Toward Common Patterns for Distributed, Concurrent, Fault-Tolerant Code. The RAMCloud developers ran smack into these problems, and felt that pain:

For example, when a server failure notification is received by a server, several of its segments are affected, and each affected segment replica can be in a different state. Some replicas may have an RPC outstanding to the failed server and must abort the RPC and restart replication elsewhere instead of expecting a response. Other affected replicas may not be consistent with their counterparts; such replicas require contacting the cluster coordinator before starting recreation in order to prevent inconsistencies.

Now, what the RAMCloud team propose is not so new or trendy; again, they look back two decades, to the dawn of object-oriented design, and the "patterns" approaches that proved so successful in the 1990's:

Although the implementations were different in many respects, we eventually noticed a common theme; each of these modules contained a set of rules that could trigger in any order. We gradually developed a pattern for DCFT code based on three layers: rules, tasks, and pools. This particular pattern has worked for a variety of problems in RAMCloud. We believe that this pattern, or something like it, might provide a convenient way of structuring DCFT modules in general.

It's important to look to the future, but it's important to learn from and build on the past. Some old techniques are sound, and we shouldn't jettison them just because they're old (of course, remember you're getting this from a 52-year-old systems programmer who still codes 10 hours a day in C!).

So keep pushing the envelope, but let's not excoriate those who try to help us learn from proven decades-old approaches and bring that wisdom to the problems of modern software.

The Legend of 1900: a very short review

Fifteen years late, we stumbled across The Legend of 1900.

I suspect that 1900 is the sort of movie that many people despise, and a few people really enjoy.

Count me on the "really enjoy" side. Just knowing that the lead character's name was

Danny Boodman T.D. Lemons Novecento

was probably enough to hook me.

Like any good fantasy, the movie is a parable: about war, about immigration, about society, and perhaps most of all, about the longing that drives people to travel. As 1900 says in a crucial monologue:

I think land people waste a lot of time wondering why. Winter comes and you can't wait for summer, summer comes and you never can wait for winter. That's why you never tire of traveling or chasing some place far away, where it's always summer.

I can't tell what sort of reaction you might have to this movie, but if you find yourself on a quiet evening looking for something just a bit unusual to watch, you might give The Legend of 1900 a try.