A few interesting articles

At least for the time being, the base-board installation upstairs is complete, so it's back to surfing the Internet...

• The Weird Stuff Warehouse is where old tech goes to retire

  Tucked neatly between Yahoo! headquarters and Lockheed Martin is a row of unmarked warehouses. To the common passersby, it's nothing more than an office park surrounded by perfectly coiffed lawns. But to those who are in on the secret, there's a place full of technology treasures waiting to be unearthed. It's called the Weird Stuff Warehouse and for more than 27 years it's been providing the Bay Area with a surplus of old and new technology.

• Making of White One

  A 4k intro is a executable file of at most 4 kilobytes (4096 bytes) that generates video and audio. That is, it puts something moving on your screen and something audible on your speakers. The finished product runs for a few minutes, has some coherent theme and design and ideally, sound and visual effects complement each other. On top of that, it's a technical challenge: It's impossible to store 3D models, textures or sound samples in 4 kilobytes, so you have to generate these things at runtime if you need them.

• Revenge, ego and the corruption of Wikipedia

  Taken all together, the edits strongly suggest a focused attempt to diminish Hannah’s legacy. But why? Who was Qworty and what axe did he have to grind with Hannah?

• With Ubiquity, Sears is Turning Shuttered Stores into Data Centers

  The first Ubiquity project will be a Sears store on the south side of Chicago, nestled alongside the Chicago Skyway. The 127,000 square foot store is closing at the end of June, and will be retrofitted as a multi-tenant data center. Farney says he already has a commitment for the first tenant at the site on East 79th Street, which has 5 megawatts of existing power capacity and the potential to expand.

• Finding peace in post-disaster Haiti

  In fact, Grand Rue melees were the exception, not the rule. Just as in New York after Sandy, responders and many journalists looking back on the postquake moment would highlight the lack of unrest in their after-action reports, often crediting their own presence, such as the UN adviser who told the Los Angeles Times: “There has been no rioting over food, and we avoided people dying of hunger or thirst. This is no small accomplishment.” As Auf der Heide has written, “Even when looting is not actually observed, that fact is often attributed to the extraordinary security measures that have been taken rather than the fact that such behavior is inherently uncommon.”

• ABC: Always Be Coding. I like:

  2. Be passionate. If you don’t care, then nobody else will. Be passionate about something. It might be programming, but what about it? Do you enjoy building compilers in your spare time? Do you build and fly RC helicopters? It doesn’t really matter because if you’re passionate about it, then you can make it interesting.

• Statistical Formulas For Programmers

  Most of these formulas can be found in Wikipedia, but others are buried in journal articles or in professors' web pages. They are all classical (not Bayesian), and to motivate them I have added concise commentary. I've also added links and references, so that even if you're unfamiliar with the underlying concepts, you can go out and learn more. Wearing a red cape is optional.

Buildings, rising and falling

Like almost every other engineer on the planet, I woke up this morning fascinated by the bridge collapse in Washington State.

It's interesting that it seems like the incident was at least partly a human factors issue: Skagit River bridge collapse: Looking for a temporary fix to get traffic moving

The bridge has a maximum clearance of about 17 feet – higher than the truck’s load in this case – but the clearance curves down to 14 feet 5 inches along the sides, where the collision occurred. The tractor-trailer was hauling drilling equipment southbound.

The article makes much of the idea of putting a "temporary" bridge in place while a replacement is studied. Well, all structures are temporary, of course; the question is just how long do we expect they will last?

A friend, traveling to Boston, remembered that we spent our first years together there, and sent us a picture of the beautiful Zakim Bunker Hill Bridge. It is indeed beautiful, though we've never seen it; when we lived in Boston we used to drive over the Charlestown High Bridge, which we called the "Fitzgerald Bridge", and which surely must be in the running for Ugliest Bridge Ever.

Getting back to the Skagit River Bridge collapse, any engineering failure is important and needs to be studied. Even if the issues here are pretty simple, in other cases they are more complex. Another Christian Science Monitor article points out that studying the decay of structures is a constant process: Collapse of I-5 bridge in Washington State: no fatalities, many questions

The bridge was not classified as structurally deficient, but a Federal Highway Administration database listed it as being "functionally obsolete" – a category meaning that the design is outdated, such as having narrow shoulders and low clearance underneath.

The bridge is also classified as "fracture critical" by the National Bridge Inventory. That means the bridge is designed so that a failure in any one part of the bridge can collapse the entire span. There are some 18,000 fracture-critical bridges nationwide, of which 8,000 are also "structurally deficient," according to a 2012 review of Federal Highway Administration records by Bloomberg News.

Some failures are far more complex and need much more analysis. In the case of the horrible disaster in Bangladesh a month ago, the investigation will probably take many months or even years: Bangladesh factory collapse probe uncovers abuses

The man in charge of the investigation, Mainuddin Khandker, told BBC Bangla on Wednesday that "extremely poor" construction materials were used in the building and said the report identified five causes of the collapse.

"A portion of the building was also constructed on land which had been a body of water before and was filled with rubbish," he told the Associated Press news agency.

The 400-page report was submitted to the government on Wednesday.

Closer to home, our own bridge is still under construction, yet already we are studying its decay: Lawmakers seek probe as doubts are raised on Bay Bridge opening

"This issue does not affect the safety, strength or the lifespan of the Skyway," Caltrans said in a written statement Monday. "... Corrosion of steel tendons inside the bridge was extremely limited and was addressed."

But the construction record shows that Caltrans managers did not address frequent warnings from the agency's bridge inspectors until it was too late to head off substantial corrosion. Thomas Devine, a UC Berkeley engineering professor and a leading metallurgist, disagreed with Caltrans claims that its examination of the issue proved that the corrosion was insignificant.

Numerous scientific and methodological errors made the Caltrans findings "meaningless" and "essentially useless," Devine told The Bee.

Like other lawmakers, Cannella expressed frustration about the slow trickle of information coming out of Caltrans. "It's frustrating to me, as a lawmaker, to keep getting my information from The Sacramento Bee," Cannella said.

Sometimes a structure collapses, but the story has a happy ending: The remarkable twist to the tale of Oklahoma's tornado-hit hospital

Surveying the shredded shell of the medical centre, the two-floor structure torn apart as if a bomb had struck, it is hard to believe that there were no casualties inside.

Yet the accounts of those who rode it out inside make the story of survival all the more extraordinary. They describe people lifted off their feet and carried tumbling down hallways by screaming gusts; the thump of cars smashing into the building; shards of doorways, picture frames and tiles turned into projectiles and hurtling past them; the air choked with dust; and that all-encompassing noise -- a jet engine meeting a freight train.

That nobody suffered more than cuts and bruises is down to a combination of detailed planning, calm heads, quick thinking, selfless bravery and plain luck.

Meanwhile, new buildings continue to be built, even when it seems like there's no need: Sky High and Going Up Fast: Luxury Towers Take New York

Ultraluxury housing and construction is booming across Manhattan, which is now beginning to rival London in popularity with the world’s wealthy. The number of condominium buildings in the borough with apartments selling for more than $15 million has risen to 49, up from 33 in 2009, according to CityRealty.

...

As with many of these buildings, only about a quarter of the units will be occupied at any one time.

Software, of course, doesn't decay (although disk drives and flash memory do wear out, a phenomenon that we call "bit rot", though that's a gross over-simplification). But software does undergo constant change, which can introduce new problems even as we fix the old ones. So we software engineers love to study how other fields of engineering deal with failures.

And, of course, modern architecture and engineering design wouldn't be possible without software, since nearly everything is simulated and analyzed ahead of time on computers. Even software that was once considered to be just a toy is now used for serious architectural study: SimCity: An Interview with Stone Librande

Manaugh: Now that the game is out in the world, and because of the central, online hosting of all the games being played right now, I have to imagine that you are building up an incredible archive of all the decisions that different players have made and all the different kind of cities that people have built. I’m curious as to what you might be able to make or do with that kind of information. Are you mining it to see what kinds of mistakes people routinely make, or what sorts of urban forms are most popular? If so, is the audience for that information only in-house, for developing future versions of SimCity, or could you imagine sharing it with urban planners or real-life Mayors to offer an insight into popular urbanism?

Librande: It’s an interesting question. It’s hard to answer easily, though, because there are so many different ways players can play the game. The game was designed to cover as many different play patterns as we could think of, because our goal was to try to entertain as many of the different player demographics as we could.

Well, enough of all that. It's a beautiful day and time's a-wasting. We're off to go enjoy our weekend.

Stuff I'm reading on a May afternoon

Looking forward to that long weekend? Wishing you had something to read? Try some of these...

• What the State Birds Should Be

  Everyone knows that state birds are a big joke. There are a million cardinals, a scattering of robins, and just a general lack of thought put into the whole thing.

  States should have to put more thought into their state bird than I put into picking my socks in the morning.

• Networks, Crowds, and Markets: Reasoning About a Highly Connected World

  Networks, Crowds, and Markets combines different scientific perspectives in its approach to understanding networks and behavior. Drawing on ideas from economics, sociology, computing and information science, and applied mathematics, it describes the emerging field of study that is growing at the interface of all these areas, addressing fundamental questions about how the social, economic, and technological worlds are connected.

  The book is based on an inter-disciplinary course entitled Networks that we teach at Cornell.

• Bayesian Programming and Learning for Multi-Player Video Games: Application to RTS AI

  we review the current solutions to problems raised by multi-player game AI, by outlining the types of computational and cognitive complexities in the main gameplay types. From here, we sum up the transversal categories of problems, introducing how Bayesian modeling can deal with all of them. We then explain how to build a Bayesian program from domain knowledge and observations through a toy role-playing game example. In the second part of the thesis, we detail our application of this approach to RTS AI, and the models that we built up. For reactive behavior (micro-management), we present a real-time multi-agent decentralized controller inspired from sensory motor fusion. We then show how to perform strategic and tactical adaptation to a dynamic opponent through opponent modeling and machine learning (both supervised and unsupervised) from highly skilled players’ traces.

• Where in the World is Satoshi Nakamoto?

  Then, in April 2011, Satoshi said he had "moved on to other things." Satoshi has not been heard from since then. But the quest to find Satoshi continues. This is that story.

• When We Held Kings: The oral history of the 2003 World Series of Poker

  Poker went from a game understood by few and played in smoky backrooms to a television staple. In this 10th-anniversary oral history, more than 30 people who were part of the event explain what happened and what it meant for the poker business.

• A Virtual Weimar: Hyperinflation in a Video Game World

  A culmination of a series of unanticipated circumstances — and, finally, a most unfortunate programming bug — has over the last few weeks produced a new and unforeseen dimension of hellishness within Diablo 3: hyperinflation.

• Distributed Systems Reading Group reading list

  (note: we probably won’t read anything already covered here: http://pdos.csail.mit.edu/6.824/schedule.html)

• 6.824: Distributed Systems: Spring 2013

  It will present abstractions and implementation techniques for engineering distributed systems. Major topics include fault tolerance, replication, and consistency. Much of the class consists of studying and discussing case studies of distributed systems.

• 6.828: Operating System Engineering

  You will study, in detail, virtual memory, kernel and user mode, system calls, threads, context switches, interrupts, interprocess communication, coordination of concurrent activities, and the interface between software and hardware. Most importantly, you will study the interactions between these concepts, and how to manage the complexity introduced by the interactions.

• Learning NVP, Part 1: High-Level Architecture

  This blog post kicks off a new series of posts describing my journey to become more knowledgeable about the Nicira Network Virtualization Platform (NVP). NVP is, in my opinion, an awesome platform, but there hasn’t been a great deal of information shared about the product, how it works, how you configure it, etc.

• The Oldest Algorithmic Patent?

  While doing some research on cryptographic history, I stumbled on what may be one of the oldest algorithmic patents. That is, what is patented is an algorithm, rather than a physical, biological, or electronic device. It's US patent 1,356,546, filed on December 4, 1918, and issued on October 26, 1920.

Enjoy your Memorial Day weekend.

The intriguing theory of Expert Beginners

I've been enjoying Erik Dietrich's series of articles on the Daedtech blog about a phenomenon he calls the "Expert Beginner".

Dietrich's observation is that, while learning a new skill (such as computer programming, say), you might accidentally think that you had become an expert when in fact you were actually doing it all wrong. You just hadn't noticed that you were doing it wrong, and nobody was around to point that out to you.

For whatever reason, software appears to be a field that is particularly prone to this sort of Expert Beginner situation, perhaps because we software types are often learning things on our own, rather than having the opportunity to be taught from an established body of knowledge.

Not only is the Expert Beginner full of bad habits, but even worse: should he at some point encounter somebody who has greater expertise, and can help educate the Expert Beginner toward improvement, this assistance is likely to be met by resistance, as the Expert Beginner will have to start by un-learning all of his bad habits, which he is naturally reluctant to do (since they have served him well so far).

Dietrich is an entertaining writer and I recommend the essays:

• How Developers Stop Learning: Rise of the Expert Beginner

  The Expert Beginner has nowhere to go because progression requires an understanding that he has a lot of work to do, and that is not a readily available conclusion. You’ll notice that the Expert Beginner is positioned slightly above Advanced Beginner but not on the level of Competence. This is because he is not competent enough to grasp the big picture and recognize the irony of his situation, but he is slightly more competent than the Advanced Beginner due mainly to, well, extensive practice being a Beginner.

• How Software Groups Rot: Legacy of the Expert Beginner

  Perhaps it’s a lack of automated testing. Giant methods/classes. Lots of copy and paste coding. Use of outdated or poor tooling. Process. It can be any number of things, but the common thread is that you have a person or people in positions of authority that have the culturally lethal combination of not knowing much, not knowing what they don’t know, and assuming that, due to their own expertise, anything they don’t know isn’t worth knowing. This is a toxic professional culture in that it will force talented or ambitious people either to leave or to conform to mediocrity.

• How Stagnation is Justified: Language of the Expert Beginner

  The Expert Beginner believes that he and his ‘fellow’ Expert have a simple difference of opinion among ‘peers.’ While it may be true that one Expert speaks at conferences about source control best practices and the other one runs the IT for Bill’s Bait Shop and has never used source control, either opinion is just as valid.

• Up or Not: Ambition of the Expert Beginner

  An Expert Beginner’s entire career is built on a foundation of cognitive dissonance. Specifically, they believe that they are experts while outside observers (or empirical evidence) demonstrate that they are not. So an Expert Beginner is sentenced to a life of believing himself to be an expert while all evidence points to the contrary, punctuated by frequent and extremely unwelcome intrusions of that reality.

• Self-Correcting Organizations: Fall of the Expert Beginner

  Following the career arc of Expert Beginners is really quite sad. In the early stages, one feels annoyed and a little indignant at advancement by luck instead of competence. As things progress, real damage is caused by poor implementation and wrong-headed approaches, resulting for a lot of people in stress, frustration, failure, and at times even lost jobs and failed ventures. And, in the end, the fate of the one that caused these things is probably poetically just, but hard to find happiness in. A person ill-suited for a role assumed it, caused problems, and then suffered personal hardship. It’s not a great story.

To my mind, what this boils down to, for a person who's concerned with continuing to develop their own skills indefinitely, is: in order to improve my skills, I need to find somebody who can coach me.

You can always get better; you can always learn more. It's just a matter of finding somebody who can help you improve, and putting in the effort to learn from that person.

Just don't get stuck being an Expert Beginner.

Some benchmarking principles

Here's a nice slide deck on general benchmarking principles that I came across recently: Tokutek benchmarking principles

For hard-core perf types, there's not much new here, but I thought it was worth passing along in case some found it interesting.

Among my favorite observationss from the slide deck:

• benchmark frequently, to catch performance regressions soon after they occur
• keep a benchmark history, for analysis of data over time
• use graphs and plots to help with data interpretation
• share benchmarking data widely ("developers love data")

At various companies, at various times, I've worked with benchmarking teams, and it's nice to see the slow-but-steady "systematization of knowledge" going on here, since some of these are hard lessons and it's worth your time not to re-learn them yourself.

HotOS: Hot or not?

I didn't go to the biennial HotOS conference, which was held last week in New Mexico.

I am, however, extremely grateful to USENIX and to the sponsors and participants for making the technical session materials freely available to all.

There's a lot of dig through in the conference, and I'll share my thoughts on some of the work once I've had more of a chance to read it.

Meanwhile, Matt Welsh has stirred up a fair bit of discussion with his post-conference blog post: What I wish systems researchers would work on.

Of the 27 papers presented at the workshop, only about 2 or 3 would qualify as bold, unconventional, or truly novel research directions. The rest were basically extended abstracts of conference submissions that are either already in preparation or will be submitted in the next year or so. This is a perennial problem for HotOS, and when I chaired it in 2011 we had the same problem. So I can't fault the program committee on this one -- they have to work with the submissions they get, and often the "best" and most polished submissions represent the most mature (and hence less speculative) work.

Welsh has been involved with the conference for years; moreover, given his well-known move from member of the faculty of Harvard's Computer Science department to lead researcher at Google, he has a fascinating background in both the academic and industrial research arenas.

So his thoughts are worth considering.

And yet, I must say that I find Welsh's criticisms to ring hollow.

It doesn't seem right to critique a topic by whether it is brand new, or whether it "has been a recurring theme" that involves "problems that are 25 years old." In some ways, I think it is a mark of Computer Science's maturity that we've stopped completely changing our entire perspective every 18 months, and are instead returning to deep problems of enduring interest.

For example, the observation by an IBM research team that the last two decades have seen dramatic progress in network I/O performance but a dramatic lack of progress in storage I/O performance seems like a great topic for the operating systems community to be discussing. Why, just last week I was digging into continued research in the network I/O world, while the disk storage world appears to be still stuck in bloated, layered, horrifically complex implementations from gigantic enterprise companies that ladle on the complications while ratcheting up the price: it's not uncommon for an enterprise SAN device to cost 7 or 8 digits, as much as 1000 times the cost of the servers that are trying to process that imprisoned data.

And when Welsh outlines the areas he considers important, he notes that:

A typical Google-scale system involves many interacting jobs running very different software packages each with their own different mechanisms for runtime configuration: whether they be command-line flags, some kind of special-purpose configuration file (often in a totally custom ASCII format of some kind), or a fancy dynamically updated key-value store.

and also that:

the modes of interaction are vastly more complex and subtle than simply reasoning about state transitions and messages, in the abstract way that distributed systems researchers tend to cast things.

I couldn't agree more with Welsh's points, which makes me wonder how he reacted to the session from the RAMCloud team: Toward Common Patterns for Distributed, Concurrent, Fault-Tolerant Code. The RAMCloud developers ran smack into these problems, and felt that pain:

For example, when a server failure notification is received by a server, several of its segments are affected, and each affected segment replica can be in a different state. Some replicas may have an RPC outstanding to the failed server and must abort the RPC and restart replication elsewhere instead of expecting a response. Other affected replicas may not be consistent with their counterparts; such replicas require contacting the cluster coordinator before starting recreation in order to prevent inconsistencies.

Now, what the RAMCloud team propose is not so new or trendy; again, they look back two decades, to the dawn of object-oriented design, and the "patterns" approaches that proved so successful in the 1990's:

Although the implementations were different in many respects, we eventually noticed a common theme; each of these modules contained a set of rules that could trigger in any order. We gradually developed a pattern for DCFT code based on three layers: rules, tasks, and pools. This particular pattern has worked for a variety of problems in RAMCloud. We believe that this pattern, or something like it, might provide a convenient way of structuring DCFT modules in general.

It's important to look to the future, but it's important to learn from and build on the past. Some old techniques are sound, and we shouldn't jettison them just because they're old (of course, remember you're getting this from a 52-year-old systems programmer who still codes 10 hours a day in C!).

So keep pushing the envelope, but let's not excoriate those who try to help us learn from proven decades-old approaches and bring that wisdom to the problems of modern software.

The Legend of 1900: a very short review

Fifteen years late, we stumbled across The Legend of 1900.

I suspect that 1900 is the sort of movie that many people despise, and a few people really enjoy.

Count me on the "really enjoy" side. Just knowing that the lead character's name was

Danny Boodman T.D. Lemons Novecento

was probably enough to hook me.

Like any good fantasy, the movie is a parable: about war, about immigration, about society, and perhaps most of all, about the longing that drives people to travel. As 1900 says in a crucial monologue:

I think land people waste a lot of time wondering why. Winter comes and you can't wait for summer, summer comes and you never can wait for winter. That's why you never tire of traveling or chasing some place far away, where it's always summer.

I can't tell what sort of reaction you might have to this movie, but if you find yourself on a quiet evening looking for something just a bit unusual to watch, you might give The Legend of 1900 a try.

Gluttony

Definition: Fringe Binge. A Fringe Binge is what happens when Netflix releases Season Four of Fringe on streaming, and so you watch nine episodes in three nights.

It means your daily conversations are punctuated with fragments like:

Wait, hon, I missed something: which alternate universe is this?

Wild: a very short review

Cheryl Strayed's Wild: From Lost to Found on the Pacific Crest Trail is an unusual book.

It is a memoir, written 17 years after the fact, of a truly life-changing event.

In 1995, with her life in a shambles (a situation so horrible that the oft-overused "hit rock bottom" actually seems appropriate), Strayed decided to take action, and set out to turn the page, embarking upon a solo backpacking trip:

It was a world I'd never been to and yet had known was there all along, one I'd staggered to in sorrow and confusion and fear and hope. A world I thought would both make me into the woman I knew I could become and turn me back into the girl I'd once been. A world that measured two feet wide and 2,663 miles long.

A world called the Pacific Crest Trail.

That's a tall order for a trail to fill.

However, I'm lucky enough to have walked several portions of the Pacific Crest Trail myself, and have even camped along some of its highest and most remote stretches, and I can confirm: it is no ordinary trail.

Strayed's book is startlingly raw and honest. She hides nothing, reveals all, lets us deep down into her torments and the agonies she felt as she struggled along. Before beginning her trip, she had never backpacked a day in her life:

I'd gone to an outdoor store in Minneapolis called REI about a dozen times over the previous months to purchase a good portion of these items.

...

By the time I made the decision about which backpack to purchase -- a top-of-the-line Gregory hybrid external frame that claimed to have the balance and agility of an internal -- I felt as if I'd become a backpacking expert.

She hadn't though; it's not the sort of thing you can learn in a store, from a book, or by listening to people, and it's not long before the realities of the trail begin to pile up and she is forced to adapt. But she is so remarkably determined and focused that she simply wills herself to overcome obstacles:

I knew that if I allowed fear to overtake me, my journey was doomed. Fear, to a great extent, is born of a story we tell ourselves, and so I chose to tell myself a different story from the one women are told. I decided I was safe. I was strong. I was brave. Nothing could vanquish me. Insisting on this story was a form of mind control, but for the most part, it worked.

And so she goes, walking along, changing her life:

I had to change was the thought that drove me in those months of planning. Not into a different person, but back to the person I used to be -- strong and responsible, clear-eyed and driven, ethical and good. And the PCT would make me that way. There, I'd walk and think about my entire life. I'd find my strength again, far from everything that had made my life ridiculous.

It's a great plan, but she soon finds out that the trail doesn't care what she's thinking:

As I hiked, I moaned again and again, as if that would provide some cooling relief, but nothing changed. The sun still stared ruthlessly down on me, not caring one iota whether I lived or died. The parched scrub and scraggly trees still stood indifferently resolute, as they always had and always would.

Though barrier after barrier confronts her, she surmounts each one, and slowly her self-chosen therapy begins to bear fruit:

Uncertain as I was as I pushed forward, I felt right in my pushing, as if the effort itself meant something. That perhaps being amidst the undesecrated beauty of the wilderness meant I too could be undesecrated, regardless of what I'd lost or what had been taken from me, regardless of the regrettable things I'd done to others or myself or the regrettable things that had been done to me. Of all the things I'd been skeptical about, I didn't feel skeptical about this: the wilderness had a clarity that included me.

The wilderness includes not just clarity, but also a different scale, and a different sense of time:

Foot speed was a profoundly different way of moving through the world than my normal modes of travel. Miles weren't things that blazed dully past. They were long, intimate straggles of weeds and clumps of dirt, blades of grass and flowers that bent in the wind, trees that lumbered and screeched. They were the sound of my breath and my feet hitting the trail one step at a time and the click of my ski pole. The PCT had taught me what a mile was.

The PCT, indeed, has much to teach (as I said, it is no ordinary trail), and Strayed is eager to soak it all up, as she follows in the footsteps of the pioneering naturalists who worked to save and create the trail:

It had only to do with how it felt to be in the wild. With what it was like to walk for miles for no reason other than to witness the accumulation of trees and meadows, mountains and deserts, streams and rocks, rivers and grasses, sunrises and sunsets. The experience was powerful and fundamental. It seemed to me that it had always felt like this to be a human in the wild, and as long as the wild existed it would always feel this way. That's what Montgomery knew, I supposed. And what Clarke knew and Rogers and what thousands of people who preceded and followed them knew. It was what I knew before I even really did, before I could have known how truly hard and glorious the PCT would be, how profoundly the trail would both shatter and shelter me.

Like the Pacific Crest Trail, Cheryl Strayed's Wild is powerful and fundamental, and it will both shatter and shelter you. You may not be a backpacker; you may not be a woman in the depths of despair; but if you take the time to enter Strayed's world, and go with her on her adventure, you will enjoy it, you will learn from it, and you won't regret a moment of the time you spent seeing the world through her eyes.

C10M Resources

So I happened to spend some time listening to this video today, which led me to: C10M

Scalability. It's the question that drives us. It's an anomaly inherent to the programming of the matrix.

The essays were interesting; I'm looking forward to more of his articles.

The articles Graham has already written led me to Packet Processing on Intel® Architecture

Packet processing while executing other workloads on an Intel processor reduces hardware costs, simplifies the application development environment, and reduces time to market. The Intel DPDK is also playing a critical role in Software-Defined Networking (SDN) and Network Functions Virtualization (NFV)

I also thought that Performance Optimization might be pretty interesting, but attempting to read more kept running me into "register with our marketing department" pages...

Anyway, it was interesting enough that I thought I'd pass it all along and see what sort of "yeah, that's cool, but check *this* out!" sort of feedback I got.

Andrew Simpson

The investigations into what happened last Thursday continue, and it will undoubtedly be weeks or months before we learn much more about the details, but what we know now is this: while practicing on the San Francisco Bay, just after 1:00 PM, the Artemis Racing AC72 "Big Red" capsized, and champion sailor Andrew Simpson died.

Wired summarizes the event:

Yesterday afternoon, both Artemis and Cup defender Team Oracle were on San Francisco Bay for a practice run. The teams stayed close together — they weren’t racing, but paying close attention to how the boats stacked up in similar winds. At 1 p.m., the boat neared the center of the triangle created by Angel Island, Alcatraz, and Treasure Island. During a bear away, the same common but tricky maneuver that led Oracle’s AC72 to capsize in October, Artemis’ boat went into a nosedive.

Although sailing is a very dangerous sport, it is an extremely rare occurrence for a top-level professional athlete to have a fatal accident while practicing his sport; it is hard to overstate what a shock it is for this to occur.

The Artemis Racing team is superb; they are well-managed and well-trained, they have the very best equipment available, and they are extremely careful about what they do. I'm biased, of course, since the team is located in my hometown, but I'm not lying: the best sailors in the world were out on the bay last Thursday, sailing the fastest boats on the planet in the best sailing spot on the planet.

But I've been out on the central bay at 1:00 PM on a May afternoon, and it can be a terrifying and intimidating place. In a great piece a few years ago, Captain Ray Wichmann of Olympic Circle explains the meteorology behind "the slot":

This San Francisco Bay summer wind is the result of a unique geographical phenomenon. At 38˚ north of the Equator, the Bay Area is in the path of the prevailing westerlies. This means that most of the time the wind comes from the west. Just 60 miles inland from the Bay is the Great Central Valley, with its extremely high summer temperatures. From Bakersfield in the south to Redding in the north, afternoon summer temperatures often reach 110˚F and sometimes even higher. This is the only place on earth that I know of where you can have a 50˚F temperature difference within 50 miles, without any elevation change. It can be 55˚F in San Francisco and, at the same time, 105˚F in Vacaville.

As this very hot air rises, cooler air is pulled in to replace it. The hotter the afternoon temperature in the Central Valley, the more rapidly the heated air rises and the stronger the inward air flow. The Sierra Nevada Mountains, a 10,000 foot wall to the east, make it unlikely the replacement air will come from that direction. There are, however, many low places between the Central Valley and the Pacific Ocean through which this inrushing air can pass easily. Only the Golden Gate, however, goes all the way down to sea level. Because of this, the Gate is the funnel for the primary source of replacement air for the Central Valley. Since air is fluid, it accelerates as it is forced to squeeze through narrow openings. As this cooler replacement air squeezes through the Golden Gate, it is accelerated directly down ‘The Slot, setting up the Bay’s exhilarating winds of summer.

The Independent reports that several investigations are underway:

The first will be led by the San Francisco police, who investigated an April 2012 accident in a race off California and Mexico, when four crew of a 37-foot Aegean yacht died.

A second investigation will be led by the America's Cup director of racing, the Australian Iain Murray, a former America's Cup yacht designer and skipper.

The San Francisco Chronicle includes some additional details about the Murray investigation:

"It appears Bart was trapped under some of the solid sections of the yacht," Murray said. "Unfortunately, they were unable to retrieve him, and he passed."

Murray said each crew member was equipped with oxygen, a helmet and an inflatable lifejacket. He said he wasn't sure what safety gear Simpson was using or if he was tethered to the boat at the time of the crash.

San Francisco officials said Simpson was stuck under water for about 10 minutes before he was pulled onto a support boat. There, city firefighters attempted CPR but were unable to revive him.

Nothing that can happen now will bring Simpson back, but I hope that the investigations will be productive, and lead to improvements in gear, in technique, in preparation, and, eventually, in safety. I'll keep an eye on these investigations to see how they turn out.

Weekend reading

Happy Mother's Day weekend!

In between whatever else is going on in your life, if you're looking for some things to read, try some of these:

• My Mom told me that git doesn't scale. This is easily the best technical talk I've watched in 2013. A little bit of naughty language, but we're all grownups, yes?

  With over 2 million and a half repositories, GitHub is the world's largest source code host. Since day one, we've faced an unique engineering problem: making terabytes of Git data always available, either directly or through our website. This talk offers a hopefully insightful view into the internals of Git, the way its original design affects our scalable architecture, and the many things we've learnt while solving this fascinating problem.

• The Prophets of Oak Ridge

  When she and Michael retell their story, which they do often to eager foot soldiers of social justice groups, they put it in tidy parable form on PowerPoint: We saw injustice at Y-12, so we broke in to bring truth and attract the world’s attention, and here is the U.S. government’s seven-decade budget for nuclear-weapons infrastructure.

• WIRED 20th Anniversary

  On this, our 20th anniversary, the time has come to reflect on this generation of leaders, thinkers, and makers. These people, their companies, and their ideas have shaped the future we live in today. Below, we've gathered stories for, by, and about the people who have shaped the planet's past 20 years—and will continue driving the next.

• Depression Part Two. Amazing. There should be a Pulitzer Prize for web writing, and this article should win it.

  I'll just say this: Nobody can guarantee that it's going to be okay, but — and I don't know if this will be comforting to anyone else — the possibility exists that there's a piece of corn on a floor somewhere that will make you just as confused about why you are laughing as you have ever been about why you are depressed.

• Historical maps overlaid on Google Maps

  View the maps from the David Rumsey Historical Map Collection overlaid on their locations on Google Maps.

• How To Survive a Ground-Up Rewrite Without Losing Your Sanity

  If you're considering launching a major rewrite, or find yourself as the tech lead on such a project in flight, or are merely toiling in the trenches of such a project, hoping against hope that it will someday end... this post is for you.

• Buridan's Principle

  I had no idea where to publish the paper, so I let it drop. In 2011, a reader, Thomas Ray, suggested submitting it to Foundations of Physics. I did, and it was accepted. For the published version, I added a concluding section mentioning some of the things that had happened in the 25 years since I wrote the original version and citing this entry as a source of information about the paper's history.

• How the Syrian Electronic Army Hacked The Onion

  From examining the details of this incident, as well as those effecting the AP, Guardian and others, it’s clear that the SEA is not using complex methods of attack. All of the hacks so far have been a result of simple phishing, or possibly dictionary attacks—all of which are preventable with a few simple security measures.

• This Is What One Half Second of High Speed Trading Looks Like

  Each box represents one exchange. The SIP (CQS in this case) is the box at 6 o’clock. It shows the National Best Bid/Offer. Watch how much it changes in a fraction of a second. The shapes represent quote changes which are the result of a change to the top of the book at each exchange. The time at the bottom of the screen is Eastern Time HH:MM:SS:mmm (mmm = millisecond). We slow time down so you can see what goes on at the millisecond level. A millisecond (ms) is 1/1000th of a second.

Bruce Sterling lights it up

One of the best things I read in April was the transcript of Bruce Sterling's closing remarks at the 2013 South By South West conference.

Sterling is always a fascinating and insightful observer of modern society, and he really was on target this time.

Sterling starts by taking the long view:

I’m out in Arizona visiting a certain desert canyon. I always wanted to see it: Walnut Canyon National Monument.

Walnut Canyon, an extremely Southwestern place. It happened to have a little civilization in it once, from about 1100 A.D. to maybe 1250 A.D. The most high-tech guys in the Southwest.

Now the interesting thing about these ancient cliff-dweller guys is that they were much, much more high tech than South By South West.

He contemplates the events at Walnut Canyon in the context of the passing of the "Personal" computer:

Then we look back in nostalgia at the Personal Computer world. It’s not that we were forced out of our stone boxes in the canyon. We weren’t driven away by force. We just mysteriously left. It was like the waning of the moon.

They were too limiting, somehow. They computed, but they just didn’t do enough for us. They seemed like a fantastic way forward, but somehow they were actually getting in the way of our experience.

All these machines that tore us away from lived experience, and made us stare into the square screens or hunch over the keyboards, covered with their arcane, petroglyph symbols. Control Dingbat That, backslash R M this. We never really understood that. Not really.

Sterling's point is that he's trying to understand the social aspects of technological change:

Computers were really, truly disruptive. Mobile devices are so radically disruptive that they even disrupted computers. They’re a bigger deal then the dead bookstores. We’ve got guys who own cell phones in this world who can’t even read.

And I’m very intimate with this spectacle. I’m very keen on all its little ins and outs.

The thing that bugs me about your attitude toward it is that you don’t recognize its tragic dimension.

This is something that literature has always been very keen on, that technology never gets around to acknowledging. The cold wind moaning through the empty stone box.

One of Sterling's points is that human beings should be aiming for quality rather than quantity:

A crapject is what happens when you give somebody access to the cheap means of production, and they just start LOLCatting with physical objects. They’re just emitting jokes as real things, you know? And they’re like crap, and they can’t be sold, and they basically have the same value as any other content on the Internet. They just happen to have been made of plastic, or birchwood, or foamcore, or concrete, or whatever you’ve managed to drag into the means of production.

And his other point is that technologists need to have a more developed system of ethics:

When you disrupt the stone box, the stone box goes empty. It’s not merely irritated or disturbed, it’s dead. It’s dead media. It’s dead, it has been killed, and to be a phoenix you have to admit your complicity in the barbecue fire.

It’s your fire, it’s not somebody else’s. Yes, we killed the past. We didn’t pull the trigger on it directly, but it died for our benefit, it died through things we did.

Own up to that. Own up to that: yes, we burned it up. No one is historically innocent. Yes, we are carnivores at this barbecue. Yes, it died, we roasted it, we ate it.

You may not agree with everything Sterling says; I'm certainly not a complete believer.

But his points are powerful, well-considered, and forcefully made.

You won't regret spending a few minutes reading his entire essay, or listening to the speech.

Tinker Tailor Soldier Spy (2011): a very short review

We watched the 2011 film adaptation of LeCarre's Tinker Tailor Soldier Spy.

So brilliantly acted! So beautifully filmed! So carefully adapted!

So incredibly dull.

Sigh.

Happily, with Netflix, you send that one back and try another. Perhaps another day it will appeal more. Perhaps one day I will care enough to read the book first.

Or perhaps not.

The Theoretical Minimum: a very short review

I plowed through Susskind and Hrabovsky's The Theoretical Minimum: What You Need to Know to Start Doing Physics over the last month.

Here's the publisher's blurb:

The Theoretical Minimum is a book for anyone who has ever regretted not taking physics in college—or who simply wants to know how to think like a physicist. In this unconventional introduction, physicist Leonard Susskind and hacker-scientist George Hrabovsky offer a first course in physics and associated math for the ardent amateur. Unlike most popular physics books—which give readers a taste of what physicists know but shy away from equations or math—Susskind and Hrabovsky actually teach the skills you need to do physics, beginning with classical mechanics, yourself. Based on Susskind’s enormously popular Stanford University-based (and YouTube-featured) continuing-education course, the authors cover the minimum—the theoretical minimum of the title—that readers need to master to study more advanced topics.

An alternative to the conventional go-to-college method, The Theoretical Minimum provides a tool kit for amateur scientists to learn physics at their own pace.

The publisher's description is accurate, and the book delivers on the promise. It is well-written, clear, beautifully typeset, and covers the topics it sets out to cover.

The problem with this book, I think, is similar to that faced by the MOOC techniques (Udacity, etc.): it is simply very hard for students, even extremely motivated students, to master this material by reading a book. To really learn to do physics, you need to:

• Hear it explained, in a variety of ways, from a variety of perspectives
• Ask questions; stop and get a tough point explained in a different manner when the first explanation didn't click
• Discuss the topics with other students
• Attempt homework problems on your own; have them graded; return to the questions you got wrong, attempt them again, and continue until you can produce the solution yourself.
• See examples of how to apply these techniques to a variety of problems

And so forth.

Perhaps a reasonable approach would be to combine a text such as The Theoretical Minimum with Professor Susskind's video lectures, then blend in something like Professor Michael Scatz's Physics I class on Coursera, and also add in a more in-depth textbook for additional substance.

But time is short; I enjoyed reading Susskind and Hrabovsky, and I'm now moving on to whatever piques my interest next.

Gaming

I'm having an absolute blast playing The Cave with my grand-daughter. I'm sure it can't be easy to design a video game that 52-year-olds and 9-year-olds agree upon, but somehow Ron Gilbert and the team have done so.

Meanwhile, I'm really enjoying the series of articles that have been appearing in the New York Times over the last few months about video games and the gaming industry:

• The Nerd as Auteur in BioShock Infinite

  The word “care,” however, doesn’t come close to capturing the effort that went into BioShock Infinite, an unusually cinematic shooting adventure set in a floating city in which the ultimate goal is to rescue a mysterious young woman. Mr. Levine and a team of 200 have been toiling on the game, the third BioShock installment, for more than four years.

• Same Story, Different Screens: A Video Game That’s Also a Series

  In an unusual arrangement, neither the show nor the video game is a spinoff of the other. Instead, in a process that took five years and three contracts between Syfy and Trion Worlds, the fictional world of Defiance is a collaboration between the companies. The series and the game were developed simultaneously. Each is designed to be a stand-alone piece of entertainment. Yet they also intersect. Events in the show will influence the game. And events in the game will influence the show.

• Thrown for a Curve in Rhode Island

  “It just felt really good, when this all started, to have the sexy sports celebrity from Boston who seemed to like Rhode Island and showed up in Rhode Island, and who built this exotic new business, even though no one knew what it was,” says the historian Ted Widmer, who grew up in Providence and works at Brown. “It seemed like the digital economy, or biotech, or whatever. But then it turned out that it wasn’t the new digital economy. It was some 13-year-old’s medieval fantasy.”

• Where the Artists Are the Superheroes

  The company is the world’s fourth-largest game maker, ranked by sales, after Activision Blizzard, Electronic Arts and Nintendo, according to Michael Pachter, an industry analyst at Wedbush Securities. But Ubisoft’s video game production studio in Montreal, where Mr. Measroch works, is one of the world’s largest, with a staff of 2,500. It is here where an overwhelmingly male staff of writers, producers, coders, directors, animators, artists and others come together to create the fantasy worlds of games like Assassin’s Creed and Far Cry, into which millions of people escape.

All these articles are good, and worth reading, and there are more linked from the top-level page above.

I hope the Times continues this recent activity of publishing well-written, well-researched, enlightening articles about an industry that many are completely unaware of.

Meanwhile, since I'm on this topic, let me also point to The quest for Shadow of the Colossus' last big secret

Ozzy turned to forums filled with likeminded, passionate devotees. At first he was looking for canny ways to defeat colossi faster, but he soon discovered something far more mesmerising. The hidden garden at the top of the Shrine of Worship, glimpsed during Shadow of the Colossus' final cutscene, was accessible in-game. This is Shadow Of The Colossus' biggest Easter Egg, teased by the mossy growths, handholds and ledges that weave around the exterior of the structure, but not actually reachable until you've completed the game multiple times. The Secret Garden, as it became known, is a final reward for the most dedicated of colossi-hunters: one last challenge and a glimpse of verdant green beauty in a starkly austere land. But it wasn't enough for Ozzymandias and his fellow fans.

"Are there other places you can get to?" Ozzymandias wanted to know. "What else is there?" he asked. "What else can you do?"

Well-built games have always inspired this level of devotion; I remember obsessing over hidden corners of The Bard's Tale back in 1985. And, hey, Bethesda, what's next?

But for now, the weekend is nearly here, and that means my grand-daughter will be over and we'll be playing The Cave...

Dune, the mini-series: a very short review

Growing up, Frank Herbert's Dune was one of the books that made a huge impression on me. I think I read the book when I was 11 or 12 years old, and I was totally enthralled by it.

Interestingly, when I discuss the book with others, it's odd how we each have different memories of it. For me, I remember being just fascinated by the details of the people surviving in the desert: the special clothing they wore to minimize loss of water, the recycling and re-use of water, locating their houses underground, etc.

Other people were fascinated by the political intrigue (the maneuvering of the Great Houses).

Others were fascinated by the love stories, and by the relationship of women in the book (the main female characters are concubines).

Anyway, it's been decades since I read the book, but I just loved it.

So when I stumbled across the SciFi Channel mini-series on Netflix not too long ago, I decided to give it a try.

It was surprisingly enjoyable. What's a bit odd about the mini-series is that the producers clearly decided to allocate their budget so that they spent most of the money on screen-writing, and much less of the money on special effects, music, etc. For a show made in the year 2000, the visuals are laughable; much of the time it's obvious that the actors are standing in front of large painted backdrops.

However, since Dune is, at its heart, a story about people, not (just) a space fantasy, the result is oddly pleasing. The richness of the original novel comes through strongly, and the story is given a telling that honors it.

As I watched the mini-series, it struck me how much modern works have built on and learned from Dune. Let's see, just for an example:

• The story is set on a faraway planet
• Rapacious industrialists have arrived, bent on scouring the planet of its scarce and unique natural resource, and taking that resource elsewhere to sell
• The resource is so critical that enormous sums of money are spent, and many parties are fighting over the proceeds
• But there are indigenous life forms, and an indigenous population, who love the planet, use its resources wisely, and are prepared to fight to the death to defend it.
• Though the colonizers have superiority in weapons, the locals are fighting for their lives, and their homes, and they drive the outsiders out.

No, not Dune, I'm talking about Avatar.

Of course, I suppose I could be talking about Lawrence of Arabia, or the Last Mohican, or ...

Anyway, cheesy though it is, the SciFi Channel mini-series adaptation of Dune wasn't the worst thing I've watched on Netflix this spring...