The intriguing theory of Expert Beginners

I've been enjoying Erik Dietrich's series of articles on the Daedtech blog about a phenomenon he calls the "Expert Beginner".

Dietrich's observation is that, while learning a new skill (such as computer programming, say), you might accidentally think that you had become an expert when in fact you were actually doing it all wrong. You just hadn't noticed that you were doing it wrong, and nobody was around to point that out to you.

For whatever reason, software appears to be a field that is particularly prone to this sort of Expert Beginner situation, perhaps because we software types are often learning things on our own, rather than having the opportunity to be taught from an established body of knowledge.

Not only is the Expert Beginner full of bad habits, but even worse: should he at some point encounter somebody who has greater expertise, and can help educate the Expert Beginner toward improvement, this assistance is likely to be met by resistance, as the Expert Beginner will have to start by un-learning all of his bad habits, which he is naturally reluctant to do (since they have served him well so far).

Dietrich is an entertaining writer and I recommend the essays:

• How Developers Stop Learning: Rise of the Expert Beginner

  The Expert Beginner has nowhere to go because progression requires an understanding that he has a lot of work to do, and that is not a readily available conclusion. You’ll notice that the Expert Beginner is positioned slightly above Advanced Beginner but not on the level of Competence. This is because he is not competent enough to grasp the big picture and recognize the irony of his situation, but he is slightly more competent than the Advanced Beginner due mainly to, well, extensive practice being a Beginner.

• How Software Groups Rot: Legacy of the Expert Beginner

  Perhaps it’s a lack of automated testing. Giant methods/classes. Lots of copy and paste coding. Use of outdated or poor tooling. Process. It can be any number of things, but the common thread is that you have a person or people in positions of authority that have the culturally lethal combination of not knowing much, not knowing what they don’t know, and assuming that, due to their own expertise, anything they don’t know isn’t worth knowing. This is a toxic professional culture in that it will force talented or ambitious people either to leave or to conform to mediocrity.

• How Stagnation is Justified: Language of the Expert Beginner

  The Expert Beginner believes that he and his ‘fellow’ Expert have a simple difference of opinion among ‘peers.’ While it may be true that one Expert speaks at conferences about source control best practices and the other one runs the IT for Bill’s Bait Shop and has never used source control, either opinion is just as valid.

• Up or Not: Ambition of the Expert Beginner

  An Expert Beginner’s entire career is built on a foundation of cognitive dissonance. Specifically, they believe that they are experts while outside observers (or empirical evidence) demonstrate that they are not. So an Expert Beginner is sentenced to a life of believing himself to be an expert while all evidence points to the contrary, punctuated by frequent and extremely unwelcome intrusions of that reality.

• Self-Correcting Organizations: Fall of the Expert Beginner

  Following the career arc of Expert Beginners is really quite sad. In the early stages, one feels annoyed and a little indignant at advancement by luck instead of competence. As things progress, real damage is caused by poor implementation and wrong-headed approaches, resulting for a lot of people in stress, frustration, failure, and at times even lost jobs and failed ventures. And, in the end, the fate of the one that caused these things is probably poetically just, but hard to find happiness in. A person ill-suited for a role assumed it, caused problems, and then suffered personal hardship. It’s not a great story.

To my mind, what this boils down to, for a person who's concerned with continuing to develop their own skills indefinitely, is: in order to improve my skills, I need to find somebody who can coach me.

You can always get better; you can always learn more. It's just a matter of finding somebody who can help you improve, and putting in the effort to learn from that person.

Just don't get stuck being an Expert Beginner.

Some benchmarking principles

Here's a nice slide deck on general benchmarking principles that I came across recently: Tokutek benchmarking principles

For hard-core perf types, there's not much new here, but I thought it was worth passing along in case some found it interesting.

Among my favorite observationss from the slide deck:

• benchmark frequently, to catch performance regressions soon after they occur
• keep a benchmark history, for analysis of data over time
• use graphs and plots to help with data interpretation
• share benchmarking data widely ("developers love data")

At various companies, at various times, I've worked with benchmarking teams, and it's nice to see the slow-but-steady "systematization of knowledge" going on here, since some of these are hard lessons and it's worth your time not to re-learn them yourself.

HotOS: Hot or not?

I didn't go to the biennial HotOS conference, which was held last week in New Mexico.

I am, however, extremely grateful to USENIX and to the sponsors and participants for making the technical session materials freely available to all.

There's a lot of dig through in the conference, and I'll share my thoughts on some of the work once I've had more of a chance to read it.

Meanwhile, Matt Welsh has stirred up a fair bit of discussion with his post-conference blog post: What I wish systems researchers would work on.

Of the 27 papers presented at the workshop, only about 2 or 3 would qualify as bold, unconventional, or truly novel research directions. The rest were basically extended abstracts of conference submissions that are either already in preparation or will be submitted in the next year or so. This is a perennial problem for HotOS, and when I chaired it in 2011 we had the same problem. So I can't fault the program committee on this one -- they have to work with the submissions they get, and often the "best" and most polished submissions represent the most mature (and hence less speculative) work.

Welsh has been involved with the conference for years; moreover, given his well-known move from member of the faculty of Harvard's Computer Science department to lead researcher at Google, he has a fascinating background in both the academic and industrial research arenas.

So his thoughts are worth considering.

And yet, I must say that I find Welsh's criticisms to ring hollow.

It doesn't seem right to critique a topic by whether it is brand new, or whether it "has been a recurring theme" that involves "problems that are 25 years old." In some ways, I think it is a mark of Computer Science's maturity that we've stopped completely changing our entire perspective every 18 months, and are instead returning to deep problems of enduring interest.

For example, the observation by an IBM research team that the last two decades have seen dramatic progress in network I/O performance but a dramatic lack of progress in storage I/O performance seems like a great topic for the operating systems community to be discussing. Why, just last week I was digging into continued research in the network I/O world, while the disk storage world appears to be still stuck in bloated, layered, horrifically complex implementations from gigantic enterprise companies that ladle on the complications while ratcheting up the price: it's not uncommon for an enterprise SAN device to cost 7 or 8 digits, as much as 1000 times the cost of the servers that are trying to process that imprisoned data.

And when Welsh outlines the areas he considers important, he notes that:

A typical Google-scale system involves many interacting jobs running very different software packages each with their own different mechanisms for runtime configuration: whether they be command-line flags, some kind of special-purpose configuration file (often in a totally custom ASCII format of some kind), or a fancy dynamically updated key-value store.

and also that:

the modes of interaction are vastly more complex and subtle than simply reasoning about state transitions and messages, in the abstract way that distributed systems researchers tend to cast things.

I couldn't agree more with Welsh's points, which makes me wonder how he reacted to the session from the RAMCloud team: Toward Common Patterns for Distributed, Concurrent, Fault-Tolerant Code. The RAMCloud developers ran smack into these problems, and felt that pain:

For example, when a server failure notification is received by a server, several of its segments are affected, and each affected segment replica can be in a different state. Some replicas may have an RPC outstanding to the failed server and must abort the RPC and restart replication elsewhere instead of expecting a response. Other affected replicas may not be consistent with their counterparts; such replicas require contacting the cluster coordinator before starting recreation in order to prevent inconsistencies.

Now, what the RAMCloud team propose is not so new or trendy; again, they look back two decades, to the dawn of object-oriented design, and the "patterns" approaches that proved so successful in the 1990's:

Although the implementations were different in many respects, we eventually noticed a common theme; each of these modules contained a set of rules that could trigger in any order. We gradually developed a pattern for DCFT code based on three layers: rules, tasks, and pools. This particular pattern has worked for a variety of problems in RAMCloud. We believe that this pattern, or something like it, might provide a convenient way of structuring DCFT modules in general.

It's important to look to the future, but it's important to learn from and build on the past. Some old techniques are sound, and we shouldn't jettison them just because they're old (of course, remember you're getting this from a 52-year-old systems programmer who still codes 10 hours a day in C!).

So keep pushing the envelope, but let's not excoriate those who try to help us learn from proven decades-old approaches and bring that wisdom to the problems of modern software.

The Legend of 1900: a very short review

Fifteen years late, we stumbled across The Legend of 1900.

I suspect that 1900 is the sort of movie that many people despise, and a few people really enjoy.

Count me on the "really enjoy" side. Just knowing that the lead character's name was

Danny Boodman T.D. Lemons Novecento

was probably enough to hook me.

Like any good fantasy, the movie is a parable: about war, about immigration, about society, and perhaps most of all, about the longing that drives people to travel. As 1900 says in a crucial monologue:

I think land people waste a lot of time wondering why. Winter comes and you can't wait for summer, summer comes and you never can wait for winter. That's why you never tire of traveling or chasing some place far away, where it's always summer.

I can't tell what sort of reaction you might have to this movie, but if you find yourself on a quiet evening looking for something just a bit unusual to watch, you might give The Legend of 1900 a try.

Gluttony

Definition: Fringe Binge. A Fringe Binge is what happens when Netflix releases Season Four of Fringe on streaming, and so you watch nine episodes in three nights.

It means your daily conversations are punctuated with fragments like:

Wait, hon, I missed something: which alternate universe is this?

Wild: a very short review

Cheryl Strayed's Wild: From Lost to Found on the Pacific Crest Trail is an unusual book.

It is a memoir, written 17 years after the fact, of a truly life-changing event.

In 1995, with her life in a shambles (a situation so horrible that the oft-overused "hit rock bottom" actually seems appropriate), Strayed decided to take action, and set out to turn the page, embarking upon a solo backpacking trip:

It was a world I'd never been to and yet had known was there all along, one I'd staggered to in sorrow and confusion and fear and hope. A world I thought would both make me into the woman I knew I could become and turn me back into the girl I'd once been. A world that measured two feet wide and 2,663 miles long.

A world called the Pacific Crest Trail.

That's a tall order for a trail to fill.

However, I'm lucky enough to have walked several portions of the Pacific Crest Trail myself, and have even camped along some of its highest and most remote stretches, and I can confirm: it is no ordinary trail.

Strayed's book is startlingly raw and honest. She hides nothing, reveals all, lets us deep down into her torments and the agonies she felt as she struggled along. Before beginning her trip, she had never backpacked a day in her life:

I'd gone to an outdoor store in Minneapolis called REI about a dozen times over the previous months to purchase a good portion of these items.

...

By the time I made the decision about which backpack to purchase -- a top-of-the-line Gregory hybrid external frame that claimed to have the balance and agility of an internal -- I felt as if I'd become a backpacking expert.

She hadn't though; it's not the sort of thing you can learn in a store, from a book, or by listening to people, and it's not long before the realities of the trail begin to pile up and she is forced to adapt. But she is so remarkably determined and focused that she simply wills herself to overcome obstacles:

I knew that if I allowed fear to overtake me, my journey was doomed. Fear, to a great extent, is born of a story we tell ourselves, and so I chose to tell myself a different story from the one women are told. I decided I was safe. I was strong. I was brave. Nothing could vanquish me. Insisting on this story was a form of mind control, but for the most part, it worked.

And so she goes, walking along, changing her life:

I had to change was the thought that drove me in those months of planning. Not into a different person, but back to the person I used to be -- strong and responsible, clear-eyed and driven, ethical and good. And the PCT would make me that way. There, I'd walk and think about my entire life. I'd find my strength again, far from everything that had made my life ridiculous.

It's a great plan, but she soon finds out that the trail doesn't care what she's thinking:

As I hiked, I moaned again and again, as if that would provide some cooling relief, but nothing changed. The sun still stared ruthlessly down on me, not caring one iota whether I lived or died. The parched scrub and scraggly trees still stood indifferently resolute, as they always had and always would.

Though barrier after barrier confronts her, she surmounts each one, and slowly her self-chosen therapy begins to bear fruit:

Uncertain as I was as I pushed forward, I felt right in my pushing, as if the effort itself meant something. That perhaps being amidst the undesecrated beauty of the wilderness meant I too could be undesecrated, regardless of what I'd lost or what had been taken from me, regardless of the regrettable things I'd done to others or myself or the regrettable things that had been done to me. Of all the things I'd been skeptical about, I didn't feel skeptical about this: the wilderness had a clarity that included me.

The wilderness includes not just clarity, but also a different scale, and a different sense of time:

Foot speed was a profoundly different way of moving through the world than my normal modes of travel. Miles weren't things that blazed dully past. They were long, intimate straggles of weeds and clumps of dirt, blades of grass and flowers that bent in the wind, trees that lumbered and screeched. They were the sound of my breath and my feet hitting the trail one step at a time and the click of my ski pole. The PCT had taught me what a mile was.

The PCT, indeed, has much to teach (as I said, it is no ordinary trail), and Strayed is eager to soak it all up, as she follows in the footsteps of the pioneering naturalists who worked to save and create the trail:

It had only to do with how it felt to be in the wild. With what it was like to walk for miles for no reason other than to witness the accumulation of trees and meadows, mountains and deserts, streams and rocks, rivers and grasses, sunrises and sunsets. The experience was powerful and fundamental. It seemed to me that it had always felt like this to be a human in the wild, and as long as the wild existed it would always feel this way. That's what Montgomery knew, I supposed. And what Clarke knew and Rogers and what thousands of people who preceded and followed them knew. It was what I knew before I even really did, before I could have known how truly hard and glorious the PCT would be, how profoundly the trail would both shatter and shelter me.

Like the Pacific Crest Trail, Cheryl Strayed's Wild is powerful and fundamental, and it will both shatter and shelter you. You may not be a backpacker; you may not be a woman in the depths of despair; but if you take the time to enter Strayed's world, and go with her on her adventure, you will enjoy it, you will learn from it, and you won't regret a moment of the time you spent seeing the world through her eyes.

C10M Resources

So I happened to spend some time listening to this video today, which led me to: C10M

Scalability. It's the question that drives us. It's an anomaly inherent to the programming of the matrix.

The essays were interesting; I'm looking forward to more of his articles.

The articles Graham has already written led me to Packet Processing on Intel® Architecture

Packet processing while executing other workloads on an Intel processor reduces hardware costs, simplifies the application development environment, and reduces time to market. The Intel DPDK is also playing a critical role in Software-Defined Networking (SDN) and Network Functions Virtualization (NFV)

I also thought that Performance Optimization might be pretty interesting, but attempting to read more kept running me into "register with our marketing department" pages...

Anyway, it was interesting enough that I thought I'd pass it all along and see what sort of "yeah, that's cool, but check *this* out!" sort of feedback I got.