Are automated tools for testing distributed systems becoming a reality?

These recent publications would suggest that the time has finally come to deploy serious test tools for bullet-proofing large scale distributed systems.

Exciting!

• Challenges in Designing at Scale: Formal Methods in Building Robust Distributed Systems

  PlusCal and TLA+ have proven very effective at establishing and maintaining the correctness through change of the fundamental components on which DynamoDB is based.

• Using TLA+ for teaching distributed systems

  TLA is a tool for specifying distributed algorithms/protocols and model checking them.

• My experience with using TLA+ in distributed systems class

  Integrating TLA+ to the class gave students a way to get a hands-on experience in algorithms design and correctness verification.

• Combining static model checking with dynamic enforcement using the Statecall Policy Language

  The Statecall Policy Language (SPL) was designed to make model checking more accessible to regular programmers.

• SAMC: Semantic-aware model checking for fast discovery of deep bugs in cloud systems

  It was used to test 10 versions(both old and current) of Zookeeper, Hadoop/YARN, and Cassandra across 7 different protocols (leader election, atomic broadcast, cluster management, speculative execution, read/write, hinted handoff, and gossiper).

• Lineage-driven Fault Injection

  MOLLY doesn’t blindly explore a state space, instead MOLLY reasons backwards from a successful outcome (hopefully common!), to figure out what might have caused it to fail, and then probes paths that exercise those potential causes.

• Paper review: "Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems"

  Table 5 shows that almost all (98%) of the failures are guaranteed to manifest on no more than 3 nodes, and 84% will manifest on no more than 2 nodes.

All I have time for is link dumping

Well, that and going on 3-day company events.

• Pioneer in Database Systems Architecture Receives ACM's Highest Honor: For fundamental contributions to the concepts and practices underlying modern database systems.

  His work on Ingres introduced the notion of query modification, used for integrity constraints and views. His later work on Postgres introduced the object-relational model, effectively merging databases with abstract data types while keeping the database separate from the programming language. Stonebraker's implementations of Ingres and Postgres demonstrated how to engineer database systems that support these concepts; he released these systems as open software, which allowed their widespread adoption and their code bases have been incorporated into many modern database systems.

• Code Words: Issue Two – March 2015

  A quarterly publication about programming from the Recurse Center

• Coq used to prove that false is true

  It is always worth remembering that all computer aided mathematics programs contain bugs.

• You Cannot Have Exactly-Once Delivery

  Every major message queue in existence which provides any guarantees will market itself as at-least-once delivery. If it claims exactly-once, it’s because they are lying to your face in hopes that you will buy it or they themselves do not understand distributed systems.

• What Went Wrong with the FISA Court

  The Foreign Intelligence Surveillance (FISA) Court is no longer serving its constitutional function of providing a check on the executive branch’s ability to obtain Americans’ private communications. Dramatic shifts in technology and law have changed the role of the FISA Court since its creation in 1978 — from reviewing government applications to collect communications in specific cases, to issuing blanket approvals of sweeping data collection programs affecting millions of Americans.

  Under today’s foreign intelligence surveillance system, the government’s ability to collect information about ordinary Americans’ lives has increased exponentially while judicial oversight has been reduced to near-nothingness. This report concludes that the role of today's FISA Court no longer comports with constitutional requirements, including the strictures of Article III and the Fourth Amendment. The report lays out several steps Congress should take to help restore the FISA Court’s legitimacy.

• 2600 miles in 4 minutes: A time-lapse of Andy's Pacific Crest Trail hike

  The process of stopping to take a picture every mile had a big impact on the actual experience of doing the hike. I had to be well-aware of where I was at all times, which was quite distracting in that it took me out of the moment and made it difficult to maintain good momentum. Though I suppose it was nice to have something to occupy my thoughts with as well. In retrospect, I am happy to have done it. It provided a good conversation starter with so many hikers that I would meet along the way. And I now love seeing so many wonderful faces popping up in pictures in unexpected places.

More link dumping

Happy second day of spring.

• Classp: a “classier” way to parse

  Recently, we opened up the source code for Classp, a side-project a few of us have been working on that demonstrates it’s possible to have an automatic parser generator that is not based on formal grammars. Instead of grammar productions, you write classes similar to C++ or Java and you write class patterns to define the syntax. Although there are libraries like Boost.Spirit and Scala Parsers that give you a nice way to write a grammar in the programming language itself, in the end you are still writing a grammar. Even though Classp looks a lot like C++ or Java, it is not just a C-like way to write a grammar. It’s an entirely different way to specify syntax.

• Greece’s debt crisis: How did we get here?

  Our story begins in June 2000, a year and a half before all the Deutschmarks and French francs and Italian lire and Spanish pesetas were to be eradicated forever, and converted into euros. That was the date at which 11 became 12: Greece was invited to be one of the select dozen countries who would be part of the euro at its inception.

• An Eleven Year-Old’s Guide to Plane-Spotting

  Plane-spotting is when you identify any sort of aircraft and, if you want, keep a log of the type of aircraft, the time, and the date, and maybe any other interesting things like a bird, or a really good apple that you’re eating (I stress again that the log is optional). If you are watching commercial aircraft you should try to identify the airline, too. I take photos of the airplanes for fun as well.

• The Longest Hearthstone Turn Ever Is Happening Right Now

  “Lorewalker Cho” is a legendary creature card that alters the rules so that when either player casts a spell, it puts a copy of that spell into the other player’s hand. And “Mind Vision,” when played, copies a random card in your opponent’s hand and puts it in your hand. If both players work together using these cards, they could generate many additional copies of cards that normally would be limited to a single use.

• Imagine a Slightly Different Announcement from Ted Cruz

  Imagine an American Muslim man stands in front of an audience of 10,000 Muslims at a college in Virginia to announce that he's running for president.

Stuff I'm reading, day off edition

There was a heavy fog overnight. The local news station was generous and labelled it "rain".

• Dear Data

  Each week we collect and measure a particular type of data about our lives, use this data to make a drawing on a postcard-sized sheet of paper, and then drop the postcard in an English “postbox” (Stefanie) or an American “mailbox” (Giorgia)!

  Eventually, the postcard arrives at the other person’s address with all the scuff marks of its journey over the ocean: a type of “slow data” transmission.

  By creating and sending the data visualizations using analogue instead of digital means, we are really just doing what artists have done for ages, which is sketch and try to capture the essence of the life happening around them. However, as we are sketching life in the modern digital age, life also includes everything that is counted, computed, and measured.

  We are trying to capture the life unfolding around us, but instead we are capturing this life through sketching the hidden patterns found within our data.

• The billionaire’s typewriter

  I re­al­ized I dis­agree deeply with Medium about the ethics of de­sign. And by ethics, I mean some­thing sim­ple: though Medium and I are both mak­ing tools for writ­ers, what I want for writ­ers and what Medium wants couldn’t be more dif­fer­ent. Medium may be avoid­ing what made the type­writer bad, but it’s also avoid­ing what made it good. Writ­ers who are tempted to use Medium—or sim­i­lar pub­lish­ing tools—should be con­scious of these tradeoffs.

• The Battle Is For The Customer Interface

  The new breed of companies are the fastest-growing in history. Uber, Instacart, Alibaba, Airbnb, Seamless, Twitter, WhatsApp, Facebook, Google: These companies are indescribably thin layers that sit on top of vast supply systems ( where the costs are) and interface with a huge number of people ( where the money is). There is no better business to be in. The New York Times needs to write, fact check, buy paper, print and distribute newspapers to get their ad money. Facebook provides a platform for us to write our own content, and Twitter monetizes the front page of newspapers, which happens to now be the Twitter feed.

• Inside the Mad, Mad World of TripAdvisor

  For travelers the impact has been equally profound. What begins as a simple search-engine query becomes an epic fact-finding mission that leaves no moldy shower curtain unturned, a labyrinthine choose-your-own-adventure—do you read the one-bubble rant?—in which the perfect hotel always seems just one more click away. For all the power of the service, it raises deep questions about travel itself, including, most pressingly, who do we want—who do we trust—to tell us where to go? “The future,” Don DeLillo once wrote, “belongs to crowds.” Are we there yet?

• Beware of Airbnb entering the hyperlocal travel guide business

  Airbnb wants to build a community of users, itself structured in homogenous layers (e.g. young families looking for budget rentals, yuppies aiming at trendy places…) There’s even the growing crowd of professionals who prefer an Airbnb apartment free of the check-in/out hassles of hotels, and who will gladly trade unexciting room service for a super-fast DSL connection. (I’m told a growing number of Googlers do so for their business trips, with their employer’s blessing.) Each of these sub-communities will be far more likely to trust their peers than the usual travel guides

• California's About to Run Out of Water. We Have to Act Now

  Today, most of the state’s agricultural land is used to grow high-end crops like pistachios, almonds, and wine grapes, which require water every year. Without H2O, these crops die and farmers lose out on millions of dollars. “Twenty years ago I could talk to a farmer and say, ‘Take a year off farming and we’ll pay you double and give us the water,’” says Kightlinger. “Now they can’t take a year off production. It’s made it much more difficult to free up water in a drought.”

• Andrew Bate’s Swarmfarm Robotics finds a more efficeint way to spray weeds

  As he thought about the challenges ahead, Bates realised that bigger was not necessarily better when it came to equipment. He began thinking about using new technology in the form of small unmanned robots. Digital smart automation, he believed, could hold the key to new and improved farming systems. Instead of using one large expensive tractor, farmers could use swarms of small, simple, clever robotic machines. He and his wife Jocie, an agricultural economist, set up Swarmfarm Robotics. Working with Sydney University’s Centre for Field Robotics and Queensland University of Technology, he developed a new robot to spray for weeds. Weighing only 200kg, and measuring about five metres across, the self-propelled device can be powered by either an electric or a diesel motor.

• Warren Buffett Just Wrote His Best Annual Letter Ever

  I have read all 50 of Warren’s letters and feel this is the most important one he has ever written. It’s really three letters in one. First there’s his usual look back at Berkshire’s performance in 2014. Then, because this is the 50th anniversary of his buying Berkshire, there’s a separate section where he goes through the history of his leadership. Finally, his colleague Charlie Munger writes his own look at the company, which is also excellent.

• We Buy Broken Gold

  The fact is, they’re already weakened: that’s what they’re doing in your store in the first place. They need the money. It’s not quite stealing gold fillings out of the pulled teeth of weary soldiers, but it’s the same principle. Use your expertise to exploit the weakness of someone who doesn’t know any better, and do it in a situation in which he believes he has reason to trust you. Why would a wealthy diamond merchant in a three-thousand-dollar suit want to cheat me out of a hundred bucks’ worth of gold?

• Reasons not to use Facebook

  Why you should not "use" (i.e., be used by) Facebook.

• Man In The Moonbase (Part 1) : The Death and Life of the Best Game You Never Played

  Moonbase would be Humongous Entertainment’s gateway game—a bridge that transitioned preteen fans of our Backyard Sports titles into more mature strategy games like Starcraft, Command and Conquer, or Cavedog’s own Total Annihilation. “It seemed foolish to capture the adoration of these young kids—to really build a reputation for quality with them by working hard—and then to just let them drop off the map. We had to evolve, we had to find new opportunities.”

• In fantasy worlds, historical accuracy is a lie

  But though almost anything's possible within Dragon Age's beloved world of Thedas, something feels off. Although Dragon Age is a fantasy roleplaying game, Thedas is overlaid with a faux-European sociopolitical landscape -- and that means there are few people of color among its citizenry. Why do the sinister old arguments of "historical accuracy" still apply to this fantasy world?

Stuff I'm reading, March Madness edition

Big 12's gotta be cringing, half their teams are already gone?

But meanwhile, back in the office...

• Best of VLDB 2014

• Worship Maintainers

  We need to place a much higher value on maintenance and maintainers. We need to worship maintainers with the same fervor we worship creators. We need to recognize that not having emergencies can mean more despite being less visible than a well-handled emergency. We need to value continuity (in a bumpy world) as much as we value growth. We need to recognize that growth is pointless unless we provide it with a solid foundation.

• Pesticides and Bees: It's Complex

  It’s not that there are no smoking guns; there are hundreds of smoking guns, all of which contribute to the decline of bees. The consensus among bee scientists is that honey bee declines are the result of multiple factors, working independently or synergistically.

• A software engineer’s role traversal

  Q: Any final thoughts on the developer job market, or where you think it’s going?

  A: Jason Myers: I think the market is going to continue to be challenging for employers and great for developers. There are still tons of beginners around filling that need, and I’m hoping more companies will pick them up, and devote resources to leveling them up more. It’s still easy for mid to senior level engineers to move to other companies as they wish.

• Signals from the O’Reilly Software Architecture Conference 2015

  The old notion of a software architect being a non-coding, post-useful deep thinker is giving way to something far more interesting, says Neal Ford. “Architecture has become much more interesting now because it’s become more encompassing … it’s trying to solve real problems rather than play with abstractions.”

• The Dark Web's Top Drug Market, Evolution, Just Vanished

  Over the past weekend, the massive anonymous market known as Evolution halted withdrawals of bitcoin from its website, telling users that it was dealing with technical difficulties. Then on Tuesday evening, both its market and user forum went offline, with no opportunity for drug buyers and sellers to pull out the funds they had stored in their Evolution accounts. The result has been a wave of panic that’s shaken the online black market economy as much as any of the law enforcement drug busts of the last two years.

• Ungrateful

  Indeed, the very qualities that have defined Deadheads—a resistance to corporatization, a suspicion of institutions, a strong sense of ownership over the band—have conspired to make these particular fans among the most likely to be skeptical of an enterprise on the scale of Fare Thee Well. “The Dead are unique,” said Joel Makower, author of Woodstock: The Oral History. “The Dead encouraged bootlegs. They built a fan base around thwarting the Man.” But the Man is harder to avoid now than ever, noted Makower: “If Woodstock were to happen now, it would be ‘Chevy Presents Woodstock.’ ”

Thomas Ptacek on how to recruit and select programmers

Thomas Ptacek, formerly of Matasano Security, and now at Starfigher, has written a long and detailed article about how he goes about trying to recruit and select programming talent: The Hiring Post.

Ptacek's article isn't the typical four paragraph overview that you'll read in your weekly business magazine; it's detailed and born of years of pain trying to hire security software experts.

But, his essay is so good that I think it will end up on the short list of articles about the software engineering hiring process that are worth going back to over and over, like Joel Spolsky's The Guerrilla Guide to Interviewing, Michael Lopp's A Glimpse and a Hook and The Sanity Check, or Laurie Voss's This is why you never end up hiring good developers.

Let's use Ptacek's own table of contents to break it down:

• The telling success story that motivated me to write this.

  Ptacek starts out by noting how pleased he is with a recent Matasano hire, and how he realized that they had only managed to connect because he'd changed the way their hiring process works:

  A few years ago, Matasano couldn’t have hired Alex, because we relied on interviews and resumes to hire. Then we made some changes, and became a machine that spotted and recruited people like Alex: line of business .NET developers at insurance companies who pulled Rails core CVEs out of their first hour looking at the code. Sysadmins who hardware-reversed assembly firmware for phone chipsets. Epiphany: the talent is out there, but you can’t find it on a resume.

• The part where I join the chorus of people telling you why interviews are terrible.

  Ptacek bemoans the lack of consistency and objectivity of the interview process

  Driven in part by an oral tradition of how the last 20 years of technical job interviews has resulted in terrible hires, interviewers try to assess for “smart and gets things done”. In other words: “subjective and even more subjective x-factor”.

• But at least read this bad thing about interviews, please.

  Ptacek makes an observation I haven't seen widely highlighted, though it should be: the hostility and stress of the technical interview is directly damaging to the potential value of the interview:

  I walked into the conference room to meet him. He appeared to be physically shaking. You didn’t need to be a psychologist to detect the nervous energy; it radiated from him, visibly, like in an R. Crumb cartoon.

  Engineering teams are not infantry squads. They aren’t selected for their ability to perform under unnatural stress. But that’s what most interview processes demand, often —– as is the case with every interview that assesses “confidence” —– explicitly so.

  As Ptacek observes, you don't want to hire people who are good at interviewing; you want to hire people who are good at writing software. So, how do you do that? Well, says Ptacek, the first thing you must do is realize you have a problem:

  For every genuinely competent and effective developer who can ace a tough dev interview, there are many more genuinely competent and effective developers who can’t. While we’re selecting for the ability to control a conversation, we’re missing ability to fix broken 2-phase commits.

• The warm-up trick.

  Ptacek suggests that the first thing you can do is to be open about the process, and about your expectations:

  Not knowing what to expect makes candidates nervous. That’s a pointless handicap.

  At my last firm, we had the “first-call” system. Every serious applicant got, on average, 30-45 minutes of director-level time on the phone before any screening began.

• The work-sample trick.

  Ptacek makes an ever-more-common suggestion. Instead of trying to ask people questions about programming, ask them to do some programming. But as Ptacek notes, this doesn't mean tinker-toy "reverse a string at the whiteboard" gimmicks, but rather a realistic project which you have to carefully design and provide:

  You can’t do this with a trial period where a candidate gets paid to fix whatever random bugs are in the issue-tracker. Your goal is to collect data you can use for apples-apples comparisons, which means every candidate has to be working on the same problems. You have to design a test, create a scoring rubric, and iterate.

• One simple trick that will make your team hate you.

  In this section, Ptacek continues to hammer on the "objective, reproducible, and reliable" aspect of the hiring process:

  Interviewers hate it. Thomas’s First Law Of Interviewing: If interviewers enjoy it, you’re doing something wrong. But we kept doing it, because we found that we were comfortable with “why” we were making hire/no-hire decisions when we had facts to look at. We could look at the results of an interview and compare them to the same data generated by successful team members.

• Grade your hiring process.

  Lastly, Ptacek reminds you not to sit back and relax, but to work to constantly improve your hiring process, just like you'd work to constantly improve any other process at your organization.

  Ask yourself some questions about your hiring process.

  Are you learning both from your successes and failures?

Ptacek really cares about this, so much so that he left Matasano to start a new company to develop his ideas into a realistic hiring tool: Announcing Starfighter

I'm not sure how Starfighter will do; for one thing, it's clearly aimed directly at the "penetration tester" sub-specialization, which is certainly an important job category, but it's just one of thousands of software specialties. I am skeptical that the CTF style of programming will be applicable to other software specialties, but these are some very smart people, and they really believe in their idea, so I'm interested to see how it all goes.

What I'm reading, mid-March edition

The fruit trees are all happy, everything is in bloom.

And I'm reading a completely random collection of stuff...

• Disambiguating Databases

  The scope of the term database is vast. Technically speaking, anything that stores data for later retrieval is a database. Even by that broad definition, there is functionality that is common to most databases. This article enumerates those features at a high level. The intent is to provide readers with a toolset with which they might evaluate databases on their relative merits.

• There is No Now

  One of the most common mistakes in building systems—especially, but not only, distributed-computing systems on commodity machines and networks—is assuming an escape from basic physical realities. The speed of light is one such reality, but so is one that is more pernicious and just as universal: we cannot make perfect machines that never break. It is the combination of these realities, of asynchrony and partial failure, that together make building distributed systems a difficult pursuit. If we do not plan and account for failures in individual components, we all but guarantee the failure of combined systems.

• The future of Team Foundation Version control

  We've been investing heavily in Git because there's a ton of work to do to bring it up to parity with what we can do with TFVC. I think people get confused for a number or reasons. We talk about our progress on Git a lot. The industry talks about Git a lot. And, if you are watching, you'll hear more and more about teams inside Microsoft adopting Git. My own team has moved a bunch of stuff to Git.

• How GitHub Conquered Google, Microsoft, and Everyone Else

  Google Code is dying because most of the open source world—a vast swath of the tech world in general—now houses its code on GitHub, a site bootstrapped by a quirky San Francisco startup of the same name. All but a few of those thousand projects are now on GitHub.

• Open-sourcing Pinball

  After experimenting with a few open-source workflow managers we found none of them to be flexible enough to accommodate the ever-changing landscape of our data processing solutions. In particular, current available solutions are either scoped to support a specific type of job (e.g. Apache Oozie optimized for Hadoop computations) or abstractly broad and hard to extend (e.g. monolithic Azkaban). With that in mind, we took on the challenge of implementing a highly customizable workflow manager build to survive the evolution of the data processing use cases ranging from execution of basic shell commands to elaborate ETL-style computations on top of Hadoop, Hive and Spark.

• The sad state of sysadmin in the age of containers

  Nobody seems to know how to build Hadoop from scratch. It's an incredible mess of dependencies, version requirements and build tools.

  None of these "fancy" tools still builds by a traditional make command. Every tool has to come up with their own, incomptaible, and non-portable "method of the day" of building.

  And since nobody is still able to compile things from scratch, everybody just downloads precompiled binaries from random websites.

• Booster Test

  NASA socials give some inside access to people like me (and you, if you live in the USA and want to sign up next time) who have no official connection to the space program. Yesterday we got to tour the plant where the boosters are made. It was great to learn about techniques for mixing, casting, and curing huge amounts of propellant without getting air bubbles or other imperfections into the mix and without endangering workers. The buildings in this part of ATK have escape slides from all levels and are surrounded by big earthworks to deflect potential explosions upwards. It was also really cool to see the hardware for hooking boosters to the main rocket, for vectoring nozzles, and things like that. Alas, we weren’t allowed to take pictures on the tour.

• Wild-Winter Whodunnit—Climate Change Over the U.S. With a Slow Jet Stream?

  It’s a notion so vast in scale it’s difficult to imagine, yet the mechanics are fairly simple. The jet stream is generated by a combination of Earth’s rotation and the flow of air down atmospheric gradients between high-pressure, mid-latitude warmth and low-pressure Arctic cold. Over the last several decades, the Arctic has warmed faster than any other region; during periods of especially heightened warming, as occurs when melting sea ice exposes dark, sunlight-absorbing waters, the north-to-south temperature difference shrinks. The pressure differences flatten.

  This decreased gradient slows down the jet stream—and as it slows, it also seems to become wavier, plunging south or veering north when encountering atmospheric obstacles it would once have coursed straight through.

• The 2015 California Earthquake Forecast

  We can’t deal with the situation using simple, linear computer models based on one idea of Earth’s behavior. The third UCERF is a supple, fine-grained instrument that takes advantage of many significant advances during the last decade. When I told a USGS quake guy yesterday how much I admired the new model, his eyes twinkled. They’re proud of this.

• Great ladies of history find a new home in strategy games

  Paradox's chief operating officer is Susana Meza-Graham, and she says she loves working in the demanding strategy game space. "I love that the games we develop and publish within the strategy space matter for others," she says. "We have countless examples of teachers using our games as a way to bring history, politics and economics to life for their students."

  She shows me a TED talk from Alex Petroff, who at 6:38 minutes in says Paradox games have helped him run an NGO dedicated to getting rural people out of poverty in the Congo.

  "We've never gone out of our way to label a game as typically 'male' or 'female'," Meza-Graham says. "Instead we try to find appealing ways to communicate the merits of the game and what makes them special and worth playing."

  Crusader Kings II in particular has a higher ratio of female players than many of the company's other strategy games. Meza-Graham puts the total at about 40 percent.

Sorry, but now we know: you're a dog.

Twenty years ago, the New Yorker published Peter Steiner's wonderful send-up of the Internet, perhaps the greatest Internet cartoon ever written: On the Internet, nobody knows you're a dog

Well, twenty years have passed.

In his talk at this year's GDC, Raph Koster picks up the story, and Gamasutra's Leigh Alexander summarizes it for us

We now live in an age where the internet filters results for you based on assumptions about what you're like drawn from geographic location or other patterns. This creates a phenomenon called a "filter bubble," says Koster, where increasingly one's perception of the world is led by online targeting. Your average online user will increasingly see only those news sources and even political candidates that agree with their own views -- and anyone who's ever Facebook-blocked a relative with offensive political views has become complicit in this sort of filtering.

...

Without clear lines and well-bounded communities, people can become confused in a way that leads to conflict. For example, with Kickstarter and Early Access games, users become hostile and controversies arise because the distinction between a "customer" and a "funder", a creator and a user, is indistinct; confusion about different types of roles and relationships within the game industry gave rise to last year's "controversy that shall not be named."

When you look at modern communities, from Twitter and Reddit to Facebook and chan boards, all the best practices -- keeping identity persistent, having a meaningful barrier to entry, specific roles, groups well-bordered and full anonymity at least somewhat difficult -- have been thrown out the window, giving rise to toxicity online, the veterans say.

As Koster notes on his blog, the discussion continues across the gaming community.

But it's not just gaming. For that, we turn to the always-fascinating, if complex, Don Marti, with his latest note: Digital dimes in St. Louis.

Don points to Jason Kint's recent article: Unbridled Tracking and Ad Blocking: Connect the Dots.

As part of my presentation, I shared a couple of charts that caused a bit of a stir among the audience of media execs charged with leading their organizations digital media ad sales businesses. The fact that these particular slides triggered such a reaction struck me as particularly timely because later that day the White House released its proposal for a Consumer Privacy Bill of Rights, which would require companies to clearly communicate their data practices to consumers and give them more control over what information is collected and how it is used.

Don wonders what all this tracking technology is destroying:

So how do we keep the local papers, the people who are doing the hard nation-protecting work of Journalism, going?

Where is all this rooted? Koster suggests that it's that gruesome and awful creation of technology, the filter bubble that Eli Pariser identified some five years ago:

For example, on Google, most people assume that if you search for BP, you’ll get one set of results that are the consensus set of results in Google. Actually, that isn’t true anymore. Since Dec. 4, 2009, Google has been personalized for everyone. So when I had two friends this spring Google “BP,” one of them got a set of links that was about investment opportunities in BP. The other one got information about the oil spill. Presumably that was based on the kinds of searches that they had done in the past. If you have Google doing that, and you have Yahoo doing that, and you have Facebook doing that, and you have all of the top sites on the Web customizing themselves to you, then your information environment starts to look very different from anyone else’s. And that’s what I’m calling the “filter bubble”: that personal ecosystem of information that’s been catered by these algorithms to who they think you are.

It's all terribly complex; it's not precisely clear where the slippery slope began, and how we crossed the line.

But the first step to getting out of the hole is to stop digging.

And to do that, you have to have the discussion; you have to know you're in the hole.

Thank you, Messrs Koster and Marti and Kint and Pariser. It's not a happy story to shout to the world, but you must keep telling it.