The bug it took three dozen engineers to fix

Don't miss this nicely presented story about a subtle but devastating performance bug in a database connection pool cache eviction policy choice which was finally solved by a task force of Facebook engineers: Solving The Mystery of Link Imbalance: a Metastable Failure State at Scale

Carlson prevails again

A beautiful win for Carlson in game eleven and he retains the championship.

Shall I make a prediction? Next up is caruana.

Turkey trot

It seems like we are at that time of the year when everybody is incredibly busy.

And there's so much to read!

• If you only watch one technical video this year, make it James Hamilton: AWS Innovation at Scale

  This session, led by James Hamilton, VP and Distinguished Engineer, gives an insider view of some the innovations that help make the AWS cloud unique. He will show examples of AWS networking innovations from the interregional network backbone, through custom routers and networking protocol stack, all the way down to individual servers. He will show examples from AWS server hardware, storage, and power distribution and then, up the stack, in high scale streaming data processing. James will also dive into fundamental database work AWS is delivering to open up scaling and performance limits, reduce costs, and eliminate much of the administrative burden of managing databases.

• Google to Quadruple Computer Science Prize Winnings to $1 Million

  The Turing Award had carried prize money of $250,000 and was jointly underwritten by Google and Intel since 2007. But Intel decided to step away as a funder, and Google stepped up and upped the ante.

• Microsoft Releases Emergency Security Update

  “The attacker could forge a Kerberos Ticket and send that to the Kerberos KDC which claims the user is a domain administrator,” writes Chris Goettl, product manager with Shavlik. “From there the attacker can impersonate any domain accounts, add themselves to any group, install programs, view\change\delete date, or create any new accounts they wish.

• Compiler Design in C

  Compiler Design in C is now, unfortunately, out of print. However, you can download a copy

• Keeping Secrets

  the conference featured the work of a group from Stanford that had drawn the ire of the National Security Agency and the attention of the national press. The researchers in question were Martin Hellman, then an associate professor of electrical engineering, and his students Steve Pohlig, MS ’75, PhD ’78, and Ralph Merkle, PhD ’79.

  A year earlier, Hellman had published “New Directions in Cryptography” with his student Whitfield Diffie, Gr. ’78. The paper introduced the principles that now form the basis for all modern cryptography

• Building a complete Tweet index

  In this post, we describe how we built a search service that efficiently indexes roughly half a trillion documents and serves queries with an average latency of under 100ms.

• Why I'm not signing up for Google Contributor (or giving up on web advertising)

  People say all kinds of stuff. You have to watch what they do. What they do, offline, is enjoy high-value ad-supported content, with the ads. Why is the web so different? Why do people treat web ads more like email spam and less like offline ads? The faster we can figure out the ad blocking paradox, the faster we can move from annoying, low-value web ads to ads that pull their weight economically.

• Your developers aren’t slow

  Feel like your team isn’t shipping fast enough? Chances are, your developers aren’t to blame.

  What’s really slowing down development?

  If it’s not your developers, what’s slowing down development? Here’s a hint: it’s your process.

• Cache is the new RAM

  You know things are really desperate when “less painful than writing it yourself” is the main selling point.

• Delayed Durability in SQL Server 2014

  With delayed durability, the transaction commit proceeds without the log block flush occurring – hence the act of making the transaction durable is delayed. Under delayed durability, log blocks are only flushed to disk when they reach their maximum size of 60KB. This means that transactions commit a lot faster, hold their locks for less time, and so Transactions/sec increases greatly (for this workload). You can also see that the Log Flushes/sec decreased greatly as well, as previously it was flushing lots of tiny log blocks and then changed to only flush maximum-sized log blocks.

• Delayed Durability in SQL Server 2014

  Like many other additions in recent versions of SQL Server (*cough* Hekaton), this feature is NOT designed to improve every single workload – and as noted above, it can actually make some workloads worse. See this blog post by Simon Harvey for some other questions you should ask yourself about your workload to determine if it is feasible to sacrifice some durability to achieve better performance.

• The Programmer's Price: Want to hire a coding superstar? Call the agent.

  Hiring computer engineers used to be the province of tech companies, but, these days, every business—from fashion to finance—is a tech company. City governments have apps, and the actress Jessica Alba is the co-founder of a startup worth almost a billion dollars. All of these enterprises need programmers. The venture capitalist Marc Andreessen told New York recently, “Our companies are dying for talent. They’re like lying on the beach gasping because they can’t get enough talented people in for these jobs.”

• git-p4 - Import from and submit to Perforce repositories

  Create a new Git repository from an existing p4 repository using git p4 clone, giving it one or more p4 depot paths. Incorporate new commits from p4 changes with git p4 sync. The sync command is also used to include new branches from other p4 depot paths. Submit Git changes back to p4 using git p4 submit. The command git p4 rebase does a sync plus rebases the current branch onto the updated p4 remote branch.

Fun and Games

With two games to go, Carlsen still maintains his 1 point advantage over Anand. But two games left means nothing is yet decided.

• Did champ fall asleep during chess title match game?

  The chess world is still buzzing over images posted to social media that appear to show world chess champion Magnus Carlsen of Norway actually falling asleep at the board during this week’s Game 8 of his world championship match with Indian challenger Viswanathan Anand of India in Sochi, Russia.

• Cool Chess Photo Trivia: Name event, Year!

  Did both the players know back then that they would meet in Sochi! Can you name the chess event and the year?

• Ten Years of World of Warcraft

  These days, so many more people have passed through the gates of Azeroth than ever played its antecedents that many don’t even know the deep wellspring sources from which it came. Most of the defining characteristics of WoW are from a long tradition that started around 1990. WoW represents the (perhaps final) evolution of the DikuMUD model.

• The Unbalanced Design of Super Smash Brothers

  I think a large part of what enables this depth to be found in Smash Brothers is that the game isn't balanced. It's not a small roster of perfectly tuned characters. It's a big game with lots of characters that creates a huge and unexplored problem space. It's up to the players to explore the nooks and crannies and see what treasures they can find. I find that incredibly exciting and compelling.

• Handmade Hero

  Handmade Hero is an ongoing project to create a complete, professional-quality game accompanied by videos that explain every single line of its source code.

• Finally, a New Clue to Solve the CIA’s Mysterious Kryptos Sculpture

  Is a Clock a Clock?

  Sanborn told WIRED that he’s always been fascinated by Berlin’s many clocks but the Berlin Clock in particular has intrigued him the most. The clock, also known as the Berlin Uhr or Set Theory Clock, was designed in the 1970s by inventor and tinkerer Dieter Binninger. It displays the time through illuminated colored blocks rather than numbers and requires the viewer to calculate the time based on a complex scheme.

• Boxer-Clad Coders Adorn Silicon Valley’s Billboard Boom

  Surging billboard demand has led to a jarring sight for those driving on San Francisco’s main highway these days -- a 50-foot picture of a pasty software engineer, lying provocatively on his side, showing a bit of chest hair and wearing only his underwear.

  “Find the hottest tech talent,” reads the billboard for technology jobs website Dice.com. The ad went up last month to tout new services for recruiting technology workers.

• Lavish Perks Spawn New Job Category: At Tech Companies, Aim-to-Please Specialists Provide Yoga Classes, Jell-O Shots; ‘We’re Like Little Elves’

  “We are just providing basic standards,” says Ms. Nguyen, 40 years old, whose title is head of workplace. Free lunch, dinner, snacks and events like a Jell-O shot-making “studio night” are a big part of what it takes to keep Pinterest’s roughly 450 employees productive and happy, she adds.

• Bandstand

  On the whole, if there were different line-ups of bands, the classic line-up of a band is shown. (Obviously, there might be times when what I consider a classic line-up is different to what other people might consider a classic line-up.) Stage positions are approximated.

• Mango’s Tropical Café Owners Unveil Track Design for World’s Tallest Rollercoaster – The Skyscraper™

  The track design of Skyscraper is totally unique to Orlando and was created specifically to begin and end inside SKYPLEX. The $250 million entertainment complex comprises approximately 495,000 square feet located on 12 prime acres in the attractions corridor. SKYPLEX will feature the area’s largest indoor theme park complete with an upscale arcading, simulators, rock climbing, bars, and a host of rides and games, as well as additional opportunities for attractions, interactive venues, restaurants and retail.

Is it an omen? The squirrel that occupies our front yard stole the lemon off our lemon bush.

Interpret that!

Other people discuss things I don't understand

I should understand these things.

But I don't.

But I must try to.

• Uber Executive Suggests Digging Up Dirt On Journalists

  Over dinner, he outlined the notion of spending “a million dollars” to hire four top opposition researchers and four journalists. That team could, he said, help Uber fight back against the press — they’d look into “your personal lives, your families,” and give the media a taste of its own medicine.

  Michael was particularly focused on one journalist, Sarah Lacy, the editor of the Silicon Valley website PandoDaily, a sometimes combative voice inside the industry. Lacy recently accused Uber of “sexism and misogyny.” She wrote that she was deleting her Uber app after BuzzFeed News reported that Uber appeared to be working with a French escort service. “I don’t know how many more signals we need that the company simply doesn’t respect us or prioritize our safety,” she wrote.

• The Coming Era of Privacy Scandals: Why the Uber story is certain to recur

  The data revolution has happened; the toothpaste can’t be put back into the tube. It’s not just the NSA which has access to enormous amounts of personal data on us; it’s any number of companies big and small, which you may or may not have ever heard of. If that data can be subpoenaed, and most of it can be, then it can also be accessed without a subpoena by people within the firm, who might have nothing better to do during their lunch hour than look up celebrities, or friends, or enemies and see what they can find. Such behavior is ignoble, to be sure – but it is going to happen.

• Uber's data makes a creepy point about the company

  A couple of years ago, there was an entry on the company's blog titled "Rides of Glory." The company examined its rider data, sorting it for anyone who took an Uber between 10 p.m. and 4 a.m. on a Friday or Saturday night. Then it looked at how many of those same people took another ride about four to six hours later – from at or near the previous nights' drop-off point.

  Yes, Uber can and does track one-night stands.

• Some thoughts on App Based Car Services

  Cars are already becoming generic. And already we have a generation coming up that gives a much smaller damn about driving than did previous ones — at least in the U.S. All that aspirational stuff about independence and style doesn’t matter as much as it used to. How long before GM, Ford and Toyota start making special models just for Uber and Lyft drivers?

• What to do about Uber

  So here's a modest suggestion: treat the city's roads like a traditional public utility, or a public resource like cell spectrum. Invite Uber, Lyft and other ride-sharing companies to tender for a license as the city's preferred ride-sharing service. Let them compete to show how they'll treat drivers and riders fairly, in their terms and conditions; and let them bid for the right to this future profit stream.

• Nick Denton On How Gawker, Uber and Facebook Will Save Humanity

  Gawker Media’s sites have also been aggressive in calling out the many supposed sins of Uber and its CEO, Travis Kalanick. But Denton has praise for the car-summoning service — high praise indeed. “Uber may do more for the world than foreign aid workers in Mozambique because at some point some version of Uber will allow for more efficient use of resources and a better standard of living,” he says.

• Stuck in the middle

  There is nothing surprising or wrong with the idea of a tech company investigating reporters. The idea that only reporters have the ability to publish is a 20th century idea. Now anyone who wants to speak can start a blog or a podcast and get up and speak.

Drip drip drip

It's snowing in Truckee!.

And it rained about half an inch, maybe more, here in the Bay Area, today.

But oh-my-goodness is it dry: 203 drought maps reveal just how thirsty California has become.

More than 80% of California is in extreme drought, according to the U.S. Drought Monitor, and the state's condition isn't expected to improve in the near future.

I'm not sure I'd want to change places with Buffalo, New York, though: Jets-Bills Game Will Not Be Played In Buffalo Due To Massive Snowfall.

A league spokesman says a severe lake-effect snowstorm hitting Buffalo has led the NFL to relocate the Bills home game against the New York Jets on Sunday.

But hopefully we can at least continue to get some rain...

Things you can put into your roasted vegetables dish

For example, if you're making roasted vegetables for the company potluck.

• Sweet Potatoes
• Yams
• Potatoes
• Parsnips
• Beets
• Carrots
• Celery
• Leeks
• Butternut Squash
• Acorn Squash
• Kabocha Squash

Clean; peel; chop.

Toss with olive oil, thyme, rosemary, salt and pepper.

Bake at 350 degrees for 45 minutes or so.

Attend company potluck.

Read, read, read

My youngest child is 23 years old!

Does this mean I am finally a grown-up?

• Alexander Grothendieck 1928–2014

  So what can all this mean for us who work in what Grothendieck described as a “mansion” in which “the windows and blinds are all closed,” while he was one of those “whose spontaneous and joyful vocation it has been to be ceaseless building new mansions”? At least he did not call our dwelling a cave. However, in complexity theory we have it worse than Plato’s cave-prisoners in not merely missing the blinding world outside, but sensing its impact as a negative image in our present ignorance of lower bounds.

• Andrew Odlyzko Strikes Again

  Last year I blogged about Andrew Odlyzko's perceptive analysis of the business of scholarly publishing. Now he's back with an invaluable, must-read analysis of the economics of the communication industry entitled Will smart pricing finally take off?.

• A Worm's Mind In A Lego Body

  The nematode worm Caenorhabditis elegans (C. elegans) is tiny and only has 302 neurons. These have been completely mapped and the OpenWorm project is working to build a complete simulation of the worm in software. One of the founders of the OpenWorm project, Timothy Busbice, has taken the connectome and implemented an object oriented neuron program.

• Keep Daly City Uncool

  Admittedly, the rumor that a proposed move to Daly City caused dissent among Reddit's ranks is met with a sense of relief around here where there's reason to fear that all those nearby gobs of money will ooze out of San Francisco and Atherton and engulf Val's, Joe's and our twin Targets like ravenous blob fueled by Bitcoin and fully-vested stock options.

• Introducing “A Field Guide to the Distributed Development Stack”: Tools to develop massively distributed applications

  In addition to being a (hopefully) useful framework, the Guide is also meant to be a living resource. So, we’ve put the source on GitHub and invite you to contribute. If you feel like we’ve missed a tool (which we most certainly have, since new things are popping up every day) or a major theme, then fork the repo and send me a pull request. We’ll be keeping this document up to date and republishing it as we watch this trend continue to grow. We’ll use O’Reilly Atlas to pull in the contributions and periodically republish the guide.

• Binary artifact management in Git

  Management of large binaries is still an unsolved problem in the Git community. There are effective alternatives and work-arounds but it’ll be interesting to see if anyone tries to solve the problem more systematically.

• Beating Ebola Means Drinking, Last Thing Patient Wants to Do

  “We will halve the mortality by firstly just stopping anti-inflammatories and giving hydration, and really pushing it,” Mardel said. “I want every man and woman in Sierra Leone to know this. I want sports personalities to be talking about it. I want everybody to be talking about it.”

• This Mod Is the Absolute Best Way to Play XCOM

  XCOM: Enemy Unknown is a great tactical game from Firaxis games. Its first expansion pack, Enemy Within, added so much amazing content that it made playing the game an entirely new experience. And now there’s a mod that’s so good it makes the expansion pack feel like a tired retread.

This time, the buzzing was not in my head

Reason number forty nine to have an unnecessary panic attack:

The other evening, I was going through my house, getting ready for bed, turning off lights, closing doors, making all my regular routine rounds.

I check the garage to make sure the light is off, but when I open the door, there is a strange sound.

It's sort of a buzzing sound, sort of a clicking sound, sort of a humming sound.

I wander around the garage for a while, trying to find the source of the sound: it's clearly coming from the car.

The car is switched off, but I wonder: did I perhaps leave the key in the ignition? leave a switch at a funny setting?

No.

The sound is coming from under the back of the car, so I get down on my tummy, shine my flashlight around, stick my arm in and try to touch various parts of the car to see if any of them are vibrating.

It's completely baffling.

So, I do the Only Sensible Thing: I go back in to the house, sit down at my computer, and search the Web.

Sure enough, there are dozens of forums filled with discussions of this behavior, and after clicking a few times, one of the messages sends me back out to the car, to look in the glove compartment, in the Owner's Manual, where, on page 9, I find:

Noise from under the vehicle.

NOTE: You may hear a noise from under the vehicle approximately 5 to 10 hours after the engine is turned off. However, this does not indicate a malfunction. This noise is caused by the operation of the fuel evaporation leakage checking system and is normal. The noise will stop after approximately 15 minutes.

And it did.

Carlsen wins game 6!

After 6 rounds, it's Magnus Carlsen 3.5, Viswanathan Anand 2.5.

Superb chess, even though it's nearly all over my head. You can read lots of analysis on the web, including here.

Until I read the analysis, I didn't understand exactly where Anand lost, but I'm comforted that after I read the analysis, I understood it.

Me, I just love the fact that they both play 1. e4.

Stuff I'm reading, mid-November edition

Today we once again came to the conclusion that the middle of November is the perfect time to go to Morgan Territory Park, and that Morgan Territory Park is the perfect place to be a dog.

• IBM X-Force Researcher Finds Significant Vulnerability in Microsoft Windows

  Cutting to the chase, VBScript permits in-place resizing of arrays through the command “redim preserve.” This is where the vulnerability is.

  redim preserve arrayname( newsizeinelements )

  VBScript.dll contains a runtime evaluation method, CScriptRuntime::Run(VAR *), which farms out the SafeArray redimension task to OleAut32.dll with the SafeArrayRedim(…) function. Essentially, what happens is that fairly early on, SafeArrayRedim() will swap out the old array size (element count) with the resize request. However, there is a code path where, if an error occurs, the size is not reset before returning to the calling function, VBScript!CScriptRuntime::Run().

• Microsoft Fixes Bug That Dates Back to Windows 95

  According to Microsoft, the bug resides in the Windows Object Linking and Embedding (OLE) technologies that debuted in the 1990s as a way for applications to share information with each other.

• Unintended Consequences of the Information Economy

  The DoD can’t just dial up more innovation capacity by throwing money at the problem, like they did in WWII. Nor, in a free country, can the U. S. government just mandate for whom companies choose to work. Innovation capacity requires not only brilliant engineers, who are hard enough to come by, and who cannot be easily identified in the job market, but also a willingness to accept a lot of risk: to try and perhaps to fail, over and over. To old school 20th century managers, this looks a lot like waste, but in fact it’s a necessary part of the innovation process. The economics of conflict is changing just like the economics of everything else is changing.

• Wil Wheaton: Anonymous trolls are destroying online games. Here’s how to stop them.

  It’s time to break this cycle—and to teach gamers that they can compete without being competitive, that they can win and lose without spewing racist, misogynist, homophobic bile at their fellow gamers. But doing so requires casting off the cloak of anonymity.

• Social media is broken

  Anonymity can be very important for the marginalized, for whistleblowers, etc. But within their communities of trust they build reputation, including pseudonymous reputation. The real issue is feeling free of reputation, which equals feeling free of consequence. That is where bad behavior comes from.

• Why I love databases

  To many of my colleagues and friends, the database is a magical black box system, too scary and complicated to understand. I wanted to change that.

  While talking about databases, the topic of distributed systems cannot be ignored. Most modern databases are distributed, either implicitly (distributed clustered databases) or externally (a single application connected to multiple databases via application-level sharding).

  This post is a confession of my love for databases and distributed systems. It is mostly targeted towards programmers like me, application developers who regularly interact with databases.

• Climbing off the CTO ladder (Before We Fall Off)

  Spotify has recently started thinking about the non-management career track and we are working to see if there is a program we would like to adopt. We’ve had feedback from engineers and managers alike that we need to add a tech ladder; however, I am not convinced that adding a ladder fixes everything. The second ladder is a blunt instrument that can easily add some clarity to our growth ambitions, but it is not a complete solution. Part of management is optimizing for the productivity of the team, not our own work streams. This should not be taken lightly as it will have an effect on the organization’s culture. Understanding this may help invest in a technology ladder that works for the culture of your organization.

• Optimizing Disk IO and Memory for Big Data Vector Analysis

  For “big data” datasets where the size of data is significantly larger than the size of memory, the most common bottleneck is disk bandwidth. Disk bandwidth of the highest-end disks remain on the order of hundreds of megabytes per second, while memory bandwidth is usually at least an order of magnitude faster. Furthermore, very little work is required of the CPU per record (just two predicate evaluations and a sum) --- database queries tend to be far less CPU-intensive than other domains (such as graphics rendering or scientific simulations). Hence, step 1 is often a bottleneck.

• Intro to Distributed Hash Tables (DHTs)

  Lets assume that you want to get to some place, and you are not sure where it is. A good idea would be to ask someone how to get there. If you are very far from your destination, most likely the person you asked will give you a very vague description of how to get there. But it will get you starting in the correct direction.

  After you advance a while, you can ask somebody else. You will get another description, this time more a detailed one. You will then follow this description, until you get closer.

  Finally when you are really close, you will find someone that knows exactly where is that place you are looking for. Then your search will end.

• Yelp Prison Review Faxbot

  Fletcher Bach and I recently discovered that there are reviews of prisons on Yelp. Some of these reviews are snarky one-liners -for example, one yelper describes Rikers Island as a “great island getaway right in my own backyard”. Other reviews appear to be honest first person accounts. Some people review what it’s like to visit the prison; others describe their experiences as inmates.

• Android 5.0 Lollipop, thoroughly reviewed

  Android 5.0 Lollipop is at least the biggest update since Android 4.0, and it's probably the biggest Android release ever. The update brings a complete visual overhaul of every app, with a beautiful new design language called "Material Design." Animations are everywhere, and you'd be hard-pressed to find a single pixel from 4.4 that was carried over into 5.0—Google even revamped the fonts.

• Google's New, Improved Android Will Deliver A Unified Design Language

  As Android lead designer Matias Duarte demoed it on stage, he explained that it moved with the physics of card stock, but also splash with your touch, like “ink rippling in a pond.” He clearly put it better than I can, though I’d add that Android‘s core UI has long been cleanly designed, but was always a bit cold. Material Design adds a bit of human warmth back to the equation.

• Cooking the Alinea cookbook

  Allen Hemberger cooked his way through one of the most complex cookbooks out there, the Alinea cookbook. Aside from the chefs who work in the kitchen there, Hemberger's probably the only person to have made every single recipe. These recipes aren't easy; look at the last one he prepared...he even struggled to find the correct ingredients.

The Owens Valley dust bowl

Can plowing the soil actually REDUCE the dust that blows away?

And can it be possible to reduce the dust pollution in the Owens Valley while also INCREASING water deliveries to the Los Angeles area?

My mom sent me an absolutely fascinating story from the Los Angeles Times: New dust-busting method ends L.A.'s longtime feud with Owens Valley

The story describes a technique which certainly falls into the "seems too good to be true" category:

The new solution is relatively inexpensive and nearly waterless, DWP officials said. It involves using tractors to turn moist lake bed clay into furrows and basketball-sized clods of dirt. The clods will bottle up the dust for years before breaking down, at which point the process will be repeated.

The method was first tested in the early 1990s, then tabled out of concern the furrows and clods would disintegrate after a few rains. Two years ago, the DWP resurrected the idea and tested it on several acres of lake bed, but on a much larger scale, with furrows 2 to 3 feet deep. The results showed promise, provided the treated area has clay soil and flooding infrastructure in place.

Don't underestimate the stakes here: this is big business, and big money. And the other techniques that have been attempted over the previous quarter-century cost much, much more:

The new method will cost DWP customers about $1 million per square mile — three times less than shallow flooding. The cost of reducing dust with gravel, which has been applied to swaths of the lake bed, is about $25 million per square mile, officials said.

The utility has already spent $1.3 billion in accordance with a 1997 agreement to combat dust over a 40-square-mile area, reducing particle air pollution in the region by 90%.

This represents dramatic progress for the Owens Valley, which has been struggling with these issues for 100 years.

Perhaps more importantly, the Owens Valley is probably the most-watched, most-studied, most-analyzed, and most-fought-over area in the water battles that occupy the entire Western United States.

So what happens in the Owens Valley doesn't just affect the Owens Valley; it affects fully one third of the country.

For example, consider this discussion about water issues in southern Nevada: Las Vegas and the Groundwater Development Project, which has an entire chapter entitled "Remember Owens Valley".

Not much grows on the exposed lakebed, and that’s where the trouble starts. There’s dust, lots of it. Sweeping winds come roaring down the valley and create tremendous dust storms. At times there’s so much dust and the visibility is so poor the locals call it the “Keeler fog” after the small remnant of a town on what used to be the lake’s eastern shoreline. Recently, a pilot followed a dust plume from the lake-bed all the way into the Grand Canyon.

Nothing about this process has been easy. A year ago, ARID Journal covered the controversy in detail: Particulate Matters: Settling the Dust on the Owens Dry Lakebed.

For the first 50 years that it diverted Owens River water to Los Angeles, the LADWP denied that dust was a problem on the river’s former lakebed. When in 1976 scientists at China Lake Naval Weapons Center in Ridgecrest, California, photographed clouds carrying an estimated 40,000 metric tons of fine alkali grit billowing out of the Sierra into the neighboring Mojave Desert foothills of Kern County, the LADWP claimed that Inyo County had some of the best air in the country and that, “there has been no substantiation of adverse health effects of alkali dust.” In 1987 the U.S. Environmental Protection Agency deemed the severity of the fine-grain pollution issuing from Owens dry lake as the worst in the country, outside of forest fire smoke, as often as 24 days a year. LADWP was on record that the land impacted by its water exports was “such a small area we think it is insignificant.

Over decades of study, multiple techniques were attempted:

More than a dozen suppression methods were considered in the lead-up to the first mitigation projects, including covering the lakebed in used automobile tires, but only three were eventually approved for widespread use: gravel cover, plants, and shallow flooding. Gravel, at $33 million per square mile to install, was deemed prohibitively expensive. Plant cover, most of which had to be salt grass, cost $15 million per square mile to install, then it needed irrigating. By far the cheapest immediate fix for a water company was to install bubblers to provide shallow flooding. LADWP estimates that the up-front cost of this was more like $12.9 million per square mile.

And the battle went on in the courts:

When a new dust abatement notice for 2.93 additional square miles arrived that summer, Nichols called in the lawyers in what has proved a sustained assault on the 1998 dust deal. Schade and Great Basin also went to the courthouse, filing suit against LA for non-compliance on an outstanding order. By October 2012, over in federal court, LADWP was suing Schade’s department, naming him personally as a capricious and rogue regulator, and also naming the California Air Resources Board, the US Environmental Protection Agency, the California State Lands Commission, and the federal Bureau of Land Management as colluders.

So when research by an Orange County landscape architecture firm named Nuvis came up with a new idea, it must have seemed like a fantasy at first:

Those decisions include widespread use of an as yet un-validated waterless dust control method called “tillage,” which will have to be approved by regulators including Schade before the meandering furrows shown in the Nuvis schematics could be plowed into the lakebed. “We’re hoping that tillage, basically like farm tillage, will be approved,” says Adams. “It costs about 10% of what it costs to do flooding. It’s a huge savings for rate payers.”

So, let's hope this this new technique actually works.

And let's hope that agencies across the west are able to learn from this, and spend less time (and money) fighting in the courts, and more time (and money) figuring out how to use water effectively without destroying the land.

And let's all say a big thanks to Ted Schade.

Game on!

It's going to be a competition this time -- Anand won game 3, to even the score at 1.5 to 1.5.

Super!

An early lead for Carlsen

After two games in the 2014 World Chess Championship, Magnus Carlsen has won game two.

In game one, Anand had white, and although it was hard-fought, Carlsen held the draw.

In game two, Carlsen had white, and won.

The details of the games, of course, are far above my ability to comprehend, but they are certainly beautiful to watch.

Carlsen Anand II

There's just three days until Carlsen-Anand begins.

The match will be held in Sochi, Russia: here's the official site.

The games are played at 3 PM Sochi time; according to WorldTimeBuddy, this will be 4 AM my time.

So I can wake up each day with the very best of chess!

Michael Aigner has a nice preview here.

3 tools for system software testing

I very much enjoyed reading three papers which all happened to be part of the same session of the 11th USENIX Symposium on Operating Systems Design and Implementation.

The session that captivated me was called "Pest Control," and it included these papers:

• Torturing Databases for Fun and Profit

  Here we propose a method to expose and diagnose violations of the ACID properties. We focus on an ostensibly easy case: power faults. Our framework includes workloads to exercise the ACID guarantees, a record/replay subsystem to allow the controlled injection of simulated power faults, a ranking algorithm to prioritize where to fault based on our experience, and a multi-layer tracer to diagnose root causes.

• All File Systems Are Not Created Equal: On the Complexity of Crafting Crash-Consistent Applications

  We find that applications use complex update protocols to persist state, and that the correctness of these protocols is highly dependent on subtle behaviors of the underlying file system, which we term persistence properties. We develop a tool named BOB that empirically tests persistence properties, and use it to demonstrate that these properties vary widely among six popular Linux file systems. We build a framework named ALICE that analyzes application update protocols and finds crash vulnerabilities, i.e., update protocol code that requires specific persistence properties to hold for correctness.

• SKI: Exposing Kernel Concurrency Bugs through Systematic Schedule Exploration

  In this paper, we propose SKI, the first tool for the systematic exploration of possible interleavings of kernel code. SKI finds kernel bugs in unmodified kernels, and is thus directly applicable to different kernels. To achieve control over kernel interleavings in a portable way, SKI uses an adapted virtual machine monitor that performs an efficient analysis of the kernel execution on a virtual multiprocessor platform. This enables SKI to determine which kernel execution flows are eligible to run, and also to selectively control which flows may proceed.

In a lot of ways, these papers are all very similar. They all examine the very challenging problem of finding bugs in extremely complex system software.

And they all take the approach of building a tool to find such bugs.

And all of their tools use lower-level capabilities to examine and interact with the software under test: one tool captures the SCSI commands that are sent to the storage system, and is capable of then manipulating those captures to simulate various power failures and disk errors that can occur; another tool captures the system calls that are made from the application to the operating system, and can manipulate those system calls in various ways; the third tool uses virtualization infrastructure to capture and manipulate the actions between the operating system and the (virtual) hardware.

This sort of tool-based testing is wonderful, I think.

I've seen similar tools (in concept) that do things with captured/replayed network traces; that do things with captured/replayed transaction logs; that do things with captured/replayed web server logs; etc.

It's extremely hard to go "the last mile" when testing complex system software, so I'm always enthusiastic when I see people building powerful testing tools.

How can it be November already?

Must be all that Halloween candy, it affected my brain somehow...

• The Window, the Patch, & the Artifact

  Earhart and Noonan departed Miami for San Juan, Puerto Rico early on the morning of Tuesday, June 1, 1937. A Miami Herald photo of the plane taxiing out for takeoff shows that some time between whenever the photo with Nilla Putnam was taken on Saturday and the departure for San Juan on Tuesday, the window had been replaced by a shiny aluminum patch.

• Why the Chess Computer Deep Blue Played Like a Human

  The extent to which these results speak to human analogizing is unclear. But they open the possibility that our process of analogy making may be even less rational and more stochastic than we suspect, and that the deep archetypes we match against in our brain might bear far less relationship to reality than we might think. Underneath our apparent rationality may lie neurobiological processes that look considerably closer to random trial and error. In this view, human creativity and randomness go hand in hand.

• Pro Git, 2nd Edition

  Written by Git pros Scott Chacon and Ben Straub, Pro Git (Second Edition) builds on the hugely successful first edition, and is now fully updated for Git version 2.0, as well as including an indispensable chapter on GitHub. It’s the best book for all your Git needs.

• Writing Reviewable Code

  This document describes a strategy for structuring changes used successfully at Facebook and in Phabricator.

• The ASF @ 15 -- Chairman's Statement

  Apache has seen amazing success over the last 15 years. Not only do ASF projects impact almost every area of computing, but the Apache License, our Contributor License Agreements (CLAs), and our pattern of open, collaborative development (often known as "The Apache Way") continue to influence Open Source projects outside of the ASF.

• Wolf Wars

  In February, after the Lost Creek pack loped past the border of Yukon–Charley, state biologists shot all 11 wolves from a helicopter, wiping out 20 years of research in a single day. Had it been a few years earlier, the state agents charged with predator control would’ve seen Burch’s radio collars and spared at least some of the Lost Creek pack. But no longer, Burch says: “There’s no negotiations anymore. They kill almost all the wolves they can find. These last two winters they’ve pretty well gotten most of them.”

• Conference writeups, HotNets 2014

A standardized mistake is still a mistake

The software industry, like most technical industries, is plagued with the necessary evil of standardization.

Standards certainly are necessary. Without them, the software industry would not be making much progress.

But they can be very evil, too.

Consider, for example, the ancient (in software terms) LDAP standard, most recently re-issued about ten years ago as RFC 4510 and its sub-standards (4511 through 4519).

These standards cover a piece of software called a "directory server".

• A directory is a fairly simple database, and is most commonly used to hold a database of usernames and other information about those users, such as their passwords.
• And a directory server is most commonly used to provide an organization-wide security feature in which you can use the same password to log in to multiple pieces of software, as long as all of those pieces of software authenticate against the same directory server.

So, it's a fairly nice thing to have, and so it's become standarized and common.

But evil hides within this standardized mechanism. Lots of evil.

Consider one particular bit of evil, as discussed in Section 5.1 of RFC 4513, Simple Authentication Method.

This section describes how you can use a directory server to authenticate a user:

Servers that map the DN sent in the Bind request to a directory entry with an associated set of one or more passwords used with this mechanism will compare the presented password to that set of passwords. The presented password is considered valid if it matches any member of this set.

That's pretty good. In fact, that's the meat of the whole feature, right there: you can use this "directory server" thing to authenticate your users, by making a Bind request with the user name and the password that they gave you, and find out if their password is valid.

That's a really important feature. You shouldn't build your own password management and authentication functionality, after all; you should use an established, standardized mechanism.

So, cool.

But, uh-oh, look at what the standard contains, right here, in Section 5.1.2, Unauthenticated Authentication Mechanism of Simple Bind:

An LDAP client may use the unauthenticated authentication mechanism of the simple Bind method to establish an anonymous authorization state by sending a Bind request with a name value (a distinguished name in LDAP string form [RFC4514] of non-zero length) and specifying the simple authentication choice containing a password value of zero length.

What's that?

IT'S A HAND GRENADE!

What this says, in simpler words, is:

If you try to use a directory server to authenticate your users, it is YOUR JOB to make sure that the password that the user types in is at least 1 letter long, because the directory server doesn't check that!

That is, in even simpler words:

WARNING! There is a HORRIBLE bug here, but it's been here since the beginning and so we standardized the bug.

I mean, really, it should have been obvious, when the committee sat down to write this manual, when somebody typed in the words:

Unauthenticated Authentication Mechanism

That should have been a full blocker right there. I mean, can you imagine? Just say those words out loud to yourself:

Unauthenticated Authentication Mechanism

Wow.

Now, the authors of RFCS 4513 do at least try to make this clear (on page 14 of a 33-page document that is 1 of 10 documents that define how directory servers work):

The distinguished name value provided by the client is intended to be used for trace (e.g., logging) purposes only. The value is not to be authenticated or otherwise validated (including verification that the DN refers to an existing directory object). The value is not to be used (directly or indirectly) for authorization purposes.

Unauthenticated Bind operations can have significant security issues (see Section 6.3.1). In particular, users intending to perform Name/Password Authentication may inadvertently provide an empty password and thus cause poorly implemented clients to request Unauthenticated access. Clients SHOULD be implemented to require user selection of the Unauthenticated Authentication Mechanism by means other than user input of an empty password. Clients SHOULD disallow an empty password input to a Name/Password Authentication user interface.

This is the language of a standards committee waving their hands.

This is the language of a standards committee saying:

If you have the bad luck to try to write a program which uses a directory server to authenticate users, and you haven't had the good fortune to stumble across these few paragraphs on page 14 of document 4 of an 10-volume manual set written many years ago, you're almost certainly going to build a gigantic horrible super-critical security bug into your program.

Sorry about that.

And this isn't just a theoretical issue; these bugs exist, and they're super-critical, and they're being introduced all the time! Here's one: ldap authentication with blank password

When using LDAP authentication, providing a blank password is treated as a successful login. The cause is that the back-end LDAP server treats a bind with an empty/blank password as an anonymous bind, and I suspect the LDAP code in artifactory simply looks for a successful bind. This probably only happens with certain LDAP server implementations, but SunONE LDAP (which we use at my site) does this.

These bugs, unfortunately, are all over the place.

Here's one.

Here's one.

Here's one.

Here's one.

Believe me: there are tens of thousands of these bugs; they might be getting created faster than they are being found and fixed. There are so many of these bugs, that people write entire research papers just on the topic of how this introduces so many bugs..

The people who provide directory servers, of course, aren't terribly proud of this, and many modern directory servers do at least default to having this feature turned off, which helps a little bit. But they generally don't make a lot of noise about MASSIVE SECURITY HOLES IN THEIR OWN SOFTWARE WHICH THEY WERE REQUIRED TO PROVIDE BECAUSE IT'S PART OF THE STANDARD. Instead, they say things like:

the default behavior is for server to return a unwillingToPerform result code when someone tries to bind using a null password.

We won't go any deeper into this 'feature', those interested in the rational behind it and the associated drawbacks can read the following links :

This is documentation written by a programmer who's trying to tell you:

I didn't want to provide this feature.

Please don't ever turn this feature on.

If you ever think you want to turn this feature on, go read this 33-page manual that says why I had to let you turn this feature.

It's just a mess, and has led to the world being sprinkled with well-meaning but confusing web pages hinting at the evil that lurks within:

WARNING: An attempt to bind with a blank password always succeeds because the LDAP protocol considers this to be an "anonymous" bind, even though a username is specified. Always check for a blank password before binding.

It's hard to write software, and it's hard to write standards, and it's certainly hard to write software standards.

But if you ever find yourself on a standards committee, working on a standard, try not to be That Guy.

Try not to standardize a mistake.

Just pretend you're Gandalf, standing there as the evil approaches, and scream at the top of your lungs:

You Shall Not Pass!