On the great joy of reproducing a bug

Dave Winer completely nails it: Reproducible

This is the kind of "bug report" that makes programming for users so frustrating. You hear a user is losing data, but have absolutely no idea how to reproduce the problem. It's something you've never seen the software do, can't imagine it actually doing. The description sounds like magic, but nothing in software happens by magic, all programmers will tell you. I believe it's happening but I have no idea why.

And then, after plenty of work and sweat and tears

we had arrived at the Holy Moment in debugging -- reproducibility. I told him "reproducible" is the programmer's favorite word. If you can tell me the steps to reproduce the problem, then I can find it and fix it. Until it's reproducible all I can do is share your frustration.

Amen.

Warriors advance in the playoffs

Having swept New Orleans, the Warriors now face Memphis, a much stronger team.

Bleacher Report breaks it all down in tremendous detail: Grizzlies Will Present Brand-New Playoff Challenge for Hungry Warriors Squad

But the NBA is about matchups, and not all matchups are equal. Just as the Grizzlies will have to solve, or at least partially impede, Curry and Thompson, the Warriors will have to slow down Memphis' post play.

And while it's easy to game-plan rotations and plan for various scenarios, on-court situations usually lend themselves to chaos. Will the Grizzlies' perimeter players be able to knock down shots and punish Golden State for overhelping? Will Green and Bogut handle their business enough to prevent the need for help in the first place?

Game 1 is Sunday, May 3rd.

Ta-Nehisi Coates on Baltimore

In The Atlantic, Ta-Nehisi Coates: Nonviolence as Compliance.

Now, tonight, I turn on the news and I see politicians calling for young people in Baltimore to remain peaceful and "nonviolent." These well-intended pleas strike me as the right answer to the wrong question.

...

When nonviolence is preached as an attempt to evade the repercussions of political brutality, it betrays itself. When nonviolence begins halfway through the war with the aggressor calling time out, it exposes itself as a ruse. When nonviolence is preached by the representatives of the state, while the state doles out heaps of violence to its citizens, it reveals itself to be a con. And none of this can mean that rioting or violence is "correct" or "wise," any more than a forest fire can be "correct" or "wise."

Fragility

Programmers have an expression for a particular type of flaw in program code.

That's fragile, they say.

When a programmer points to a segment of code and calls it fragile, they mean something very specific.

They mean that a minor, innocent-looking, apparently-unrelated change to the overall program at some later date would cause this bit of code to unexpectedly fail without warning.

It might be because this code is in fact linked to the behavior of some other part of the system, but the linkage is not made explicit. For example, you might have an array which needs to be the same size as some other data structure elsewhere in the system, but there is nothing that is checking that those two conditions match, so changing the other part of the system would break an assumption that this code doesn't check.

In programming, this sort of programming mistake is typically called the Don't Repeat Yourself principle, but there are lots of other particular ways to make "fragile code.

I was thinking about this the other day when I read an intriguing essay on the Nautilus site: Why the Flash Crash Really Matters.

The essay talks about the financial system, and compares it to various other situations in which an otherwise minor event had catastrophic consequences:

This is why asking whether the Waddell & Reed sale, or the behavior of a manipulative trader, really caused the crash is a mistake. The disparate “causal” explanations of the crash can’t be reconciled with each other for a simple reason: They aren’t in conflict. The Flash Crash was an emergent phenomenon. Just as any grain of sand might cause the sandpile to collapse, and as Three Mile Island’s meltdown could be attributed to a failed pump, stuck valve, or operator error, the trigger for the Flash Crash could have been related to Sarao, Waddell & Reed, or something else entirely. The true roots were in the complexity of the system itself.

In large-scale systems programming (database systems, distributed systems, web servers, network file systems, etc.), there is a problem that arises when the system reaches a certain size and complexity: you can no longer hold the whole thing in your head. If you aren't careful, when your system reaches that size, you will find it completely breaking down: bugs crop up left and right, unexpectedly, faster than you can fix them; you feel like you're playing "Whack-a-Mole" with the breakages in your software.

The only way out of this is to graduate to a whole new type of system design and implementation. Your overall system must be componentized; individual modules must have clear responsibilities and clean interfaces; the boundaries between modules must be well-known to all the teams; extra care must be taken with code paths which cross module boundaries; system interfaces must check their parameters, assert their pre-constraints, and generally validate that the overall rules of the system are being obeyed.

Is the modern international financial system effectively a giant software system? The authors of the Nautilus essay clearly think so:

In the years since the Flash Crash, the SEC has implemented measures to reduce tight coupling in the markets, and exchanges now pause trading if there are drastic price moves in individual securities. These measures help, but are they enough? The fundamental interactive complexity of the market and the unpredictable and difficult-to-observe interactions between software components, trading models, and market participants remain in place.

"Tight coupling," eh? Yes, that's a code smell.

It won't be easy to re-architect the world's financial system, to remove its fragility and make it more scalable and less susceptible to catastrophic failures.

But, somehow, it needs to be done.

On testing strategies, and end-to-end testing

The straightforwardly-named Google Testing Blog is an instance of the "group blog" category, where a collection of people, all of them Google employees (as far as I know), publish articles about the art and science of software testing.

I read the Google Testing Blog faithfully, partly because I'm rather obsessive about software testing.

Computer programming is a profession that appeals to and rewards the obsessive, and within that broader group of people, the still more obsessive sorts like me often obsess about testing.

Computer programmers obsess about testing the way that car lovers obsess about oil changes, the way that NBA athletes obsess about free throws, the way that sushi chefs obsess about knife selection. Testing is a tool in the programmer's toolkit, but you when you see it wielded with experience and training, it is an amazingly powerful tool.

Anyway, I found myself reading a recent article on the Google Testing Blog: Just Say No to More End-to-End Tests.

The article sets up a strawman to attack:

Testers can invest their time in writing many types of automated tests, including unit tests, integration tests, and end-to-end tests, but this strategy invests mostly in end-to-end tests that verify the product or service as a whole. Typically, these tests simulate real user scenarios.

I'm not sure where the author came up with this strawman, frankly. In all my decades of professional software development, it's been a long, long time since I've been around anyone who's suggested that we invest "mostly in end-to-end tests that verify the product or service as a whole." However, I'm sure there could be such people, and indeed we see that there are entire books on the subject (of course: there are entire books on any subject).

Anyway, back to the Google Testing Blog. The author then proceeds to relate a

composite sketch based on a collection of real experiences familiar to both myself and other testers

The article uses these "real experiences" to demolish the idea of end-to-end testing with a scenario so bizarre, so fanciful, so implausible that I can hardly believe it:

Let's assume the team already has some fantastic test infrastructure in place. Every night:

• The latest version of the service is built.
• This version is then deployed to the team's testing environment.
• All end-to-end tests then run against this testing environment.
• An email report summarizing the test results is sent to the team.

The deadline is approaching fast as our team codes new features for their next release. To maintain a high bar for product quality, they also require that at least 90% of their end-to-end tests pass before features are considered complete. Currently, that deadline is one day away:

Days Left Pass % Notes
1 5% Everything is broken! Signing in to the service is broken. Almost all tests sign in a user, so almost all tests failed.

Uhm, what?

This is wrong on so many levels that it's hard to know where to start.

• Did any of these tests ever pass?
• When was the last time they passed?
• Did they all pass with 2 days left?
• What did the team do on the day when the "email report summarizing the test results" first reported that "almost all tests failed"?

Whatever went wrong with this project, though, one thing is very clear to me:

The testing strategy is not the problem here.

Any team which allowed themselves to think they were at a point where the "deadline is one day away" and yet "almost all tests failed" is so poorly managed, so inexperienced, so lacking in commonsense that they aren't going to solve their problems by blaming the testers or their test strategy.

Tests are one barometer of project progress, but there are many other metrics that any successful software development project uses, combined with that innate sense that experienced software developers acquire that tells them just how close they are to something that is actually ready.

And if you choose to ignore the information that's available to you, that's your fault. If the testers had followed a different strategy, and produced a different set of tests, the team could just as well have ignored that data entirely, too.

Trying to put aside the invective of the article, and its caricatured depiction of a project wildly out of touch with reality, what is the article really trying to say?

I think the points they wish to make are:

• The sooner you can receive feedback from your tests, the sooner you can act on it
• Smaller, more focused tests are cheaper to write and faster to run
• If your tests run fast, you can run them very often
• If you run your tests very often, they will more clearly point to the instant when a problem was introduced into the code
• But if you test only individual components or modules, problems can creep in where the modules and components must be assembled into larger software systems, so don't entirely omit complete system tests.

Really, this is well-trodden ground. Every time I see an article like this, I bemoan the fact that Martin Fowler's work on Continuous Integration is now 15 years old, yet seems to be so little-known to so many people.

Let's look in detail at what Fowler wrote, way back in September 2000, in the original Continuous Integration article:

An important thing to decide is what makes a successful build. It may seem obvious, but it's remarkable how this can get muddy. Martin once reviewed a project. He asked if the project did a daily build and was answered in the affirmative. Fortunately Ron Jeffries was there to probe further. He asked the question "what do you do with build errors?" The response was "we send an e-mail to the relevant person". In fact the project hadn't succeeded in a build for months. That's not a daily build, that's a daily build attempt.

And Martin Fowler isn't the only one who's been talking about these basic principles for decades. For example, consider Joel Spolsky's Daily Builds Are Your Friend (again, nearly 15 years old):

If a daily build is broken, you run the risk of stopping the whole team. Stop everything and keep rebuilding until it's fixed. Some days, you may have multiple daily builds.

The Google testing team's project didn't have a daily build, they had a daily build attempt. What they needed was the classic CruiseControl lava lamp. What they needed was a team-wide commitment to treat breaking the build seriously:

On large teams, one good way to insure that breakages are fixed right away is to do the daily build every afternoon at, say, lunchtime. Everyone does as many checkins as possible before lunch. When they come back, the build is done. If it worked, great! Everybody checks out the latest version of the source and goes on working. If the build failed, you fix it.

I think that the Google Testing Team are primarily trying to convey the notion that different types of tests are useful for different purposes, and you need to have a complete collection of tests, using lots of different testing approaches, to consider your testing strategy complete.

In fact, they even discuss the notion of the "testing pyramid" at the end of their article, though it's a shame that they don't point to the original source of the "testing pyramid" notion, Mike Cohn's The Forgotten Layer of the Test Automation Pyramid

I'm pleased that the Google Testing Blog is publishing articles on testing, I just wish they'd dig a bit deeper into the history of the field, and take a more modern approach, rather then putting up strawmen that haven't been in favor in decades and then tearing them down as if they'd just had a bold new vision of how to build quality software.

Mission Burrito

If I ever moved to the East Coast, how would I get my Mission Burrito?

I guess I'd just have to hope that The Alameda-Weehawken Burrito Tunnel was operational.

I got tele-scammed

The other day, my mobile rang.

I was in a meeting, so I just sent the call to voicemail.

Later, I listened to the voicemail. A robotic voice droned:

Hello. We have been trying to reach you. This call is officially a final notice from IRS, Internal Revenue Services. The reason of this call is to inform you that IRS is filing lawsuit against you. To get more information about this case file, please call immediately on our department number NNN NNN NNNN.

It was chilling. Lawsuits? The IRS is filing a lawsuit against me?

But something about the call didn't sound right.

Well, actually, MANY things about the call didn't sound right:

• It was a robot, not a person
• It didn't greet me by name
• It was full of awkward, incorrect English ("Internal Revenue Services"?!!)

Something clicked in my brain and I remembered reading something last fall: Scam Phone Calls Continue; IRS Unveils New Video to Warn Taxpayers.

The new Tax Scams video describes some basic tips to help protect taxpayers from tax scams.

These callers may demand money or may say you have a refund due and try to trick you into sharing private information. These con artists can sound convincing when they call. They may know a lot about you, and they usually alter the caller ID to make it look like the IRS is calling. They use fake names and bogus IRS identification badge numbers. If you don’t answer, they often leave an “urgent” callback request.

Yep, that matched, quite well.

A nice article at the South Bend Tribune was helpful, too, as it even included the same fake phone number that had appeared on my phone. Credit 'charge' appears very real

I have repeatedly written about IRS scam telephone calls but I am doing so again as your BBB continues to receive many questions from area residents who are concerned about receiving such calls. Caller IDs are showing all kinds of phone numbers, which pretty much indicates the numbers are being spoofed. Some have reported their Caller ID shows 585-310-3870, 725-422-5697 and 726-597-6584, but the IRS impersonator provides different numbers on the message.

Most recipients are being told “this is your final notice from the IRS” and “a lawsuit is being filed against you for failure to pay taxes.” Some are saying if the taxes are not paid at once, a warrant will be issued for your arrest and the police will be coming after you. Consumers are then told taxes must be paid “immediately.” Instructions are given to wire the money via Western Union or get an advance cash card such as Green Dot MoneyPak from your local drugstore or retailer namely Wal-Mart, Kmart or Target.

I read about lots of scary, annoying stuff, but rarely do I actually get one of these myself.

In a weird way, it was good to get one; it kind of was a tune-up, a practice exam, a drill.

A good reminder that it's a strange world out there, and you should stay on your toes and not fall for the nasty scam.

Oh, and yes: I simply deleted the voicemail (though I did file a complaint on the FTC's website for reporting telescams, and I did re-check that my phone is on the do-not-call list, which it has been for years).

Fess up

OK.

Come on, admit it.

Be honest.

We just don't have a freaking clue.

• Delayed flash crash arrest may herald future spoofing detection woes

  Tim Massad, the head of the Commodity Futures Trading Commission, which oversees the trading of futures and swaps, said on Wednesday that it took so long to charge Sarao because of the size and complexity of U.S. derivatives markets. "These are huge markets," he said. "There's a lot going on."

• Trading at the speed of light

  The faster your trading system, the quicker you can take advantage of those discrepancies. But that is not all the high-frequency traders can do. They can “front-run” news events, jumping on anything that might move a price, and get in and out of a stock before anyone else knows that something has happened. By trading at such speed, they can even get into an equity in the gap between the placing of a large buy order and the execution of it, and thereby buy just before the price rises.

• ‘Flash crash’ charges spark alarm over regulation of US markets

  Sherrod Brown, the top Democrat on the Senate banking committee, told the Financial Times: “It’s encouraging that the Justice Department and [Commodity Futures Trading Commission] are pursuing this case, but troubling that it has only come to light now with the help of a whistleblower who invested substantial time in putting the pieces together.”

• 'Flash Crash' arrest shakes investors' confidence

  Adding to concerns, the CFTC was alerted to Sarao's alleged misdeeds by a whistle-blower, who has not been identified, according to Shayne Stevenson, who represents the whistle-blower through Hagens Berman law firm in Seattle. Stevenson said his client brought "high-quality information" about "market manipulation" to the CFTC, which alerted the DOJ.

• Accused British 'flash crash' trader fights extradition to U.S.

  A British man accused of market manipulation that contributed to the May 2010 Wall Street "flash crash" said he opposed being extradited to the United States, while the operator of the market where he traded sought to rebut prosecutors' suggestion that futures helped cause the crash.

• Roots of 'flash crash' go back further than you thought at CME

  More than a year before the May 6, 2010 "flash crash," CME Group noticed questionable trading in its E-mini market by a particular electronic trader who was placing orders and cancelling them.

  As the crash whipsawed the futures and stock markets in 2010, CME saw the suspicious activity again and warned the trader that day that orders must be placed “in good faith,” without an intent to cancel. The trader responded two weeks later: “Kiss my ass.”

• How computerized trading in the hands of a nobody in Britain allegedly crashed the stock market

  Not everyone agrees.

• A Sweatpants-Wearing Rando Might Have Caused the Flash Crash

  He seems to have executed his trades out of a modest, semi-detached house under the Heathrow flight path that he shared with his parents. He used off-the-shelf software that he souped up to make his bets. He named one shell company "Nav Sarao Milking Markets Ltd." The guy showed up to court in a pair of white sweatpants.

• 'Flash Crash' Arrest Raises More Questions Than Answers

  But surely, since then regulators went through the action that took place on and around the day of the Flash Crash with a very fine comb? And if so, why did Sarao's alleged actions not jump at them as strange at the time?

• Why Did It Take So Long To Catch The ‘Flash Crash’ Futures Trader?

  The CME Group declined to comment, citing the ongoing investigation. But analysts worry that the CME’s revenue model interferes with its motivation to police trading. The more trades that zip through the exchange, the more money it makes. That means it could be disincentivized from tackling manipulative traders who still bring valuable liquidity to the market.

• Spoofing the Stock Market -- Here’s One Way to Stop It

  It may be a bit more complicated than that, but not much.

• Junk bonds really to blame for Flash Crash

  Junk Debt collapsed a full hour before the stock market Flash Crash took place (and I would add that utilities and Treasurys were leading prior, indicating a VIX spike was possible). If junk debt collapses, how can stocks not collapse afterward when in a bankruptcy proceeding, junk debt has a higher claim on assets than equity? What caused the Flash Crash in stocks was a Flash Crash in junk debt, which was like a 1987 style crash in credit.

It's not like this is anything important.

It's not like it's the entire world economy.

Who cares whether we understand it or not?

Oh, dear.

Flash Crash news

Well, this is interesting: CFTC Charges U.K. Resident Navinder Singh Sarao and His Company Nav Sarao Futures Limited PLC with Price Manipulation and Spoofing.

In particular, the CFTC release notes:

in or about June 2009, Defendants modified a commonly used off-the-shelf trading platform to automatically simultaneously “layer” four to six exceptionally large sell orders into the visible E-mini S&P central limit order book (the Layering Algorithm), with each sell order one price level from the other. As the E-mini S&P futures price moved, the Layering Algorithm allegedly modified the price of the sell orders to ensure that they remained at least three or four price levels from the best asking price; thus, remaining visible to other traders, but staying safely away from the best asking price. Eventually, the vast majority of the Layering Algorithm orders were canceled without resulting in any transactions. According to the Complaint, between April 2010 and April 2015, Defendants utilized the Layering Algorithm on over 400 trading days.

The Complaint alleges that Defendants often cycled the Layering Algorithm on and off several times during a typical trading day to create large imbalances in the E-mini S&P visible order book to affect the prevailing E-mini S&P price. Defendants then allegedly traded in a manner designed to profit from this temporary artificial volatility. According to the Complaint, from April 2010 to present, Defendants have profited over $40 million, in total, from E-mini S&P trading.

As others quickly pointed out, the notion that "layering" is involved in these wild price swings is being studied by multiple agencies. For example: Exclusive: SEC targets 10 firms in high frequency trading probe - SEC document.

The SEC has been seeking evidence of abuse of order types, as well as traditional forms of abusive trading like "layering" or "spoofing" and other issues relating to high-frequency trading that might be violations of the law, SEC Director of Enforcement Andrew Ceresney told Reuters in May (reut.rs/1kwSqF5).

Spoofing and layering are tactics where traders places orders that they cancel before they are executed to create the false impression of demand, aiming to trick others into buying or selling a stock at the artificial price.

I'm pleased that investigators continue to investigate.

On the other hand, even after 5 years the investigators still appear to be uncertain as to exactly what happened and why.

It's disturbing news, all around.

A hodge of podge

Boy, my mind has really been wandering recently.

Must be the weather.

• The Town That Creep Built

  “Fault creep” is a condition that results when the underlying geology is too soft to get stuck or to accumulate tectonic stress: in other words, the deep rocks beneath Hollister are slippery, more pliable, and behave a bit like talc. Wonderfully but unsurprisingly, the mechanism used to study creep is called a creepmeter.

  The ground sort of oozes past itself, in other words, a slow-motion landslide at a pace that would be all but imperceptible if it weren’t for the gridded streets and property lines being bent out of shape above it.

• Seveneves

  The moon blew up without warning and for no apparent reason. It was waxing, only one day short of full. The time was 05:03:12 UTC. Later it would be designated A+0.0.0, or simply Zero.

• Linux 4.0

  But "Hurr durr I'ma sheep" trounced "I like online polls" by a 62-to-38% margin, in a poll that people weren't even supposed to participate in. Who can argue with solid numbers like that? 5,796 votes from people who can't even follow the most basic directions?

• Linux 4.0 released

  Looking at just the statistics in git, this release is not just when we cross half a million commits total, but also cross the 4 million git object limit. Interestingly (if you look for numeric patterns), Linux 3.0 was when we crossed a quarter million commits and 2 million git objects, so there's a nice (and completely unintentional) pattern there when it comes to the kernel git repository.

• Almonds in California: They use up a lot of water, but they deserve a place in California's future.

  The problem is that, thanks to the current drought, the water supply is going away faster than expected. The almond industry is an indicator of how difficult it might be to adapt to climate change, economically and environmentally.

  What we’re witnessing in California right now is a glimpse into the future. California has now endured drought in 11 of the last 15 years, and there’s every reason to believe this is just the beginning.

• Math for eight-year-olds: graph theory for kids!

  This morning I had the pleasure to be a mathematical guest in my daughter’s third-grade class, full of inquisitive eight- and nine-year-old girls, and we had a wonderful interaction. Following up on my visit last year (math for seven-year-olds), I wanted to explore with them some elementary ideas in graph theory, which I view as mathematically rich, yet accessible to children.

• Replicating SQLite using Raft Consensus

  I decided to build a distributed replication layer using the Raft consensus protocol, which gives me effective replication without the hassle of running a much heavier solution like MySQL. It provides all the advantages of replication, with the data modelling functionality of a relational database, but with the convenience of a single-file database.

• A Million Little Boxes

  While crossword construction is unavoidably linguistic, constructors these days rely heavily on data and programming. Take, for instance, Puzzle 5, always the hardest of the tournament. This year, Jeff Chen was the sadist behind the curtain. An aspiring novelist from Seattle, he runs the crossword database XWord Info and has authored or co-authored 37 New York Times puzzles.

  Chen’s Puzzle 5 was titled “Attention, Newbies!” The conceit was adding new B’s (get it?) to familiar phrases. “Vocalist” became “vocab list,” “alloy” became “ballboy,” and so on. To find workable and interesting phrases like this, Chen wrote a Python script and applied it to his master word and phrase list. Using this program, he realized “caroms” could become “car bombs,” for example.

• From the vault: Watching (and re-watching) “The Mother of All Demos”

  To give an idea of the scope of the demo, Engelbart demonstrated an early look at word processing, windowing, hypertext, and dynamic file linking, as well as using graphics in a computer program. It was also the first time many of the attendees had seen a mouse, although work on the mouse began in 1963.

• Arcology: cutaways of the future city-hives that never were

  Soleri taught architecture at ASU, but his real classroom was Arcosanti: "an experimental town and molten bronze bell casting community" a few hours from Phoenix, which was supposed to have 5,000 residents by now, demonstrating "lean" city living in high-density complexes that combined super-efficient usage of space with stylized, naturalistic exteriors that made each building part of the landscape.

• The Cult of Work You Never Meant to Join

  I take walks. I leave my phone in my pocket when I’m out with friends or eating my meals. I spend a fair amount of time on my hobbies, like writing and hunting for the world’s best cheeseburger.

• Code Quality

  It's like a salad recipe written by a corporate lawyer using a phone auto-correct that only knew Excel formulas.

Edward Capriolo's NoSQL blog series

One of the things about computing that hasn't changed much over the decades is that it is often best to learn by doing. Many of the original computing pioneers were hobbyists, who built their own computers for themselves, programmed them for fun and for mental stimulation, and who traded notes about what they were doing, what worked and what didn't, and what they had learned along the way.

So along comes Edward Capriolo, who has entertained himself over the winter writing his own NoSQL data store, and has entertained and educated us, along the way, with his series of blog posts:

• Building a Column Family store for fun and enjoyment
• Index Files - Building a column store - Part 2
• Key Cache - Building a column family store - Part 3
• Memtable, Flushing, and SSTables - Building a column family store - Part 4
• SSTables and Compaction - Building a column family store - Part 5
• Commitlog - Building a Column Family store - Part 6
• Bloom Filters - Building a Column Family store - Part 7
• Auto Clustering and gossip - Building a Column Family store - Part 8
• Request Routing - Building a NoSQL Store - Part 9
• Request coordination, eventual consistency, and CAP - Building a NoSQL store - Part 10
• Battling entropy with Hinted Handoff - Building a NoSQL store - Part 11
• Elastic node scale up - Building a NoSQL store - Part 12
• Cleanup compaction - Building a NoSQL store - Part 13

Thank you, Edward, for the time and energy you spent helping us all get smarter, it is much appreciated.

Playing the lottery

So the big news broke late last week: Grateful Dead sets two shows at Levi’s Stadium

A statement by Weir, Hart, Kreutzmann and Lesh said “it has become clear to us that we first need to return to our beginnings, where we first said hello — to each other and to all of you. And so it is that we have decided to plug in for two additional shows on June 27 and 28 at Levi’s Stadium in Santa Clara, California — just a dozen miles south of where Dana Morgan’s Music Store once stood.”

Fans wanting tickets for the two shows, priced at $59.50-$199.50, can sign up for an online ticket lottery that the band installed on its website, www.dead50.net. The sign-up will be in place through April 14 and those who place requests — with a maximum of four tickets per person, per show — will learn if they have received tickets on April 19, according to the band’s website. The announcement said 65,000 tickets would be sold for each show.

One of my grandfather's favorite sayings was:

Well, you pay your money and you take your chances.

And so we did.

I'm thinking lucky thoughts for April 19.

Hild: a very short review

I was, perhaps, the perfect target audience for Nicola Griffith's Hild.

I was fascinated by the subject matter; I love to read historical fiction; I love to be swept away into another world.

I just didn't expect that it would be so much effort to read this book.

Griffith decided to aim for a high level of authenticity, which is wonderful, and is one of the reasons I wanted so much to read Hild. Griffith is thorough in her descriptions, accurate in her dialogue, and detailed in her choice of things like the names of the characters and the vocabulary used to describe the things they are doing in their day to day lives.

And, as this book is set one thousand five hundred years ago, her decision means that there is a considerable amount of distance between the characters, the objects, the events, and the language of that time, and of this one.

But that means you end up with a book which has a Dictionary and a Guide to Pronunciation at the end, and is packed with passages like this:

The high men of the isle gathered to seek favour and pay homage to Edwin and his new queen. Every evening, arriving at the beat of a drum or the ripple of a lyre, a handful of brightly cloaked men, wearing enough gold to dazzle a jay, would swing into the hall and bend their proud heads to the high table. Bryneich from the north, with their short hair, red mouths, and enamelled brooches, under Coledauc king -- who bowed to Hild and gave best wishes from Prince Morcant. The piglet, Hild remembered. Men from Rheged, under Rhoedd the Lesser, Rhoedd's sister-son and little Uinniau's older brother, styling himself prince and bearing gifts from Rhoedd for the king and queen -- and a beautiful double pin inlaid with garnet from the princess Rhianmelldt for the princess Hild. Coelgar, returned from Lindsey, with half a dozen Lindsey thegns at his back and a kinglike bearing. And Dunod, lord of Craven, whom some called king.

Do you see what I mean? It's somehow wonderful and exhausting at the same time.

Eventually you become, if not fluent, at least competent in the language, so when every other sentence talks about a "seax", or a "gesith", or a "wealh", or a "gemaecce", or a "cyrtel", or a "haegtes", or an "aethling", you can start, slowly, to follow the underlying story and events.

But the overall effect, although brilliant, fascinating, and immersive, is also jarring, otherworldly, and uninviting.

Yes, uninviting.

And I so wanted this book to be inviting.

Follow-up on the GitHub DDOS

Brian Krebs follows up on the GitHub DDOS that I wrote about last week, in his essay: Don’t Be Fodder for China’s ‘Great Cannon’.

Krebs points to a superb, long, detailed, in-depth analysis at the CitizenLab website: China’s Great Cannon.

Several previous technical reports have suggested that the Great Firewall of China orchestrated these attacks by injecting malicious Javascript into Baidu connections. This post describes our analysis of the attack, which we were able to observe until April 8, 2015.

We show that, while the attack infrastructure is co-located with the Great Firewall, the attack was carried out by a separate offensive system, with different capabilities and design, that we term the “Great Cannon.” The Great Cannon is not simply an extension of the Great Firewall, but a distinct attack tool that hijacks traffic to (or presumably from) individual IP addresses, and can arbitrarily replace unencrypted content as a man-in-the-middle.

The operational deployment of the Great Cannon represents a significant escalation in state-level information control: the normalization of widespread use of an attack tool to enforce censorship by weaponizing users. Specifically, the Cannon manipulates the traffic of “bystander” systems outside China, silently programming their browsers to create a massive DDoS attack. While employed for a highly visible attack in this case, the Great Cannon clearly has the capability for use in a manner similar to the NSA’s QUANTUM system, affording China the opportunity to deliver exploits targeting any foreign computer that communicates with any China-based website not fully utilizing HTTPS.

There's lots to dig into here. Thanks, CitizenLab, for sharing what you've learned.

Stuff to think about

If there was a description of perfect weather, it would be days like today.

• How to Survive a Mega-Drought: Questions of drought ethics and etiquette answered

  California’s water-rights system is almost incomprehensibly complicated, but the result is that many farms that have depended for decades on an allotment of water from the state supply — their share of a public commodity — are now receiving nothing. Not a drop. Because there’s not enough to go around.

• Scenes from the New American Dustbowl

  We meet in a little café and the two of them offer me a crash course in California Water 101. Even in non-drought years the logistics are complex. Snowpack runoff is captured in reservoirs. Rivers and lakes are dammed. Canals snake across the state. Some water is managed at the federal level, some at the state. There are 500 public water districts, each with local ordinances. There are senior water rights, junior rights, riparian rights.

• Don't Think Open Access Is Important? It Might Have Prevented Much Of The Ebola Outbreak

  Part of the problem is that none of these articles were co-written by a Liberian scientist. The investigators collected their samples, returned home and published the startling results in European medical journals. Few Liberians were then trained in laboratory or epidemiological methods. Even today, downloading one of the papers would cost a physician here $45, about half a week’s salary.

• Sheikh of the Skies

  Falconry essentially is the art of managing a falcon’s appetite, he said. “You’re standing here on the beach, when the falcon starts disappearing around the back of the hotel, you’ve now lost visual contact, your communication is gone, your mobile signal is gone. If the bird now sees a pigeon across the road, what’s stopping the bird from deciding to chase that pigeon? Nothing really, other than the bird’s loyalty to fly around the building and seek comfort in seeing Pete again. In an urban environment, the margin for error is huge, you’re radically increasing the possibility of something going wrong.”

Popular music as cultural analysis

We're going to be seeing The Decemberists in a few weeks, and I'm starting to get really excited about the show.

I had listened to them, on and off, but hadn't paid enough attention, and with the upcoming tour as inspiration I've been really paying a lot more attention to them.

And they're fascinating.

Musically, I started listening to them because I picked up Long Live The King due to its inclusion of a cover of a wonderful Grateful Dead song, Row Jimmy.

Then I moved on to The King Is Dead, their blockbuster, which of course I adored because of Peter Buck.

So at first What a Terrible World, What a Beautiful World freaked me out a little bit, because it's considerably different from The King Is Dead. But over the last few months I've grown to love WATWWABW at least as much as TKID, if not more.

Besides just their music, one of the interesting things about The Decemberists is how much people like to talk about them as a way of talking about the world at large. This is true of many popular artists, but it is particularly true about The Decemberists, perhaps because their songs get people thinking about larger topics.

So, for example, we have Colin Meloy being interviewed: YA Books, RPG and the New Decemberists LP: Colin Meloy Rolls the Dice:

That relationship between bands or singers and their audience, it's kind of a funny relationship and abusive in its own right, going both ways. I shouldn't say abusive, but it can be antagonistic. I think that it's an odd relationship, and it's just that particular singer trying to come to terms with that aspect of it. Having an audience, you may want to continue doing things on your own terms, but that becomes more challenging when there are expectations. And audiences have more of a voice than ever with the advent of the Internet.

Over at Slate, Carl Wilson takes an even broader view: Against indie: New albums from Modest Mouse, Sufjan Stevens, and more show it's time to eliminate the racist term for good.

Other music listeners might ask if bands of the Decemberists’ vintage can change enough to feel pertinent in 2015. A decade ago, music blogs, film and TV music supervisors, Pitchfork, and other new media outlets boosted “indie” to a rare visibility. Now, many of those acts are returning from long absences to quite an altered atmosphere.

Wilson goes on to explain why he uses the powerful term "racist" in this situation:

Few of them claim to be fighting any kind of battle against pop anymore—fans are almost always worse than artists on that count. But this decade has also seen a more widespread suspicion and critique of the workings of social privilege, and “indie” has a problem there—because its creators and listeners seem so disproportionately white, male, and upper-middle-class.

Later, Wilson more directly skewers The Decemberists for what he sees as their failings:

Likewise I am a bit skeptical that without “indie,” the Decemberists could even exist. If there were then still a call for a post-modern folk-rock Gilbert and Sullivan, it would have to have more of the courage of its strangeness. The band’s hiatus has done it some good, and the songwriting is more grounded on this year’s What a Terrible World, What a Beautiful World. But I still find Meloy’s unrelenting streams of conceits wearying, like a prog concept album from 1975 without even the gonzo musicianship to liven up the occasion.

More than any other band, they bring me back to the self-regarding turn that America made in the 2000s—the post-9/11 world-wariness and self-soothing. It would be too much to say that’s what made it an ideal period for “indie.” But when I listen to the Decemberists, I’m tempted.

I am, indeed, white, male, and upper-middle-class. So, guilty as charged. But does that mean I'm somehow committing a social offense by being a Decemberists fan?

I'll have to spend more time listening to their music before I can come to a more considered opinion about whether they are letting us down.

But it also seems like Wilson is asking Meloy and company to fight Wilson's battles, which is unfair. As Meloy says,

I just like stories. I like people telling stories.

Seems fair enough, to me. I'll keep listening, and hopefully I'll enjoy going to their show and meeting the people I meet there.

Sometimes art can just be entertainment, after all; it doesn't always have to change the world. That's a lot to ask, of anybody.

Ten years of git

Linux.com is running an interesting short interview with Linus Torvalds about git.

Q: Does Git last forever, or do you foresee another revision control system in another 10 years? Will you be the one to write it?

Torvalds: I'm not going to be the one writing it, no. And maybe we'll see something new in ten years, but I guarantee that it will be pretty "git-like." It's not like git got everything right, but it got all the really basic issues right in a way that no other SCM had ever done before.

Over at the Atlassian web site, there's a pretty little animated page: "only a mere few days later, the world was given the gift of Git".

The "10 year anniversary" is based on this, I believe.

If you have any knowledge of git, and git internals, gotta love this:

"write-tree etc. by hand" (!)

Can Linux network service names contain periods?

So I was doing some testing of inetd and xinetd.

inetd is just like xinetd except that xinetd has an 'x' in it. (Just joking. Here's a slightly better answer).

So I wanted to test two different versions of my service, and the machine I had root access to was using xinetd, so I followed some RedHat documentation I found on the net.

The files in the /etc/xinetd.d/ directory contains the configuration files for each service managed by xinetd and the names of the files correlate to the service. As with xinetd.conf, this file is read only when the xinetd service is started. For any changes to take effect, the administrator must restart the xinetd service.

I was a little bit confused about "the names of the files correlate to the service", so I read further in the RedHat docs:

service — Defines the service name, usually one listed in the /etc/services file.

I looked in my /etc/xinetd.d directory, and sure enough I had some existing files:

$ ls /etc/xinetd.d
chargen daytime discard echo time

Sounds pretty good, so I created two new files:

$ sudo vim /etc/xinetd.d/server.1 /etc/xinetd.d/server.2

I put in some simple configuration, restarted xinetd, and ...

...

... nothing happened.

After a bunch of flailing around, I found /var/log/syslog, where xinetd reported that it loaded the configuration files /etc/xinetd.d/chargen, /etc/xinetd.d/daytime, /etc/xinetd.d/discard, /etc/xinetd.d/echo, and /etc/xinetd.d/time.

There weren't any other error messages or complaints.

It just quietly omitted my service configuration files.

A colleague, stopping by (mostly to stop the full-volume stream of profanities I was directing at my computer screen), looked over my shoulder and said:

Hmm.. I wonder if period is an illegal character in a service name?

WHAT?

Well, sure enough, I renamed those two files from "server.1" and "server.2" to "server1" and "server2", and restarted xinetd, ...

... and everything worked fine.

So I poked around in places like Linux Network Administrators Guide and Linux Networking HOWTO, and read:

service specifies the service name

and

name

A single word name that represents the service being described.

I guess Real Linux System Administrators don't need things like this documented, and don't need their tools to print syntax error messages when they make simple syntax errors in naming their network services.

I'll go back to getting useful work done. Bummer about those 90 minutes of my life; I could have maybe used them for something useful.

Uncomplicated praise for Roads & Kingdoms

In the last few months, I have fallen completely in love with the Roads & Kingdoms website.

Here's how they describe themselves:

Roads & Kingdoms is an independent journal of food, politics, travel and culture. It was recently voted the Gold Winner for Best Travel Journalism Site by the Society of American Travel Writers. The magazine first launched in Myanmar as a Tumblr that became a home for reports on everything from Burmese civil war to dissident MCs to the perils of rancid crab. R&K is now a fulltime digital magazine based in New York and Barcelona, publishing longform dispatches, interviews and global ephemera daily.

I love it all: I love the longform dispatches; I love the interviews; I love the global ephemera.

Everything about the site is well-done. The writing is clear and vivid (and well-edited). The photography and presentation is marvelous. And the topics are both fascinating and relevant.

I'm not sure where they've found the funding for the site, which is blissfully short on advertising and other annoyances. Perhaps this will change, as all great websites seem to eventually succumb to reality.

But for now, I'll keep loving Roads & Kingdoms, and I'll keep reading them as often as I can.

GitHub DDOS

Why did the attack start?

Who did it?

What caused it to stop?

• Large Scale DDoS Attack on github.com

  We are currently experiencing the largest DDoS (distributed denial of service) attack in github.com's history. The attack began around 2AM UTC on Thursday, March 26, and involves a wide combination of attack vectors. These include every vector we've seen in previous attacks as well as some sophisticated new techniques that use the web browsers of unsuspecting, uninvolved people to flood github.com with high levels of traffic. Based on reports we've received, we believe the intent of this attack is to convince us to remove a specific class of content.

• GitHub battles “largest DDoS” in site’s history, targeted at anti-censorship tools

  The attack started on Thursday morning (March 26), and has continued unabated since then, evolving several times to circumvent GitHub's defenses.

• Massive denial-of-service attack on GitHub tied to Chinese government

  the two GitHub pages are constantly loaded and reloaded by millions of computer users inside and outside of China, an endless loop that left unmitigated outages not just on the two targeted pages but throughout GitHub's entire network. Exhibit A in the case in which China is involved are the two specific GitHub pages targeted: one hosts anti-censorship service GreatFire.org while the other hosts a mirror site of The New York Times' Chinese edition. The targets suggest the attackers are sympathetic to the vast censorship apparatus known as the Great Firewall of China.

• China's Man-on-the-Side Attack on GitHub

  In short, this is how this Man-on-the-Side attack is carried out:

  1. An innocent user is browsing the internet from outside China.
  2. One website the user visits loads a javascript from a server in China, for example the Badiu Analytics script that often is used by web admins to track visitor statistics (much like Google Analytics).
  3. The web browser's request for the Baidu javascript is detected by the Chinese passive infrastructure.
  4. A fake response is sent out from within China instead of the actual Baidu Analytics script. This fake response is a malicious javascript that tells the user's browser to continuously reload two specific pages on GitHub.com.

• Chinese authorities compromise millions in cyberattacks

  On March 17th 2015, our websites and partner websites came under a DDoS attack. We had never been subjected to an attack of this magnitude before. This attack was unusual in nature as we discovered that the Chinese authorities were steering millions of unsuspecting internet users worldwide to launch the attack. We believe this is a major cyber-security and economic threat for the people of China.

• Hackers Attack GreatFire.org, a Workaround for Websites Censored in China

  GreatFire.org’s mirroring services provide unrestricted access within China to a range of websites, including itself and the Chinese language version of The New York Times, which has been regularly blocked in China. Some of the others are Deutsche Welle, BBC News, China Digital Times, Google.com, and Boxun, a Chinese-language news website. GreatFire.org says it does not mirror The Wall Street Journal. GreatFire.org works directly with some, but not all, of the websites it mirrors.

  GreatFire.org is partly funded by Open Technology Fund, a United States government-financed initiative under Radio Free Asia. Last year it provided $114,000 in funding, according to its website. Mr. Smith declined to comment on any financial backing.

• Evidence links China to GitHub cyber-attack

  "The upshot is that people from around the world... had their traffic redirected to swamp GitHub," Prof Alan Woodward of the University of Surrey told the BBC after verifying the research.

• The Attack on GitHub Must Stop

  According to Insight Labs, Internet traffic within China is being manipulated, such that users are essentially attacking GitHub.

• Baidu’s traffic hijacked to DDoS GitHub.com

  What is happening here is pretty clear now: A certain device at the border of China’s inner network and the Internet has hijacked the HTTP connections went into China, replaced some javascript files from Baidu with malicious ones that would load

  ["https://github.com/greatfire/", "https://github.com/cn-nytimes/"]

  every two seconds.

• Did China Just Launch a Cyber Attack on GitHub?

  Github’s whole site uses HTTPS encryption, so when a Chinese netizen visits content hosted on the site, Chinese censors can only see that the user is visiting github.com, but not the full URL address within GitHub. So China can't selectively block just some content on GitHub without blocking the entire site.

• Man-on-the-side attack

  Instead of completely controlling a network node as in a man-in-the-middle attack, the attacker only has regular access to the communication channel, which allows him to read the traffic and insert new messages, but not to modify or delete messages sent by other participants. The attacker relies on a timing advantage to make sure that the response he sends to the request of a victim arrives before the legitimate response.

• Using Baidu to steer millions of computers to launch denial of service attacks

  The attackers have implemented a sneaky mechanism that allows them to manipulate a part of the “legitimate traffic” from inside and outside China to launch and steer Denial of Service attacks against Cloudfront and the Greatfire.org's anti censorship project.

• China's Great Firewall Turned Around: Why China Wants To Censor Global Internet

  Because the overall internet is too important to block, and because some sites are necessary (like Github) there are always holes in the system. Add in a useful dose of encryption (yay!) and the ability to control everything that's read in one particular country becomes increasingly difficult. You might hope the response would be to give up attempts to censor, but China isn't likely to give up just like that. So, instead, it's basically trying to censor the global internet, by launching a high powered attack on the site that is the problem, while basically saying "get rid of these projects and we'll stop the attack."

Brian Krebs is doing marvelous work reporting on tax fraud

Brian Krebs has been publishing some amazing stories on his website recently about the sophistication of computer-assisted tax fraud cases.

To wit:

• Sign Up at irs.gov Before Crooks Do It For You

  Recently, KrebsOnSecurity heard from Michael Kasper, a 35-year-old reader who tried to obtain a copy of his most recent tax transcript with the Internal Revenue Service (IRS). Kasper said he sought the transcript after trying to file his taxes through the desktop version of TurboTax, and being informed by TurboTax that the IRS had rejected the request because his return had already been filed.

• States: Spike in Tax Fraud Against Doctors

  In this increasingly common crime, thieves steal or purchase Social Security numbers and other data on consumers, and then electronically file fraudulent tax returns claiming a large refund. The thieves instruct the IRS to send the refund to a bank account that is tied to a prepaid debit card, which the fraudster can then use to withdraw cash at an ATM

• Crimeware Helps File Fraudulent Tax Returns

  “We were later notified that the breach was much deeper than originally suspected, which included all of our beneficiaries, their personal information, my life insurance policy, 401-K stuff, and our taxes,” Doug said. “My sister-in-law is an accountant, so I raced to her and asked her to help us file our taxes immediately. She pushed them through quickly but the IRS came back and said someone had already filed our taxes a few days before us.”

  Doug has since spent many hours filling out countless forms with a variety of organizations, including the Federal Trade Commission, the FBI, the local police department, and of course the Internal Revenue Service.

• Citing Tax Fraud Spike, TurboTax Suspends State E-Filings

  “During this tax season, Intuit and some states have seen an increase in suspicious filings and attempts by criminals to use stolen identity information to file fraudulent state tax returns and claim tax refunds,” the company said in a statement.

• The Rise in State Tax Refund Fraud

  States allow unlinked returns because most taxpayers owe taxes at the federal level but are due refunds from their state. Thus, unlinked returns allow taxpayers who owe money to the IRS to pay some or all of that off with state refund money.

It's great of Krebs to keep a light on this subject, because it's complex and frustrating.

Happily, slow progress appears to be being made, as Krebs covers in his latest column:

• Tax Fraud Advice, Straight from the Scammers

  those involved in tax refund fraud shifted more of their activities away from the Internal Revenue Service and toward state tax filings. This shift is broadly reflected in discussions on several fraud forums from 2014, in which members lament the apparent introduction of new fraud “filters” by the IRS that reportedly made perpetrating this crime at the federal level more challenging for some scammers.

  ...

  According to a January 2015 GAO report (PDF), the IRS estimated it prevented $24.2 billion in fraudulent identity theft refunds in 2013. Unfortunately, the IRS also paid $5.8 billion that year for refund requests later determined to be fraud. The GAO noted that because of the difficulties in knowing the amount of undetected fraud, the actual amount could far exceed those estimates.

Lastly, and again kudos to Krebs, he's collected and summarized his advice for protecting yourself here: What Tax Fraud Victims Can Do.

Ugh.

Thank you, Brian Krebs, for all your hard work.