Serious infrastructure issues

Wow! I hadn't been paying any attention to this story until now: India blackout, on second day, leaves 600 million without power.

India’s blackout spread to cover more than half the country on Tuesday, as the electrical grid collapsed in 14 states in the country’s north and east, depriving at least 600 million people of power, many for a second day.

What's maybe the most interesting part is how nobody can figure out what's wrong:

Power Minister Sushilkumar Shinde said the government had not yet not been able to detect the reason for the grid breakdown.

...

“We are absolutely clueless why this has happened again today,” Shakti Sinha, principal secretary in the power department of the Delhi government, said in a telephone interview.

It will be interesting to keep an eye on this story in the future, to see what is learned from these events.

Drone pilots in the NYT

You should read this article in the New York Times: A Day Job Waiting for a Kill Shot a World Away.

The story spends most of its word-count focusing on the soldiers who operate the remotely-piloted aircraft:

By 2015, the Pentagon projects that the Air Force will need more than 2,000 drone pilots for combat air patrols operating 24 hours a day worldwide. The Air Force is already training more drone pilots — 350 last year — than fighter and bomber pilots combined. Until this year, drone pilots went through traditional flight training before learning how to operate Predators, Reapers and unarmed Global Hawks. Now the pilots are on a fast track and spend only 40 hours in a basic Cessna-type plane before starting their drone training.

Interviewing actual drone pilots, the author investigates the "roiling emotions after they fire a missile":

When he was deployed in Iraq, “you land and there’s no more weapons on your F-16, people have an idea of what you were just involved with.” Now he steps out of a dark room of video screens, his adrenaline still surging after squeezing the trigger, and commutes home past fast-food restaurants and convenience stores to help with homework — but always alone with what he has done.

The article is firmly rooted in the present reality of drones deployed in Asian combat areas. I wish the author had spent more time investigating the near future, in which drones are deployed to non-combat areas, and their pilots are surveilling ordinary civilians suspected only of unpopular opinions, as the Christian Science Monitor did last month in their article: Drones over America. Are they spying on you?, noting that the FAA is “streamlining the process for public agencies to safely fly UAS in the nation’s airspace.”

Ugh. Our dystopian future draws ever nearer.

Nice Olympics sailing video

These Elliot 6 meter boats look like they'd be incredibly fun (if a bit wet) to sail around in! Wired magazine has a nice video interview with U.S. contestant Anna Tunnicliffe.

Inside the Coursera contract

I've been interested in the various new "online university" projects that are underway: Udacity, Coursera, EdX, etc.

This month, the Chronicle of Higher Education has an interesting article that looks inside the contract that Coursera is signing with its member universities: Inside the Coursera Contract: How an Upstart Company Might Profit From Free Courses.

The article claims that the efforts are still working to build their programs, and their audiences, and are not yet trying to address the "should we try to make a profit" question that is hanging over every effort like a giant storm cloud:

Coursera is following an approach popular among Silicon Valley start-ups: Build fast and worry about money later. Venture capitalists—and even two universities—have invested more than $22-million in the effort already. "Our VC's keep telling us that if you build a Web site that is changing the lives of millions of people, then the money will follow," says Daphne Koller, the company's other co-founder, who is also a professor at Stanford.

But the article explores a number of the possible ways that these programs might make money, or at least might earn enough money to defray their costs:

Coursera's leaders say they are actively pursuing only two of the moneymaking ideas on the list: charging students who pass the courses a small fee for a certificate, and serving as a matchmaker between students looking for jobs and companies seeking qualified employees.

Ah yes, employment: everyone wants to cash in on LinkedIn's success.

Other ideas cited by the article include:

• Selling Courses to Community Colleges.
• Charging Tuition.
• Offering "Secure Assessments."

At least up to this point, the article suggests, these new programs are mostly about money:

When and if money does come in, the universities will get 6 to 15 percent of the revenue, depending on how long they offer the course (and thus how long Coursera has to profit from it). The institutions will also get 20 percent of the gross profits, after accounting for costs and previous revenue paid. That means the company gets the vast majority of the cash flow.

The article ends up concluding that the academic institutions are motivated mostly by fear, not by greed:

College officials, for their part, seem more motivated by fear than by the promise of riches. "Most of us are thinking this could be a loss of revenue source if we don't learn how to do it well," says Mr. Rodriguez, of the University of Virginia. "These are high-quality potential substitutes for some of what universities do."

It's a sad thing that efforts such as these, with such potential to bring unbelievable opportunities, benefits, and knowledge to the world, are apparently mired in an oscillation between fear and greed instead.

From what little I have seen so far, the potential is tremendous: the content is strong, the presentation tools work well, and the interest is there. Let's hope that wise minds prevail and these efforts work to provide a greater good for humanity, not simply a gusher of money for some well-heeled investors.

Networking: theory and practice

I came across two nice papers recently with some interesting synergy.

First, on the theory side, BBN's Craig Partridge offers his suggestions for Forty Data Communications Research Questions. Each question is briefly summarized, with lots of pointers to further research for the curious. The one that interested me the most was:

22. Are there cooperative protocols above the physical layer? There’s a substantial body of work showing that wireless networks can perform better if nodes help each other[51] – e.g. a node off to the side echoes what it hears from a sender in order to improve reception at the receiver. For the most part, these are physical layer improvements (perhaps coordinated with the media access layer). Is there a role for similar cooperation at higher layers? At first this may seem impractical, as most higher layer communications is point-to-point. Yet many servers often work together to deliver a web page and many servers work together to deliver an email. The step from collaboration to cooperation would seem small.

Secondly, on the practical side, Adam Langley has posted his notes from his HOPE9 talk: Living with HTTPS. I wasn't familiar with the H.O.P.E. conferences, but it seems like HOPE9 was a very interesting conference -- just browse through some of the abstracts and shake your head in wonder at the range of topics and viewpoints!

Langley's talk, meanwhile, is interesting from start to finish, with plenty of practical advice about how to secure your site, pointers to tools to help you assess where you may still have problems, as well as ideas about how we can continue to improve our security protocols and techniques in the future.

Security is a never-ending struggle, and you have to be constantly considering it. For example, Langley notes the risks involved in mixed-mode sub-resources:

When you load sub-resources over HTTP, an attacker can replace them with content of their choosing. The attacker also gets to choose any page on your HTTPS site with the problem. That includes pages that you don't expect to be served over HTTPS, but happen to be mapped. If you have this problem anywhere, on any HTTPS page, the attacker wins.

Read more, learn more, be smarter. Good ideas for the weekend!

The other Eagle

When I read that Orange Coast College is selling the Alaska Eagle, their instructional big boat, which has a long, rich, and much heralded history, I was initially a bit confused.

I thought they meant another Eagle, a really big boat, with its own much-heralded history as an instructional vessel.

Oh, well, they are both beautiful sailboats, and here's to sailing academies of all flavors!

Meanwhile, closer to home, a train arrived the other day.

A train full of boats.

The America’s Cup Express, a train stretching 1.5 miles long, rolled into San Francisco. The 121 railroad cars are carrying the material required to move the America’s Cup World Series and the teams into the host city of the Cup ahead of racing in August and October.

The cargo includes the race boats – the powerful AC45 wing-sailed catamarans that have been thrilling crowds and challenging the best sailors in the world as the Series has moved around the world over the past year. Now, the racing is coming to San Francisco for the first time.

Some photos of the train, and old unused Pier 80, are here, here, and here.

Soon those boats will be in the water of the bay!

Precious Reynolds and the Milwaukee Protocol

Don't miss this great story in Wired magazine: Undead: The Rabies Virus Remains a Medical Mystery.

Thousands of years of medical history say that Precious should not have survived. And indeed, some of the world’s foremost rabies experts refuse to concede that Willoughby’s theory of the disease is sound—or even that his protocol is actually saving patients.

Trying to understand the evolution of finance

Although I'm far from an expert in finance, I'm unavoidably a part of the financial system. I pay my taxes; I try to support my family; I live in a country where I must, for the most part, arrange my life to achieve the successful financing of my own retirement, my healthcare, my children's education, etc.

So I try to pay a certain amount of attention to the overal financial system of the country, and of the world.

Recently my attention was drawn by a number of related matters:

• Writing on his excellent blog, Felix Salmon draws attention to the ongoing discussion about the role of government oversight and regulation in the prevention of financial fraud: Why finance can’t be fixed with better regulation.

  Jim Surowiecki and John Kay both have columns today looking at the way in which regulatory structure failed to stop abuses in the financial-services industry, and wondering how we might be able to do better in future.

  The Surowiecki column is here: Bankers Gone Wild, while the Kay column is here: Finance needs Stewards, not Toll Collectors.

  Surowiecki takes a full swing against those who claim the financial world is doing just fine with self-regulation:

  Even in the absence of market discipline, self-regulation could work if institutions had strong internal safeguards against corruption. But while every institution says that it has these norms—that’s why scandals like LIBOR are always blamed on a “few rogue traders”—the track record of the banking industry over the past two decades doesn’t inspire confidence in its devotion to the truth or to the public interest.

  Meanwhile, Kay swings hard, too, but in (more or less) the opposite direction, claiming that government has no hope, and regulators are doomed to be as corrupt as those they regulate:

  Regulators come to see the industry through the eyes of market participants rather than the end users they exist to serve, because market participants are the only source of the detailed information and expertise this type of regulation requires. This complexity has created a financial regulation industry – an army of compliance officers, regulators, consultants and advisers – with a vested interest in the regulation industry’s expansion.

  I think that Kay oversteps with his broad condemnation of the corruption of government civil servants. Certainly there are problems, but in my experience there is a vast civil service full of district attorneys, police detectives, FBI agents, IRS examiners, and so forth, who are honest, ethical, have high moral standards and are truly trying to do the right thing. James Kwak generally agrees, saying

  The only thing I would disagree with is the characterization of U.S. regulators as “crudely corrupt.” Yes, many Congressmen are “in thrall to Wall Street money,” but when it comes to the staff at the regulatory agencies I think the picture is more complex and closer to the “intellectual capture” that Kay describes in Europe.

  But as Salmon observes, overall these bitter conclusions leave us in a very tough spot:

  financial-industry scandals will continue to arrive on a semi-regular basis. When they do, they will always be accompanied by calls for stronger regulation: rules-based, or principles-based, or some combination of the two. But the real problem here isn’t regulatory, and as a result there isn’t a regulatory solution. The real problem is deeply baked into the architecture of too-big-to-fail banks.

• For a detailed look at one such regulator, and how it is coping with this problems, you might find this recent article by Ben Protess and Jessica Silver-Greenberg interesting: New York Fed Faces Questions Over Policing Wall Street:

  But there are limits to its power. Despite its leading role in policing the banks, the New York Fed cannot levy fines. When examiners do detect questionable behavior, they often push the company to adopt changes. If the wrongdoing persists, officials can pass along the case to the Federal Reserve board in Washington.

  It is up to the central bank to take action. The Fed, which can impose fines and cease-and-desist orders, filed 171 enforcement actions last year. The cases are down 44 percent from the year before, but the actions have increased sharply from the precrisis era.

• Much of the discussion about the future of financial institutions, including all the essays in the previous paragraphs, involves the topic of trust: how does the public come to trust their government, and their corporations, and their leaders?

  The Kay Report, published this week by the UK Department for Business Innovation and Skills, takes this question as its principal topic:

  Financial intermediation depends on trust and confidence: the trust and confidence that savers who invest funds have in those they choose to manage these funds, and the trust and confidence of investors in the businesses they support. Trust and confidence are the product of long-term commercial and personal relationships: trust and confidence are not generally created by trading between anonymous agents attempting to make short term gains at each other’s expense.

  Trust and confidence, or their absence, are the product of the prevailing culture. Incentives matter: not because, as some people crudely think, financial rewards are the only human motivation – although there are some people of whom that is true, and many of them are to be found in the financial sector. Most people have more complex goals, but they generally behave in line with the values and aspirations of the environment in which they find themselves. We must create cultures in which business and finance can work together to create high performing companies and earn returns for savers on a sustainable basis

  The report is long and detailed, with many specific recommendations, though I'm not sure when I might find time to read them all.
• On this side of the pond, a recent column by Professor Luigi Zingales of the University of Chicago Business School has been receiving a lot of attention, at least partly for its attention-grabbing headline: Do Business Schools Incubate Criminals?.

  Although the headline may be outlandish, Zingales' article is clear and direct. He takes business schools strongly to task:

  Most business schools do offer ethics classes. Yet these classes are generally divided into two categories. Some simply illustrate ethical dilemmas without taking a position on how people are expected to act. It is as if students were presented with the pros and cons of racial segregation, leaving them to decide which side they wanted to take.

  Others hide behind the concept of corporate social responsibility, suggesting that social obligations rest on firms, not on individuals. I say “hide” because a firm is nothing but an organized group of individuals. So before we talk about corporate social responsibility, we need to talk about individual social responsibility. If we do not recognize the latter, we cannot talk about the former.

  and dismisses those schools which give only lip service to ethics as being worse than those who say nothing at all:

  The way to teach these ethics is not to set up a separate class in which a typically low-ranking professor preaches to students who would rather be somewhere else. This approach, common at business schools, serves only to perpetuate the idea that ethics are only for those students who aren’t smart enough to avoid getting caught.

  Zingales' conclusion is a call to action:

  The daily scandals that expose corruption and deception in business are not merely the doing of isolated crooks. They are the result of an amoral culture that we -- business-school professors -- helped foster. The solution should start in our classrooms.

• Finally, and on a more theoretical bent, I've been wondering for a while when the academic side of the finance profession would start to weigh in on the theoretical questions surrounding the Great Recession of 2008, questions like:
  • How exactly did mortgage securitization affect the crisis?
  • How did credit default swaps fit into the picture?
  • How are the changing non-bank entities affecting financial markets?
  and so forth. So, I was quite pleased to learn, recently, that the Federal Reserve Bank of New York is in fact sponsoring exactly this sort of research. A recent press release explains some of their efforts in detail: New York Fed Launches Blog and Economic Policy Review Series Examining the Evolution of Banking.

  Today, the New York Fed’s Liberty Street Economics blog begins a seven-part series that explores the changing role of banks in the financial intermediation process. A series of Economic Policy Review articles will accompany the blogs. Both series discuss the complexity of the credit intermediation chain associated with securitization and note the growing participation of nonbank entities within it. These series also discuss implications for monitoring and rulemaking going forward.

  You can find some of the early articles on the bank's website, where you can find articles such as: The Evolution of Banks and Financial Intermediation: Framing the Analysis

  These papers are not easy for the non-practitioner to read; even the high-level summaries are full of pronouncements such as:

  The two hypotheses imply a specific relationship between the amount of enhancement afforded and the ex post performance of the security. Namely, buffering would lead one to expect higher enhancements among more poorly performing securities, while the signaling hypothesis would imply instead that high enhancements are associated with high performance. The authors’ econometric analysis suggests that buffering investors is in fact the main motivation behind the provision of enhancement in asset securitization, thus corroborating the underlying argument that banks have played a fundamental role in supporting the modern intermediation process.

  But it's great to see that the institutions are taking a serious, detailed, and thorough look at the financial industry and how it operates. Hopefully these studies will pay off.

Who invented the Internet, part 94

Duncan Black makes a really good point.

It wasn't so much the technology, though it was that too, it was the idea, and it isn't an idea a for-profit company would have pursued in nearly the same way.

I'm (just barely) old enough to remember all this. I started seriously programming computers in 1981, I had my first email address in 1983, and I can remember when my 1985-vintage email address was spelled decvax!cca!bryanp.

Thirty years, sadly, seems just long enough to forget all that important history. Thankfully, people like Tim Lee are around to help us remember.

Annie Hall, now and then

I love this "now and then" essay about the locations used in the movie "Annie Hall": The Filming Locations of Annie Hall, Part 1 – New York, You’ve Changed.

Annie Hall is one of those few Woody Allen movies that I actually sort-of "got", so I remember it quite well.

It's interesting that some areas have dramatically changed in 36 years, others not at all.

We've noticed the same thing in our neighborhood, though we've only been here 24 years. Some areas are completely un-recognizable from how they were a quarter century ago, while others haven't changed a bit.

Simple hardware lessons for a software guy, part 3

Good things come in threes, right? So this poor software guy is happy that he's had some great opportunities to learn about hardware over the last few days.

The third of three is Robin Harris's fascinating article: The post-RAID era begins.

Harris starts out by pointing out that simple RAID 5 algorithms are failing to scale due to the immense growth in disk capacity.

He proceeds to discuss the substantial progress that is being made in "erasure coding", which involves new ways of encoding stronger parity checks into your data.

In the ’90s a new form of erasure coding was developed that enabled developers to create codes with an arbitrary level of redundancy – survive 4 failures? 10? Pick a number! – called fountain or rateless erasure codes.

Harris suggests that in the (near) future, storage systems will no longer be using simple RAID techniques, but instead will be using the more sophisticated erasure coding technology, and references a number of startups who are active in this area.

Harris also links to a nice presentation by Professor Jim Plank of the University of Tennessee: Erasure Codes for Storage Applications.

The Point of This Tutorial

• To introduce you to the various erasure coding techniques.
  • Reed Solomon codes.
  • Parity-array codes.
  • LDPC codes.
• To help you understand their tradeoffs.
• To help you evaluate your coding needs.
• This too is not straightforward.

Professor Plank is not kidding about the coding needs being "not straightforward": by the fifth page of his paper A Tutorial on Reed-Solomon Coding for Fault-Tolerance in RAID-like Systems we're into arithmetic over Galois Fields...

Professor Plank's web site is loaded with lots of additional information about erasure coding.

Thus ends our short three-part series of the software guy learning more about hardware...

Simple hardware lessons for a software guy, part 2

It's more than just networking hardware that mystifies software guys like me. Take, for example, the incredibly sophisticated memory caches that are found on modern CPUs.

I recently came across a superb paper by Paul McKenney of IBM's Linux Technology Center: Memory Barriers: a Hardware View for Software Hackers.

McKenney's paper sets out to teach software types like myself just enough about the design and implementation of cache coherency algorithms in modern multi-core processors so that we can write software that is both correct and efficient.

The hardware designers cannot help directly here, since the CPUs have no idea which variables are related, let alone how they might be related. Therefore, the hardware designers provide memory-barrier instructions to allow the software to tell the CPU about such relations. The program fragment must be updated to contain the memory barrier

The paper is clear and straightforward, with great diagrams and examples. I particularly enjoy the "quick quiz" format, in which McKenney intersperses review questions in the text, and provides answers at the end of the paper so you can periodically stop, take the quiz, check your answer, and keep going.

If you've ever wondered about the MESI protocol, or about store buffers, or about writeback, snooping, and invalidation, or how the DEC Alpha processor impacted the Linux kernel memory-ordering primitives, don't hesitate, dig right in!

This is the sort of paper that every system software engineer should read, understand, and periodically re-read. Thank you IBM and Paul McKenney for making it available!

Simple hardware lessons for a simple software guy

Being a software guy, I don't know much about the hardware that I use. Oh, I understand it at a conceptual level, but I sure wouldn't know anything about how to choose the right line card for a Cisco 6500.

So I'm a software guy, I admit it.

But I still find it helpful to know something about hardware, and so I love to find useful helpful information like Greg Ferro's great article: Top 20 Practical Cabling Tips for Network Engineers – Part One.

It's easy to read, and full of obvious-in-retrospect-but-maybe-you-never-thought-about-it advice like:

Hanging cables within the rack means that means that gravity will induce physical stress on the copper core which will stretch and distort over time. A longer patch lead is heavier and will cause more shear stress on the cable. This will create signal degradation over time and leads to intermittent failures over time.

Greg Ferro is a great author to follow, if you aren't doing so already; keep those tips coming, Greg!

The bubble is back!

Wow, 1.25 Billion?!! Read about it here.

It's not just a game ...

... it's a reason for Wheatley to go to space!.

Random stuff I'm reading currently

Herewith, some things that you might or might not find interesting:

• Jacqueline Piatigorsky, one of the great benefactors of chess in my lifetime, has died at the age of 100. Randy Hough has a nice memorial essay about her here.

  Mrs. Piatigorsky's name is forever linked to the Piatigorsky Cup GM tournaments held in Los Angeles in 1963 and 1966. Participants included world champions Petrosian, Spassky, and Fischer. She oversaw all the details of these events, including the relay of moves to the analysis room, and personally designed the Cup. She even placed the demonstration boards on the stage herself once when none of the helpers were present and none of the masters would deign to help. These two tournaments, universally considered to be the strongest in the U.S. since New York 1924, did much to enhance the stature of chess in the U.S. and generated still-popular tournament books.

• The Olympics are starting in just a couple days. Are you ready? LongForm.org has just what you need: Longform’s Olympics Primer. I particularly enjoyed Will You Medal in the Morning? and the story of Penny Brookes from Much Wenlock

  Then de Coubertin hied to the Olympian Fields and saw the Games for real. Yes, it was only Much Wenlock, one little town in the Midlands, and the Olympians were mostly just Shropshire lads, but now it wasn’t a dream. Right before his eyes, the baron could see athletes running and jumping, with laurel wreaths placed upon the victors’ brows and brotherhood upon the horizon of sport.

• Everybody's talking about the fascinating discussion between Eric Schmidt and Peter Thiel that occurred at the Fortune Brainstorm Tech conference in Aspen. Happily, Fortune magazine have made a full transcript of the debate available on their website. In general, I'm usually more on Schmidt's side than Thiel's, but in this particular case I think Theil made some great points, and Schmidt really ought to think about some of what Thiel had to say. An excerpt:

  ADAM LASHINSKY: You touched on the employment issue earlier, the New York Times did a major piece suggesting that Apple doesn't account for the same kind of employment in the United States that General Motors did in the 1950s. And you make a similar case, Twitter, Facebook, as great as they are, 140 characters is neat, but they don't' employ a lot of people.

  PETER THIEL: They're all great companies.

  ADAM LASHINSKY: In that they're profitable.

  PETER THIEL: But, they're not ones that are able to basically ‑‑ you know, they're sort of the exception to the rule that we don't have enough innovation. So, you have to avoid confusing the specific and the general. Google is a great company. It has 30,000 people, or 20,000, whatever the number is. They have pretty safe jobs. On the other hand, Google also has 30, 40, 50 billion in cash. It has no idea how to invest that money in technology effectively. So, it prefers getting zero percent interest from Mr. Bernanke, effectively the cash sort of gets burned away over time through inflation, because there are no ideas that Google has how to spend money.

  Thank you, Fortune, for making the transcript available. Go read it!
• Staying on the social aspects of computing angle for just a bit, don't miss this fascinating short essay in the Atlantic: Rich Kids of Instagram Epitomize Everything Wrong with Instagram

  The very basis of Instagram is not just to show off, but to feign talent we don't have, starting with the filters themselves. The reason we associate the look with "cool" in the first place is that many of these pretty hazes originated from processes coveted either for their artistic or unique merits, as photographer and blogger Ming Thein explains: "Originally, these styles were either conscious artistic decisions, or the consequences of not enough money and using expired film. They were chosen precisely because they looked unique—either because it was a difficult thing to execute well (using tilt-shift lenses, for instance) or because nobody else did it (cross-processing)," he writes. Instagram, however, has made such techniques easy and available, taking away that original value.

• Jeff Atwood is back with a nice essay about the ongoing introduction of new programmer jargon: New Programming Jargon. I loved "Common Law Feature", "Smug Report", and "Yoda Conditions".
• Except for the persistent mis-spelling of "yacht", this article from the OCSC Sailing Blog is fun: Fun and interesting places to sail in SF Bay. Meanwhile, the first of the new America's Cup boats has launched, and what a boat it is!
• Oracle have shut down the long-running Sun Labs research effort called Project Fortress: Fortess Wrapping Up

  over the last few years, as we have focused on implementing a compiler targeted to the Java Virtual Machine, we encountered some severe technical challenges having to do with the mismatch between the (rather ambitious) Fortress type system and a virtual machine not designed to support it (that would be every currently available VM, not just JVM). In addressing these challenges, we learned a lot about the implications of the Fortress type system for the implementation of symmetric multimethod dispatch, and have concluded that we are now unlikely to learn more (in a research sense) from completing the implementation of Fortress for JVM.

• Perhaps unsurprisingly, mobile devices are already starting to bump up against the limits of a 32-bit implementation, so soon there will be 64-bit ARM processors. Over at Linux Weekly News, Jon Corbet has a nice summary of the work underway to add 64-bit ARM support to Linux: Supporting 64-bit ARM systems

  The 64-bit ARM instruction set is completely different from the 32-bit variety, to the point that there is no possibility of writing assembly code that works on both architectures. The system call interfaces also differ significantly, with the 64-bit version taking a more standard approach and leaving a lot of legacy code behind.

• VMWare are open-sourcing an intriguing new database engine that they call Affinity. Apparently this work comes out of a company that VMWare bought, called Pi Computing; VMWare's Richard McDougall explains a bit of the history here. It looks like a very complete effort; they have a full transactional model with snapshot isolation.

  Affinity uses the Read-Only multiversion [ROMV] protocol for read-only transactions, thus enabling non-blocking reads. This only applies to transactions that are explicitly specified as read-only by the caller. These transactions read a snapshot that will correspond to the effect of committed write transactions up to the point the read-only transaction started (n.b. if the number of snapshots grows very large at any point in time, an older [already allocated] snapshot may be used). In contrast, plain read-write or read-only transactions (not explicitly marked as read-only by the caller) progress according to the 2-phase-locking protocol, and may in some cases read data items in a "newer" state than the corresponding explicit read-only transaction would.

  It's interesting that the most complete and informative page in their documentation set is what they call their Terminology Page. Getting terminology clear is a great first step!
• I'm just totally digging Werner Vogels' new series on "Back-to-Basics", in which he picks one or two classic papers a week to re-read and re-discover. I hope he keeps this up! Here are the first two episodes:
  • Back-to-Basics Weekend Reading - Hints for Computer Systems Design
  • Back-to-Basics Weekend Reading - Virtualizing Operating Systems
  Interestingly, I just re-read Lamport's paper myself a few weeks back.
• This is an interesting effort: Best Paper Awards in Computer Science
• Sam Overton is proposing to add virtual nodes to Cassandra. Read his proposal here. The Acunu team continue to generate interesting ideas and systems.
• I'm not sure I understand what DBToaster is, but it looks interesting.

  The DBToaster compiler accepts queries written in SQL, and generates query engine code that can be incorporated directly into any C++ or Scala project

  Their white paper adds considerable additional details.
• OSCON 2012 has just finished, and it looks like it was another good year for the conference. Here's the proceedings.

Yeah, you're right, this was some pretty random stuff. Well, don't say I didn't warn you!

Solomon's Oak: a very short review

Over the spring, I read a very enjoyable book by Jo-Ann Mapson named Solomon's Oak.

My mother says she loves reading books that are set in places she knows, or has lived in, or is currently visiting.

Solomon's Oak, for me, is such a book: it is set in or near the tiny town of Jolon, California, in the hills of the California Coast Range, near where the Mission San Antonio de Padua is located.

I've been there; it's a very nice place, and it's exactly as Mapson describes it.

Now, Solomon's Oak is not actually about Jolon, but rather about some people, each with their own unique and individual story, who are drawn together because of a unique oak tree.

It's one of those books that sounds like nothing when it's described, but is thoroughly enjoyable to read. The people are intriguing, the situations they encounter are compelling, and the book is ever so well-written. It flows along, and you just find yourself turning page after page, wanting to know how it's all going to come out.

Oh, by the way, there are dogs, and horses, and a pirate-themed wedding, and lots of other things in the book.

And, at just $2.99 for your Kindle, it's a great bargain.

Looking for some summer Kindle reading? Try Solomon's Oak, you'll (probably) really like it.

Postcards from the high end...

These two items are not really related, but consider them two postcards from the high-end of database-y things:

• Firstly, Amazon rolled out this week what they call their "High I/O" instance configuration: New High I/O EC2 Instance Type - hi1.4xlarge - 2 TB of SSD-Backed Storage:

  we are introducing a new family of EC2 instances that are designed to run low-latency, I/O-intensive applications, and are an exceptionally good host for NoSQL databases such as Cassandra and MongoDB.

  The instance is certainly impressive, as described by Werner Vogels: Expanding The Cloud – High Performance I/O Instances for Amazon EC2

  The hi1.4xlarge has 8 cores and 60.5GB of memory. Most importantly it has 2 SSDs of 1 TB each and a 10 Gb/s Ethernet NIC that using placement groups can be directly connected to other High I/O instances.

  The SSDs will give you very high IO performance: for 4k random reads you can get 120,000 IOPS when using PV and 90,000 when using HVM or Windows. Write performance on SSDs is more variable depending on, among other things, the free space the disk, the fragmentation and the type of filesystem. With PV virtualization we are seeing between 10,000 and 85,000 IOPS for 4k random writes and with HVM between 9,000 and 75,000.

  Distributed filesystem guru Jeff Darcy gives the new instance a spin, concluding that they indeed deliver tremendous performance, but noting that they are also quite costly to provision compared to lower-end instance.

• Meanwhile, I happened to see some fascinating information about a company I've been intermittently following, Clustrix, who are rolling out their latest high-end system, Clustrix 4.0. Curt Monash covers the announcement: Clustrix 4.0 and other Clustrix stuff

  Clustrix technical notes include:

  • Clustrix is MVCC (Multi-Version Concurrency Control).
  • Clustrix exploits MVCC to allow online, lockless schema changes. Clustrix says these changes are typically single-column, for example an add or a widening/datatype change.
  • Clustrix indexes are a mix of b-trees and log-structured merge files.
  • Clustrix sounds like it’s paid attention to being multi-core. For example, DR replication is via parallel, multi-core log streaming, going single-core only when transactions have the potential to influence each other.
  • MySQL features Clustrix lacks include triggers and XML support.
  • Clustrix uses MLC flash.

  Note that Clustrix is way not cheap: their pricing sheet starts at 150,000, although that price includes both the software and the hardware, I believe.

Anyway, enjoy some o' that Big Data info!

Some Dallas phone numbers are not Dallas phone numbers

Here's a short, well-told programming story about when a Dallas phone number is not actually a Dallas phone number: Confessions: The Phone Number.

Perforce 2012.2 has entered beta testing

I'm excited that the next release of the Perforce server is now available for beta testing.

I was pleased to be able to contribute to this release in several ways, and I'm quite looking forward to seeing what sort of feedback it gets as it enters use.

This is the fourth release of the Perforce server I've been part of; I've said it before, but it remains true, that one of the most exciting things about being a software engineer is when somebody else tries your software, and you find out whether it actually works or not!

Have you tried the new Perforce server? Let me know what you think!

Good enough for GitHub

Great article from the team at Netflix about the power of open source techniques to improve their overall development process: Open Source at Netflix.

In particular, I love this quote:

we’ve observed that the peer pressure from “Social Coding” has driven engineers to make sure code is clean and well structured, documentation is useful and up to date. What we’ve learned is that a component may be “Good enough for running in production, but not good enough for Github”.

The usefulness and effectiveness of the review process is nothing new, of course; what is new is the emerging understanding that a strong open source community can provide a healthy open review process in situations where a closed organization is not mature enough or capable of providing that review process internally.

The Coursera Tsunami

As his institution announces its enrollment in the burgeoning Coursera consortium, Professor Dick Lipton of Georgia Tech offers some thoughts about how Coursera might play out for GT.

Are we as educators going to disappear? Why take my course on combinatorics if you can have Bob Sedgewick, one of the world experts, give the lectures?

...

Will universities as we know them today survive? Will some disappear, will some thrive; will all have to change and adapt?

The New York Times article covers several interesting takes on the discussion, including the difficulty of translating raw enrollment numbers into tangible benefits, and the enormous peer pressure that institutions are feeling to establish an online presence, whether it be Coursera, Udacity, edX, or some other similar arrangement.

As you'll recall, I took both the Coursera and Udacity cryptography classes last spring, and have been (slowly) working my way through Sebastian Thrun's Statistics class on Udacity this summer. So I found it quite interesting to see my Coursera instructor, Professor Dan Boneh of Stanford, discussing how the online course materials could benefit the in-class students as well:

Professors say their in-class students benefit from the online materials. Some have rearranged their courses so that students do the online lesson first, then come to class for interactive projects and help with problem areas.

“The fact that students learn so much from the videos gives me more time to cover the topics I consider more difficult, and to go deeper,” said Dan Boneh, a Stanford professor who taught Coursera’s cryptography course.

As the Times points out, it's far from clear how this whole experiment will go, and how long these marvelous free-for-all education resources will remain available. But for now, it's a wonderful thing, and it looks like there are more great courses coming online soon!

A collection of AWS post-mortems

So, of course, there was another AWS outage at the end of June. Happily, I was on vacation and lying on the beach and didn't even notice it!

But others noticed it, and thought about it, and wrote up some rather interesting observations about it.

I spotted a few, and thought I'd call attention to them, in case you found them as interesting as I did:

• First, start with Amazon's own writeup
• I found it interesting to see how Netflix perceived it
• Here's some information about how Heroku perceived it
• And another post-mortem, from vendor CloudBees

In case you're wondering about the "leap second" reference in the CloudBees writeup, a decent overview is in Jonathan Corbet's Linux Weekly News article: Leaping seconds and looping servers.

Did I miss any good post-mortems? Let me know!

Java code coverage and thrown exceptions

Recently, I've been working with various Java code coverage tools, such as Emma and JaCoCo.

Generally, these tools are quite powerful and sophisticated.

However, I recently learned of a rather fundamental limitation: they only measure a block as "covered" if that block terminates normally; if the block of code terminates due to a thrown exception, the block is not marked as covered.

The Emma documentation discusses exceptions in sections 2.5, 2.6, and 2.7 of the FAQ:

Thus, EMMA does not attempt to track the success or failure of every single bytecode instruction: instead, it marks a basic block covered when the control reaches its last instruction. A covered basic block is thus guaranteed to have executed without failures at least once in a given coverage run session.

...

the motivation behind this has been that most legitimate code is written so that methods return normally (without throwing an exception).

I'm sure I wasn't the only person unaware of this limitation.

There doesn't appear to be a way around this. So, for the time being, just because your Emma or JaCoCo coverage report indicates that a particular block of code is uncovered, you can't immediately conclude that it isn't invoked at all.

It might have been invoked, but terminated due to a thrown exception.

How is this situation handled in other code coverage tools, for other languages in which exceptions are a first-class part of the language?

Google Summer of Code and the University of Moratuwa

This is the eighth year of operation for Google Summer of Code, Google's highly unusual open source summer internship program for computer science students from around the world.

Google Summer of Code is a global program that offers post-secondary student developers ages 18 and older stipends to write code for various open source software projects. We have worked with open source, free software, and technology-related groups to identify and fund projects over a three month period. Since its inception in 2005, the program has brought together over 6000 successful student participants and over 3000 mentors from over 100 countries worldwide, all for the love of code.

I've been associated with GSoC for six of its eight years, the last four years as mentor, working with the Apache Derby project.

Interestingly, three of the four students I've worked with were undergraduates at a Computer Science school that you've probably never heard of: the University of Moratuwa, in Sri Lanka. Although the university is relatively new, its roots go back more than 100 years, as it was founded as an offshoot of the Government Technical School in Columbo.

I can imagine that it must be rather challenging for students at the University of Moratuwa to interact with other computer scientists around the globe, so the Google Summer of Code program has turned out to be a tremendous resource for the program. The school was once again the top participant in this year's GSoC program, and has placed more students with GSoC than any other institution.

It has been nothing but a pleasure to work with the students of the University of Moratuwa. They have all been hard-working, dedicated, serious students of computer science, eager to get a chance to learn more about the practicalities and issues of real-world software, and hopeful of turning their interest in computer science into a career.

I think that the Google Summer of Code program offers a significant benefit to these students. Often, an academic program in computer science is necessarily limited: the restrictions of time in the classroom make it hard to treat issues in depth, and the conversations are generally theoretical, as is appropriate for the academic setting.

But modern computer science involves tremendous practical challenges as well: engineers must become comfortable working as part of a bigger team, on large, complex, mature code bases, written by others who have since moved on, and may have left little in the way of documentation; successful engineers have to learn how to communicate well, be thorough about their work, attend to details throughout the project, and address all facets of a problem, including such aspects as testing, documentation, and bug-fixing.

The world of open source offers a great opportunity for gaining such experience. Apache Derby, for example, is a 15-year old codebase, worked on by hundreds of engineers over the years, with deep and extensive functionality. Moreover, as a full-featured DBMS implementation, Derby concerns itself deeply with issues of reliability, robustness, and maintainability, which are crucial for real-world projects. Lastly, the Derby community is an open, supportive, and welcoming group, with a great team of programmers who are always happy to discuss and explain the inner workings of Derby.

So, Apache Derby plus Google Summer of Code is a great fit.

Congratulations to the students of the University of Moratuwa for another successful year working with the Google Summer of Code: I hope all the students are benefiting from the program.

I've never had a chance to visit Sri Lanka; hopefully I will, someday. If I do, I will surely take the time to visit the University of Moratuwa and learn more about this institution.

Construction Everywhere

I happened to go for a walk along San Francisco's Embarcadero the other day, and noticed several significant construction projects underway:

• At Pier 15, work is well underway to build the new home of the Exploratorium museum. There is detailed information about the project in this article: Exploratorium’s New Waterfront Home on Track for a Spring 2013 Opening.

  Indoor space of 330,000 square feet, includes quintessential Exploratorium exhibits in four galleries; space for professional teacher training, after-school programs, educational camps and lifelong learning; a theater; the Exploratorium store; and a café.

  The Exploratorium team also have a nifty collection of videos about the project. Apparently they are intending to produce a complete documentary once the work is complete. The videos are here: Creating the Campus. (Of course, I liked the demolition videos the best!)

  I've always enjoyed my visits to the Exploratorium, and I think this new location will be superb and should bring them much higher visibility and usage.

• A few blocks north, at Pier 27, construction is well underway on the new cruise ship terminal: James R. Herman Cruise Terminal Project at Pier 27. There's a lot less information about this project, but it still looks like an exciting new project for the Embarcadero waterfront.

  The Port desires to develop a world-class home port cruise terminal that reflects the values of San Francisco residents and the City’s leadership, meets international cruise terminal standards, and complies with regulatory requirements, including the Port’s commitment to the stewardship of waterfront historic resources within the Embarcadero Historic District.

  The project update page has more detail, including a timeline and progress reports.
• And, over on my side of the bay, BART officials gave a progress report on the Oakland Airport Connector:

  The controversial Oakland Airport Connector, once plagued by funding problems that put the project's future in doubt, is now "on time and on budget" at the halfway point of construction, BART officials said Friday.

  The BART project page has more information. It's interesting that the connector trains will be driver-less:

  An Automated Guideway Transit (AGT) system was determined to be the preferred technology for the OAC project. The AGT concept refers to an array of transit technologies, the common elements being that they operate within their own guideways, would have stations physically integrated with the Coliseum/Oakland Airport BART Station and the airport terminal, and do not require a vehicle operator.

  I've used the BART connector to SFO several times, and I think it's wonderful! It's much nicer to take BART to the airport than it is to drive. Hopefully the OAK connector will see similar success.

  Since the BART project is very close to my house, we drive by it periodically. It can be hard to tell what's happening in a construction project; sometimes nothing seems to be going on, then a week later there appears to be a lot of progress.

All the new construction is quite exciting; these all seem like valuable and important projects, and they all seem to be progressing well, so far as I can tell.

MMDS 2012

This week, across the bay, is MMDS 2012. Workshop on Algorithms for Modern Massive Data Sets

Admit it: just reading the titles of these talks (e.g., "Affiliation Network Models for Densely Overlapping Communities in Networks") makes you drool!

Of course, since it's a workshop, you really had to be there (and I wasn't), since all that's available online are the abstracts of the talks.

But still, it's more work to keep an eye out for...

Tips for security conference week

I love these tips from the SpiderLabs team at Trustwave as we head towards security conference week in Las Vegas (BlackHat and DEF CON).

"3-2-1" will help you survive and is easy to remember:

• 3 hours of sleep per night
• 2 meals
• 1 shower

Read the tips. Be prepared.

GSoC 2012: halfway there!

We've reached the midway point in this year's Google Summer of Code.

The student with whom I am working, Mohamed Nufail of the University of Moratuwa in Sri Lanka, has been working with the Apache Derby team on improving our regression test suites.

As Apache Derby is an extremely sophisticated and mature DBMS implementation, thorough and detailed regression testing is extremely important to the community.

We've been using several code coverage tools, named Emma and JaCoCo, to try to evaluate and improve the quality of our regression test suites.

Currently, we're deep in a puzzling problem which started out as an apparent gap in our test suites (we weren't covering the client-server aspects of JDBC4 testing thoroughly), but might additionally reveal either a bug in Derby's test automation or perhaps even a bug in the Emma tool. Certain classes are clearly invoked by the test suites (we can see output from print statements in the class's methods in the test output), yet Emma reporting is claiming that the class's code was not run.

That's the way it is with software: you start out looking for something, then you find something, and then when you dig into it, it becomes still more interesting!

A great thing about GSoC is that it provides an opportunity for interested young computer scientists to get involved with complex and intricate pieces of software (e.g., Apache Derby) in the framework of an established team that works together cooperatively to improve that software (e.g., the Apache Derby development community), and thus helps bring real-world experience to the academic process.

Various database topics

I was very hopeful when I started looking at this article: Why the days are numbered for Hadoop as we know it, but the article turns out to promise much more than it delivers.

It starts out promising to tell us "what's next" after Google File System and Google Map/Reduce.

But in the end, what it mostly tells us is that it's getting harder and harder to choose a code-word for your project.

So if you want to know about Dremel, Pregel, and Giraph, read the article for some fairly interesting summaries of those projects.

Meanwhile, for something a bit more relevant, and a bit more technical, DBMS industry analyst Curt Monash has updated his top-level survey of database diversity: Database diversity revisited.

Monash's article is filled with links; to follow all of his analysis, you'll spend a lot of time clicking. But it's very much worth the trouble, because he's got lots of thorough summaries of the incredibly rich DBMS ecosystem.

Lastly, can't post about databases without some Postgres goodness, so:

• Some examples of manipulating date/time data and timezones
• Some info about Foreign Data Wrappers. Foreign Data Wrappers allow external data sources to be integrated into Postgres databases and behave somewhat like normal tables.
• A fairly detailed example/tutorial of setting up Slony replication
• The latest from Robert Haas: Absurd Shared Memory Limits. Postgres 9.3 will contain a revised implementation of shared memory. By using anonymous inherited memory segments, most kernel re-configuration is now avoided, and Postgres 9.3 will be one step easier to install, and require one fewer reboot, yay!

A troubling article on scientific careers

In this weekend's Washington Post: U.S. pushes for more scientists, but the jobs aren’t there:

Michelle Amaral wanted to be a brain scientist to help cure diseases. She planned a traditional academic science career: PhD, university professorship and, eventually, her own lab.

But three years after earning a doctorate in neuroscience, she gave up trying to find a permanent job in her field.

Dropping her dream, she took an administrative position at her university, experiencing firsthand an economic reality that, at first look, is counterintuitive: There are too many laboratory scientists for too few jobs.

The article suggests that the problem is a combination of cuts in academia occurring at the same time as cuts in industry:

Since 2000, U.S. drug firms have slashed 300,000 jobs.

...

Traditional academic jobs are scarcer than ever. Once a primary career path, only 14 percent of those with a PhD in biology and the life sciences now land a coveted academic position within five years

The author suggests that, overall, this is substantially influenced by changes in federal government policy:

One big factor: Since 2004, federal research spending across all agencies has stagnated relative to inflation, according to an analysis by the American Association for the Advancement of Science.

Interesting, and worrisome.

ubuntu tracker and locate

I wouldn't mind uninstalling or disabling both tracker and locate. I don't use either of them (as far as I know) and would be plenty happy to do without their presence.

Every time I restart my computer it's useless for five minutes or more while these beasts ravage my hard drive.

So I'm ready to shut them off, but how? Should I just uninstall them? Is there some preferences file to set? Is there a simple how-to somewhere?

Anybody know?

So that's what it takes to find a Higgs Boson!

Can you imagine seeing one of these rolling down your street?

And now we will know for sure

From Soccer Finally Comes to Its Senses With Goal-Line Tech:

After years of discussion and debate, the sport has at long last approved the use of goal-line technology at all levels of the game.

The FIFA press release says:

It’s not a dangerous development as long as it remains focused on goal-line technology. I don’t want technology anywhere else. I want football to maintain its human face, this will be my goal as long as I am the President of FIFA. We will still have to see how it works. It is only to help the referee, we must remember that.

Interestingly, as far as I can tell, FIFA did not pick a single winner: both the camera-based technology (Hawk-Eye) and the magnetic field technology (Goal-Ref) are now approved for use.

Silicon Valley craziness

Elaine Wherry's essay: the Recruiter Honeypot is fascinating.

What started out as an odd idea about how to recruit some better software recruiters turned into an odyssey of discovery about how software recruiting works nowadays.

In late 2009, I created an online persona named Pete London – a self-described JavaScript ninja – to help attract and hire the best JavaScript recruiters. While I never hired a recruiter from the experiment, I learned a ton about how to compete in today’s Silicon Valley talent war.

Software recruiters, often called headhunters, are independent contractors who attempt to match companies wishing to hire software specialists with employees wishing to find a better job.

For this service, headhunters receive a fee, typically paid by the hiring company upon the successful filling of a position.

The fee can be substantial, which leads, naturally, to practices that are less than ideal.

The article is great fun; Wherry's tips and lessons are revealing, and the entire essay is filled with hard-earned valuable wisdom.

And speaking of Silicon Valley craziness, don't miss Crammed Into Cheap Bunks, Dreaming of Future Digital Glory, a story in today's New York Times about "Hacker Hostels" that have opened to house the next generation of would-be Zucks:

These are not so different from crowded apartments that cater to immigrants. But many tenants are here not so much for the cheap rent — $40 a night — as for the camaraderie and idea-swapping. And potential tenants are screened to make sure they will contribute to the mix.

Of course, all this craziness has been true for at least 20 years.

But is it affecting the core values of Silicon Valley? Are there even such things? Bill Davidow asks the question: What Happened to Silicon Values?. The piece is mostly a "wasn't it great when we were young" sort of piece, but there are some important criticisms included.

Many other things have changed in the valley over the past five decades. I've become increasingly concerned about one thing that is seldom discussed: the valley is no longer as concerned about serving the customer, and even sees great opportunity in exploitation. We are beginning to act like the bankers who sold subprime mortgages to naïve consumers. In such an environment, we are less likely to create the role models of the past who guided the valley to its future.

Here in Silicon Valley, there's lots to think on.

Matthew Yglesias on patent "reform"

Matthew Yglesias has a good essay up on Slate: America’s Patent Crisis.

The problem is that the underlying diagnosis is wrong. All else being equal, it would be nice for the government to do its paperwork faster, but there’s little reason to believe the United States is underpatented.

The only people getting rich in this system are the lawyers.

Post-vacation link dump

I guess that if you go on vacation, the rest of the world doesn't stop. Here's a bit of what's been going on while I was sitting on the beach:

• Derby 10.9 has been released! Knut Anders surveys the release on his blog
• Cormac Herly of Microsoft Research digs deep into a question that's more interesting than you might think: Why do Nigerian Scammers say they are from Nigeria?

  Since gullibility is unobservable, the best strategy is to get those who possess this quality to self-identify. An email with tales of fabulous amounts of money and West African corruption will strike all but the most gullible as bizarre. It will be recognized and ignored by anyone who has been using the Internet long enough to have seen it several times. It will be figured out by anyone savvy enough to use a search engine

• A fairly interesting discussion about Computer Science research, and whether it is better done by academic institutions, by industrial research labs, or by some combination:
  • Startup University: "The academic research process is incredibly inefficient when it comes to producing real products that shape the world."
  • An open letter to Matt: "Forget efficiency. At least, forget it on the timescale you measure in your current job. Instead, aspire to do work that is as groundbreaking and important as the best work in the history of the field."
  • Defending Matt Welsh’s 'Startup University' Post: "most universities do have some number of applied fields, and measuring impact in applied fields is often initiated by looking at real-world deployments of the research ideas"
  • The PhD Grind, and Why Research Isn’t Like Sex: "The research process (and particularly its funding aspect) is unfortunately very conservative. But this doesn’t necessarily mean that a person should accept defeat. Rather, take it as a challenge. "
  • Google’s Hybrid Approach to Research: "The empirical components are clearly on the upswing, in part because the computer systems we construct have become so large that analytic techniques cannot properly describe their properties"
• Bruce Sterling knocks the ball way out of the park with his superb and brilliant Turing Centenary speech

  If we’re willing to learn from Alan Turing’s life experience, we should be devoting some thought to a suicidal Artificial Intelligence. Nobody does this, because they somehow imagine that code can’t simulate abject despair.

  Vaclav Havel — because I’m a writer, I admire this writer guy a lot — he said, “Sometimes I wonder if suicides aren’t in fact sad guardians of the meaning of life.” We may be keen on thinking machines. Yet we don’t allow ourselves a feminine Artificial Intelligence, and we also overlook a suicidal Artificial Intelligence, even though the very founder of the concept was an effeminate, suicidal guy.

• Yay for the LHC, finding the Higgs particle!
• The latest from Quinn Norton on Anonymous

  In the beginning, Anonymous was just about self-amusement, the “lulz,” but somehow, over the course of the past few years, it grew up to become a sort of self-appointed immune system for the Internet, striking back at anyone the hive mind perceived as an enemy of freedom, online or offline. It started as a gang of nihilists but somehow evolved into a fervent group of believers.

• As if I haven't nattered on about the Euro 2012 tournament enough: Graph theory applied to soccer
• Michael Paquier ruminates about how you might extend ALTER TABLE to support re-sharding a distributed table:Postgres-XC: online data redistribution
• A fun piece from the May issue of Wired about counterfeiting
• The Grateful Dead archive is online!
• A multi-part series covering the development of the new game Sir, You Are Being Hunted: Tagged: Sir, You Are Being Hunted. I particularly liked If I May Speak Procedurally, Sir: Building The British Countryside Generator

  we identified a number of features in the countryside that typify the aesthetic we wanted, and seem to be quintessential in British rural environments. Possibly the most important element is the ‘patchwork quilt’ arrangement of agricultural land, where polygonal fields are divided by drystone walls and hedgerows. These form recognisable patterns that gently rise and fall across the rolling open countryside, enclosing crops, meadows, livestock and woodlands. This patchwork of different environmental textures is something that is very stereotypically part of the British landscape. I looked for a mathematical equivalent we could use to simulate this effect and quite quickly decided upon using Voronoi diagrams.

• The robot of the future is going to explore Mars: The Robot of the Future That's About to Explore the Deep Past of Mars

  With a rover like this, you can pick up a rock and analyze it, but you might burn a week or two figuring out if there's any scientific value in it. A human can look at a rock and be able to tell, instantly, whether it deserves further study. The instruments on Curiosity are really great, but the human brain is the best instrument of all. The human brain can solve problems intelligently and quickly, whereas robots rely on us here on Earth -- and that takes a lot of time.

• Surprisingly, there don't seem to be anywhere near as many articles about this year's Google I/O as there were in previous years. Was the conference a bust? Am I just looking in the wrong places? Here's one of the few I found: Google IO 2012 Notes – Lots of Them

So, there you go.

A week in Kauai

It was my lucky week, as my wife treated me to a week in Kauai.

This was my fourth trip to Hawaii, and my first repeat visit; we've previously been to the Big Island of Hawaii, and to Maui, and we visited Kauai 4 years ago. Each island has a different "personality", and it takes a while to get accustomed to the unique aspects of each. Kauai, which many call the "garden island," is lush, quiet, and relaxed.

From Kauai2012

We stayed in the resort community of Princeville, on Kauai's North Shore. As with the other Hawaiian Islands, the northern and eastern shores of Kauai are cooler and wetter, while the southern and westen shores are warmer and drier. Princeville is the richest shade of green that you could possibly imagine! The plants seem to grow two inches every day.

From Kauai2012

Kauai is quieter than some of the more-developed islands. We spent our days on the beach, or kayaking, or hiking, or playing marathon cribbage matches by the pool.

From Kauai2012

Of course, one day was reserved for driving up to visit Waimea Canyon, the spectacular 3,000-foot-deep canyon on Kauai's west side. On the way back, we stopped at the Kauai Coffee Company's visitor center, which is located near the center of the coffee plantation, allowing you the interesting opportunity to drive through several miles of coffee trees on your way to the visitor center and back.

From Kauai2012

Our favorite place to go snorkeling is "Tunnels", more precisely called Makua Beach, which is an unbelievably perfect place to swim, snorkel, and beachcomb. Of course, even though the beach is protected by a large protective reef, and the water is warm, calm, and clear, you must treat this beach, like any beach, with respect. Although Tunnels is known for being a great place to swim with the turtles, I didn't venture into those areas of the beach: for one thing, I'm not the strongest swimmer, but at least partly it's because, 10 years later, I'm still affected by Bethany Hamilton's experience. So I stayed in the shallow, peaceful, calm areas, and enjoyed the tangs, jacks, wrasses, needlefish, pufferfish, and, of course, the humuhumunukunukuapuaa, which were plentiful and as beautiful as ever.

From Kauai2012

We shared our section of Tunnels with a Hawaiian Monk Seal. Monk Seals are tremendously endangered: there are barely 1,000 of these animals left alive, and the North Shore of Kauai is one of the only places you can ever see them, so we were very lucky to have the chance to splash in the surf just 10 yards away from our resting friend.

From Kauai2012

One of our favorite things to do is to go kayaking on the rivers in Kauai, so in case you ever get the chance, here's how to Do It Right:

• Pack yourself a nice bag, with a solid picnic lunch (you're going to work up a hunger!), lots of sunscreen, your camera, and some bottles of water to drink while you're paddling.
• Get yourself down to Kayak Kauai in Hanalei. They have the best location for launching onto the river: their office is right on the riverbank, with its own kayak dock. They'll fix you up with kayaking gear in a jiffy and you'll be out on the water.

• From Kauai2012

  First, head up the river, toward the mountains. Paddle upstream; it's not too hard in the gentle current of the Hanalei River. Enjoy the scenery, and make sure your camera is handy!

• From Kauai2012

  Try to get at least up to the famous single lane bridge over the Hanalei River. Once you're past the bridge, delight in the most beautiful view you'll ever see from a kayak. (You did bring that camera, right?)

• From Kauai2012

  Once you're sufficiently far past the bridge to enjoy the view, give yourself a pat on the back, and apply a bunch more sunscreen!
• Now you can float gently back down the river. Don't go back to dock yet, though! You're just getting started!
• Continue paddling downriver, past the town of Hanalei, and on down to the mouth of the Hanalei River, where it meets Hanalei Bay

• From Kauai2012

  At the mouth of the river, you'll find a large stretch of shallow sandy beach to land your kayak. Pull the kayak up safely onto the sand, just in case the tide is coming in, and oh, before you forget, apply some more sunscreen!
• The park at the mouth of the Hanalei River is the perfect place for that picnic lunch. Rest, relax, enjoy the views of the surfers and other watersports, do some beach-combing, take some more pictures. The park has full facilities, so this is a nice time to take that early-afternoon bio break.
• When you're rested and rejuvenated, and you've had all the beach time you can take, it's time to pack back up and re-board your kayak. Paddle back upstream, and you'll be back at the dock before you know it.
• After you've checked all your gear back in, this is a great time to un-kink those sore muscles with a peaceful walk around Hanalei and its shops and restaurants. There's something for everybody here!

Lastly, a few miscellaneous recommendations, should you find yourself in Kauai any time soon:

• Fun South Shore experience: lunch at Keoki's Paradise in Poipu. Sit at a waterside table under the awning and don't miss the fresh fish sandwich
• Fun North Shore experience: the newly-opened Kauai Mini Golf has a very entertaining course, and is planted with thousands of exotic tropical plants
• Fun Shave Ice: we had ours in Kilauea, where we also made a return visit to see Daphne the Umbrella Cockatoo
• Fun Mai-Tai: we liked Neide's and Bouchon's restaurants in Hanalei, as well as Keiko's Paradise in Poipu
• Fun shopping: Bryan likes Havaiki in Hanalei; Donna likes Banana Patch Shop in Kilauea

I'm back online

Had a wonderful vacation!

I'm still working on the notes for that vacation, so for now you'll have to read something else.

Try this: The 120,000-Foot Leap

On the morning of March 15, Felix Baumgartner scooted forward, stood up, and grabbed the handrails on either side of the door he’d just opened. From where he was standing, he could see all of New Mexico and parts of Texas, Arizona, and Colorado. He was 71,581 feet — 13 and a half miles — above Earth, dangling in a small capsule from a balloon the size of a 16-story building. Facing the cameras mounted on the capsule, he gave a little salute, a bit awkward in his full-pressure spacesuit, and stepped off.