Journal of a Programmer

DERBY-5584

2012-01-28T14:35:00.000-08:00

This week I've been spending a bit of time working on DERBY-5584.

This is a bug that I introduced into Derby in October, 2009, when I re-wrote Derby's GROUP BY engine to add multi-level aggregation support (DERBY-3002).

The new bug is rather complicated, because it involves (a) multiply-scanned result sets and (b) DISTINCT aggregates.

Let's see if I can explain those two concepts, starting first with DISTINCT aggregates.

Aggregates are a feature of the SQL language. The SQL statement normally reports on individual rows in your table(s): each row in your result can be traced back to a particular row in a particular table (possibly multiple tables, joined together). Aggregates, however, are functions which collapse the detail of individual rows of data into coarser-grained summarizations.

The classic aggregates in the core SQL language are: MIN, MAX, COUNT, SUM, and AVG, and these have been part of the language for decades. Most DBMS implementations add additional aggregate functions, but it's sufficient for our purposes to consider the base 5.

An example of an aggregate query would be to figure out the average mileage of flights between San Francisco and San Antonio:


SELECT AVG(flight_miles) 
   FROM flights
   WHERE (origin = 'SFO' AND destination = 'SAT')
      OR (destination = 'SFO' AND origin = 'SAT')

When performing these sorts of aggregate queries, it's often quite useful to break things down by grouping the results based on some other fields. For example, suppose we wanted to know whether the distance flown tends to vary depending on the day of the week (do weekend flights get to take a shorter flight path?):


SELECT departure_day_of_week, AVG(flight_miles) 
   FROM flights
   WHERE (origin = 'SFO' AND destination = 'SAT')
      OR (destination = 'SFO' AND origin = 'SAT')
   GROUP BY departure_day_of_week

Executing a query such as this is performed (in Derby, as in many other databases), by sorting the data on the grouping column(s) and then processing it in order. So we'd: (a) find all the rows matching the WHERE clause, (b) pull the departure_day_of_week and flight_miles columns out of each row, (c) sort the rows by departure_day_of_week, and then pump all that data through the aggregation engine.

The aggregation engine would then be able to see the rows in a single pass: first all the "FRI" rows, then all the "MON" rows, then "SAT" rows, "SUN" rows, "THU" rows, and "TUE", and lastly the "WED" rows. For each group of rows, the engine computes the AVG(flight_miles) in the obvious straightforward fashion (computing the total sum and the number of instances and dividing). The result of the query is 7 rows, with the average for each day.

Note that the author of the query generally includes the grouping columns in the result, to make the result easy to read.

You can GROUP BY multiple columns, but the overall principle is the same: the columns used to sort the records are exactly the columns used to group the results.

DISTINCT aggregates add some additional complexity. Suppose that, if two flights happen to have exactly the same flight_miles, we only want to consider the flight in the average once; that is, we only want to compute the average over all the unique values of flight_miles for that particular departure_day_of_week.

(I admit, this is very contrived, but hopefully it's still obvious what we're talking about here.)

In this case, the programmer can use the word DISTINCT to specify this:


SELECT departure_day_of_week, AVG(DISTINCT flight_miles) 
   FROM flights
   WHERE (origin = 'SFO' AND destination = 'SAT')
      OR (destination = 'SFO' AND origin = 'SAT')
   GROUP BY departure_day_of_week

This adds a small additional wrinkle into the query execution, as we need to ensure that we can refine the results to consider each unique combination of destination_day_of_week and flight_miles only once.

Derby accomplishes this by including the DISTINCT aggregate column into the sorting process, so that the rows are sorted by the compound key (departure_day_of_week, flight_miles). This way, all the rows which continue duplicate values for this pair of columns arrive together, and we can consider only the first such row and discard the others.

When I re-wrote the GROUP BY engine as part of DERBY-3002, I considered this problem, and implemented it, but I made a mistake. Note that, above, I observed that "the columns used to sort the records are exactly the columns used to group the results". However, with DISTINCT aggregates, this isn't precisely true, as there is one extra column in the sort key (the DISTINCT aggregate column) which isn't used to group the results, just to sort them.

In DERBY-3002, I handled that special case by: after the sort, but before the grouping pass, I removed the DISTINCT aggregate column from the sort key.

This worked, but ...

For most queries, the natural flow of query execution processes each intermediate result once. This is efficient and so the database works hard to do this whenever possible.

However, there are some cases in which an intermediate result must be processed multiple times. One example is the so-called "Cartesian Product" query:


SELECT a.*, b.* FROM a, b

In this query, the database produces all possible combinations of rows from tables a and b: for each row in table a, and each row in table b, there is a row in the result containing those values from a's row and b's row.

In such a query, Derby uses a brute force technique and simply implements the query as: for each row in a, read each row of b, and emit the values.

This means that we read the inner table (table b), multiple times, once per row in table a.

This was where my bug came in: it is possible that the inner table is a GROUP BY query of its own, which includes a DISTINCT aggreate.

When this happens, my code that removed the DISTINCT aggregate column from the sort key was causing problems. The first time we read the inner table, everything was fine, but then the next time, the ordering/grouping/sorting columns were all messed up (I removed the column from the key, but didn't add it back for the next go-round).

In DERBY-5584, you can see a clear and simple example of a query which demonstrates the bug.

The fix, I believe, is that instead of physically removing the column from the sort key, we need to instead teach the grouping logic that it may need to consider only a leading subset of the columns in the sort key as the grouping column.

I've implemented that change and am hoping it solves the bug.

It's not just a game, it's an obsession

2012-01-27T18:42:00.000-08:00

The search

https://www.google.com/#q=skyrim+site:youtube.com

now returns more than 24 million videos, nearly all of them fan-contributed.

For example, here's a particularly nice one: http://www.youtube.com/watch?v=IL5K09mqwZc.

Now, back to that bandit camp...

Advances in weather instrumentation

2012-01-27T15:38:00.000-08:00

I dug this great post from Cliff Mass about the state of modern weather observation instrumentation.

As I read it, I was remembering a discussion I'd had with my sailing friend Nick about whether the real-time wind reports at http://www.sailflow.com could/should include information from the boats themselves (to start with, from large commercial vessels or government vessels such as the USCG small boats, but eventually might each sailboat itself report on the wind conditions it's observing)?

So it was a kick to read about how large-scale weather forecasting in fact does include such information, and to come across this subtle observation about the feedback loop that occurs with that process:

As part of the Volunteer Observing System (VOS), mariners take observations every six hours. The light blue dots on this chart show where ships were reporting at one particular time:
...
[But] as forecasts get better, the ships avoid the areas we really need data---in the middle and near major storms. When forecasts were bad, ships would get trapped in dangerous conditions--now they can get out of the way.

Technology improves, and the way that humans use that technology improves as well, and the whole inter-connected world evolves.

"Then the screams began."

2012-01-26T09:16:00.000-08:00

All right, I know, I said I was not writing about these topics, but, really, have you read this report in today's New York Times about the conditions in the Chinese factories that build the world's high-tech gadgetry?

Those accommodations were better than many of the company’s dorms, where 70,000 Foxconn workers lived, at times stuffed 20 people to a three-room apartment, employees said. Last year, a dispute over paychecks set off a riot in one of the dormitories, and workers started throwing bottles, trash cans and flaming paper from their windows, according to witnesses. Two hundred police officers wrestled with workers, arresting eight. Afterward, trash cans were removed, and piles of rubbish — and rodents — became a problem.

It's a long and detailed article; you need to read it all. But it concludes with this observation:

“You can either manufacture in comfortable, worker-friendly factories, or you can reinvent the product every year, and make it better and faster and cheaper, which requires factories that seem harsh by American standards,” said a current Apple executive.
“And right now, customers care more about a new iPhone than working conditions in China.”

Go. Read. The. Article.

Also worth reading: Making It in America, by Adam Davidson of NPR's Planet Money team.

These meetings can lead the company to move dozens of jobs to another country or, in some cases, to create new jobs in the U.S. When Standard decided to increase its fuel-injector production, it chose to do that in the U.S., and staffed up accordingly (that’s how Maddie got her job). Standard will not drop a line in the U.S. and begin outsourcing it to China for a few pennies in savings. “I need to save a lot to go to China,” says Ed Harris, who is in charge of identifying new manufacturing sources in Asia. “There’s a lot of hassle: shipping costs, time, Chinese companies aren’t as reliable. We need to save at least 40 percent off the U.S. price. I’m not going to China to save 10 percent.” Yet often, the savings are more than enough to offset the hassles and expense of working with Chinese factories. Some parts—especially relatively simple ones that Standard needs in bulk—can cost 80 percent less to make in China.

These are complex issues. I'm pleased that journalists are taking the time to really dig into them, and to help educate us all, for as Dan Lyons said: "The problem I’m having isn’t with Apple, but with me."

Location, location, location

2012-01-25T11:35:00.000-08:00

In computing, as in everything else, we care about physical location:

First, do you know where, I mean exactly where, the Web was invented? In case you don't, David Galbraith helps us track down the precise location, complete with a Google-mapped satellite view and a short interview with TBL himself:
I wrote the proposal, and developed the code in Building 31.I was on the second (in the European sense) floor, if you come out of the elevator (a very slow freight elevator at the time anyway) and turn immediately right you would then walk into one of the two offices I inhabited. The two offices (which of course may have been rearranged since then) were different sizes: the one to the left (a gentle R turn out of the elevator) benefited from extra length as it was by neither staircase nor elevator.The one to the right (or a sharp R turn out of the elevator) was shorter and the one I started in. I shared it for a long time with Claude Bizeau.I think I wrote the memo there.
I love the photo of the nice young fellow who currently occupies the office...!
Secondly, this is all-too-faddish, but you've got to check out this delightful video full of the latest Silicon Valley cliches.
"Who has a party in Palo Alto?"
...
"It's like Pandora ... for cats!"
Great stuff, and it really is quite accurate and well-written.
Lastly, this brought back great memories of my time in New England just after we got married. We lived in Boston (Brighton, to be precise); I worked in Kendall Square in Cambridge; many of my friends switched jobs and worked at "the old Lotus building", which at the time was the new Lotus building, back when Lotus was its own company and was the hot new place to be; I rode my bike to and from the office along Western Avenue and through Central Square; and I went to grad school 5 stops down the Red Line at UMass Boston (while my wife was taking a film studies class at BU).

I miss those days in Boston.

But the point of Feld's essay, and the point, really, of all three essays/films, is that location matters. It really does. I loved my time in Boston, and even though Brad Feld is right that East Cambridge is one of the most remarkable and wonderful spots in the country, the Bay Area is even better, which is why we (reluctantly) switched coasts in 1988.

Back to the hard-core tech stuff ...

2012-01-24T16:44:00.000-08:00

Here are three pointers that are (hopefully) as interesting to you as they were to me:

Over at Real World Technologies, David Kanter has posted this great writeup of Intel's latest mobile-oriented microprocessor platform: Medfield, Intel's x86 Phone Chip
Medfield is a credible SoC for smartphones and is good enough to begin the process of building vendor and carrier relationships for Intel. This is particularly true, given Intel’s attractive roadmap. In 2013, Intel will ship a 22nm FinFET SoC with the new, power-optimized Silvermont CPU and the recently announced PowerVR Series 6 graphics. The rest of the world will ramp 20/22nm in 2014 at the earliest, a gap of 6-12 months. Judging by Intel’s plans for 14nm SoCs based on the Airmont CPU core in 2014, this process technology advantage is only likely to grow over time. Whether that advantage will yield a significant smart phone market share for Intel is uncertain, but Medfield clearly demonstrates that it is possible.
Even if Intel doesn't succeed, their continued presence in the marketplace and competition for market share will spur the big players (Qualcomm, TI, ARM, etc.) to improve their own systems, not just rest on their laurels.
The HTTP Working Group, which is (I think) a joint effort of the W3C and the IETF, is considering a new effort to try to define a HTTP 2.0 specification.
Why here? This mailing list is the best approximation of the HTTP community; it has participation (or at least presence) from most implementations, including browsers, servers, intermediaries, CDNs, libraries, tools, etc. I firmly believe that as HTTP evolves, it needs to accommodate the entire community, not just the selected needs of a subset, so rather than creating a new WG or having a private collaboration, it should happen here.
This won't be easy, but it's great to see them trying. I suspect that these topics are some of what they'll be talking about.
And if you don't have enough to read yet, here's a very nice, compact, and well-rounded Distributed Systems Reader. As it turns out, I was already familiar with all but two of these papers, but hey! Two new papers on the foundations of Distributed Systems; what's not to like!

CAPTCHA factories

2012-01-24T15:37:00.000-08:00

It seems to be the week for me to write about social issues in computing, so I'll toss one more in. Then, I promise, I'm done with that slant for a while, because it's really not my sort of thing.

Anyway, I can't help but note the fascinating (in a can't-look-away-from-the-abject-horror sort of way) recent research done by a team at the University of California San Diego: Re: CAPTCHAs -- Understanding CAPTCHA-Solving Services in an Economic Context.

CAPTCHAs, of course, are those strange little pictures full of numbers and letters, generally semi-garbled or semi-distorted, that various web pages ask you to type in in order to prove you're a human being, not a robot. The idea is to dissuade people who are writing programs to manipulate web pages that are only intended to be manipulated by human beings. The acronym stands for "Completely Automated Public Turing test to tell Computers and Humans Apart".

Do they work? Well, according to the UCSD study, they actually do work fairly well, according to the strict interpretation of their stated goal. After testing several specialized programs designed to be able to "break" CAPTCHAs, the researchers found that the automated solvers generally were not successful:

We observed an accuracy of 30% for the 2008-era test set and 18% for the 2009-era test set using the default setting of 613 iterations, far lower than the average human accuracy for the same challenges (75–90% in our experiments).

So CAPTCHAs are working, right?

Well, not so fast.

It turns out that there is an immense industry of CAPTCHA-solving, and the solvers are actual human beings, not computer programs:

there exists a pool of workers who are willing to interactively solve CAPTCHAs in exchange for less money than the solutions are worth to the client paying for their services.

These people apparently sit in front of computers for hours at a time, doing nothing but solving CAPTCHAs that are displayed in front of them by Internet-based solving services that then turn around and sell these solutions to clients willing to pay for CAPTCHA solutions:

Since solving is an unskilled activity, it can easily be sourced, via the Internet, from the most advantageous labor market—namely the one with the lowest labor cost. We see anecdotal evidence of precisely this pattern as advertisers switched from pursuing laborers in Eastern Europe to those in Bangladesh, China, India and Vietnam

How much do these people end up getting paid? Almost nothing, but still enough to attract workers:

on Jan. 1st, 2010, the average monthly payout to the top 100 earners decreased to $47.32. In general, these earnings are roughly consistent with wages paid to low-income textile workers in Asia [12], suggesting that CAPTCHA-solving is being outsourced to similar labor pools

What do the authors conclude from all of this? The answer is that you can view the whole arrangement in an economics framework:

Put simply, a CAPTCHA reduces an attacker’s expected profit by the cost of solving the CAPTCHA. If the attacker’s revenue cannot cover this cost, CAPTCHAs as a defense mechanism have succeeded. Indeed, for many sites (e.g., low PageRank blogs), CAPTCHAs alone may be sufficient to dissuade abuse. For higher-value sites, CAPTCHAs place a utilization constraint on otherwise “free” resources, below which it makes no sense to target them. Taking e-mail spam as an example, let us suppose that each newly registered Web mail account can send some number of spam messages before being shut down. The marginal revenue per message is given by the average revenue per sale divided by the expected number of messages needed to generate a single sale. For pharmaceutical spam, Kanich et al. [14] estimate the marginal revenue per message to be roughly $0.00001; at $1 per 1,000 CAPTCHAs, a new Web mail account starts to break even only after about 100 messages sent.

It's a cold, calculating, unfeeling analysis, but it's an absolutely fascinating paper, easy to read and full of lots of examples and descriptions of the details behind this corner of the Internet. I never knew this existed, and I'm wiser now that I do.

However, I still felt the irresistible urge to go and wash my hands after learning all this. :(

Some ideas on what to do with your pitchforks

2012-01-23T13:33:00.000-08:00

Here are two suggestions, from two very smart people:

David Pogue: Put Down the Pitchforks on SOPA
In this case, the solution is to work on the language of the bills to rule out the sorts of abuses that the big Web sites fear. (And to fix the other minor point, which is that the bills won’t work. For example, they’d make American Internet companies block your access to domain names like “piracy.com,” but you’d still be able to get to them by typing their underlying numerical Internet addresses, like 197.12.34.56. In other words, anybody with any modicum of technical skills would easily sidestep the barriers.)
Clay Shirky: Pick up the pitchforks:David Pogue underestimates Hollywood
This is a general problem — there is a reasonable conversation to be had about sites set up for large, commercial operations that are designed to violate copyright. And because there’s a reasonable conversation to be had, Pogue (and many others) simply imagine that the core of SOPA must therefore be reasonable. Surely Hollywood wouldn’t try to suspend due process, would they? Or create a parallel enforcement system? Or take away citizen recourse if they were unfairly silenced?

I think that the overall discussion around SOPA/PIPA/etc. has been valuable and I have certainly learned a lot by following it. I hope that the discussion continues, and more importantly I hope that the discussion continues in the open: as Mike Masnick points out on TechDirt, a major problem with the legislative process here is that our representatives constructed the bill in secret, rather than via open debate.

no hearings, no debate, no discussion. It was a seven minute session that wasn't recorded or available to the public. That's a sign that the fix is in, not that the public is being represented.

Hopefully, the major lesson that everybody learns from this is that things are better when discussion and debate occur openly. Isn't that what open government is supposed to be?

No, it's not breathtaking, it's horrific

2012-01-23T09:16:00.000-08:00

I had the same reaction that Dan Lyons did to this phenomenal expose published in Sunday's New York Times, but I couldn't put it into words.

Thank you, Mr Lyons, for expressing what I wanted to say.

Update: See also: Mr Daisey and the Apple Factory.

"A Bug Hunter's Diary": simple idea, effectively executed

2012-01-22T15:49:00.000-08:00

Tobias Klein's A Bug Hunter's Diary is a simple idea which Klein carries through to execution quite well.

The book is structured as a series of 7 separate chapters; each chapter relates the story of how Klein:

Searches for a vulnerability
Isolates the vulnerability
Develops a demonstration of the vulnerability
Refines the demonstrated vulnerability to produce an exploit of the bug

Each chapter follows roughly the same structure, but the particulars and details of the vulnerability and its exploit are different each time.

Along the way, Klein includes information on extremely valuable tools and techniques, such as: how to use various debuggers to observe software in action; the various types of vulnerabilities such as stack overflows, heap overruns, out-of-range data, etc; how to find sample data files to use as input sources; how and when to write your own quick-and-dirty programs to enumerate possibilities or search for weaknesses, and how to disassemble code to understand its behavior.

Klein's choice of subjects is also impressively broad. The operating systems include Windows, Solaris, Mac OS X, Linux, and iOS; the vulnerable software packages include operating systems, browsers, image processing libraries, and device drivers. This wide ranging approach might be rather overwhelming for a beginning programmer, but this book is not intended for a novice audience; as Klein states at the outset, "you should have a solid grasp of the C programming language and be familiar with x86 assembly."

Klein also provides thorough references and material for additional study; each chapter ends with detailed references and notes to enable the reader to pursue these topics more deeply. Indeed, since this sort of work is best done "hands on", Klein has, commendably, taken the time to precisely note the exact versions of the software he works with so that you can "follow along" on your own machine, setting up the vulnerable software and watching it crash, just as he did.

If you're tired of ordinary programming books, and looking for something a little different, this might be a good book to try. It's got lots of code to read, lots of bugs to understand, and lots of tools and techniques on display. Among all of this, I'm confident that you will find much to learn from, and you'll finish the book resolving, as I did, to practice these skills and improve your programming ability.

It's no Hugo Cabret, but ...

2012-01-22T09:38:00.000-08:00

... Brian Selznick's Wonderstruck is a warm and entertaining book.

It's written in the same style as Hugo Cabret: neither graphic novel nor ordinary prose, it's something in between, with the book's story interspersed between the author's own pencil drawings. The book is doubly-spun, for it tells two inter-related stories, 70 years separated, as it goes.

Although the book touches many subjects, I was particularly drawn to its observations about how we organize our knowledge, whether it be in museums, atlases, and libraries, or just in the spaces we inhabit.

In a way, anyone who collects things in the privacy of his own home is a curator. Simply choosing how to display your things, deciding what pictures to hang where, and in which order your books belong, places you in the same category as a museum curator.

The urge to collect, to organize, and to understand is so universal; it provides a delightful underlying motif in the book.

He noticed a discarded map of the museum next to the sink. He unfolded it and read the names of the halls: Meteorites, Gems and Minerals, Man in Africa, Northwest Coast Indians, Biology of Birds, Small Mammals, Earth History. Like his mom's library, the entire universe was here, organized and waiting.

Selznick particularly captures the plight of the young mind, exposed to so much knowledge, trying to drink it all in:

Ben wished the world was organized by the Dewey decimal system. That way you'd be able to find whatever you were looking for, like the meaning of your dream, or your dad.

Of course, your dad isn't cataloged in a library anywhere; some things we each discover on our own.

What would it be like to pick and choose the objects and stories that would go into your own cabinet? How would Ben curate his own life? And then, thinking about his museum box, and his house, and his books, and the secret room, he realized he'd already begun doing it. Maybe, thought Ben, we are all cabinets of wonders.

If you've got a young reader, just setting forth on his own voyage of discovery and knowledge, reading and exploring and collecting and questioning, I think the two of you will very much enjoy reading Wonderstruck together.

Crypto class delayed

2012-01-21T08:22:00.001-08:00

From an email to registered students:

But since we don't have a firm timeline right now, we'd rather leave this open and get back to you with a definitive date soon (rather than just promise you a date that's far enough in the future that we can feel confident about it). We'll let you know a firm date as soon as we possibly can.

Bummer ... but hopefully it's not long!

Herb Sutter article on exception-safe function calls

2012-01-20T14:08:00.000-08:00

Fascinating, detailed article by Herb Sutter about exception safety, object allocation, and function calls: GotW #102: Exception-Safe Function Calls.

With some humor, too!

This code attempts to “throw unique_ptr at the problem.” Many people believe that a smart pointer is an exception-safety panacea, a touchstone or amulet that by its mere presence somewhere nearby can help ward off compiler indigestion.

Exception safety is so tricky. I'm reminded of the nest of vipers that is JDBC exception handling, in which code which might encounter a JDBC exception tries to be a good citizen and clean up its resources, but the cleanup processing itself may encounter exceptions, and these exceptions may then conceal the original problem that led to the first exception, making your application an un-diagnosable mess.

Not to mention that all your code starts to look like this or this.

But now we've wandered far afield from C++ exception handling, and Sutter's great article (which you should read).

A nicely-presented example of dynamic algorithm interactions

2012-01-19T18:22:00.000-08:00

Raymond Chen has a great example of dynamic algorithm interactions on his blog: Don't try to allocate memory until there is only x% free.

The diagrams are very helpful and the description is quite clear, even amusing:

Program 20 is now curled up in the corner of the computer in a fetal position. Program 10 meanwhile continues allocating memory, and Program 20, having shrunk as much as it can, is forced to just sit there and whimper.

A commenter notes that this is very similar to the dynamic pricing feedback loop noticed by Michael Eisen a year ago: Amazon’s $23,698,655.93 book about flies. Right down to the humor:

Peter Lawrence can now comfortably boast that one of the biggest and most respected companies on Earth valued his great book at $23,698,655.93 (plus $3.99 shipping).

All kidding aside, it is in fact tremendously hard to write software that uses the available machine resources efficiently and effectively, while still being a "good citizen" when other programs are trying to use the machine, too.

I have a feeling my great-grandchildren will still be trying to design algorithms to do this well...

A scattered collection of database and filesystem news

2012-01-19T17:27:00.000-08:00

These things aren't fundamentally related, but they are all interesting in rather similar ways, so I'm noting them all in a single post. Certainly they each deserve much more detailed analyses; who has the time?

First, some filesystem notes:

Microsoft are getting a lot of attention for their announcement about ReFS.. Two points that particularly struck me:
Optimize for extreme scale. Use scalable structures for everything. Don’t assume that disk-checking algorithms, in particular, can scale to the size of the entire file system.
Never take the file system offline. Assume that in the event of corruptions, it is advantageous to isolate the fault while allowing access to the rest of the volume. This is done while salvaging the maximum amount of data possible, all done live.
Paul Thurrott says that the new filesystem will be in the server edition of Windows only (at least for now).
By the way, does one say "Ree Eff Ess"? Or does one say "reffs"?
Many people remarked that ReFS seemed to be "bringing ZFS to Windows", so it's interesting to see this recent work on bringing ZFS to Mac OS X. They seem to have spent a lot of time on their web page, but it's hard to find much about the underlying technology. But they're apparently a small young company, so let's give them time.

Now, to complement your filesystems news, here's some database news:

First and most important, don't miss all the talk about Amazon's new DynamoDB:
- Jeff Barr's notes on the AWS blog: Amazon DynamoDB - Internet-Scale Data Storage the NoSQL Way
- James Hamilton's observations: Amazon DynamoDB: NoSQL in the Cloud.
- Werner Vogels's observations: Amazon DynamoDB – a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications
- Sid Anand's observations on the state of major NoSQL systems as we enter 2012: The State of NoSQL in 2012
And vastly more. There is a huge amount to read about the new DynamoDB work; it will take a while for us all to digest it and understand it. But, as usual with Amazon, there is a lot of "meat" there!
And on another, related, front, I see that Google have open-sourced their leveldb library. This, too, is a "NoSQL-style" database:
- Keys and values are arbitrary byte arrays.
- Data is stored sorted by key.
- Callers can provide a custom comparison function to override the sort order.
- The basic operations are Put(key,value), Get(key), Delete(key).
- Multiple changes can be made in one atomic batch.
- Users can create a transient snapshot to get a consistent view of data.
- Forward and backward iteration is supported over the data.
- Data is automatically compressed using the Snappy compression library.
There is additional information to read about leveldb in their documentation pages.

I hope to find time to plow through much of this material; there sure is a lot to learn about in the world!

Sluggish computers are so annoying

2012-01-18T07:36:00.000-08:00

I despise the following experience, which happens to me all too frequently:

I bring up a web site in my browser
The web site starts to load, but initially it is fragmentary, with lots of hidden frames, images, etc. still left to draw.
As the various bits and pieces start to fill in, the page shifts around and adjusts, redrawing some sections, moving and re-shaping other areas.
By this time, I've already spotted the place I want to click on, to get to the next web page, so I click on it
But by the time the computer can process my click and decide what I was clicking on, the web page has re-drawn itself, and the computer decides I've clicked on a different part of the page and takes me somewhere else.

Interestingly, this seems to happen most commonly with ads, so the result is that the computer decides I've clicked on some ad, instead of the useful link that was sitting right next to the ad. (The ad, of course, takes up lots of screen real estate, while the useful link is tiny. So the odds are much higher that the computer interprets my click as being destined for the ad.)

Of course, I didn't want to click on the ad, so I end up going "back", re-clicking on the correct link, and going to the actual page I wanted.

But it makes me wonder how many apparently "real" clicks on ads are actually sluggish-computer-misunderstood-my-click clicks, and how much that is distorting the online advertising industry.

Is it true that, in practice, most of you use AdBlock or or a proxy server or something similar, so this doesn't happen to you because the ads get filtered out before they ever get to your browser?

Is there an AdBlock that works reliably in Google Reader?

Stanford online cryptography class draws closer

2012-01-17T16:24:00.000-08:00

According to the web site, the crypto class will start next Monday!

tick ... tick ... tick ...

Have you listened to Cory Doctorow's 28th CCC keynote yet?

2012-01-17T10:03:00.000-08:00

In all the hubbub of the holidays, the always fascinating Cory Doctorow traveled to Germany to give the keynote address at the 28th Chaos Communications Conference.

Doctorow is an entertaining speaker, and he's delivered variants of this speech before, but he does a particularly good job with the speech this time.

He opens by observing that general-purpose computers, like other very general-purpose tools, provide a power than can be hard to comprehend by policymakers:

General-purpose computers are astounding. They're so astounding that our society still struggles to come to grips with them, what they're for, how to accommodate them, and how to cope with them.
...
The important tests of whether or not a regulation is fit for a purpose are first whether it will work, and second whether or not it will, in the course of doing its work, have effects on everything else. If I wanted Congress, Parliament, or the E.U. to regulate a wheel, it's unlikely I'd succeed. If I turned up, pointed out that bank robbers always make their escape on wheeled vehicles, and asked, “Can't we do something about this?", the answer would be “No". This is because we don't know how to make a wheel that is still generally useful for legitimate wheel applications, but useless to bad guys. We can all see that the general benefits of wheels are so profound that we'd be foolish to risk changing them in a foolish errand to stop bank robberies. Even if there were an epidemic of bank robberies—even if society were on the verge of collapse thanks to bank robberies—no-one would think that wheels were the right place to start solving our problems.

But as Doctorow goes on to explain, general-purpose computers are even more powerful than other general-purpose technologies (such as the wheel), because a general-purpose computer can, in our modern world, become any other sort of tool:

The world we live in today is made of computers. We don't have cars anymore; we have computers we ride in. We don't have airplanes anymore; we have flying Solaris boxes attached to bucketfuls of industrial control systems. A 3D printer is not a device, it's a peripheral, and it only works connected to a computer. A radio is no longer a crystal: it's a general-purpose computer, running software.

Doctorow's core point is that we need to be very careful about how we regulate computers, because the general-purpose computer is much more important than many of the social problems that have been ascribed to it so far:

Regardless of whether you think these are real problems or hysterical fears, they are, nevertheless, the political currency of lobbies and interest groups far more influential than Hollywood and big content. Every one of them will arrive at the same place: “Can't you just make us a general-purpose computer that runs all the programs, except the ones that scare and anger us? Can't you just make us an Internet that transmits any message over any protocol between any two points, unless it upsets us?"

Here's how to find Doctorow's speech:

Bonus: After you've listened to Doctorow, wander over to Freedom-to-Tinker and read this fascinating interview with Harvard professor Jonathan Zittrain: Applications and Appliances: A Conversation with Jonathan Zittrain.

JZ: Nothing’s inherently wrong with single-purpose devices. The worry comes when we lose the general-purpose devices formerly known as the PC and replace it with single-purpose devices and “curated” general-purpose devices.

My car switched over to binary notation this week ...

2012-01-16T07:53:00.001-08:00

... I happened to look down at the odometer a few days ago, and it read: 110010.

A few days later, I looked again, and it read: 110101.

I was starting to get a little bit worried, but it seems to be back to normal now.

A few thoughts on The Hunger Games

2012-01-14T10:43:00.000-08:00

Over the holidays I had the chance to read Suzanne Collins's trilogy: The Hunger Games. I'm sure you know about these books; they are an international phenomenon, even if I wasn't paying attention for the first couple years of their existence.

The books in the series are:

These books are extremely powerful. I finished all three in about 10 days of fevered reading, brushing aside friends and family, staying up late and waking up early in order to continue reading, dwelling on them whenever I wasn't actually reading them. It took me several days after finishing the series before I was even willing to start thinking and reflecting about the books, and several weeks more before I was calmed down enough to start writing about them.

I can't remember the last time I was simply desparate to find out what happened next. Captivating, gripping, enthralling: these are words that describe these books well.

It's interesting that the books are published by Scholastic and positioned as "young adult" novels. Although the books are clearly written for a 12-15 year old girl, the topics that are covered are universal: war, discrimination, poverty, alcoholism, government policy, and death are all central to the story.

Certainly there are plenty of teenage girl aspects to the books (the three biggest motifs in the books are: food and our feelings about it; makeup and fashion; relationships with boys), but I never for an instant felt like I was reading "some girl book". After all, it's not like the rest of us don't care about these things.

But, reader bewarned: this is not the sort of book you'll pick up for bed-time reading with your elementary school child. Rather, what you should have in mind is: "what would The Lord of the Flies be like if Stephen King wrote it, and the heroine was a 16 year old girl?"

I feel the comparison to King is apt, for, like him, Collins is a master of the craft: her characters are vivid, her dialogue is precise, and her sense of the pacing and flow of storytelling is ideal. Nothing is awkward, nothing is out of place.

And in case the phrase "young adult fiction" brings to mind watered-down vocabulary, simplified sentences, and an absence of skill and technique, banish that thought from your mind, for this is literature at the highest level. Across the planet, I'm sure that tens of thousands of literature students are hard at work on theses with topics such as: the metaphoric use of fire (has there ever been such a perfect description of puberty as "Catching Fire"?); the linguistic skill and innovation in coinages such as "muttation" (the perfect word for this creature!); and on the symbolic use of the Jabberjay as representative of the modern nation-state's control over the means of communication and the domain of social discourse.

These are amazing books, and Collins is a superb writer. Although they are not for everyone, I hope they find their way into many lives, and are the basis of many discussions and debates about man's inhumanity to man and about what is really important in life.

I know that I will be thinking about Katniss Everdeen, Peeta Mellark, Haymitch Abernathy, and Gale Hawthorne for many years to come; perhaps you will be, too!

Strange economics

2012-01-12T18:59:00.001-08:00

I'm still learning about my new Kindle, and things still puzzle me.

For example: why does this book cost $10 to buy in paperback, but cost $12 to buy on the Kindle?

Strange...

Update: Perhaps I am seeing this effect. If so, it worked: I didn't buy either the ebook or the paperback (is that what the publisher wanted?).

Bill Slawski's series of articles on SEO-related patents

2012-01-12T16:41:00.000-08:00

Bill Slawski, the Search Engine Optimization expert who does business at SEO By The Sea, has recently been publishing a fascinating series of articles entitled The 10 Most Important SEO Patents.

As you'll discover as you start to read the articles, the series title isn't perfectly accurate, since these aren't necessarily all "SEO patents", but it's no matter, for the articles themselves are extremely interesting.

He's written 6 articles so far:

Part 1 - The Original PageRank Patent Application
This provisional patent may not have the weight or legal value of the continuation patents that followed it, but it captures the excitement and personality of its inventor, Larry Page, in a manner that those patents missed. It also provides head-to-head examples of search results from both Google and AltaVista for specific queries to illustrate how the link analysis involved in what Page was doing with PageRank made a difference.
Part 2 - The Original Historical Data Patent Filing and its Children
The impact of this patent goes on today, likely responsible for the recent Google freshness update, the possible impact on the rankings of a site as content upon pages change and anchor text pointing to that page no longer matches up well, whether Google might consider some pages as doorway pages when they are purchased and links are added to pages and topics of those pages change, and more.
Part 3 - Classifying Web Blocks with Linguistic Features
The patent does provide us with an idea of how a search engine might understand the different blocks that if finds on a page, and use those when it indexes, analyzes and classifies content on that page. For example, a section of a page that contains every short phrases, with each word capitalized, and each phrase a link to another page on a site, that appears near the top of the page or in sidebar to the left of the page might be the main navigation for that page.
Part 4 - PageRank Meets the Reasonable Surfer
the algorithm behind the Reasonable Surfer model might determine that even though the link is prominently placed and stands out from the rest of the text in an important part of a page, the text of the link has nothing to do with the content of the rest of the page, and that text evidences a very commercial intent.
Part 5 - Phrase Based Indexing
This phrase-based indexing system provided a way to defeat Googlebombing, and to determine how much anchor text relevance should be passed along with links.
Part 6 - Named Entity Detection in Queries
Google, Bing, and Yahoo all look for named entities on web pages and in search queries, and will use their recognition of named entities to do things like answer questions such as “where was Barack Obama born?”

These are really wonderful articles, full of detail, clearly-written, packed with examples, and loaded with link after link to chase and study.

If you are interested in how the world's top search engines work "under the covers", you won't want to miss this series. Thanks Bill for writing such a great set of posts!

Code smells

2012-01-10T16:01:00.000-08:00

Here's a nice article on code smells over at the Atlassian developer blog: Testing and Bad Smells: when to investigate potential bugs.

once in a while, you investigate and find something more. The bad smell is merely a symptom of a larger issue that was otherwise unnoticeable… or, at least, unnoticed. By investigating the smell, you’ve prevented a much bigger issue from shipping

I liked the way the author took the "bad smells" metaphor and worked with it to provide some nice rules of thumb for how to refine your instincts so that you have a better sense of when to dig deeper, and when it would be more efficient to just move along.

You're not crazy ...

2012-01-10T09:58:00.000-08:00

... the weather really is different this winter.

This winter's remarkable AO/NAO pattern stands in stark contrast to what occurred the previous two winters, when we had the most extreme December jet stream patterns on record in the opposite direction (a strongly negative AO/NAO). The negative AO conditions suppressed westerly winds over the North Atlantic, allowing Arctic air to spill southwards into eastern North America and Western Europe, bringing unusually cold and snowy conditions. The December Arctic Oscillation index has fluctuated wildly over the past six years, with the two most extreme positive and two most extreme negative values on record. Unfortunately, we don't understand why the AO varies so much from winter to winter, nor why the AO has taken on such extreme configurations during four of the past six winters.

Mathematics Journalism

2012-01-09T16:49:00.000-08:00

It turns out there is an annual award for Mathematics Journalism. I did not know this before, but I think it is an excellent thing; it is great to encourage this quite challenging of subfields.

I learned about the award because this year's winner, Dana Mackenzie, wrote about it on his chess blog, which I follow.

The prior winners of the award look like a fascinating list, including of course Martin Gardner, but there are a number of other authors on that list that I should follow up on. Somehow Martin Gardner was not the first winner of the award, but you certainly can't take anything away from the actual first winner (James Gleick); I've enjoyed several of his books.

Is it just me, or is Rachel calling a lot more frequently nowadays?

2012-01-09T13:50:00.000-08:00

It seems like Rachel from Cardholder Services is calling constantly nowadays.

Am I the only one having this problem?

This rather unhelpful story in the Los Angeles Times says that I'm not alone. About the best it can offer, though, is to suggest that you put yourself on the Do Not Call Registry even though the article notes that this doesn't work.

These rather more helpful articles in the St Louis Beacon and the Wisconsin State Journal seem to indicate that the FTC is aware of these problems, and apparently is even trying to do something about them.

Apparently, though, this is far from the first time the FTC has tried to prosecute such activity. The FTC press release notes that "Cox resides in California but runs his allegedly illegal operation through multiple foreign corporations purportedly in countries such as Panama, Hungary, Argentina, and the Republic of Seychelles" which may be at least part of the reason it's hard to stop.

Well, for the time being I guess I'll just continue to hang up on these clowns :(

MI MITM

2012-01-07T09:22:00.000-08:00

The "Man in the Middle Attack" is a security vulnerability which has to do with intercepting communications without being observed. It has been around, oh, at least 500 years or so.

MITM attacks are always interesting. Did you see the recent MI (Mission Impossible) movie? There's a great MITM attack in that movie (on the 130th floor of the tallest building in the world, no less!).

In the MI movie, two different criminal organizations are conducting a business transaction, in which one organization is selling information and the other is paying money. The MI team arranges a clever deception, pretending to be the seller of the information to the buyer, and pretending to be the purchaser of the information to the seller. The trick works because neither the buyer nor the seller know each other ahead of time, and cannot successfully authenticate each other, so they fall for the trick.

It's classic MI stuff; they used to pull it off in the original TV series oh-so-many years ago. But I find that it's often easier to recognize techniques like this when they are portrayed in an entertaining fashion, as opposed to the more dry, if more technically correct, format in which they are usually discussed.

So, I happened to be reading Andrew ("bunnie") Huang's fascinating paper on MITM attacks in HDCP video transmission. It's deep and complex work, and not easy going; the slides are a much easier way to get an overview of what's going on here.

As bunnie observes, this particular attack is less about cryptography (though there is plenty of that going on here), than about understanding the policy and cultural frameworks within which cryptography and digital rights management are used:

While the applications of video overlay are numerous, the basic scenario is that while you may be enjoying content X, you would also like to be aware of content Y. To combine the two together would require a video overlay mechanism. Since video overlay mechanisms are effectively banned by the HDCP controlling organization, consumers are slaves to the video producers and distribution networks, because consumers have not been empowered to remix video at the consumption point.

Reading bunnie's work is always engrossing, even though most of it goes way over my head. That's the thing about trying to learn stuff: often you work and work and work and maybe you just get a little bit smarter, but that's certainly better than not getting smarter at all.

So, thank you bunnie, for the informative writeup of your work!

Open Access and the Research Works Act

2012-01-07T08:13:00.000-08:00

This fall, all the Internet policy debate was about SOPA/PIPA.

But there's another issue that you should be paying attention to: Open Access and the Research Works Act.

Start by reading Rebecca Rosen's essay in The Atlantic: Why Is Open-Internet Champion Darrell Issa Supporting an Attack on Open Science?. She calls the bill "a direct attack on the National Institutes of Health's PubMed Central, the massive free online repository of articles resulting from research funded with NIH dollars. ", and gives some great background and pointers to the previous incarnations of this legislation, and why they were problematic.

Move on to a nice essay by John Dupuis of York University in Toronto: Scholarly Societies: It's time to abandon the AAP over The Research Works Act, where he challenges the scientific societies of the world to explain why this act supports their stated mission:

These societies will certainly have among their vision and mission statements something about advancing the common good, promoting the scholarly work of their membership and scholarship in their fields as a whole.
To my mind, The Research Works Act is directly opposed to those goals.

Michael Eisen of U.C. Berkeley comments on the poisonous influence of money in these discussions: Elsevier-funded NY Congresswoman Carolyn Maloney Wants to Deny Americans Access to Taxpayer Funded Research.

This industry already makes generous profits charging universities and hospitals for access to the biomedical research journals they publish. But unsatisfied with feeding at the public trough only once (the vast majority of the estimated $10 billion dollar revenue of biomedical publishers already comes from public funds), they are seeking to squeeze cancer patients and high school students for an additional $25 every time they want to read about the latest work of America’s scientists.

Saving the best for last, make sure you don't miss Danah Boyd's superb rant: Save Scholarly Ideas, Not the Publishing Industry. She concedes that it's no surprise that the corporate publishers and their lobbyists are using their money and prestige to squeeze the politicians:

the scholarly publishing industry is in the midst of complete turmoil. Its business model is getting turned upside down and some of these organizations are going to die. So I get why their lawyers are trying to grab any profit by any means necessary, letting go of the values and purpose that drove their creation.

But what really bothers Boyd is the behavior of the academics themselves:

How did academia become so risk-adverse? The whole point of tenure was to protect radical thinking. But where is the radicalism in academia? I get that there are more important things to protest in the world than scholarly publishing, but why the hell aren’t academics working together to resist the corporatization and manipulation of the knowledge that they produce? Why aren’t they collectively teaming up to challenge the status quo? Journal articles aren’t nothing… they’re the very product of our knowledge production process.

Readers of my blog know that I've written about this issue before, and I've pointed to important academics in my field, such as Matt Welsh, who have taken highly visible positions on these topics.

I've mostly been following the debate as it relates to the Computer Sciences field, but it seems like the issue is much more intense and controversial in the Biological Sciences field, where the major institution is the U.S. Government's PubMed Central:

PMC is a free full-text archive of biomedical and life sciences journal literature at the U.S. National Institutes of Health's National Library of Medicine (NIH/NLM).

One of the journalists who's doing a good job of covering these issues is Glyn Moody of the web site TechDirt; among his recent articles have been a detailed examination of copyright license clauses in major online science libraries, and a thorough exploration of the financial aspects of major science publishers.

It's a tough and challenging problem, and with all the debate and activity over SOPA/PIPA it isn't getting a lot of attention. But now you know more than you did before.

Update: Stevan Harnad has an interesting article here, specifically discussing the issues revolving around peer review, while Joshua Gans weighs in with his views here, specifically providing an economics-based perspective.

Mining dark fiber

2012-01-06T14:48:00.000-08:00

Something about this story in Information Age seems "off" to me.

The story, entitled Mining dark fibre, describes how there are firms that specialize in locating hitherto-unused fiber optic cables and putting them to use where there is unmet demand for bandwidth.

Luckily for Ritchie, there are other opportunities to improve latency times, in the form of unused cabling infrastructure around the world. "We're finding unused cables all [the] time, everywhere [from] China and Russia to parts of Brazil," he explains.
There are a number of reasons why unused, or "dark", fibre optic cable might be lying around, he says. "Quite often, when electricity lines are put down, there's underlying optical fibre as well, because if you're digging a hole you may as well whack as many services in there as possible."

So far, this makes plenty of sense to me. I don't find it at all unremarkable that there is plenty of unused fiber optic cable around the world, and keeping track of where it is and who might be able to use it could be a perfectly reasonable and money-making "match-maker" sort of job.

After all, the great Neal Stephenson described all of this more than 15 years ago, in his epic article for Wired: Mother Earth Mother Board.

But then the article goes a little wonky:

sometimes when military objectives change, all of a sudden a bunch of infrastructure becomes available
...
Why would there be a secret substation on the Russian border? You would need to ask the Chinese government.
...
Ritchie is not prepared to reveal precisely how one goes about 'discovering' an unused fibre optic cable in the Mongolian desert. "Are we telling everybody how we do it? No."

What the ? Military objectives? Secret substations? Skull-duggery in the Mongolian desert?

Did the author of the otherwise-bland article just want to spice it up a bit?

Or is there something much darker going on here, slipping by just out of reach from me?

Windows 8 Storage Spaces whitepaper

2012-01-06T07:08:00.000-08:00

It looks like the Storage Spaces feature of Windows 8 will be very powerful.

Fundamentally, Storage Spaces virtualizes storage in order to be able to deliver a multitude of capabilities in a cost-effective and easy-to-use manner. Storage Spaces delivers resiliency to physical disk (and other similar) failures by maintaining multiple copies of data. To maximize performance, Storage Spaces always stripes data across multiple physical disks. While the RAID concepts of mirroring and striping are used within Storage Spaces, the implementation is optimized for minimized user complexity, maximized flexibility in physical disk utilization and allocation, and fast recovery from physical disk failures.

My family have been using a Data Robotics Drobo device for more than six months now and it's been very successful. These types of low-end, easy-to-use, simple solutions for home storage are wonderful; it's great to see that Microsoft will be providing this as part of their core operating system.

Fonts on shirts!

2012-01-05T14:38:00.000-08:00

In life, it is always amazing to me to see that, wherever you look, there are people who have spent far more time than you might expect thinking about a problem that you hadn't even thought existed.

Such as, for example, this fascinating essay on font design for professional soccer jerseys: Fixing Major League Soccer’s Font Problem

Major League Soccer’s uniform font persists, blocking the orderly progression of good design.
North American leagues largely allow teams leeway in customizing the look of their numbers and letters; that’s how you get Red Sox “3”s, Laker “3”s old Blue Jay “3”s and Steeler “3”s. Thousands of colleges and minor league teams have unique takes too. On the world stage, soccer is far from uniform, and international competitions showcase a range of interesting font design. Conversely, the English Premier League has been perfecting the font lockup for at least 15 years.

Gotta love that double entendre on "uniform font" :)

I actually own one shirt (a bit of swag from my day job) with my name on the back shoulders; tonight I must remember to go have a look at the font...

The Beautiful and the Damned: a very short review

2012-01-05T07:38:00.000-08:00

At some point this fall I zipped through The Beautiful and the Damned: A Portrait of the New India by Siddhartha Deb.

It's an extremely quick read, short and fascinating, vivid and well-written, but oh my is it grim and depressing. The book is structured as five stories, about five representative individuals that the author meets and gets to know in some detail: a businessman, an engineer, a farmer, a factory worker, and a waitress. Each story is compelling and intriguing.

But the book is really about India as a whole, as the subtitle suggests, and unfortunately India as a whole is described as a terrifyingly awful place:

The data collected by the Indian government, which has been subject to some controversy for its tendency to downplay the number of poor people and the extent of their destitution, is nevertheless stark. In 2004-5, the last year for which data was available, the total number of people in India consuming less than 20 rupees (or 50 cents) a day was 836 million -- or 77 per cent of the population.
...
They live in slums, are expected to be available to work around the clock, and are denied access to the ration cards that would allow them to buy subsidized food from what remains of the country's public distribution system. And although they are everywhere -- huddled in tents erected on pavements and under flyovers in Delhi; at marketplaces in Calcutta, where they sit with cloth bags of tools ready for a contractor to hire them for the day; gathered around fires made from rags and newspapers in the town of Imphal, near Burma; and at train stations everywhere as they struggle to make their way into the 'unreserved' compartments offering human beings as much room as cattle trucks taking their passengers to the slaughterhouse -- they are invisible in the sense that they seem to count for nothing at all.

If you are interested in India, this is a great book, but prepare to do some serious soul-searching about the future of the human race as you read it.

It's amazing how much my mood improves ...

2012-01-05T07:12:00.000-08:00

... as the days start to get longer, and it's no longer dark when I get off work, and it's no longer dark when I am walking the dog in the morning. Yay for longer days!

Now if only the rains would start; the rainy season is already overdue by a month...

The language of stamps

2012-01-04T16:16:00.000-08:00

Here's a great article about the various subtexts and hidden meanings that have, over the years, been conveyed through the selection and positioning of postage stamps on postcards and envelopes.

I'm not sure I completely believe all the various claims that the article makes, but it's well-researched and well-documented and very fun to read.

The custom of the language of stamps reached different ages in different countries. In Russia, where it was a great fashion, no such postcard was published after the revolution, just as in the socialist countries after 1945. On the one hand, etiquette itself was considered a bourgeois left-over, and on the other hand the power did not tolerate any encoded message either. In western European countries, however, we find its instances as long as the end of the sixties.

Woe unto those who thoughtlessly allowed their eight-year-old granddaughter to affix the stamp to the envelope, thus unintentionally sending the message: "I have discovered your deceit", when they of course meant to convey: "Many thanks for your kindness"!

A few links on the teaching of programming

2012-01-04T07:23:00.000-08:00

As I (slowly) emerge from my holiday stupor, another short post on a collection of related topics:

CodeAcademy, an interesting new startup backed by Fred Wilson's Union Square Ventures, is seeing impressive signups for their new online programming course: Code Year. For the time being, I'm not doing much web programming, so I didn't sign up, but if you did, let me know what you think.
Meanwhile, Christian Heilmann is concerned about the quality of introductory programming language instruction, and writes a detailed critique of one such beginners's tutorial: Teach Them How To Hit The Ground Running And Faceplant At The Same Time?. Heilmann follows up his critique with another posting on his personal blog a few days later: Beginner tutorials who don’t help beginners?. In the two articles, Heilmann makes several points, but the ones which stuck with me were: (a) worry more about teaching people than about making them happy, and (b) don't take short-cuts at the start which will need to be un-learned soon afterwards:
“Quick tutorials for beginners” are killing our craft. Instead of pointing to existing documentation and keeping it up to date (in the case of the wiki-based docs out there) every new developer turned to an author wanting the fame for themselves. And a lot of online magazines cater to these to achieve “new” content and thus visitors. We measure our success by the number of hits, the traffic, the comments and retweets. And to get all of that, we want to become known as someone who wrote that “very simple article that allowed me to do that complex thing in a matter of minutes”.
As Heilmann points out, Smashing Magazine is generally not known for these sorts of things, and is usually a great source of quality educational material. It's pleasing to see that the Smashing Magazine editors are hosting this discussion on their site, clearly they agree that it's an important problem.
Lastly, the amazing John Resig, author of the JQuery Javascript toolkit, writes about his work at Khan Academy on using Javascript as a language for teaching programmers: JavaScript as a First Language. Resig enumerates a number of reasons why he feels that Javascript is appropriate for use as a first programming language, and notes the importance of teaching good habits from the start, but also observes that full instruction will require more than just a single language:
It should be noted that while we're starting with JavaScript as a first language - largely due to its ubiquity, desirability in the larger workforce, lack of prior installation requirements, and ability to create something that's easy to share with friends - we're not going to be myopic and only focus on JavaScript. There are so many things that can be learned from other languages, not to mention entire skillsets that aren't terribly relevant to in-browser JavaScript, that it behooves us to try and bring as much of them into our curriculum as possible.

I don't know what the future will bring, but I am extremely confident that it will bring a need for more programmers, and for more people who at least understand programming, even if they don't practice it full-time themselves.

So it's great to see the field continuing to discuss, take responsibility for, and invest in the teaching and training of new programmers.

Update: "employees have been known to sign up for courses on their own". Horrors!

Beyond the Uncanny Valley indeed!

2012-01-03T15:51:00.000-08:00

Have you seen the new Tintin movie yet?

Kevin Kelly is absolutely correct: the best movie-makers have now moved solidly Beyond the Uncanny Valley.

I saw the movie in 3D and loved it, but it clearly would have been just as good, and possibly better, in 2D.

I found many parts to love about the movie (I am a huge Herge's Adventures of Tintin fan!), but to pick one particularly special part, I love the scene early in the movie where Tintin is getting his picture painted by a street artist in the market, and as the portrait is completed we see that the picture is the exact image of Tintin from Herge's original books, and then the camera pulls away to show a wall where dozens of street portraits have been pinned up, each one of them a character from the books, done in the original Herge style.

It's just an elegant and graceful nod to the fact that the Spielberg/Jackson team are aware that yes, they have made a hyper-realistic movie of a comic book.

Oh stop reading my blog already and go see the movie! (But go read the book, first!)

Mapping out the new year

2012-01-02T08:21:00.000-08:00

Here are four great map articles to start off your New Year:

In Slate Magazine, Seth Stevenson's article is an homage to a gorgeous new wall map of the U.S.A.: The Greatest Paper Map of the United States You’ll Ever See.
the big mapmaking corporations of the world employ type-positioning software, placing their map labels (names of cities, rivers, etc.) according to an algorithm. For example, preferred placement for city labels is generally to the upper right of the dot that indicates location. But if this spot is already occupied—by the label for a river, say, or by a state boundary line—the city label might be shifted over a few millimeters. Sometimes a town might get deleted entirely in favor of a highway shield or a time zone marker.
...
But Imus—a 35-year veteran of cartography who’s designed every kind of map for every kind of client—did it all by himself. He used a computer (not a pencil and paper), but absolutely nothing was left to computer-assisted happenstance. Imus spent eons tweaking label positions. Slaving over font types, kerning, letter thicknesses. Scrutinizing levels of blackness.
In Cartastrophe, Daniel Huffman offers his critique of Rick Aschmann's complex and sophisticated dialect map: North American English Dialects, Based on Pronunciation Patterns. Huffman's conclusion:
The more complexity you can show, the richer the story and the more versatile the product. The map quickly begins to be more than the sum of its parts. Putting two thematic layers on a map gives you three data sets — one each for the layers, plus allowing you to visualize the relationship between the two layers. One plus one equals three. But all of this is worthless if it becomes so complex as to be unclear. A map with one clear data set is worth more than a map with fifteen data sets you can’t read.
Over at BldgBlog, Geoff Manaugh offers an intriguing look at the complexities of collecting mapping data on ice floes: Ice Island Infrastructure.
In other words, they want to turn icebergs into floating science research stations, mapping earthquakes at sea.
...
While the authors compare this, briefly, to using buoys—and, thus, this method is not all that different from any other free-floating oceanographic instrumentation system—the transformation of icebergs into scientifically useful platforms is a compelling example of how a natural phenomenon can become infrastructure with even the smallest addition of equipment. The iceberg has literally been instrumentalized: a temporary archipelago, too short-lived to appear on maps, turned into a scientific instrument.
Lastly, since so many of us nowadays think that mapping begins and ends with Google Maps on your smartphone: Galaxy Nexus Power Analysis: Why chargers can't keep up with navigation. (Plus, black screens draw less power than white... sometimes!).
After personally experiencing having my phone end up LESS charged than I started out with on a 1 hour trip with navigation on while plugged in, I decided to run some tests.
The first was to see how much power the Galaxy Nexus could draw from the USB connection. This turns out to be about 4.5W. (0.9A at 5V)
...
Now the real test... Google Maps.
This varied a LOT. Minimum was 4W with some steady peaks of 5.1W!

The Disneyland art of Claude Coats

2011-12-31T09:01:00.000-08:00

I'm just old enough to remember growing up in Orange County (well, Whittier, but it was the same thing), in the early 1970's, just as Disneyland was building out and completing the most important of the signature rides: Pirates of the Caribbean, Submarine Voyage, and of course the Haunted Mansion.

So I loved this fabulous essay about the artist Claude Coats, who started his career in the movie-making side of Walt Disney Studios (Pinocchio, Fantasia), then made the transition to artist and set designer at Disneyland: Long-Forgotten: Claude Coats: The Art of Deception and the Deception of Art.

The essayist makes the point that the skill of the background painter is to create the world that the other artists will fill with music, animation, and story:

You the viewer are invited to imagine yourself on the other side of the frame (the opposite dynamic of Davis). You see yourself as the character in the landscape. Though never intended for public display, those sketches are among the most beautiful and seductive examples of Mansion art. Who wouldn't want go exploring in this?

I love the description of the Disneyland ride as a voyage through a painting:

As you look at some of those Coats backgrounds up above, like Gepetto's cottage and the Sorcerer's Apprentice interiors, you almost wish you could step into them and look around, so inviting are they. With Rainbow Caverns, Coats finally enabled you do just that: ride right through one of his moody, atmospheric paintings.

And of course, this is the basis of the oft-remarked "suspension of disbelief":

The whole drawing depicts a dissolve between there and here, inside and outside, human artifice and wild nature. This is not an exit point for characters stepping over into our presence; this is a place that invites you to enter.

The essay is filled with gorgeous sketches and paintings, so go have a look!

Prince of Persia: still alive after 25 years!

2011-12-28T16:47:00.000-08:00

I've been having fun looking through the notes put together by "mrsid", a programmer who took up the challenge of re-implementing the classic Apple II game "Prince of Persia", by reverse-engineering a running copy of the game, while simultaneously reading Jordan Mechner's original diary and design notes:

In the meantime I found Jordan Mechner's blog. He had the courage and insight to post all of his old journals from the 1980s. He meticulously kept a log of his daily work. What a great read that was. Just a few days before I started looking for Prince of Persia information Jordan also posted this article on his blog. It contained a link to a PDF, which turned out to be the Prince of Persia source code documentation.
I was amazed. The source was lost on Apple II disks, but the document written just a few days after the release in 1989 was there, with all kinds of juicy little details about the graphics engine, the data structures, lists of images, and more. It was like someone had handed me the key to a long lost treasure.

Mrsid presents the notes as follows:

Hopefully there will be more essays posted in the future, as the complete notes are not yet available.

In the meantime, though, it's great fun to read Jordan's original notes, as well as mrsid's reverse engineering analysis, and it's also quite cool to see the discussion back-and-forth between the two of them in the comments on the blog.

I still vividly remember my eldest daughter playing Prince of Persia in the early 1990's on our fresh new Mac IIsi -- my how she loved that game, and how we loved that computer!

It's not just a game ...

2011-12-26T07:45:00.000-08:00

... it's a new way to celebrate the holidays.

Sometimes I think I understand computers ... sometimes not

2011-12-24T18:54:00.000-08:00

I spent 4 hours trying to set things up so that my Windows 7 laptop could print to a USB-attached printer on my Ubuntu Linux desktop.

Most of it went pretty easily: ensure that CUPS and Samba were installed and configured on the Linux machine, and verify that the Samba configuration allowed printer sharing.

But then, no amount of fiddling with the Add Printer wizard on the Windows 7 machine was finding success.

Finally, this weird sequence worked:

Choose Add Printer
Choose Add a local printer. Ignore all the warnings about how you should only do this if you have a locally-attached, non-USB cabled printer. :)
Choose Create a new port.
Choose Local Port. Click Next.
When prompted to enter a port name, type in \\computername\printername

It is so weird that in order to print to a printer on another machine, you have to (a) tell Windows to define a locally-attached printer, and then stuff a remote machine network address into the 'local port' field.

But hey, it worked...

Great investigation of a Google synonym query

2011-12-24T16:57:00.000-08:00

This in-depth exploration of an unexpected Google query result is fascinating.

But that’s the thing, what seems easy and straightforward to us is actually quite difficult for a machine.

Indeed.

DVCS and change authenticity

2011-12-24T08:45:00.000-08:00

In the world of version control, distributed version control systems such as Git and Mercurial are all the rage.

These systems are indeed extremely powerful, but they all suffer from a fundamental issue, which is how the various nodes in the distributed system can establish the necessary trust to verify authenticity of push and pull requests.

(Disclosure: at my day job, we make a version control system, which has a centralized architecture and a wholly different trust and authentication mechanism. So I'm more than just an interested observer here.)

Now, this issue has been known and discussed for quite some time, but it has acquired greater urgency this fall after a fairly significant compromise of the main Linux kernel systems. As Jonathan Corbet notes in that article

We are past the time where kernel developers are all able to identify each other. Locking down kernel.org to the inner core of the development community would not be a good thing; the site is there for the whole community. That means there needs to be a way to deal with mundane issues like lost credentials without actually knowing the people involved.

The emerging proposal to deal with this problem includes several new features in Git:

Signed commits
and Pulling Signed Tags,
both of which are now operational in the development mainline of the Git trunk.

I suspect that this problem is a deep and hard and fundamental one. It seems to me that the DVCS infrastructure is building a fairly complex mechanism: here's how Linus will use this technology to ensure the integrity of the Linux kernel, as described by Junio Hamano (the lead Git developer):

To make the whole merge fabric more trustworthy, the integration made by his lieutenants by pulling from their sub-lieutenants need to be made verifyable the same way, which would (1) make the number of signed tags even larger and (2) make it more likely somebody in the foodchain gets lazy and refuses to push out the signed tags after he or she used them for their own verification.

But reading this description, I'm instantly reminded of a very relevant observation made by Moxie Marlinspike in the context of the near-complete-collapse of the SSL Certificate Authority chain of trust this spring:

Unfortunately the DNSSEC trust relationships depend on sketchy organizations and governments, just like the current CA system.
Worse, far from providing increased trust agility, DNSSEC-based systems actually provide reduced trust agility. As unrealistic as it might be, I or a browser vendor do at least have the option of removing VeriSign from the trusted CA database, even if it would break authenticity with some large percentage of sites. With DNSSEC, there is no action that I or a browser vendor could take which would change the fact that VeriSign controls the .com TLD.
If we sign up to trust these people, we're expecting them to willfully behave forever, without any incentives at all to keep them from misbehaving. The closer you look at this process, the more reminiscent it becomes. Sites create certificates, those certificates are signed by some marginal third party, and then clients have to accept those signatures without ever having the option to choose or revise who we trust. Sound familiar?

I'm not saying I have the answer; indeed, the very smartest programmers on the planet are struggling intensely with this problem. It's a very hard problem. As the researchers at the EFF recently noted:

As currently implemented, the Web's security protocols may be good enough to protect against attackers with limited time and motivation, but they are inadequate for a world in which geopolitical and business contests are increasingly being played out through attacks against the security of computer systems.

Returning to the world of DVCS systems, for a moment, I've just felt, all along, that the fundamental weakness of DVCS systems was going to turn out to be their weak authenticity guarantees; indeed, this is the core reason that organizations like Apache have been very reluctant to open their infrastructure up to DVCS-style source control, even given all its other advantages.

And it seems like the people who are trying to repair the Certificate Authority technology are also skeptical that a 100% distributed solution can be effective; as Adam Langley says:

We are also sacrificing decentralisation to make things easy on the server. As I've previously argued, decentralisation isn't all it's cracked up to be in most cases because 99.99% of people will never change any default settings, so we haven't given up much. Our design does imply a central set of trusted logs which is universally agreed. This saves the server from possibly having to fetch additional audit proofs at runtime, something which requires server code changes and possible network changes.

And the EFF's Sovereign Keys proposal has a similar semi-centralization aspect:

Master copies of the append-only data structure are kept on machines called "timeline servers". There is a small number, around 10-20, of these. The level of trust that must be placed in them is very low, because the Sovereign Key protocol is able to cryptographically verify the important functions they perform. Sovereign Keys are preserved so long as at least one server has remained good. For scalability, verification, and privacy purposes, lots of copies of the entire append-only timeline structure are stored on machines called "mirrors".

With the new Git technology, as I understand it, the user who accepts a pull request from a remote repository now faces a new challenge:

The integrator will see the following in the editor when recording such a merge:
The one-liner merge title (e.g 'Merge tag rusty-for-linus of git://.../rusty.git/');
The message in the tag object (either annotated or signed). This is where the contributor tells the integrator what the purpose of the work contained in the history is, and helps the integrator describe the merge better;
The output of GPG verification of the signed tag object being merged. This is primarily to help the integrator validate the tag before he or she concludes the pull by making a commit, and is prefixed by '#', so that it will be stripped away when the message is actually recorded; and
The usual "merge summary log", if 'merge.log' is enabled.

This will be a challenging task to require of all developers in this chain of trust. Is it feasible? One thing for sure, the Git team are to be commended for facing this problem head on, for openly discussing it, and for trying to push the problem forward. It is exciting to watch them struggle with the issues, and I've learned an immense amount from reading their discussions.

So I think it will be very interesting to see how the Git team fares with this problem, as they, too, have some wonderfully talented people at work on the problems.

Holiday weekend link dump

2011-12-23T17:21:00.000-08:00

As always, apologies in advance for the link dump. It's just been so busy recently, that I haven't found the time to explore things in detail.

Still, if you're looking for a few holiday-weekend things to read, try these:

If you haven't been paying much attention to the Carrier IQ controversy, Rich Kulawiec over at TechDirt has a great summary of what's been going on, with an amazing number of links to chase and study. As Kulawiec puts it:
Debate continues about whether Carrier's IQ is a rootkit and/or spyware. Some have observed that if it's a rootkit, it's a rather poorly-concealed one. But it's been made unkillable, and it harvests keystrokes -- two properties most often associated with malicious software. And there's no question that Carrier IQ really did attempt to suppress Eckhart's publication of his findings.
But even if we grant, for the purpose of argument, that it's not a rootkit and not spyware, it still has an impact on the aggregate system security of the phone: it provides a good deal of pre-existing functionality that any attacker can leverage. In other words, intruding malware doesn't need to implement the vast array of functions that Carrier IQ already has; it just has to activate and tap into them.
Many of us may be taking some holiday break, but the busy beavers at CalTrans are embarking on the final major step of the Bay Bride reconstruction: threading the suspension cable over the top of the main bridge tower.
The cable is 2.6 feet in diameter and nearly a mile long. It weighs 5,291 tons, or nearly 10.6 million pounds, and is made up of 137 steel strands, each one composed of 127 steel wires.
The strands will go up to the top of the center tower and down to the San Francisco side of the span, where they will be looped underneath the deck of the bridge, then threaded back up to the tower and back down to the Oakland side of the bridge. There, crews will anchor the other end of the strands.
I found it interesting to read about how they finish the cable installation:
Ney said it will take a few months to complete the installation. Once all the strands are installed, crews will bind them together and coat them with zinc paste.
I'm familiar with the notion of sacrificial zinc anodes in sailboats, where the zinc is used to avoid destruction of a more valuable metal part (such as your stainless steel propellor). Is the zinc paste on the cable used for the same purpose?
The bubble is back! Everywhere you look, there is article after article after article after article about the desparate competition for software engineers that's underway right now.
Of course, at least part of the problem is that it's still the case that, all too frequently, people who think they can program, actually can't. Contrary to many people, I'm not in a hurry to blame our education system for this. I think programming is very hard, and it doesn't surprise me that there's a high failure rate. Some would say that anybody can learn to program, but I think there's a real underlying talent at issue, and just like I would be a lousy lawyer or a lousy surgeon or a lousy soprano, some people will have more aptitude for writing software than others.
Anyway, I can attest to a fair amount of the insanity, though happily I'm pretty well insulated from it. But we have clearly entered into a very exciting new time in software, with a variety of hot technologies, such as cloud computing and mobile applications, providing the fuel. Though I think Marc Andreessen may be a bit too giddy about the prospects for it all, I have to agree with his assessment that:
Six decades into the computer revolution, four decades since the invention of the microprocessor, and two decades into the rise of the modern Internet, all of the technology required to transform industries through software finally works and can be widely delivered at global scale.
A large part of that transformational technology is, once again, being driven by Amazon. As they continue releasing new features at breakneck speed, Wired magazine takes a step back and wonders: what does it mean that Amazon Builds World’s Fastest Nonexistent Supercomputer. I remember my first introduction to system virtualization, when I got to use IBM's VM software back in the early 1980's; it definitely takes a while to get your head around what's really going on here!
Moving on to something entirely unrelated to software, in the world of professional football, the dispute between Uruguayan superstar Luis Suarez and French superstar Patrice Evra has been gathering a lot of attention. My friend Andrew has reprinted a well-phrased essay on the topic, which is well worth reading.
Professional sports is of course mostly entertainment, yet somehow it is more than that; it is undeniably one of the largest parts of modern life. On that note, a recent issue of The New Yorker carries a fine review of the life of Howard Cosell, who played a major part in the development of professional sports into one of America's major passions.
On Monday nights, Cosell called out players for their mistakes with orotund rhetoric and moral high dudgeon. Just as music fans used to go to the New York Philharmonic to watch Leonard Berstein's gymnastics more than to hear, yet again, Beethoven's Fifth, people tuned in to hear -- and howl at -- Cosell. Even if you loathed him, his performance was what made Monday nights memorable.
And lest you think that this is just something minor, don't miss this wonderful article in The Economist: Little red card: Why China Fails at Football..
Solving the riddle of why Chinese football is so awful becomes, then, a subversive inquiry. It involves unravelling much of what might be wrong with China and its politics. Every Chinese citizen who cares about football participates in this subversion, each with some theory—blaming the schools, the scarcity of pitches, the state’s emphasis on individual over team sport, its ruthless treatment of athletes, the one-child policy, bribery and the corrosive influence of gambling. Most lead back to the same conclusion: the root cause is the system.
Is it sports, or is it life? It's definitely not "just" entertainment.
OK, getting back to things I understand better, it's been quite the year for Mozilla and Firefox. Ever since they changed their release process and their version numbering back in the spring, it's been a continual stream of Firefox releases, so it's no suprise that Firefox 11 is soon to be available, with yet more features and functionality. But the Firefox team are pushing beyond just making great browsers, branching into areas like web-centric operating systems, Internet identity management, and building entire applications in the browser, as David Ascher explains. Ascher's article notes that within the Mozilla Foundation, people are now thinking significantly "beyond the browser":
we’re now at a distinct point in the evolution of the web, and Mozilla has appropriately looked around, and broadened its reach. In particular, the browser isn’t the only strategic front in the struggle to promote and maintain people’s sovereignty over their online lives. There are now at least three other fronts where Mozilla is making significant investments of time, energy, passion, sweat & tears. They’re still in their infancy, but they’re important to understand if you want to understand Mozilla
Meanwhile, how is Mozilla handling this? Aren't they just a few open source hackers? How can they do all this? Well, as Kara Swisher points out, Mozilla has some pretty substantial financial backing:
Mozilla is set to announce that it has signed a new three-year agreement for Google to be the default search option in its Firefox browser.
It’s a critical renewal for the Silicon Valley software maker, since its earlier deal with the search giant has been a major source of revenue to date.
Meanwhile, what is all this doing to the life of the ordinary web developer? As Christian Heilmann observes, it brings not just excitement, but also stress and discomfort, but underlying this is the fact that the web is no longer just a place for experimentation, but has transitioned into being the production platform of our daily lives:
We thought we are on a good track there. Our jobs were much more defined, we got more respect in the market and were recognised as a profession. Before we started showing a structured approach and measurable successes with web technologies we were just “designers” or “HTML monkeys”.
Are you just feeling flat-out overwhelmed by all this new technology? Well, one wonderful thing is that the web also provides the technology to stay up to date:
MIT President Susan Hockfield said, “MIT has long believed that anyone in the world with the motivation and ability to engage MIT coursework should have the opportunity to attain the best MIT-based educational experience that Internet technology enables. OpenCourseWare’s great success signals high demand for MIT’s course content and propels us to advance beyond making content available. MIT now aspires to develop new approaches to online teaching.”
Now, if I can just find that free time that I misplaced...
Just because software is open source, it can still fade away into the sunset. Which is a shame, because I was really hoping to get a distribution with Issue 46 fixed, because I hit it all the time! Yes, yes, I know, I should just download the source and build it. Or find a new debugger. Or something.
Forgive the breathless style, and read the well-written summary of the Buckshot Yankee incident at the Washington Post: Cyber-intruder sparks massive federal response — and debate over dealing with threats. As author Ellen Nakashima observes, we're still struggling with what we mean when we toss about terms like "cyber war" and the new Cyber Command unit:
“Cyber Command and [Strategic Command] were asking for way too much authority” by seeking permission to take “unilateral action . . . inside the United States,” said Gen. James E. Cartwright Jr., who retired as vice chairman of the Joint Chiefs in August.
Officials also debated how aggressive military commanders can be in defending their computer systems.
“You have the right of self-defense, but you don’t know how far you can carry it and under what circumstances, and in what places,” Cartwright said. “So for a commander who’s out there in a very ambiguous world looking for guidance, if somebody attacks them, are they supposed to run? Can they respond?”
Finally (since something has to go last), Brad Feld has ended the year by winding down the story of Dick and Jane's SayAhh startup in a most surprising fashion (well, to me, at least): SayAhh Has Shut Down.
If you weren't following the SayAhh series, Feld had been writing a series of articles about a hypothetical software startup, using them to illustrate many of the perils and complexities that can arise when trying to build a new company from scratch. I'm not sure if there's a clean index to all the articles he wrote, but you can start here for the first article, and then mostly follow along via his blog. I think it's great that he ended the series in such a realistic fashion, though I'm quite interested to see how his readership feels about that!

I hope your holidays are enjoyable, safe, and filled with family and friends.

I found a new Derby bug!

2011-12-22T22:12:00.000-08:00

It doesn't happen very often that I find a bug in Derby, so it's worth noting: https://issues.apache.org/jira/browse/DERBY-5554.

I'm just enough disconnected from day-to-day Derby development at this point to not immediately understand what the bug is.

Note that the crash is in a generated method:


Caused by: java.lang.NullPointerException
at org.apache.derby.exe.acf81e0010x0134x6972x0511x0000033820000.g0(Unknown Source)

Derby uses generated methods in its query execution for tasks such as projection and restriction of queries.

So, for example, the generated method above is probably implementing part of the "where" clause of the query that causes the crash.

Derby generated methods are constructed at runtime, by dynamically emitting java bytecodes into a classfile format, and then dynamically loading that class into the Java runtime. It's quite clever, but quite tricky to diagnose, because it's hard to see the actual Java code that is being run.

A long time ago, I tracked down some debugging tips for working on crashes in generated code, and collected them here: http://wiki.apache.org/db-derby/DumpClassFile.

It's late, and I'm tired (and suffering from a head cold), but if I get some time over the holiday weekend I'll try to look into this crash some more.

Of course, perhaps somebody like Knut Anders or Rick will have already figured the problem out by then :)

Teaching yourself to become a thinker

2011-12-21T16:24:00.000-08:00

I finally got around to reading Bill Deresiewicz's fascinating lecture: Solitude and Leadership. It's over two years old now, but it has aged very well, so if you haven't yet seen it, I encourage you to wander over and give it a read.

Although Deresiewicz spends much of the lecture talking about leadership, that wasn't my favorite part of his talk. Rather, I particularly enjoyed his analysis of thinking. He proposes that modern civilization isn't doing enough to develop a culture of thinkers:

What we don’t have, in other words, are thinkers. People who can think for themselves. People who can formulate a new direction: for the country, for a corporation or a college, for the Army—a new way of doing things, a new way of looking at things.

What does Deresiewicz mean by a "thinker"? He gives as an example (he is speaking to a West Point audience in 2009, after all) General David Petraeus:

He has a Ph.D. from Princeton, but what makes him a thinker is not that he has a Ph.D. or that he went to Princeton or even that he taught at West Point. I can assure you from personal experience that there are a lot of highly educated people who don’t know how to think at all.
No, what makes him a thinker—and a leader—is precisely that he is able to think things through for himself. And because he can, he has the confidence, the courage, to argue for his ideas even when they aren’t popular. Even when they don’t please his superiors. Courage: there is physical courage, which you all possess in abundance, and then there is another kind of courage, moral courage, the courage to stand up for what you believe.

Admiring General Petraeus for his mental and moral courage is fair, but even more interesting to me is Deresiewicz's observation on what (I think) is the more important aspect of being a thinker: creativity and originality:

Thinking means concentrating on one thing long enough to develop an idea about it. Not learning other people’s ideas, or memorizing a body of information, however much those may sometimes be useful. Developing your own ideas. In short, thinking for yourself. You simply cannot do that in bursts of 20 seconds at a time, constantly interrupted by Facebook messages or Twitter tweets, or fiddling with your iPod, or watching something on YouTube.
I find for myself that my first thought is never my best thought. My first thought is always someone else’s; it’s always what I’ve already heard about the subject, always the conventional wisdom. It’s only by concentrating, sticking to the question, being patient, letting all the parts of my mind come into play, that I arrive at an original idea. By giving my brain a chance to make associations, draw connections, take me by surprise. And often even that idea doesn’t turn out to be very good. I need time to think about it, too, to make mistakes and recognize them, to make false starts and correct them, to outlast my impulses, to defeat my desire to declare the job done and move on to the next thing.

It's a wonderful observation, and it's so very, very true. Creativity, originality, and inspiration require patience, reflection, and concentration.

It brings to mind a wonderful lesson that my dear friend Neil Goodman taught me over twenty years ago, when I was still just learning to program and I was trying to understand the way that Neil approached a problem.

I was asking him how he knew when he was done with the design phase of his project, and ready to move on to the coding phase. Neil replied with an answer that was very evocative of Deresiewicz's advice to "outlast your impulses". Neil said (as best I remember):

Work on your design. At some point, you will think you are done, and you are ready, but you are not. You have to resist that feeling, and work on your design some more, and you will find more ways to improve it. Ask others to review it; re-read and re-consider it yourself. Again, you will think you are done, and you are ready, but you are not. You must resist, resist, resist! Force yourself to continue iterating on your design, paying attention to every part of it, over and over. Even if you feel that you can't possibly improve it any more, still you must return to it. Only then, will you reach the point when you are ready to write code.

Of course, for the full effect, you have to have Neil himself (in his earnest, impassioned, gentle-giant sort of way) deliver the message, but hopefully the point comes through.

Incidentally (perhaps Deresiewicz didn't get to pick his own title?), I don't think that solitude really captures the idea properly. Or maybe solitude is the right thing in a military context, but in the software engineering field, where I spend all my time, I don't think that solitude is either necessary nor useful for original, creative thinking. You need to get feedback and reactions from others, and you can't do that without communicating, and without listening. But you do need to exercise a number of activities which are certainly related to solitude: contemplation, reflection, consideration, etc. So if I had the chance to re-title his essay, I might suggest that he have a title more like "Leadership and the ability to think for yourself."

But that's quite a bit wordier :)

I've wandered quite a bit far afield, but hopefully I've intrigued you enough with Deresiewicz's essay that you'll wander over and give it a read, and maybe (hopefully) you will find it time well spent.

Two command line shell tutorials

2011-12-19T20:08:00.000-08:00

I just wouldn't be me if I didn't notice things like these and want to immediately post them to my blog...

Now get out there and open those terminal sessions!

Did Iran capture the drone by hacking its GPS?

2011-12-16T11:17:00.000-08:00

There's a fascinating report in today's Christian Science Monitor speculating that Iran used vulnerabilities in the drone's GPS technology to simply convince the drone to land in Iran:

"GPS signals are weak and can be easily outpunched [overridden] by poorly controlled signals from television towers, devices such as laptops and MP3 players, or even mobile satellite services," Andrew Dempster, a professor from the University of New South Wales School of Surveying and Spatial Information Systems, told a March conference on GPS vulnerability in Australia.
"This is not only a significant hazard for military, industrial, and civilian transport and communication systems, but criminals have worked out how they can jam GPS," he says.
The US military has sought for years to fortify or find alternatives to the GPS system of satellites, which are used for both military and civilian purposes. In 2003, a “Vulnerability Assessment Team” at Los Alamos National Laboratory published research explaining how weak GPS signals were easily overwhelmed with a stronger local signal.
“A more pernicious attack involves feeding the GPS receiver fake GPS signals so that it believes it is located somewhere in space and time that it is not,” reads the Los Alamos report. “In a sophisticated spoofing attack, the adversary would send a false signal reporting the moving target’s true position and then gradually walk the target to a false position.”

Here's the link to the ten-year-old Los Alamos National Laborary report: GPS Spoofing Countermeasures, which in turn has references to a number of other references to read.

Very interesting stuff.

Given that location-based devices have become so prevalent in our lives (smartphones, cars, etc.), it's interesting to contemplate how we might improve the reliability and trustworthiness of the location awareness of our automated assistants. The guys over at SpiderLabs Anterior had some great articles on this recently:

Bowl Season is here!

2011-12-15T17:23:00.000-08:00

College Football Bowl Game Season is upon us, filled with memorable events such as "The Famous Idaho Potato Bowl".

With 35 bowl games, you need to know what to watch, and what to miss.

So get prepared, and head on over to Yahoo! Sports's bowl game preview: Dashing through the bowls … and coaching moves.

You'll find useful advice such as this:

Fight Hunger Bowl (25)
Dec. 31
Illinois vs. UCLA
Who has momentum? This is the bowl where momentum goes to die. Bruins have lost three of their past four but still look great compared to an Illinois team that has lost six in a row.
Who has motivation? Tail-spinning teams playing for interim coaches make this a potential low-intensity debacle. But both have hired new coaches who presumably will be watching to see who wants to make an impression heading into 2012.
Who wins a mascot fight? Joe Bruin by default, since Chief Illiniwek was forcibly retired in 2007 and not replaced.
Dash fact: Illinois hasn’t scored more than 17 points in a game since Oct. 8.
Dash pick: UCLA 14, Illinois 10. It’s New Year’s Eve. Find something better to do with your time.

Intriguingly, the two best bowl games of the year look to involve Bay Area teams:

The Fiesta Bowl, on Jan 2, features Stanford and Oklahoma State
The Holiday Bowl, on Dec 28, features Texas vs. California

Both games should be well-matched, well-played, and quite fun to watch.

The bubble is back!

2011-12-15T14:40:00.000-08:00

Wow! One billion dollars is a lot of money.

Will they use some of that to start addressing the widely-reported discord?

South America joins the Amazon cloud

2011-12-15T09:43:00.000-08:00

Amazon Web Services is growing at an incredible rate. They just opened their eighth AWS region, in Sao Paolo, Brazil.

Here's a nifty interactive map of the AWS global infrastructure, so you can see the various regions and their information.

The U.S. West (Northern California) region is actually located just a few miles from my house. (I think that the city planners ran out of imagination, as it's at the corner of Investment Boulevard and Corporate Ave, not far from the Industrial Boulevard exit on the freeway.)

It's not much to look at, as a physical building; it's what's inside that counts!

Delightfully illustrated HTTP status codes

2011-12-15T06:55:00.000-08:00

The HyperText Transfer Protocol (HTTP) indicates a lot of information in the "status code". Everyone is of course familiar with codes 404 ("Not found"), 200 ("OK"), and 500 ("Internal server error"), but there are dozens more codes, each with their own precise meaning.

If you're having trouble remembering which code is which, or don't understand a particular code and would like a more "vivid" example, head over to Flickr, for this delightful set of HTTP Status Codes illustrated by cats.

This is definitely the geekiest humor of the year!

I particularly enjoy

but you should really view the entire set for the full effect (apologize to your co-workers in advance before laughing out loud!).

Raymond Chen skewers the executive re-org email

2011-12-14T07:26:00.000-08:00

Delightful!. Raymond Chen perfectly captures the classic executive reorg email.

The bit about auto-summarize at the end is pretty insightful, as well.

Looking for something to read?

2011-12-13T15:03:00.000-08:00

It's winter time, you're stuck indoors, how about something to read?

Well, you're in luck: both Longform and GiveMeSomethingToRead are just out with their end-of-the-year lists:

I like both lists; there's lots of interesting material here.

There's a fair amount of overlap, naturally, but also each list has a few gems that the other list missed.

FWIW, my favorite out of both lists is My Summer at an Indian Call Center, from Mother Jones Magazine.

Update: Here's another set of lists!

The Popular Mechanics article on Flight 447 is enthralling

2011-12-09T21:16:00.001-08:00

If you haven't yet had a chance to go read the Popular Mechanics article about Air France flight 447 then stop what you are doing right now, go sit down for 5 minutes, and read the article.

It is chock full of insight after insight after insight. Here's just one, picked almost at random:

While Bonin's behavior is irrational, it is not inexplicable. Intense psychological stress tends to shut down the part of the brain responsible for innovative, creative thought. Instead, we tend to revert to the familiar and the well-rehearsed. Though pilots are required to practice hand-flying their aircraft during all phases of flight as part of recurrent training, in their daily routine they do most of their hand-flying at low altitude—while taking off, landing, and maneuvering. It's not surprising, then, that amid the frightening disorientation of the thunderstorm, Bonin reverted to flying the plane as if it had been close to the ground, even though this response was totally ill-suited to the situation.

Although, as the article points out, there were multiple technical issues in play (weather, location, time of day, fatigue, etc.), in the end it boils down to human factors: training, experience, and, most of all, communication:

The men are utterly failing to engage in an important process known as crew resource management, or CRM. They are failing, essentially, to cooperate. It is not clear to either one of them who is responsible for what, and who is doing what.

Every sentence in this article is a nugget, with observations about human behavior, suggestions of areas for study and improvement, and, in the end, a realization that people are flawed and make mistakes, and what we need to do most of all is to think, talk, and help each other:

when trouble suddenly springs up and the computer decides that it can no longer cope—on a dark night, perhaps, in turbulence, far from land—the humans might find themselves with a very incomplete notion of what's going on. They'll wonder: What instruments are reliable, and which can't be trusted? What's the most pressing threat? What's going on?

Don't miss this incredible recap of the tragedy of Flight 447.

Following up

2011-12-08T13:11:00.001-08:00

I love follow-up. Follow-up is good: study something, then study the follow-ups, and you will learn more.

So, a few follow-ups:

I've been fascinated by the Air France Flight 447 investigation (background here and here), so if you were equally interested, don't miss this wonderful article in this month's Popular Mechanics: What Really Happened Aboard Air France 447:
Two years after the Airbus 330 plunged into the Atlantic Ocean, Air France 447's flight-data recorders finally turned up. The revelations from the pilot transcript paint a surprising picture of chaos in the cockpit, and confusion between the pilots that led to the crash.
I've been following Jim Gettys and his studies into the network queueing phenomenon known as "BufferBloat". If you've been paying attention to this, too, you won't want to miss this discussion in the ACM Queue column. Meanwhile, Patrick McManus, who is hard at work on the new SPDY networking protocol, has an essay on the topic in which he notes some recent published research, and worries that there is still more research needed:
A classic HTTP/1.x flow is pretty short - giving it a signal to backoff doesn't save you much - it has sent much of what it needs to send already anyhow. Unless you drop almost all of that flow from your buffers you haven't achieved much. Further, a loss event has a high chance of damaging the flow more seriously than you intended - dropping a SYN or the last packet of the data train is a packet that will have very slow retry timers, and short flows are comprised of high percentages of these kinds of packets.
Understanding TCP's behaviors is certainly complicated; I recently wrote about this at some length on the Perforce blog.

Problems such as the complexity of modern systems such as those controlling airplanes, nuclear reactors, etc., or the unexpected inter-actions of networking equipment across the planet, continue to be extremely hard. Only dedicated study of many years or decades is going to bring progress, so I'm pleased to note such progress when it occurs!

The astonishing sophistication of ATM skimmer criminals

2011-12-07T15:23:00.001-08:00

On his superb blog, Brian Krebs has just posted the latest entry in his astonishing series of investigative articles reporting on modern ATM card skimming criminals, and the devices they use to capture ATM data from compromised ATM machines: Pro Grade (3D Printer-Made?) ATM Skimmer.

Looking at the backside of the device shows shows the true geek factor of this ATM skimmer. The fraudster who built this appears to have cannibalized parts from a video camera or perhaps a smartphone (possibly to enable the transmission of PIN entry video and stolen card data to the fraudster wirelessly via SMS or Bluetooth).

Everything Brian Krebs writes is worth reading, but I find these ATM articles to be just gripping. Modern life is so complex!

Really slow news day

2011-12-06T06:46:00.001-08:00

One of the "AP Top Stories" this morning is headlined

Malia Obama, 13, is nearly as tall as her father.

Three practical articles, two practical books

2011-12-05T13:36:00.001-08:00

Here's a nifty ad-hoc collection of some nice practical articles and books to keep you grounded and focused on what really matters:

First, a nice review of Robert C. ("Uncle Bob") Martin's The Clean Coder: 9 things I learned from reading The Clean Coder by Robert C. Martin, on how professional developers conduct themselves . I haven't read the book (yet), but the review makes me interested enough to keep this book on the list for when I'm next above water on my technical reading. An excerpt from Christoffer Pettersson's review:
As a professional developer, you should spend time caring for your profession. Just like in any other profession, practice gives performance, skill and experience.
It is your own responsibility to keep training yourself by reading, practicing and learning - actually anything that helps you grow as a software developer and helps you get on board with the constant industry changes.
Second, a nifty essay by Professor John Regehr about the ins and outs of "testcase reduction", one of those rarely-discussed but ultra-important skills that gets far too little recognition. At my day job, I'm lucky to have a colleague who is just astonishingly good at testcase reduction; he just has the knack. An excerpt from Regehr's essay:
testcase reduction is both an art and a science. The science part is about 98% of the problem and we should be able to automate all of it. Creating a test case from scratch that triggers a given compiler bug is, for now at least, not only an art, but an art that has only a handful of practitioners.
Third, a nice article on Ned Batchelder's blog about "Maintenance Hatches", those special visibility hooks that let humans observe the operation of complex software in some high-level and comprehensible fashion, for diagnosis and support purposes. In my world, these hatches are typically trace logs, which cause the software to emit detailed information about its activity, and can be turned on and off, and ratcheted up to higher or lower levels, as needed. I like Batchelder's terminology, which conjures up the image of opening a cover to the machinery to observe it at work:
On a physical machine, you need to be able to get at the inner workings of the thing to observe it, fiddle with it, and so on. The same is true for your software.
Fourth, an interesting online book about the practicalities of building a modern web application: The Twelve-Factor App. I guess I was sold when I saw the first factor was to ensure you are using a source code control system (may I recommend one?). From the introduction to the book:
This document synthesizes all of our experience and observations on a wide variety of software-as-a-service apps in the wild. It is a triangulation on ideal practices app development, paying particular attention to the dynamics of the organic growth of an app over time, the dynamics of collaboration between developers working on the app’s codebase, and avoiding the cost of software erosion.
Last, but far from least, check out this amazing online book about graphics programming: Learning Modern 3D Graphics Programming by Jason McKesson. From the introduction:
This book is intended to teach you how to be a graphics programmer. It is not aimed at any particular graphics field; it is designed to cover most of the basics of 3D rendering. So if you want to be a game developer, a CAD program designer, do some computer visualization, or any number of things, this book can still be an asset for you.

The Internet is a wonderful place; so much stuff to read and explore!

Three papers on Bloom filters

2011-12-03T09:21:00.001-08:00

If you should find yourself noticing that:

You're interested in these mysterious things called Bloom filters, that seem to be popping up over and over,
and you'd like to learn more about how Bloom filters work, but you're not sure where to start.

then you're roughly in the position I was in a little while ago, and maybe this article will be of interest to you.

After reading a dozen or so Bloom-filter-related papers spanning 40 years of interest in the subject, I've whittled down the list to three papers that I can recommend to anybody looking for a (fairly) quick and (somewhat) painless introduction to the world of Bloom filters:

The first paper is Burton Bloom's original description of the technique that he devised back in the late 1960's for building a new data structure for rapidly computing set membership. The paper is remarkably clear, even though in those days we had not yet settled on standard terminology for describing computer algorithms and their behaviors.

The most striking part of Bloom's work is this description of its behavior:

In addition to the two computational factors, reject time and space (i.e. hash area size), this paper considers a third computational factor, allowable fraction of errors. It will be shown that allowing a small number of test messages to be falsely identified as members of the given set will permit a much smaller hash area to be used without increasing the reject time. In some practical applications, this reduction in hash area size may make the difference between maintaining the hash area in core, where it can be processed quickly, or having to put it on a slow access bulk storage device such as a disk.

This discussion of probabilistic, approximate answers to queries must have been truly startling and disturbing in the 1960's, when computer science was still very young: "falsely identified", "fraction of errors"? Horrors! Even nowadays, computer science has a hard time dealing with the notions of randomness and probabilistic computations, but we're getting better at reasoning about these things. Back then, Bloom felt the need to justify the approach by noting that:

In order to gain substantial reductions in hash area size, without introducing excessive reject times, the error-free performance associated with conventional methods is sacrificed. In application areas where error-free performance is a necessity, these new methods are not applicable.

In addition to suggesting the notion of a probabilistic solution to the set membership problem, the other important section of Bloom's original paper is his description of the Bloom filter's core algorithm:

Method 2 completely gets away from the conventional concept of organizing the hash area into cells. The hash area is considered as N individual addressable bits, with addresses 0 through N - 1. It is assumed that all bits in the hash area are first set to 0. Next, each message in the set to be stored is hash coded into a number of distinct bit addresses, say a1, a2, ..., ad. Finally, all d bits addressed by a1 through ad are set to 1.
To test a new message a sequence of d bit addresses, say a1', a2', ... ad', is generated in the same manner as for storing a message. If all d bits are 1, the new message is accepted. If any of these bits is zero, the message is rejected.

Bloom's work was, rather quietly, adopted in various areas, particularly in database systems. I'll return to that subject in a future posting, but for now, let's spin the clock ahead 25 years, to the mid-1990's, when the second paper, by a team working on Web proxy implementations, brought significant popularity and a new audience to the world of Bloom filters.

The Summary Cache paper describes an intriguing problem: suppose you want to implement a set of independent proxy caches, each one physically separate from the others, and you'd like your implementation to arrange to have a cache miss on one proxy be able to quickly decide if the item is available in the cache of another proxy:

ICP discovers cache hits in other proxies by having the proxy multicast a query message to the neighboring caches whenever a cache miss occurs. Suppose that N proxies [are] configured in a cache mesh. The average cache hit ration is H. The average number of requests received by one cache is R. Each cache needs to handle (N - 1) * (1 - H) * R inquiries from neighboring caches. There are a total [of] N * (N - 1) * (1 - H) * R ICP inquiries. Thus, as the number of proxies increases, both the total communication and the total CPU processing overhead increase quadratically.

How do they solve this problem? Well, it turns out that a Bloom filter is just the right tool for this:

We then propose a new cache sharing protocol called "summary cache." Under this protocol, each proxy keeps a compact summary of the cache directory of every other proxy. When a cache miss occurs, a proxy first probes all the summaries to see if the request might be a cache hit in other proxies, and sends a query messages [sic] only to those proxies whose summaries show promising results. The summaries do not need to be accurate at all times. If a request is not a cache hit when the summary indicates so (a false hit), the penalty is a wasted query message. If the request is a cache hit when the summary indicates otherwise (a false miss), the penalty is a higher miss ratio.

Bloom filters continued to spread in usage, and interesting varieties of Bloom filters started to emerge, such as Counting Bloom Filters, Compressed Bloom Filters, and Invertible Bloom Filters. In particular, Professor Michael Mitzenmacher of Harvard has been collecting, studying, and improving upon our understanding of Bloom filters and their usage for several decades.

About 10 years ago, Mitzenmacher collaborated with Andrei Broder, who is now at Yahoo! Research but was then with Digital Equipment Company Research to write the third paper, which is the best paper of the three papers I mention in this article. (Note that Broder was a co-author of the second paper as well.)

The Network Applications of Bloom Filters paper accomplishes two major tasks:

First, it provides a clear, modern, and complete description and analysis of the core Bloom filter algorithm, including most importantly the necessary mathematics to understand the probabilistic behaviors of the algorithm and how to adjust its various parameters.
Second, it provides a wide-ranging survey of a variety of different areas in which Bloom filters can be used, and have successfully been used.

Most importantly, for people considering future use of Bloom filters, the paper notes that:

The theme unifying these diverse applications is that a Bloom filter offers a succinct way to represent a set or a list of items. There are many places in a network where one might like to keep or send a list, but a complete list requires too much space. A Bloom filter offers a representation that can dramatically reduce space, at the cost of introducing false positives. If false positives do not cause significant problems, the Bloom filter may provide improved performance. We call this the Bloom filter principle, and we repeat is for emphasis below.
The Bloom filter principle: Whenever a list or set is used, and space is at a premium, consider using a Bloom filter if the effect of false positives can be mitigated..

So there you have it: three papers on Bloom filters. There is a lot more to talk about regarding Bloom filters, and hopefully I'll have the time to say more about these fascinating objects in the future. But this should be plenty to get you started.

If you only have time to read one paper on Bloom filters, read Broder and Mitzenmacher's Network Applications of Bloom Filters. If you have more time, also read the Summary Cache paper, and if you have gone that far I'm sure you'll take the time to dig up Bloom's original paper just for completeness (it's only 5 pages long, and once you've read Broder and Mitzenmacher, the original paper is easy to swallow).

The Foundations discussions

2011-12-01T07:07:00.001-08:00

There continues to be an active discussion over Mikeal Rogers's essay about Apache and git, which I wrote about a few days ago.

Here's (some of) what's been going on:

Simon Phipps wrote a widely read essay in Computerworld UK about the notion of an open source foundation, as separate from the open source infrastructure, and relates the story of a project which suffered greatly because it hadn't established itself with the support of a larger entity:
the global library community embraced Koha and made it grow to significant richness. When the time came, the original developers were delighted to join a US company that was keen to invest in - and profit from - Koha. Everything was good until the point when that company decided that, to maximise their profit, they needed to exert more control over the activities of the community.
A detailed article at the Linux Weekly News website provides much more of the details of this story. Phipps's point is that part of these problems arose because the developers of the project didn't engage in the open discussion of the long term management of the project that would have occurred had they hosted their project at one of the established Open Source foundations such as Apache or the Software Freedom Conservancy.
Stephen O'Grady also wrote an essay on the difference between foundations and infrastructure acknowledging that "foundations who reject decentralized version control systems will fall behind", but further asserting that:
GitHub is a center of gravity with respect to development, but it is by design intensely non-prescriptive and inclusive, and thus home to projects of varying degrees of quality, maturity and seriousness.
[ ... ]GitHub, in other words, disavows responsibility for the projects hosted on the site. Foundations, conversely, explicitly assume it, hence their typically strict IP policies. These exclusive models offer a filter to volume inclusive models such as GitHub’s.
[ ... ]If you’re choosing between one project of indeterminate pedigree hosted at GitHub and an equivalent maintained by a foundation like Apache, the brand is likely to be a feature.
Mikeal Rogers, whose original essay kicked off the entire discussion, has since followed up with some subsequent thoughts about foundations and institutions:
Simon believes it is the job of an institution (in this case a foundation) to protect members from each other and from the outside world. In the case of legal liabilities this makes perfect sense. In the case of community participation this view has become detrimental.
If you believe, as I do, that we have undertaken a cultural shift in open source then you must re-examine the need for institutional governance of collaboration. If the values we once looked to institutions like Apache to enforce are now enforced within the culture by social contract then there is no need for an institution to be the arbiter of collaboration between members.
Ben Collins-Sussman, a longtime Apache member, chimes in with his thoughts on the value of the Apache Foundation, pointing to the explicit codification of "community":
the ASF requires that each community have a set of stewards (“committers”), which they call a “project management committee”; that communities use consensus-based discussions to resolve disputes; that they use a standardized voting system to resolve questions when discussion fails; that certain standards of humility and respect are used between members of a project, and so on. These cultural traditions are fantastic, and are the reason the ASF provides true long-term sustainability to open source projects.
Jim Jagielski, another longtime Apache member, adds his thoughts, observing that it is important to not get caught up in statistics about popularity, adoption rate, etc., but to concentrate on communities, culture, and communication aspects:
The ASF doesn't exist to be a "leader"; it doesn't exist to be a "voice of Open Source"; it doesn't exist to be cool, or hip, or the "place to be" or any of that.
[ ... ]
It exists to help build communities around those codebases, based on collaboration and consensus-based development, that are self-sustaining; communities that are a "success" measured by health and activity, not just mere numbers.
Ceki Gulcu (poorly transliterated by me, sorry), a longtime Open Source Java developer, observes that what one person sees as consensus and meritocratic collaboration, another might see as endless discussion and fruitless debate:
Apache projects cannot designate a formal project leader. Every committer has strictly equal rights independent of past or future contributions. This is intended to foster consensus building and collaboration ensuring projects' long term sustainability. Worthy goals indeed! However, one should not confuse intent with outcome. I should also observe that committer equality contradicts the notion of meritocracy which Apache misrepresents itself as.
As I have argued in the past, the lack of fair conflict resolution or timely decision making mechanisms constitute favorable terrain for endless arguments.

It seems to be a fairly fundamental debate: some believe that the open source foundations provide substantial benefit, others feel that they reflect a time that no longer exists, and are no longer necessary.

Overall, it's been a fascinating discussion, with lots of viewpoints from lots of different perspectives.

I'll continue to be interested to follow the debate.

Tools and Utilities for Windows

2011-12-01T06:50:00.000-08:00

Scott Hanselman has posted a voluminous annotated list of the tools and utilities that he uses for developing software.

Most of these tools are specific to Windows 7, and more precisely to developing Web applications using Microsoft tools such as Visual Studio and C# and DotNet.

Still, it is a tremendous list, and if you're looking for a tool or utility for your personal development environment, there are a lot of references to chase here.

Wings

2011-11-30T14:08:00.001-08:00

The following sounds like a description of an airplane, like something you might hear from somebody like Bert Rutan:

when we addressed the wing, we started with a complicated rule, to limit what a designer could do. We added more and more pieces as we thought of more and more outcomes, and we came to a point where it was so complicated—and it was still going to be hard to control, because the more rules you write the more loopholes you create – that we reverted to a simple principle. Limit the area very accurately, and make it a game of efficiency.

But it's not from Rutan at all; it's an excerpt from Wings, the Next Generation, an article discussing the sailboats to be used in next summer's America's Cup qualification matches.

Now, everybody knows that sails, and airplane wings, actually have very much in common, so it really isn't surprising that this sounds like aerospace design. However, as Paul Cayard notes in the article, the wings on a competition sailboat have a few special constraints:

the America’s Cup rules don’t allow stored power, so two of our eleven guys—we think, two—will be grinding a primary winch all the race long. Not to trim, but to maintain pressure in the hydraulic tank so that any time someone wants to open a hydraulic valve to trim the wing, there will be pressure to make that happen.

It will be fascinating to see these boats in person, racing on the bay, but I'm glad I won't have to be one of those grinders!

Apache, Subversion, and Git

2011-11-29T06:41:00.001-08:00

Over the long weekend, a number of people seem to have picked up and commented on Mikeal Rogers's essay about Apache and its adoption of the source code control tool, Git. For example, Chris Aniszczyk pointed to the essay, and followed it up with some statistics and elaboration. Aniszczyk, in turn, points to a third essay (a year old), by Josh Berkus, describing the PostgresQL community's migration to git, and a fourth web page describing the Eclipse community's migration to git. (Note: Both Eclipse and PostgresQL migrated from CVS to git).

I find the essays by Rogers and Aniszczyk quite puzzling, full of much heat and emotion, and I'm not sure what to take from them.

Rogers seems to start out on a solid footing:

For a moment, let's put the git part of GitHub on the back burner and talk about the hub.
On GitHub the language is not code, as it is often characterized, it is contribution. GitHub presents a person to person communication system for contributions. Documentation, issues, and of course code, travel between personal repositories.
The communication medium is the contribution itself. Its value, its merit, its intention, all laid naked for the world to see. There is no hierarchy or politic embedded in the system. The creator of a project has a clear first mover advantage but the possibility is always there for its position to be supplanted by a fork, creating a social imperative to manage contributions in a satisfactory manor [sic] to her community.

This is all well-written and clear, I think. But I don't understand how this is a critique of Apache. In my seven years of experience with the Derby project at Apache, this is exactly how an Apache software project works:

Issues are raised in the Apache issue-tracking system;
discussion is held in the issue comments and on mailing lists;
various contributors suggest ideas;
someone "with an itch to scratch" dives into the problem and constructs a patch;
the patch is proposed by attaching it to the issue-tracking system;
further discussion and testing occurs, now shaped by the concrete nature of the proposed patch;
a committer who becomes persuaded of the desirability of the patch commits it to the repository;
eventually a release occurs and the change becomes widely distributed.

This is the process as I have seen it and participated in it, since back in 2004, and, I believe, was how it was done for years before that.

So what, precisely, is it that Apache is failing at?

Here is where Rogers's essay seems to head into the wilderness, starting with this pronouncement:

Many of the social principles I described above are higher order manifestations of the design principles of git itself.
[ ... ]
The problem here is less about git and more about the chasm between Apache and the new culture of open source. There is a growing community of young new open source developers that Apache continues to distance itself from and as the ASF plants itself firmly in this position the growing community drifts farther away.

I don't understand this at all. What, precisely, is it that Apache is doing to distance itself from these developers, and what does this have to do with git?

Rogers offers as evidence this email thread (use the "next message by thread" links to read the thread), but from what I can tell, it seems like a very friendly, open, and productive discussion about the mechanics of using git to manage projects at Apache, with several commenters welcoming newcomers into the community and encouraging them to get involved.

This seems like the Apache way working successfully, from what I can tell.

Aniszczyk's follow-on essay, unfortunately, doesn't shed much additional light. He states that "what has been happening recently regarding the move to a distributed version control system is either pure politicking [sic] or negligence in my opinion."

So, again, what is it that he is specifically concerned about? Here, again, the essay appears to head into the wilderness. "Let's try to have some fun with statistics," says Aniszczyk, and he presents a series of charts and graphs showing that:

git is very popular
lots of job sites, such as LinkedIn, are advertising for developers who know git
There is no 3.

At this point, Aniszczyk says "I knew it was time to stop digging for statistics."

But again, I am confused about what he finds upsetting. The core message of his essay appears to be:

The first is simple and deals with my day job of facilitating open source efforts at Twitter. If you’re going to open source a new project, the fact that you simply have to use SVN at Apache is a huge detterent [sic] from even going that route.
[ ... ]
All I’m saying is that it took a lot of work to start the transition and the eclipse community hasn’t even fully completed it yet. Just ask the PostgreSQL community how quick it was moving to Git. The key point here is that you have to start the transition soon as it’s going to take awhile for you to implement the move (especially since Apache hosts a lot of projects).

Once again, I'm lost. Why, exactly, is it a huge deterrent to use svn? And why, exactly, does Apache need to convert its existing projects from svn to git? Just because LinkedIn is advertising more jobs that use git as a keyword? That doesn't seem like a valid reason, to me.

Note that, as I mentioned at the start of this article, the PostgresQL team migrated from CVS to git, not from Subversion to git. I can completely understand this. The last time I used CVS was in 2001, 10 full years ago; even at that time, CVS had some severe technical shortcomings and there was sufficient benefit to switching that it was worth the effort. So I'm not at all surprised by the PostgresQL community's decision. The article by Berkus, by the way, is definitely worth reading, full of wisdom about platform coverage, tool and infrastructure support, workflow design, etc.

So, to summarize (as I understand it):

PostgresQL and Eclipse are migrating from CVS to git, successfully (although it is taking a significant amount of time and resources)
Apache is working to integrate git into its policies and infrastructure, but still uses Subversion as its primary scm system
Some people seem to feel like Apache is making the wrong decision about this

But what I don't understand, at the end of it all, is in what way this is opposed to "the Apache way?" From everything I can see, the Apache way is alive and well in these discussions.

UPDATE:Thomas Koch, in the comments, provides a number of substantial, concrete examples in which git's powerful functionality can be very helpful. The most important one that Thomas provides, I think, is this:

It is much easier to make a proper integration between review systems, Jenkins and Jira, if the patch remains in the VCS as a branch instead of leaving it.

I completely agree. Working with patch files in isolation is substantially worse than making reference to a branched change that is under SCM control. Certainly in my work with Derby I have seen many a contributor make minor technical errors while manipulating a patch file, that on the whole just adds friction to the overall process. Good point, Thomas!

Burton Bloom and the now-forgotten Computer Usage Company

2011-11-28T07:09:00.001-08:00

Burton Bloom's original paper on Bloom Filters is entitled Space/Time Trade-offs in Hash Coding with Allowable Errors, and his by-line is given as

Burton H. Bloom
Computer Usage Company, Newton Upper Falls, Massachusetts

with the additional parenthetical note that

Work on this paper was in part performed while the author was affiliated with Computer Corporation of America, Cambridge, Massachusetts.

Now, I'm quite familiar with Computer Corporation of America; I was an employee of theirs from 1985-1988, and I vividly remember my days working in the 4 Cambridge Center building.

But that was 15 years after Bloom's paper was published, and when I was there, I don't recall anything about "Computer Usage Company".

What was Computer Usage Company?

Unfortunately, a web search reveals only the slightest details:

There is, of course, a Wikipedia page
There is a Facebook page
and there is a short entry at the Computer History Museum website

But there is very little else. George Trimble's homepage no longer exists, and most of the links from the existing summary pages at Wikipedia and elsewhere point to articles in the IEEE Annals of the History of Computing, which (like Bloom's original paper at the ACM site) is protected behind a paywall and can't be read by commoners.

Computer Usage Company is credited with being "the world's first computer software company", but it seems on the verge of disappearing into dust. It's a shame; you'd think the software industry would work harder to keep information about these early pioneers alive.

I wonder if the IEEE keeps any statistics regarding how many people have actually paid the $30 to purchase this 20-year-old, five page memoir? I would have been intrigued to read it; I might even have paid, say, $0.99 or something like that to get it on my Kindle. But thirty dollars?

Ho-hum, just an 11-1 season

2011-11-26T19:18:00.001-08:00

It's amazing to me that Stanford are, at this point, clinging to hopes for a BCS at-large bid. Should it really be this hard to get two Pac-12 teams into the BCS? I guess that the SEC are still hoping they will field 3 teams in the 10 team BCS schedule...

Quote-un-Quote: 50 interviews of indie game developers

2011-11-26T09:11:00.001-08:00

Here's a great body of work: "Fifty Independent Videogame Developers; Fifty Interviews; Fifty Weeks".

The interviewer, who goes by the handle "moshboy", describes the intent of the project here:

all I wanted to do was get some words of insight out of a few independent videogame developers that weren’t known to put many of their own words ‘out there’. In the beginning, the idea was to interview those that had rarely or never been interviewed before.

His project succeeded, and produced a fascinating body of work:

Sometimes the quotes are a snapshot of a developer’s mindset from a certain time period, while most lean toward quoting some insight from their thoughts regarding videogame development.

The complete set of interviews are here.

Since I'm unfortunately not familiar with most of these developers, I found that a fun way to approach the work was just to scroll around in the list and randomly pick an interview.

Great job, moshboy, and thanks not only for embarking on the project and carrying it through, but for publishing the results for us all!

Distributed set difference computation using invertible Bloom filters

2011-11-25T08:11:00.001-08:00

Recently I've been slowly but steadily working my way through a meaty but rewarding recent paper entitled: What's the Difference? Efficient Set Reconciliation without Prior Context.

The subject of the paper is straightforwardly expressed:

Both reconciliation and deduplication can be abstracted as the problem of efficiently computing the set difference between two sets stored at two nodes across a communication link. The set difference is the set of keys that are in one set but not the other. In reconciliation, the difference is used to compute the set union; in deduplication, it is used to compute the intersection. Efficiency is measured primarily by the bandwidth used (important when the two nodes are connected by a wide-area or mobile link), the latency in round-trip delays, and the computation used at the two hosts. We are particularly interested in optimizing the case when the set difference is small (e.g., the two nodes have almost the same set of routing updates to reconcile, or the two nodes have a large amount of duplicate data blocks) and when there is no prior communication or context between the two nodes.

The paper itself is well-written and clear, and certainly worth your time. It's been particularly rewarding for me because it's taken me down a path of investigating a lot of new algorithms that I hadn't previously been studying. My head is swimming with

Invertible Bloom Filters (a variation on counting Bloom filters, which in turn are a variation on basic Bloom filters, an algorithm that is now 40 years old!)
Tornado codes
Min-wise sketches
Characteristic Polynomial Interpolation
Approximate Reconciliation Trees

and many other related topics.

I hope to return to discussing a number of these sub-topics in later posts, whenever I find the time (heh heh). One of the things that's challenging about a lot of this work is that it's based on probabilistic algorithms, which take some time getting used to. I first studied these sorts of algorithms as an undergraduate in the early 1980's, but they still throw me when I encounter them. When studying probabilistic algorithms, you often encounter sections like the following (from the current paper):

The corollary implies that in order to decode an IBF that uses 4 independent hash functions with high probability, then one needs an overhead of k + 1 = 5. In other words, one has to use 5d cells, where d is the set difference. Our experiments later, however, show that an overhead that is somewhat less than 2 suffices.

The question always arises: what happens to the algorithm in those cases where the probabilities fail, and the algorithm gives the wrong answer (a false positive, say)? I believe, that, in general, you can often structure the overall computation so that in these cases the algorithm still gives the correct answer, but does more work. For example, in the deduplication scenario, you could perhaps structure things so that the set difference code (which is trying to compute the blocks that are identical in both datasets, so that they can be eliminated from one set as redundant and stored only in the other set) fails gracefully on a false positive. Here, a false positive would need to cause the overall algorithm to conclude that two blocks which are in fact distinct, but which collide in the data structure and hence appear to be identical, are treated as distinct and retained in both datasets.

That is, the algorithm could be designed so that it errs on the side of safety when the probabilities cause a false positive to be returned.

Alternatively, some probabilistic algorithms instead fail entirely with very low probability, but fail in such as way as to allow the higher-level code to either simply re-try the computation (if it involves random behaviors, then with high probability it will work the next time), or to vary the computation in some crucial aspect, to ensure that it will succeed (which is the case in this particular implementation).

Most treatments of probabilistic algorithms describe these details, but I still find it important to always keep them in my head, in order to satisfy myself that such a probabilistic algorithm is safe to deploy in practice.

Often, the issue in using probabilistic algorithms is to figure out how to set the parameters so that the behavior of the algorithm performs well. In this particular case, the issue involves estimating the size of the set difference:

To efficiently size our IBF, the Strata Estimator provides an estimate for d. If the Strata Estimator over-estimates, the subsequent IBF will be unnecessarily large and waste bandwidth. However if the Strata Estimator under-estimates, then the subsequent IBF may not decode and cost an expensive transmission of a larger IBF. To prevent this, the values returned by the estimator should be scaled up so that under-estimation rarely occurs.

That is, in this particular usage of the probabilistic algorithms, the data structure itself (the Invertible Bloom Filter) is powerful enough that the code can detect when it fails to be decoded. Using a larger IBF solves that problem, but we don't want to use a wastefully-large IBF, so the main effort of the paper involves techniques to compute the smallest IBF that is needed for a particular pair of sets to be diff'd.

If you're interested in studying these sorts of algorithms, the paper is well-written and straightforward to follow, and contains an excellent reference section with plenty of information on the underlying work on which it is based.

Meanwhile, while wandering through Professor Eppstein's web site, I came across this nifty Wikipedia book on data structures that he put together as course material for a class. Great stuff!

Stanford crypto class

2011-11-24T09:35:00.001-08:00

I'm not sure how this will turn out, but I've signed up for Professor Dan Boneh's online Cryptography class, which starts this winter.

The recent events at U C Davis and U C Berkeley

2011-11-23T12:15:00.001-08:00

I mostly avoid political topics on my blog, but the current events on the University of California campuses are very important and need more attention. Here is a superb essay by Professor Bob Ostertag of U.C. Davis about the events of the last week, and a follow-up essay discussing ongoing events.

Meanwhile, it's interesting that some of the most compelling and insightful commentary is being published outside the U.S., for example this column and this column in the Guardian.

I don't know what the answers are. But I do know that the debate is important, and I salute the Davis and Berkeley communities for not backing down from the questions, and for opening their minds to the need to hold that debate, now. Our universities, and our children, are our future.

Danah Boyd on privacy in an online world

2011-11-21T20:52:00.001-08:00

It's somewhat of a shock to realize that it's been more than a decade since Scott McNealy made his famous pronouncement on online privacy:

You have zero privacy anyway. Get over it.

Well, people haven't actually just got over it. It's an important, complex, and intricate issue, and happily it is getting the sort of attention it needs.

So you should set aside a bit of time, and dig into some of the fascinating work that danah boyd has published recently, including:

A detailed analysis of the impact of the Children's Online Privacy Protection Act: “Why Parents Help Their Children Lie to Facebook About Age: Unintended Consequences of the ‘Children’s Online Privacy Protection Act’” in the online journal First Monday,
and her remarks prepared for the Wall Street Journal: Debating Privacy in a Networked World for the WSJ

Both articles are extremely interesting, well-written, and deeply and carefully considered. Here's an excerpt from the WSJ discussion:

The strategies that people use to assert privacy in social media are diverse and complex, but the most notable approach involves limiting access to meaning while making content publicly accessible. I’m in awe of the countless teens I’ve met who use song lyrics, pronouns, and community references to encode meaning into publicly accessible content. If you don’t know who the Lions are or don’t know what happened Friday night or don’t know why a reference to Rihanna’s latest hit might be funny, you can’t interpret the meaning of the message. This is privacy in action.

And here's an excerpt from the First Monday article:

Furthermore, many parents reported that they helped their children create their accounts. Among the 84 percent of parents who were aware when their child first created the account, 64 percent helped create the account. Among those who knew that their child joined below the age of 13 — even if the child is now older than 13 — over two–thirds (68 percent) indicated that they helped their child create the account. Of those with children who are currently under 13 and on Facebook, an even greater percentage of parents were aware at the time of account creation. In other words, the vast majority of parents whose children signed up underage were involved in the process and would have been notified that the minimum age was 13 during the account creation process.

As Joshua Gans notes in a great essay on Digitopoly, this is not an easy situation for a parent to be in, and the stakes are actually quite high:

And there are actually many reasons why I would want to allow her to do that. First and foremost, this is the opportunity for me to monitor her interactions on Facebook — requiring she be a friend at least for a few years. That allows me some access and the ability to educate. Second, all of her friends were on Facebook. This is where tween interactions occur. Finally, I actually think that it is the evolving means of communication between people. To cut off a child from that seems like cutting them off from the future.

I can entirely sympathise; my wife and I had similar deep discussions about these questions with our children (although at the time it was MySpace and AOL, not Facebook ).

They are your kids; you know them best. In so many ways, Facebook is just another part of life that you can help them with, like all those other temptations of life (drugs, sex, etc.). Talk to them, tell them honestly and openly what the issues are, and why it matters. Keep an eye on what they are doing, and let them know you'll always be there for them.

There are no simple answers, but it's great that people like boyd and Gans are pressing the debate, raising awareness, and making us all think about what we want our modern online world to be like. Here's boyd again:

We must also switch the conversation from being about one of data collection to being one about data usage. This involves drawing on the language of abuse, violence, and victimization to think about what happens when people’s willingness to share is twisted to do them harm. Just as we have models for differentiating sex between consenting partners and rape, so too must we construct models that that separate usage that’s empowering and that which strips people of their freedoms and opportunities.

This isn't going to be easy, but it's hard to think about anything that is more important that the way in which people talk with each other.

So don't just "get over it". Think about it, research it, talk about it, and help ensure that the future turns out the way it should.

Following up on Jonathan's Card

2011-11-21T07:51:00.001-08:00

This morning, the O'Reilly web site is running a condensed interview with Jonathan Stark, discussing, with the benefit of several months of hindsight, the intriguing "Jonathan's Card" events of the summer.

If you didn't pay much attention to Jonathan's Card as it was unfolding in real time, this is a good short introduction, with a summary of the events and some links to follow-up material.

The science of Maverick's

2011-11-18T16:13:00.000-08:00

Here's a wonderful multi-media piece diving deep into the earth science behind the surfing marvel that is Maverick's. Enjoy!

The Lewis Chessmen at NYC's Met

2011-11-17T16:37:00.001-08:00

Here's a nice story in the New York Times about the Lewis Chessmen, a 1000-year-old set of carved walrus tusk chess pieces, on exhibit at the Metropolitan Museum of Art in New York City.

Too bad I'm on the wrong side of the country; I'd love to see these. Unfortunately, according to the Met website,

After the showing in New York, they will return to London.

I guess I'll just have to figure out a way to travel to see them in their permanent home at the British Museum!

Software Patents, Microsoft, Android, and Barnes & Noble

2011-11-15T15:09:00.001-08:00

If you have any interest at all in the software industry, you'll be absolutely fascinated to read this detailed article at the GrokLaw website about the legal dispute between Microsoft and Barnes & Noble over Android-related patents.

It is well-known that Microsoft claims that Android infringes on Microsoft's patents; Microsoft themselves explain this on their website, saying they "simply cannot ignore infringement of this scope and scale", and that:

The Microsoft-created features protected by the patents infringed by the Nook and Nook Color tablet are core to the user experience.

and

Our agreements ensure respect and reasonable compensation for Microsoft's inventions and patent portfolio. Equally important, they enable licensees to make use of our patented innovations on a long-term and stable basis.

However, what has never been known (until now), is precisely what those patented innovations are. As Mary-Jo Foley observed more than 6 months ago, Microsoft refuses to identify the patents, and why it believes Android infringes upon them, unless a Non Disclosure Agreement is signed agreeing not to reveal that information.

Barnes & Noble apparently refused to sign that agreement, and instead found counsel to represent them, and now the information about the patents in question is no longer a secret.

According to the Barnes & Noble filings, the primary Microsoft patent which Android infringes is a 16-year-old patent (U.S. Patent 5,778,372), which patents:

A method of remotely browsing an electronic document residing at a remote site on a computer network and specifying a background image which is to be displayed with the electronic document superimposed thereon comprising in response to a user's request to browse to the electronic document.

Apparently, changing the background on your screen when a document is displayed is patented.

I understand software quite well.

I don't understand law at all, and specifically I don't understand intellectual property law.

However, I find the GrokLaw analysis of the Barnes & Noble v. Microsoft dispute absolutely fascinating.

Now THAT'S a data center!

2011-11-11T14:35:00.001-08:00

Here's a fun story about the Switch Communications "SUPERNAP" data center in Nevada. Switch claims it is "the world's best data center" and they have the stats to justify their claim.

These Internet-scale datacenters have really taken off in recent years. Last month the Open Compute community held their second Open Compute Summit, and part of that effort was the establishment of a foundation to guide the work as it moves forward; read more about that effort here. I haven't seen too much technical information flowing from the Open Compute Summit, although James Hamilton of Amazon posted his slides online here: here

Meanwhile (was this part of the summit, or independent?), the team at AnandTech have done some independent testing of the Open Compute server components; in their conclusion, they commend the Open Compute work as showing tremendous potential:

The Facebook Open Compute servers have made quite an impression on us. Remember, this is Facebook's first attempt to build a cloud server! This server uses very little power when running at low load (see our idle numbers) and offers slightly better performance while consuming less energy than one of the best general purpose servers on the market. The power supply power factor is also top notch, resulting in even more savings (e.g. power factoring correction) in the data center.
While it's possible to look at the Open Compute servers as a "Cloud only" solution, we imagine anyone with quite a few load-balanced web servers will be interested in the hardware. So far only Cloud / hyperscale data center oriented players like Rackspace have picked up the Open Compute idea, but a lot of other people could benefit from buying these kind of "keep it simple" servers in smaller quantities.

Lastly, since much of the activity in this area of computing has to do with power efficiency, let me draw your attention to this interesting work on power management in Android.

Cheaper, faster, and more power-efficient: the future of computing beckons!

Coarse language in professional writing

2011-11-09T15:53:00.000-08:00

Scott Hanselman shares his feelings about the use of coarse language.

Zach Holman responds.

Ted Dziuba says the real issue is passion and honesty versus marketing and publishing.

I can see both sides. Nothing about the language that Holman uses (or that Heinmeier Hansson does, for that matter) gets under my skin; perhaps I'm just thicker-skinned than many.

But meanwhile I know many readers who are put off by such things.

And so in my own writing I do my best to avoid such.

But I agree that you should (a) write about what you care about, and (b) write in your own words, not in the words that you think others want you to speak.

I guess I'm not adding much to the conversation, but there you go: a few pointers to some interesting articles and a bit of an observation by me.

Systems Software

2011-11-08T21:15:00.000-08:00

Here's a simple 3 step formula for determining if you are a systems software engineer (or are destined to be one):

Go to this year's agenda for the High Performance Transaction Systems workshop.
Scroll down the page, and see if you feel an irresistible urge to click on every single link, read every single slide of every single presentation, and follow every single reference.
There is no step 3.

24 hours at Fukushima

2011-11-07T20:56:00.000-08:00

This month, the IEEE's Spectrum magazine publishes its special report, Fukushima and the Future of Nuclear Power. It's an immense and detailed report, with multiple articles, multi-media presentations, and lots of material to dig through.

A good place to start is the lead article, 24 Hours at Fukushima. This article summarizes the events of the critical first 24 hours after the earthquake, with a focus on specific events and actions that seem like they represent learning opportunities.

The article has all sorts of fascinating details about the events of that day, such as the fact that it was hard to bring emergency equipment to the site when the roads were full of evacuations headed from the side; the observation that, after the earthquake but prior to the tsunami, an emergency cooling system was intentionally shut down because it was cooling the reactor too fast; the detail that, once power went out, electric security locks on building doors and fences had to be first broken before emergency equipment could be moved through them; and the observation that, when reactor 1 exploded, debris from the explosion ripped through the emergency backup power cable that had been installed to bring emergency power back to the plant.

And much, much more. There are so many small breakdowns, and decisions, and implications, that can be considered and thought about and studied.

The article notes that many of the subsequent problems arose from the fact that backup electrical power was lost, and could not be restored, and suggests several lessons that should be learned, including various ways to ensure that backup electrical power would be less likely to be lost, more likely to be subsequently restored, and perhaps even less likely to be needed; specifically, the article calls out 6 "lessons":

LESSON 1: Emergency generators should be installed at high elevations or in watertight chambers.
LESSON 2: If a cooling system is intended to operate without power, make sure all of its parts can be manipulated without power.
LESSON 3: Keep power trucks on or very close to the power plant site.
LESSON 4: Install independent and secure battery systems to power crucial instruments during emergencies
LESSON 5: Ensure that catalytic hydrogen recombiners (power-free devices that turn dangerous hydrogen gas back into steam) are positioned at the tops of reactor buildings where gas would most likely collect.
LESSON 6: Install power-free filters on vent lines to remove radio-active materials and allow for venting that won't harm nearby residents.

Not all of these lessons seem self-evidently obvious to me; for example, it seems like the recommendation to store backup power trucks "very close to the power plant site" would simply have resulted in leaving those trucks vulnerable to the same event that took out the main building power systems. As we know, the 14-meter tsunami washed away much larger and more resilient structures than backup power trucks.

Still, the lessons seem well-meant and clearly point out starting points for the discussions to come about improvements and enhancements. I love lesson 2 in particular, as it points out one of those "obvious in hindsight" mistakes that clearly represents an opportunity for all operators in every such site to review their similar equipment and ensure that it doesn't suffer from the same design flaw.

Engineering is hard. Things happen that you didn't expect, and you have to study your mistakes, learn from them, explore alternatives, test systems, and revise, revise, revise. As the article notes, we've learned a great deal from the tragedy at Fukushima, and we need to continue to learn more.

I'm not a nuclear engineer, but I am immensely grateful for efforts such as this one, to help us interested lay-people try to come to grips with what happened, and why, and what does it mean, and how do we make it better in the future.

Certainly, it makes me more motivated to return to my own designs, and to study them, and test them, and continue to learn from my own failures and make my future work better. I recommend this article highly; I think you'll find it interesting and well worth your time.

Perforce 2011.1 is out!

2011-11-07T15:48:00.000-08:00

Yay! The 2011.1 release of the Perforce server is ready! Read more about it at the Perforce site.

This is the second major release since I joined the team, and I'm quite excited about it. Although I played only a minor role in this release, I had a chance to get involved in many of the new features, and there are some really powerful enhancements in this release.

Engineers love releases (really, we do!): the whole point of writing software is to build something that gets used, and in order for it to get used, it has to get released. So even though a release is a whole lot of work, the result is that a new version of the product becomes available, and gets used, and that's always exciting!

So, congratulations, Perforce, on the 2011.1 release!

Epic Win for Anonymous

2011-11-06T18:17:00.000-08:00

I recently read Cole Stryker's Epic Win for Anonymous: How 4chan's army conquered the web. Well, that's somewhat over-stating the accomplishment: I skimmed it; I waltzed through it; I breezed over it.

This is the sort of book that takes you about 2 solid hours to read, if you try hard. And I'm not exactly sure why you would try hard, because it isn't really a book that rewards that. It is a very transparent book: it sets a simple goal, and it achieves it, completely:

If you've ever wondered, while browsing the web, "Why is this weird thing popular? Who cares about this stuff? How does this thing have so many views? Why do people waste their time with this? Where did it come from and where is it all going?" then read on.

Stryker's book succeeds: it helps you understand the concept of Internet memes; it shines a little light into the odd, strange corners of the Internet; it gives you some context for approaching some of the aspects of Internet life that probably seemed, if not downright horrifying, at least hard to comprehend:

What are memes?
Why is anonymity such a big deal on the Internet?
What are griefers, trolls, noobs?

If you've never heard of Anonymous, 4chan, lolcats, Rule 34, Star Wars Boy, or Encyclopedia Dramatica, then you should probably just pass this book by; its subject matter is of no interest.

But if you've heard of those topics, yet been slightly intimidated, and slightly unsure of how to proceed, then you might find this book helpful: it de-mystifies much of those lesser-known areas of the Internet, sets them out in plain terms and simple descriptions, and gives you at least enough knowledge to decide for yourself whether you want to know more.

As I reflected on the book, and tried to understand what I had learned, and how to summarize it, I found myself drawn to a particular passage. Stryker is describing an old (1986) computer game called Habitat, which was an early investigation of human-versus-human gaming:

One contentious game play element in Habitat was "Player vs. Player" or "PvP" killing. Experienced players were able to handily murder noobs, which made the game less fun for everyone but those who'd been there the longest. In addition, the very concept of virtual murder was controversial. It didn't take long for trolls to start randomly killing other players as they wandered around the virtual town. But if the engineers were to disallow PvP killing entirely, they would rob players of the thrill of danger and the joys of conquest. The moderators held a pool, asking if killing should be allowed in Habitat. The results were split 50/50. So they compromised. Killing would be disallowed inside the carefully manicured urban areas, but the moment you left town and headed out into the frontier, you were announcing to other players that you were down to scrap, if need be. This clever solution pleased most players, and continues to be the standard for many massively multiplayer games.
So will the Internet continue to look. Those who value safety over freedom will hang out on Facebook and other proprietary communities and mobile apps walled off with identity authentication. And those willing to brave the jungles of the open Internet will continue to spend time in anonymous IRC channels and message boards like 4chan.

It's an interesting metaphor, and I think it's insightful. In a new world, it's important to have a discussion about rules. And to have that discussion, there has to be a certain amount of discussion about where (and when) the rules apply. As Stryker notes:

/b/ is significant because it's the only board on 4chan that has no rules (the only thing prohibited is committing or plotting actual crimes, the same rules that apply to any public forum on or offline).

Actually, as it turns out, there are more rules than these, but to a certain extent in order to understand the rules, you have to be a member of the community.

The Internet is still young, and we are still learning how we want to behave in this new cyberspace. Places like 4chan, although almost certainly not your cup of tea, are still worth understanding and thinking about, and Stryker's book is a step toward opening the discussion and having those debates.

Four above 2800!

2011-11-06T14:19:00.000-08:00

As Dylan McClain observes in a column in today's NY Times, the current FIDE chess rankings list now includes four players with ratings of 2800 or higher:

Magnus Carlsen, ranking 2826
Viswanathan Anand, ranking 2811
Levon Aronian, ranking 2802
Vladimir Kramnik, ranking 2800

This is great news, and really emphasizes not just the excellence that is currently manifest in the top levels of chess, but also its growing spread, as the top list includes players from several countries not previously known for chess prowess (Carlsen is from Norway, Anand from India, and Wang Hao from China is in the top twenty as well).

But McClain goes on to claim:

But ratings inflation — caused in part by looser rules guiding them — makes it difficult to compare different eras.
The ratings system was actually never intended for such comparisons. It was created in 1960 by Arpad Elo, a physics professor, as a snapshot of each player’s ability and a tool for predicting games’ outcomes. The system has been tweaked over the years, but it has held up well.

McClain provides no evidence for this claim, which is a shame, as from what little I know, the evidence in fact shows entirely the opposite. As I described in a short blog post last summer, a fairly detailed study by students at the University of Buffalo recently concluded that

there has been little or no ‘inﬂation’ in ratings over time—if anything there has been deﬂation. This runs counter to conventional wisdom, but is predicted by population models on which rating systems have been based

Regardless of whether or not the ratings are being inflated, there is no doubt in my mind that today's chess players are playing some wonderful chess. As we look toward next year's World Chess Championship, there is lots of reason to be excited about the world of chess!

They're fish!

2011-11-04T19:02:00.000-07:00

I really enjoyed this fabulous interview with William Gibson in The Paris Review.

Gibson, of course, is one of the greatest science fiction writers ever, the man who coined the term "cyberspace", who gave us (so far) nine spectacular novels, with hopefully more coming.

What will you learn if you go read the interview? Well, all sorts of things!

You'll learn about Gibson's fascinating writing techniques: never planning past the first sentence, constantly re-working and re-considering his story:
Every day, when I sit down with the manuscript, I start at page one and go through the whole thing, revising freely.
...
I think revision is hugely underrated. It is very seldom recognized as a place where the higher creativity can live, or where it can manifest. I think it was Yeats who said that literary revision was the only place in life where a man could truly improve himself.
letting the work flow from someplace hard to describe:
I’ve never had any direct fictional input, that I know of, from dreams, but when I’m working optimally I’m in the equivalent of an ongoing lucid dream. That gives me my story, but it also leaves me devoid of much theoretical or philosophical rationale for why the story winds up as it does on the page. The sort of narratives I don’t trust, as a reader, smell of homework.
You'll learn about Gibson's view on whether science fiction writers are writing about the future, the past, or the present:
Nobody can know the real future. And novels set in imaginary futures are necessarily about the moment in which they are written. As soon as a work is complete, it will begin to acquire a patina of anachronism. I know that from the moment I add the final period, the text is moving steadily forward into the real future.
...
all fiction is speculative, and all history, too—endlessly subject to revision.
You'll learn what Gibson, surprisingly, finds to be the technology that is most characteristic of the human species:
Cities look to me to be our most characteristic technology. We didn’t really get interesting as a species until we became able to do cities—that’s when it all got really diverse, because you can’t do cities without a substrate of other technologies. There’s a mathematics to it—a city can’t get over a certain size unless you can grow, gather, and store a certain amount of food in the vicinity. Then you can’t get any bigger unless you understand how to do sewage. If you don’t have efficient sewage technology the city gets to a certain size and everybody gets cholera.
You'll get some great anecdotes that will totally drop your jaw:
For years, I’d found myself telling interviewers and readers that I believed it was possible to write a novel set in the present that would have an effect very similar to the effect of novels I had set in imaginary futures. I think I said it so many times, and probably with such a pissy tone of exasperation, that I finally decided I had to call myself on it.
A friend knew a woman who was having old-fashioned electroshock therapy for depression. He’d pick her up at the clinic after the session and drive her not home but to a fish market. He’d lead her to the ice tables where the day’s catch was spread out, and he’d just stand there with her, and she’d look at the ice tables for a really long time with a blank, searching expression. Finally, she’d turn to him and say, “Wow, they’re fish, aren’t they!” After electroshock, she had this experience of unutterable, indescribable wonderment at seeing these things completely removed from all context of memory, and gradually her brain would come back together and say, Damn, they’re fish. That’s kind of what I do.

It's a thrilling roller-coaster of an interview, with so many choice bits that you'll find yourself returning to his ideas again and again.

Just as we do with his books!

Enjoy.

Kindle Lending Library

2011-11-03T13:13:00.000-07:00

Wow! This, by itself, is just about enough to make me go buy a Kindle.

Andreesen Horowitz bloggers

2011-11-03T07:09:00.000-07:00

The blogging portfolio at the Andreesen Horowitz web site continues to grow, with Peter Levine the latest to start a blog.

I think that most of this blogging activity has been inspired by Ben Horowitz, who is by far the most active and best blogger of the bunch.

But the other blogs are quite interesting too, and there are some additional interesting materials on the site. If you're interested in the software industry, and more specifically in the VC-funded software startup industry, there is a lot of interesting material to see here, including blogs from:

I know from my own experience how much time and effort it takes to write a blog. I'm pleased that this group, who have lots of experience and knowledge, are investing the energy into sharing their thoughts and opinions; hopefully it will inspire others to do the same, and in the meantime it means more interesting essays to read!

To get you started, check out Ben's recent essay: Hiring Executives: If You’ve Never Done the Job, How Do You Hire Somebody Good?.

This is how computer software evolves

2011-11-01T19:16:00.000-07:00

Legacy software, so goes the old saying, is any software that actually works.

Of course, there is a fair nugget of truth to this aphorism, for once a system is running we start to be increasingly unwilling to change it.

Yet software is soft for a reason; it can be changed, and it can be improved.

It's always interesting to watch this process at work, for the improving of software can be a messy business, not just for technical reasons, but also because of social, cultural, and business reasons.

Today's exhibit: JavaScript, now venerable, but not so long ago it was the newest language on the Net. JavaScript has always had a history of strife and tension when it comes to how to upgrade the language, and has suffered through its bumps and bruises.

A major event recently was Google's decision to take a new direction with their proposed new Dart language. As Weiqi Gao observes, this has grown into quite the discussion, with some backers wanting to continue to improve JavaScript, while others feel that it's time to embark on designing a new language (such as Dart). Major discussion has ensued, and some of the emotions have been rather heated:

I respect the people involved and believe they’re for the most part making their own choices. But Dart and other unrelated Google agenda items do impose clear and significant opportunity costs on Google’s standards actiivities.

"Unrelated"? I think that is an overly strong critique. Surely Dart is strongly related.

What is the best way, then? Continue to improve the existing language, or work on building new languages? Can we really not do both? After all, Dart is not the only new language built on top of and closely related to JavaScript: consider CoffeeScript, for example. It is another new language which is designed for the web, and is implemented by being compiled into JavaScript. CoffeeScript is interesting; it pushes the envelope in other directions, such as borrowing the "whitespace is structure" paradigm from Python (a technique I've always found fascinating, even if I can barely manage to maintain decent indentation standards in my own code).

I think such experimentation with new languages is wonderful, which is why it's so great to see techniques such as Source Maps, which allow tools like IDEs and Debuggers that were written for one language to be used, in a semi-interpreted style, with other newer languages built atop those old languages. Clever!

So, just as Weiqi Gao does, I applaud the invention of these new languages, and I hope the experimentation continues. And I applaud the improvement of compilers and VMs and language runtimes, to enable such language experimentation and innovation. Even though I continue to earn my living coding in Dennis Ritchie's good old C language, all these new ideas and new approaches help us all think about problems in new ways, and find better techniques to solve problems.

It is, as the kids say, all good.

Big milestone for the new Bay Bridge

2011-10-29T09:12:00.000-07:00

This week, a major milestone in the construction of the new Bay Bridge: the last section of deck for the suspension segment is now in place. I like this article in the Contra Costa Times because it does a good job of explaining what's going on, and has some nice pictures of the Left Coast Lifter in action. As the CCTimes article notes:

The deck piece installed Friday is key.
"It's the place where the cable actually comes in and locks into the bridge," Caltrans spokesman Bart Ney said.
"It's self-anchored. That means the cable doesn't go into the ground, but into this piece. There's a big hole in the deck where will the cable will go down and anchor into this section."
Unlike the Golden Gate Bridge, where cables were strung before the deck was installed, the new Bay Bridge suspension span requires that the deck be installed before the cable. Temporary supports hold up the deck until the cable is tensioned.

I guess that means that the Lifter is likely to move on now ... is there another bridge somewhere on the planet awaiting its services? Check out this great picture of the Lifter setting that massive piece of roadway into the final slot just as easy as pie!

Here's a pretty computer-generated video showing what it will (soon) look like.

Next step: string the cables! I believe that starts in earnest after Thanksgiving. It should be something to see; if you look carefully at the various pictures, you can see the catwalks that the crew will walk during cable installation.

KFOG Live from the Archive -- online or in the store?

2011-10-26T16:35:00.000-07:00

The Peet's Coffee website wants $8 for shipping the KFOG Live from the Archives CD.

For that, it's worth waiting until Tuesday and walking 2 blocks down to the Peets to get my copy.

And maybe get a cup of coffee, too :)

Derby 10.8.2.2 is released

2011-10-26T07:19:00.000-07:00

A new maintenance release of Derby has been released, with the typical collection of bug fixes and small features.

What was more interesting about this release was the way that the community handled the decision-making surrounding a relatively minor performance feature: concurrent generation of new values for identity columns.

The feature went through the typical Derby development cycle: a developer, interested in the problem, created a proposal for fixing it; the patch was reviewed and discussed; feedback was addressed; the patch was committed.

Somewhat later, near the point when it was ready to be released, other developers in the community were running certain complex test suites that aren't routinely run, and these test suites were not performing well.

It wasn't obvious what was causing the test suites to mis-behave. When test suites get sufficiently complicated, diagnosing their behaviors can be as hard or harder than diagnosing the behavior of the actual product. We have lots of practice diagnosing the behaviors of the main product code, and over time we've built all sorts of debugging and diagnosis hooks into that code to assist with analysis. But tests generally are rather leaner in this area, and it's hard to tell what's going on when a complex, unfamiliar, and rarely-run test fails: is there a real problem in the product? Or is this a weak test, which was accidentally passing in the past because it wasn't written quite carefully enough?

The Derby community responded very well to these challenges, I feel.

There was no rush to judgement, no finger-pointing, no blaming. Instead, different members of the community studied the new behaviors, constructed theories, ran experiments, offered suggestions, and considered the ideas of others.

As time passed, the first release candidate missed its schedule, then was withdrawn from consideration. The suspect feature was backed out from the codeline, and testing continued. It was very hard to tell: was the new behavior more stable? Or about the same?

After considerable time, the product was felt to be ready for release, and the second release candidate was released this week.

Even though I had only a minor part in this process, I was fascinated to be involved. At a Derby community luncheon during the Oracle Java conference earlier this month, I attempted to sum up my observations about what had occurred to the group at large as follows:

Software is hard
We made a change
Something new and interesting happened in the software
We talked about it as a community
Lots of people learned new things about the software
We addressed the problem, and moved on

I know that it was quite painful for some of the participants who were more closely involved in the effort, but from my point of view, this was software engineering at its finest: a team of dedicated and talented people collaborate on an extremely complex undertaking and, together, overcome obstacles and deliver a strong new release.

Congratulations Derby on Derby 10.8.2.2!

Ubuntu 11.10 nits

2011-10-25T19:30:00.000-07:00

Since upgrading my Dell Latitude D610 to Ubuntu 11.10, two nits:

My computer seems to drop the wireless connection, and the only way to restart it is to reboot. Without warning, the system is suddenly offline, and fiddling with the Network widgets doesn't bring it back online. ifconfig down/up doesn't bring it back online, either. But rebooting it gets me back online with 100% reliability
The screen dims when the computer is idle for a certain period of time. Then, after waking it back up from screen-dim, the screen lights up, but then slowly darkens again to an almost-dim state. If I then go into the Displays control panel and fiddle with the screen resolution (e.g., set it to a different resolution but then revert to the current resolution), the screen becomes bright again.

I'm not sure what's causing either of these problems, and I'm not sure how to diagnose them further. For the time being, Ubuntu 11.10 on this computer is a bit of an annoyance.

Oh, by the way, 3 other computers that I've got which are running 11.10 are not displaying these symptoms. Just this one.

Hot Wheels!

2011-10-24T09:44:00.000-07:00

Mmmm... Hot Wheels ... I love Hot Wheels! So I completely loved this video.

The analysis of the track seems pretty reasonable to me; I think the killer question is the cost. Those booster motors aren't cheap!

I wonder if the author of the video has some special access, like he maybe works for Mattel's marketing department or something like that.

I had a childhood friend whose father worked for Mattel (he was a car designer/artist). So it's possible :)

Another possibility is that the video was spliced together out of multiple shots, and the person making the video only had to have enough track supplies for any single shot. That would make the overall resource consumption more reasonable, since no single shot had an unreasonable amount of track/turns/loops/boosters in it.

I wish the video had had more shots of the kids admiring the car running around the track. The first scene was great!

I thought the single least-likely part of the video was where the car jumped into that funnel-like thing and then got going again on the track. I suspect that particular part of the video took multiple shots...

Mojave Air and Space Port

2011-10-22T18:11:00.000-07:00

Here's a great photo-essay on the Mojave Air and Space Port. I used to pass this pretty routinely in the 1990's, when I would go visit my parents in Ridgecrest.

It's been 20 years now, but the memories are still strong, and this nice short photo-essay was fun to browse through.

Personal Bests

2011-10-21T19:43:00.000-07:00

It is no secret that my two favorite writers in The New Yorker are (in either order): Atul Gawande, and Adam Gopnik. So it was with some pleasure that I noticed that two recent issues featured articles from them: Personal Best, by Gawande, and Broken Kingdom, by Gopnik. Each essay is different, and their topics are completely unrelated, yet I find them intriguingly inter-connected.

Gopnik's article celebrates the fiftieth anniversary of a very under-appreciated classic: Norton Juster's The Phantom Tollbooth (published the year I was born!). As Gopnik says, it really means something when a children's book stands the test of time like this:

it means that the book hasn't been passed just from parent to child but from parent to child and on to child again.

Somehow, Gopnik tracks down not just Juster, but also Jules Feiffer, the illustrator, and spends several hours with them ("a pair of wryly benevolent uncles") talking about the book, about how it came to be, and about what they think it means that it has survived to the present day: "The book is made magical by Juster's and Feiffer's gift for transforming abstract philosophical ideas into unforgettable images."

Gopnik proposes that the "enduring magic" of The Phantom Tollbooth is in fact quite fundamental:

As with every classic of children's literature, its real subject is education. The distinctive quality of modern civilization, after all, is that children are subjected to year after year after year of schooling. In the best-loved kids' books, the choice is often between the true education presented in the book -- say, Arthur's through anaimals at the hands of Merlyn, in The Sword in the Stone -- and the false education of the world and school. The child being read to (and the adult reading) is persuaded that self-reliance is a better model for learning than slavish obedience.

The Phantom Tollbooth, claims Gopnik, represents the ultimate paean to curiosity and a love of all knowledge, wherever it might lie. It is:

not just a manifesto for learning; it is a manifesto for the liberal arts, for a liberal education, and even for the liberal-arts college.

Meanwhile, Gawande is thinking about learning, too, but from a quite different angle:

I've been a surgeon for eight years. For the past couple of them, my performance in the operating room has reached a plateau. I'd like to think it's a good thing -- I've arrived at my professional peak. But mainly it seems as if I've just stopped getting better.

Not surprisingly, this worries Gawande, and he tries various techniques to figure out what is wrong, but then inspiration arrives from a somewhat unlikely corner; while at a medical conference, he takes a break to practice his tennis, and works out with the club's "house pro". Afterwards he happens to be watching a tennis match, and:

I watched Rafael Nadal play a tournament match on the Tennis Channel. The camera flashed to his coach, and the obvious struck me as interesting: even Rafael Nadal has a coach. Nearly every elite tennis player in the world does. Professional athletes use coaches to make sure they are as good as they can be.
But doctors don't. I'd paid to have a kid just out of college look at my serve. So why did I find it inconceivable to pay someone to come into my operating room and coach me on my surgical technique?

The wonderful thing about Gawande, the reason that he is so incredibly inspirational and motivational, is that he doesn't just stop with this insight, he acts upon it:

I decided to try a coach. I called Robert Osteen, a retired general surgeon, whom I trained under during my residency, to see if he might consider the idea. He's one of the surgeons I most hoped to emulate in my career. His operations were swift without seeming hurried and elegant without seeming showy. He was calm. I never once saw him lose his temper. He had a plan for every circumstance. He had impeccable judgement. nad his patients has unusally few complications.

Sounds like the dream coach!

Osteen agrees to the request, and the coaching begins:

He came to my operating room one morning and stood silently observing from a step stool set back a few feet from the table. He scribbled in a notepad and changed position once in a while, looking over the anesthesia drape or watching from behind me.

Afterward, Gawande worries that it was all a waste of time:

The case went beautifully. The cancer had not spread beyond the thyroid, and, in eighty-six minutes, we removed the fleshy, butterfly-shaped organ, carefully detaching it from the trachea and from the nerves to the vocal cords. Osteen had rarely done this operation when he was practicing, and I wondered whether he would find anything useful to tell me.

Gawande need not have worried:

I'd positioned and draped the patient perfectly for me [...] but not for anyone else. [...] At one point, we found ourselves struggling to see [...] I should have made more room [...] my right elbow rose to the level of my shoulder [...] I operate with magnifying loupes and wasn't aware how much this restricted my peripheral vision [...] the operating light drifted out of the wound.

In fact, there are plenty of opportunities for improvement:

That one twenty-minute discussion gave me more to consider and work on that I'd had in the past five years.

How wonderful!

They repeat the process, multiple times, verifying that earlier mistakes are corrected, and moving on to other areas for improvement. Gawande is thrilled:

Since I have taken on a coach, my complication rate has gone down. It's too soon to know for sure whther that's not random, but it seems real. I know that I'm learning again. I can't say that every surgeon needs a coach to do his or her best work, but I've discovered that I do.

Oh, how deeply that one sentence resonates with me: "I know that I'm learning again." Is there any sensation more wonderful? Perhaps there are one or two, but this is near to the pinnacle of what it means to be a human. Will Gawande's observations lead others to find coaches? In my own field of software engineering, one of the big breakthroughs of this century is something called "Peer programming", in which engineers are challenged to work side-by-side, thinking out loud, sharing observations and ideas, listening and learning from each other constantly. It's fatiguing, but oh-so-helpful: when I get stuck, the first thing I do is call over my cube wall:

Hey Cal, are you there? Can you come be another pair of eyes? I'm just not seeing this...

And now we come full circle, back to Gopnick and Norton Juster. Thinking and thinking about The Phantom Tollbooth, Gopnik finally zeroes in on the specific insight that the book conveys, the reason that, fifty years later, it still thrills reader after reader.

"Many of the things I'm supposed to know seem so useless that I can't see the purpose in learning them at all," Milo complains to Rhyme and Reason. They don't tell him to listen to his inner spirit, or trust the Force. Instead, Reason says, "You may not see it now, but whatever we learn has a purpose and whatever we do affects everything and everyone else. ... Whenever you learn something new, the whole world becomes that much richer."

Indeed, says Gopnik:

Learning isn't a set of things that we know but a world that we enter.

Oh, dear reader: write that down and place it somewhere important, and look at it every day.

Yay yay yay yay! The malaria vaccine works (though not perfectly)

2011-10-19T10:10:00.000-07:00

Thank you Gates Foundation! Thank you GlaxoSmithKline!

It's nice to have good news like this. Keep it coming!

Flooding in Thailand could affect hard disk drive manufacturing

2011-10-19T07:04:00.000-07:00

Here's an interesting article in InfoWorld about how the severe flooding in Thailand will probably have a significant effect on the world's supply of hard disk drives.

Western Digital, the largest hard disk manufacturer, makes more than 30 percent of all hard drives in the world. Its plants in Ayutthaya's Bang Pa-In Industrial Estate and Pathum Thani's Navanakorn Industrial Estate together produce about 60 percent the company's disks. Both were shut down last Wednesday.

I didn't realize how significant Thailand had become in the manufacturing of electronic components:

Key disk component suppliers have also been hit. Nidec, which makes more than 70 percent of all hard drive motors, has temporarily suspended operations at all three of its plants in Thailand, affecting 30 percent of its production capacity. Hutchinson Technologies, which makes drive suspension assemblies, has also suspended operations due to power outages,

The Register offers some additional details, describing the extent of the flooding as "worse than feared":

"Over the weekend, rising water penetrated the Bang Pa-in Industrial Park flood defences, inundating the company’s manufacturing facilities there and submerging some equipment," WD said in a statement.

CNN has video, and Yahoo news is covering the question of whether the government initially under-reacted to the extent of the floods.

The world is very inter-connected; events in one area effect other areas. We are all one planet, one people.

Correctly handling RENAME in SCM merge processing

2011-10-18T07:05:00.000-07:00

Here's an interesting and detailed article by the team at Atlassian about the complexities of merging changes when the changes involve renamed files. In their case, they found that Git was substantially better than the older Subversion releases at handling this task, and migrating one of their projects to Git has resulted in improved handling of their merge and rename scenarios.

Perforce once suffered from similar problems, and the handling of renames has been a major focus over the past few years.

Happily, the implementation has gone well, and the new Perforce integration engine, which is now in its final testing, has received great feedback from our early users, and I'm excited about the upcoming release.

This will be the second major release of the Perforce server since I joined, and although I didn't work directly on the integration engine, I was glad to have contributed to the release in a number of other areas.

If you're struggling with how to correctly handle complicated rename and merge scenarios in your SCM procedures, check into Perforce; I think you'll be pleased with the sophistication of the Perforce server in this area!

Less than one month until The Elder Scrolls: Skyrim is released

2011-10-17T07:45:00.001-07:00

Wired Magazine gives us some details about how it will look and play.

The bubble is back!

2011-10-15T09:13:00.000-07:00

I think that this article is mostly rot and rubbish, but it's getting a bit of attention on the Netz right now, particularly because of

I recently had a sit-down chat with Ping Li, a venture capitalist at Accel Partners who does investments across the layers of the cloud stack. Over the course of our conversation, Ping expressed frustration about the difficulty of hiring and maintaining talent right now. “It’s this heated funding environment,” he said, going on to explain that all of the money sloshing around in the Valley had created a market for talent that’s just as tight as it was during the dotcom boom. What’s worse, he explained, is that the talent shortage is stifling fundamental innovation in the cloud space.

Sour grapes, I'd say.

Though it's definitely true that if Google continues adding 800 new employees a month, that will create a significant competition for employees.

However, the article goes on to say things like:

We’re written in Ruby and hosted on Heroku, a pair of technical decisions that we made so that we could easily and painlessly scale, and so that we wouldn’t have to waste resources on any sort of sysadmin work. Back in the depth of the last downturn, we were fortunate to have found a team of contract developers ...

This is the sort of rot and rubbish that drives good engineers crazy. Sure, Heroku is super-trendy and just the bee's knees right now, but any entrepreneur with any sense ought to understand that smart engineers have the ability to learn all sorts of different technologies, and make technology transitions all the time.

None of us emerged from the womb with five year's experience in iOS application design, after all :)

It's simple: if you hire for buzzwords, you get people who have learned how to shape their career around buzzwords: shallow thinkers who are adept at jumping on board the latest trend and can market themselves as experts in whatever new gewgaw is flitting about.

If you hire for intelligence, adaptability, communication skills, ability to think abstractly, experience working in a team, and, of course most important of all, whether they Get Things Done, you're building a team that matters, will last, and will actually succeed in making something new.

So I'm neither the slighest bit surprised nor the slighest bit sympathetic that the business leaders that haven't learned these lessons are failing. There's a reason that those lessons are important; it's because if you don't understand them and realize why they matter, you too will find yourself whining that " top engineers are being enticed with truckloads of money to break off and form two- and three-person startups," and not comprehending what's really wrong with your company and your hiring and retention practices.

Sorry about the mini-rant, but when I see buzzword-driven headhunters bemoaning the scarcity of some trendy technology fad or another, I know that what I'm reading is not a serious discussion about whether or not our education system is encouraging the sort of academic disciplines which lead to long-term systems thinkers who can work effectively in a world of constantly changing technology, but rather is a self-serving attention grab to pump up their own visibility by getting their name in the paper.

Now, we've spent too much time on that particular article :)

Dennis Ritchie has died?

2011-10-12T20:01:00.000-07:00

Oh, dear. Rob Pike is reporting that Dennis Ritchie has died. This is very sad news. He certainly was one of the top ten figures in Computer Science in my lifetime.

Three unrelated topics

2011-10-12T13:50:00.000-07:00

Sorry for the "link bait" aspect of this post, but I've been falling a bit behind recently and I wanted to get these items out there before I forgot about them:

There's been a fair amount of discussion over the past 4 weeks about Google's proposed new replacement for JavaScript, the programming language Dart. JavaScript is neat but I'm all in favor of improving programming languages, so I'm pleased to see Google pushing the discussion forward. If you're interested, you'll want to spend some time looking at the Technical Overview, and browsing the other online docs. If you'd like to see what others think, try starting with this comment thread at Lambda the Ultimate; or, for a contrarian perspective, try Peter-Paul Koch's take
The always fascinating Jeff Atwood has a fantastic essay online about Gamification. Atwood isn't just interested in Gamification, he's (attempting to) live it:
Stack Overflow is in many ways my personal Counter-Strike. It is a programmer in Brazil learning alongside a programmer in New Jersey. Not because they're friends -- but because they both love programming. The design of Stack Overflow makes helping your fellow programmers the most effective way to "win" and advance the craft of software development together.
There are great comments at the end of the Atwood article, and also don't miss the presentation he links to, Sebastian Deterding's presentation Meaningful Play: Getting Gamification Right
Lastly, while I've generally avoided the flood of mostly melodramatic and mawkish writing about the tenth anniversary of 9/11, let me draw your attention to this stellar piece by Ken Regan about Danny Lewin, MIT PhD student, co-inventor of consistent hashing, co-founder of Akamai Technology, and passenger on American Airlines Flight 11. Regan's article has many links to chase and many ideas to pursue, but make sure (if you're even the slightest bit interested in Computer Science) that you follow the most important link, to the paper that made the web work