Wednesday, October 31, 2012

HMS Bounty RIP

It's not been easy to find a lot of hard news about the HMS Bounty incident.

Early on, there was some good coverage in the Washington Post:

In the last 36 hours, though, there hasn't been an awful lot to go on. There's been a bit of information in the Christian Science Monitor:

Apparently, from what I've read, the ocean temperature in the area, dead smack in the middle of the Gulf Stream, is an unbelievably warm 77 degrees Fahrenheit, so the hope is that Captain Walbridge may still somehow be alive out there.

Godspeed and best luck to the SAR teams and their personnel; they did wonderful work on Monday and may they have another success yet to come.

Meanwhile, the final hours of the Bounty remain confusing. As the Monitor notes, lots of questions are unanswered:

The ship’s course out of Connecticut took it due east to try to avoid the oncoming hurricane Sandy. Early on Sunday, the crew felt it had skirted the danger: A Facebook post showed the ship’s position on a map well to the east of the storm’s fiercest winds.

They were mistaken. The ship was close to the tail end of the hurricane as it whipped up the Atlantic coast.

Details about the ship’s final hours remain sketchy. Apparently at least one generator failed, and the Bounty began taking on more water than it could safely handle.

Hundreds of miles from shore, in hurricane conditions, what navigational tools and techniques apply? Were they receiving GPS signals? Were they simply dead reckoning? What caused them to be off by dozens or hundreds of miles?

Reading between the lines, it sounds like they made a navigational error, which placed them in peril, and then suffered a power plant fault, which probably left them simultaneously unable to flee the storm, unable to orient themselves tactically to waves and wind, and unable to operate the pumps which are crucial in removing water faster than it can arrive.

I'm interested in reading more about the story; if you spot good information about what happened on the Bounty during those final 48 hours, do drop me a line and let me know...

Tuesday, October 30, 2012

Programmers and Paparazzi

You may not have heard of a little company named Cloudera, but that may change soon. Cloudera is one of the hottest startups in one of the hottest parts of the computer industry, the so-called "Big Data" space.

The management team at Cloudera at times looks like one of those 'dream team' assemblages that people put together in their fantasy football leagues; they've recruited top talent from places like Oracle, Yahoo, VMWare, Facebook, etc.

For a company like Cloudera, desparate to attract attention in a rapidly-growing, hotly-contested market, having big names is sure to help, and it's no surprise that many of Cloudera's competitors are assembling similar power-house talent pools, backed by vast sums of investment funds.

But it was somewhat of a surprise to me to see a new page turned in these talent wars over the last few years: companies now trumpet their personnel acquisitions like professional sports franchises advertise their latest trades, and so it's interesting to see Cloudera's Press Center touting this breathless love-fest article about their latest hire: World's Most Wired Software Engineer.

There's some sort of transition going on here, and I'm not quite sure what it means when publications like Wired are running articles that contain things like:

For Charles Zedlewski — Cloudera’s vice president of products — Impala shows off not only Marcel Kornacker the software engineer, but Marcel Kornacker the man. When Kornacker builds software, he builds it with an eye for the tiniest of details — and he’s intent to take it as far as he can. It’s the same way in his kitchen, on the other side of San Francisco Bay. Kornacker’s Epicurean exploits extend well beyond bread baking. “If you walked into his kitchen, you would think you walked into the set of Top Chef,” says Zedlewski.

Well, I don't watch Top Chef very much, either, so maybe I'm missing the whole point.

This whole topic is much in the light right now due to a long and fascinating essay by John Allspaw: On Being A Senior Engineer. Allspaw's well-thought-out, well-sourced, and well-presented essay has a lot to recommend it, even if I find myself agreeing with only about half of it.

But what I find most striking about Allspaw's essay, and even more so about the things that swirl around it, like the Wired programmer celebrity series, is the absence of humility.

Consider, for example, this follow-on from Adrian Cockcroft: What's a Distinguished Engineer?, advising people that the things that matter most are:

"how many of these people know who you are?". ... "how many DE and Fellows are hanging around your cube on a regular basis waiting to talk to you?"... "Do the top conferences invite you to speak?" ... "How many of the other invited speakers and conference organizers do you know?" and "how many know you?"

Well, pardon me, but all this name-dropping and publicity-grabbing is, to put it bluntly, a load of manure.

In my 30+ years in the software industry, I've known hundreds of superb, stellar engineers: people who taught me how to approach problems, how to encourage and take advantage of feedback, how to build software that lasts for decades.

And I've known my share of attention-hungry, limelight-seeking engineers: people who thrive on taking credit and being known.

And I can tell you, from long experience, that the intersection of these two groups is empty.

So please, young engineer, consider carefully the advice you're receiving from the Paparazzi; follow not the course of celebrity; strive instead for the immense pleasure and satisfaction that you will find in building software so solid, so clear, so reliable, and so robust that it lasts for years and forms the foundation of systems that make the world as we know it possible.

Monday, October 29, 2012

24 hours with the "17"

Oracle Team USA have posted a stunning 10 minute short feature about the capsize and recovery of their AC 72 "17".

The video includes the only footage I've seen of the actual collapse (wow, it happens so fast!), and then covers the recovery of the crew, the stabilization of the boat, and its eventual recovery back to dock.

We went out in the helicopter at daybreak, to see what else we could recover. Pieces were strewn everywhere, too small and moving too fast to note their coordinates. There were pieces inside the bay, under the bridge, and we saw the last piece more than six miles outside of the bridge.

Wonderful video, and it took guts for them to put it online, so big props to the team and the management.

ORACLE TEAM USA "17" Capsize - The Whole Story.

Sunday, October 28, 2012

The slow maturation of C++

As the C++ programming language prepares to begin its fourth decade (Stroustrup apparently first called it "C++" in 1983), it continues to slowly stabilize.

For one thing, compilers are starting to support significant subsets of the recently standardized "C++ 11" version of the language, as these notes from David Simmons describe. Another good source of such information is this page from Scott Meyers: Summary of C++11 Feature Availability in gcc and MSVC.

Of course, it's more than just the language and its compilers; you have to have a complete "ecosystem" in order to have a complete programming language. Over at the Google Testing Blog, Zhanyong Wan describes the various reasons why Google decided to build their own C++ unit testing framework: Why Are There So Many C++ Testing Frameworks?

You might wonder, when a language like Java was able to reach this level of maturity in half the time, why is C++ still struggling with these sorts of issues after more than 30 years of development? As the Google team point out, the comparison is somewhat unfair, since Java was in some ways created as a response to exactly this question:

Unlike Java, which has the famous slogan "Write once, run anywhere," C++ code is being written in a much more diverse environment. Due to the complexity of the language and the need to do low-level tasks, compatibility between different C++ compilers and even different versions of the same compiler is poor. There is a C++ standard, but it's not well supported by compiler vendors. For many tasks you have to rely on unportable extensions or platform-specific functionality. This makes it hard to write a reasonably complex system that can be built using many different compilers and works on many platforms.

And as the Google team point out, it's fundamentally much easier to write a testing framework for Java than for C++, for a variety of reasons, including:

Another reason is that some limitations of C++ make it impossible to implement certain features really well, and different people chose different ways to workaround the limitations. One notable example is that C++ is a statically-typed language and doesn't support reflection. Most Java testing frameworks use reflection to automatically discover tests you've written such that you don't have to register them one-by-one. This is a good thing as manually registering tests is tedious and you can easily write a test and forget to register it. Since C++ has no reflection, we have to do it differently.

I regularly write in both Java and C++. It is vastly easier to write in Java, but with care I can be quite productive in C++. Tools like testing harnesses and high quality compilers are a big part of whatever language ecosystem you're part of, so it's good to see that the C++ versions continue to move along, however slowly.

I mean, after all, it could be worse: apparently, the second most popular programming language in Japan is COBOL!!

Saturday, October 27, 2012

Kinda quiet recently...

I haven't been blogging very much recently.

Why, you wonder?

Wednesday, October 24, 2012

A random collection of random stuff

Wandering the Internet, reading all sorts of things...

  • A delightful examination of last week's Rotterdam art theft from the perspective of art history: The Art of the Art Heist
    Hilton Kramer called de Haan's “Self-Portrait” — the painting that was recently stolen — "one of his most beautifully painted pictures," and complimented de Haan's "remarkably successful attempts to paint in a Gauguinesque style." The thieves of Rotterdam seem to agree. Yet the look on the face of Meyer de Haan in his self-portrait is more than just Gauguinesque. De Haan is just barely meeting our gaze. He is troubled and self-assured at the same time. Or you could say that he is both absent and present. He is just about to look away, to have another thought. Behind him are painted splotches of red and yellow, a kind of dreamscape. It is so very Symbolist.
  • Wired Magazine brings us up to date on the life and times of the incredibly talented Peter Molyneux, the creator of Populous, perhaps my favorite computer game ever: How a Videogame God Inspired a Twitter Doppelgänger — and Resurrected His Career. Perhaps the most interesting part of the story involves how Molyneux reacted when one of his more passionate fans created a fake alter-identity on Twitter:
    Journalists like Gamasutra’s Leigh Alexander and GiantBomb’s Alex Navarro began retweeting some of his posts. Sites like Kotaku and GameSetWatch also ran coverage of the mysterious online figure, who maintained his anonymity in the press. The exposure helped @PeterMolydeux hit 23,000 followers by the end of 2011. And gamer cognoscenti weren’t reading it just as a cruel joke at Molyneux’s expense; @PeterMolydeux was becoming an outlet for the widespread dissatisfaction within the industry, a voice for the thousands of developers who had grown tired of the limitations of mainstream gaming. If anything, the Twitter feed gave some followers more respect for Molyneux.
  • Surely by now everyone has read the New York Post article about the tremendous power of a simple set of master keys: Key set available for $150 on eBay provides an all-access pass to NYC, shrieking headline and all. For a more nuanced take, try Bruce Schneier: Master Keys
    The current bit of sensationalism aside, this is fundamentally a hard problem. Master keys are only useful if they're widely applicable -- and if they're widely applicable, they need to be distributed widely. This means that 1) they can't be kept secret, and 2) they're very expensive to update.
    Or, for a view just slightly farther into the future, consider Geoff Manaugh's essay: Keys to the City
    This cinematographic duo would thus pose there looking at each other, under the control of hackers huddling in a van somewhere wearing Stadium Pals, long enough that they could 3D-map the keys from the ensuing image feed and then have accurate copies produced. Thus would your house be robbed by robot.
  • Going back to Wired Magazin, do not miss this epic article about reliability engineering, and the emerging discipline of failure analysis: Why Things Fail: From Tires to Helicopter Blades, Everything Breaks Eventually.
    Product failure is deceptively difficult to understand. It depends not just on how customers use a product but on the intrinsic properties of each part—what it’s made of and how those materials respond to wildly varying conditions. Estimating a product’s lifespan is an art that even the most sophisticated manufacturers still struggle with. And it’s getting harder. In our Moore’s law-driven age, we expect devices to continuously be getting smaller, lighter, more powerful, and more efficient. This thinking has seeped into our expectations about lots of product categories: Cars must get better gas mileage. Bicycles must get lighter. Washing machines need to get clothes cleaner with less water. Almost every industry is expected to make major advances every year. To do this they are constantly reaching for new materials and design techniques. All this is great for innovation, but it’s terrible for reliability.

    This article is so good, it's hard not to quote every bit of it. If you think you ever might want to be an engineer, or if you are an engineer and you want to learn how to be a great engineer, stop what you are doing and read this article, and then read it again, and again.

    Part of the revolution in reliability engineering comes from better tools, such as this tool from Vextec:

    The result is essentially an image of the component’s microstructure. Vextec’s algorithms then assess this microstructure: What are the grain sizes and orientations? How often do voids appear and in what shape? How frequently do particles of dust or other contaminants appear? The algorithms create a set of rules for the material—a statistical model of every aspect of the microstructure. The rules are used to create multiple virtual versions of the material whose microstructures vary within the rough range that the client could expect to see in manufacturing.

    But it's interesting that another part of the field comes from a side-effect of some of the government regulations put into effect after Enron famously collapsed:

    In the wake of the scandal that took down the energy juggernaut, the Financial Accounting Standards Board made changes to the Generally Accepted Accounting Principals—the rules that, among other things, govern how companies write financial statements. As of November 2002, companies were required to provide a detailed reckoning of their guarantees, including their warranty reserves and payments, in quarterly and yearly filings. The result was that, for the first time in history, someone could look at, and compare, how US public companies handle claims—how much they pay out, how much they hold aside for future payments.
    It's all there in the article: expect failure, design for failure, plan for failure, test to expose failure, test to evaluate the handling of failure, test your tests. Yes, this article is all about the building of cars and trucks and door hinges, but it really about what makes things reliable.

    My, perhaps it's time for me to go read the book again? I'm feeling like it's been too long...

  • Out of the ashes of Newsweek Magazine's collapse, one possible phoenix is the return from oblivion of Dan Lyons.
    We have a platform where we can do some of the things we did on Fake Steve, like create a sense of community and add a touch of irreverence to the world of tech. Now more than ever the Valley is in need of a bracing, no-bullshit shot of truth, and we are the ones to deliver it.
    Read more about Lyons's hope for the e-zine once known as ReadWriteWeb in his first editorial on the new site
    We want to turn our writers loose and let them write from the heart, in ways that are more personal, passionate, provocative and fun than ever before. We want ReadWrite to be a lively place filled with wit and energy, a place where you find great stories told in a convincing, engaging way, with brains and a point of view.
    I've always enjoyed Lyons's work, and I hope he does somehow find a way to at least partly return to the zing and verve he brought to Fake Steve.

  • Had enough of all of this? Practice arming your cubicle for the coming armageddon

Tuesday, October 23, 2012

IPv6 Summit in Slovenia

Ivan Pepelnak has a nice post summarizing his take on the highlights of a recent IPv6 conference: The Best of Last Week’s IPv6 Summit

Big thanks to the organizers of the conference, for making their videos available for the rest of us!

Instance tasting

Although I've never had occasion to employ this practice, sometimes a term is just so right, so well-coined, that you can immediately understand what's being described.

William Vambenepe explains the concept

I’ve heard people describing how they use “instance tasting”. Every time they get a new instance, they run performance tests (either on CPU like this paper, or i/o, or both, depending on how they plan to use the VM). They quickly terminate the instances that don’t perform well, to improve their bang/buck ratio.

Monday, October 22, 2012

New Linux 0, Old Laptop 1

I spent the weekend building and re-building the Linux installation on my laptop, only to decide in the end that something about the new (3.5+) Linux kernels hates my hardware.

The whole episode started when I foolishly decided to upgrade from Ubuntu 12.04 to Ubuntu 12.10. I believe that 12.10 is running a 3.5.X kernel.

The upgrade was without incident, except that when all was done, the laptop wouldn't boot! It showed a splash screen, then the screen went dark, and it was unresponsive.

Eventually I figured out that I could interrupt into the GRUB menu and choose an older kernel, and then the laptop booted, but it was very unstable and kept printing "a system error has occurred".

I happened to have a Fedora 17 CD around, so I re-built the laptop with Fedora 17 on it.

The laptop was fine, so I decided to run Software Update, which again ran for hours, and decided to take me to a 3.6.X kernel.

Once again, the laptop failed to boot, and once again if I overrode the GRUB menu and chose "Advanced Settings" and picked an older kernel, it came up, grudgingly.

So, for now, I'm running on a stock Fedora 17 installation, aware that I apparently can no longer update to the latest kernel.

At least until I can figure out what part of my hardware the new kernel hates.

Which probably means:

  1. Updating to the latest software
  2. Allowing the machine to fail to boot a few times
  3. Choosing an older kernel and getting back into the machine
  4. Trying to find some log files with some clues about what's wrong
  5. Re-installing back down to the older Linux

Ugh. I'm not really looking forward to this process.

Saturday, October 20, 2012

Quantal Quetzal

Ubuntu 12.10 appears to be out. Ars Technica covers the changes.

But my Update Manager isn't showing it as a choosable upgrade.

Normally there's a big box on Update Manager that gives me the upgrade option.

Am I just jumping the gun, and it will show up soon?

Or is there something wrong with my Update Manager database?

Friday, October 19, 2012

Photo-essay on cork production

Don't miss this delightful photo-essay on the production of cork:

Facing the tsunami

A fascinating short article appeared on the Chronicle of Higher Education's website yesterday: Minnesota Gives Coursera the Boot, Citing a Decades-Old Law.

The state’s Office of Higher Education has informed the popular provider of massive open online courses, or MOOC’s, that Coursera is unwelcome in the state because it never got permission to operate there. It’s unclear how the law could be enforced when the content is freely available on the Web, but Coursera updated its Terms of Service

Is this the beginning of the MOOC backlash?

The Internet continues to breaks down barriers; I am reminded of Stewart Brand's famous quote from 1984:

Information Wants To Be Free. Information also wants to be expensive. Information wants to be free because it has become so cheap to distribute, copy, and recombine---too cheap to meter. It wants to be expensive because it can be immeasurably valuable to the recipient. That tension will not go away. It leads to endless wrenching debate about price, copyright, "intellectual property," the moral rightness of casual distribution, because each round of new devices makes the tension worse, not better.

The age of books is not yet over

Derek Attig brings us updates about a new Bookmobile effort in Texas: Time-Traveling Librarians from Outer Spa... from Texas

It turns out that this temporal confusion isn’t an accident. Most obviously, the project takes its name from the protagonist of Kurt Vonnegut’s Slaughterhouse Five, a character who himself becomes “unstuck in time.” And like awesome librarian Doctors Who—with time all great big, wibbly-wobbly, timey-wimey, and stuff—BPTL founders aim to have their library “leap through time and space to connect the very best elements of bookmobiles with current needs.”

Thursday, October 18, 2012

Paul Allen is in the Bay Area

Since the Octopus is here, it must be just about time for the football game!.

Scott Hanselman nails it

In a nice three-part series, Scott Hanselman talks about software, and bugs, and what to do:

  • In Everything's broken and nobody's upset, he calls out the Emperor for having no clothes, and speaks the ugly truth:
    Software doesn't work. I'm shocked at how often we put up with it.

    ...

    Here we are in 2012 in a world of open standards on an open network, with angle brackets and curly braces flying at gigabit speeds and it's all a mess. Everyone sucks, equally and completely.

    As Hanselman observes, one of the issues is that many people use Google as their principal problem solving tool:

    Here's the worst part, I didn't spend any time on the phone with anyone about these issues. I didn't file bugs, send support tickets or email teams. Instead, I just Googled around
  • In A Bug Report is a Gift, Hanselman talks about the question of whether it's easy or hard for users to file a bug against your software:
    Is it easy for your users to report a bug? Does you app automatically report crashes? Is your site friendly? Are you support people friendly and engaged? Are your bug support forums filled with reports but no company representatives? Do you use GetSatisfaction or UserVoice?
  • And in Help your users record and report bugs with the Problem Steps Recorder, Hanselman talks about some specific technical solutions that you can put in place to help your users send you bug reports:
    On the subject of bug reporting, there's a wonderful gem of a program that ships with Windows 7 and Windows 8 that you and your users can use to report and record bugs. It's the Problem Steps Recorder and it's like TiVo for bugs.

I love the practical advice and extensive examples that Hanselman includes in these posts, but what I really love most of all is the perspective he brings to the entire topic:

How hard does the user have to work to file a bug? It's OUR bug but the user not only hit the bug but also has to work to report it!

Every click or manual step that our users have to invest in our bug is a click too many. A bug is the pain that hurts the user and keeps hurting as they report it! A good bug report is a gift that keeps on giving and we should treat it as such.

I'd love a world where all crashes are automatically reported and there's a "Send a Frown/Smile" button on everyone's computer that offers to record a short screencast with just a single "record/stop" button.

You can't fix the bug until you find it, so the first step toward improving software is to make it as easy as possible to find, isolate, record, report, and track those bugs.

Thanks, Scott, for the helpful and well-intended suggestions; now let's get them adopted!

Wednesday, October 17, 2012

Once there was a sailboat...

And then it met a gust...

Kimball Livingston records the results.

Wind: 1, Oracle: 0

Team Oracle's amazing new 72-foot catamaran met with an ugly incident during race training yesterday: America's Cup boat flips in S.F. Bay.

Tom Slingsby, the tactician of Oracle Team USA 17 - the 72-foot racing yacht - said the flip occurred in stiff winds of about 24 knots as he and the rest of the crew were "pushing the limits of the boat."

It's not clear how extensive the damage is:

During the crash, pieces of the enormous wing and rigging snapped off. When the craft drifted beyond the mouth of the Golden Gate around 5 p.m., it was still on its side, but then as the mast broke off, the boat turned upside down.

One thing is for sure: the puny vessels we humans construct are nothing against the power of the ocean:

The Oracle team, headquartered at Pier 80 south of the Bay Bridge, dispatched nine safety boats to try to right the craft or tow it back. But the current, driven by the strongest tide of the year, was too much, and the helpless Oracle crews could only accompany the vessel as it was swept 4 miles west of the Golden Gate.

Some pictures and more details at Kimball Livingston's blog: Oracle's Pitchpole

Update: Wired Magazine has more pictures and information here.

Presidential Precedent

Randall Munroe sets us straight.

No matter what happens over the next three weeks, one way or another, it'll be unprecedented!

Monday, October 15, 2012

Drilling down, spreading the load

A few related, but independent, postcards from the bleeding edge crossed my eyeballs the last few days:

  • The column-oriented gang at Google just keep cranking things out. There was the Dremel work of last summer, and an entirely-unrelated (I think) project called Supersonic that claims to provide
    an ultra-fast, column oriented query engine library written in C++. It provides a set of data transformation primitives which make heavy use of cache-aware algorithms, SIMD instructions and vectorised execution, allowing it to exploit the capabilities and resources of modern, hyper pipelined CPUs

    And now we have PowerDrill, introduced with the wonderful title: Processing a trillion cells per mouse click

    we present the column-oriented datastore developed as one of the central components of PowerDrill. It combines the advantages of columnar data layout with other known techniques (such as using composite range partitions) and extensive algorithmic engineering on key data structures. The main goal of the latter being to reduce the main memory footprint and to increase the efficiency in processing typical user queries. In this combination we achieve large speed-ups. These enable a highly interactive Web UI where it is common that a single mouse click leads to processing a trillion values in the underlying dataset.
  • Meanwhile, I'm quite enjoying the slide deck posted by the Tokutek team from their tutorial session at the XLDB conference: Data Structures and Algorithms for Big Databases
    The tutorial was organized as follows:
    • Module 0: Tutorial overview and introductions. We describe an observed (but not necessary) tradeoff in ingestion, querying, and freshness in traditional database.
    • Module 1: I/O model and cache-oblivious analysis.
    • Module 2: Write-optimized data structures. We give the optimal trade-off between inserts and point queries. We show how to build data structures that lie on this tradeoff curve.
    • Module 2 continued: Write-optimized data structures perform writes much faster than point queries; this asymmetry affects the design of an ACID compliant database.
    • Module 3: Case study – TokuFS. How to design and build a write-optimized file systems.
    • Module 4: Page-replacement algorithms. We give relevant theorems on the performance of page-replacement strategies such as LRU.
    • Module 5: Index design, including covering indexes.
    • Module 6: Log-structured merge trees and fractional cascading.
    • Module 7: Bloom filters.
    The slides are superb, but I bet it was even greater to attend the presentation; it seems that they packed pretty much every important topic in the last 10 years of file structure design into a single presentation!

XCOM Enemy Unknown

I don't have much time to play video games, so I tend to go many months between games, looking for just the right one.

I've been playing XCOM: Enemy Unknown for about a week now, and so far I like it quite a lot. It isn't the best game of all time, but it's a lot of fun, and the pacing is very nice.

I like the fact that, for the most part, it's turn-based, so it generally doesn't test my reflexes and controller-agility, but rather keeps me entertained with puzzles, plot, and action.

XCOM: Enemy Unknown is a bit of an unusual blend of base-building and tactical combat, with some switching back and forth that is managed quite well. Overall, I think that the Ars Technica review nails it: Review: XCOM: Enemy Unknown is a credit to the name: Firaxis' update keeps the drama and difficulty while adding accessibility.

Firaxis has taken the decision-making process that made the original so addictive and expanded on it, creating a play experience that's much more detailed than the original’s endless bug hunts. Limited cash flow can force choices between research in weapons you can’t afford to produce and research in life-saving armor you don’t have the supplies to build. You will need to go to desperate lengths to capture live aliens, take ludicrous risks to prevent civilian casualties, and charge into the teeth of withering gunfire to defuse deadly bombs. The developers obviously cared enough about the X-Com lineage to put their own mark on it, and their concern shows in every aspect of the game.

It's still a bit early to see how I will fare with this game, so check back around Thanksgiving and I'll let you know :)

Saturday, October 13, 2012

MOOCing again

I've finished the first week of An Introduction to Computer Networks, taught by Professors McKeown and Levis of Stanford, and hosted at Stanford's Class To Go site.

This is my fourth class I've taken this year (two cryptography classes, one statistics class), and now my third different provider (Coursera, Udacity, and now ClassToGo).

The format of the networking class is quite different from the other classes, at least so far: it is much less interactive, mostly watching videos and answering a few questions; the only background reading is a $150 networking textbook that I've been too cheap to purchase; the online slides are much more casual.

Still, I'm finding the videos worthwhile and intend to continue watching them (assuming they continue to post them).

While this doesn't make me an expert on MOOCs, it does at least make me a participant.

MOOCs have been much discussed this summer, and a few recent events are worth noting:

  1. Earlier this summer, Stanford President John Hennessy gave a keynote speech at the 4th Computing Research Association conference: The Coming Tsunami in Educational Technology. Hennessy proposes that University education has two functions:
    Universities perform two education functions:
    • Helping students learn: in class and outside
    • Credentialing: certifying students achieve mastery of a subject (grades & degrees )
    Hennessy suggests that the future of higher education is surprisingly murky:
    Online learning with lots of content of varying quality
    • Online courses will become the new textbooks.
    • Content-alone likely to be inexpensive
    • Replace of lecture function of instructors by master online lecturers
    Harder to predict what happens in credentialing:
    • Certification for post degree professional education has happened and has clear value in some fields
    • Likely to be high variability in quality of certification
    • Big question: what happens in the UG space?
    Lance Fortnow follows up on Hennessy's talk, observing that university educators have been surprisingly resistant to incorporating technological advances to improve their efficiency:
    one cannot increase the number of students to faculty without decreasing the quality of education. That's where MOOCs come in, supposedly the solution to allow faculty to be far more efficient in the number of students they can teach without reducing quality. Might help control college costs but could harm research at top tier universities and many other universities might cease to exist.
  2. In a widely-noted post, Amy Bruckman foretells doom for higher education: The Future of Universities: Everything a MOOC is NOT
    The trend over the last dozen or so years is for people who make money creating intellectual property to be compensated more and more poorly. Fewer people are making a living as musicians. Professional journalism is in crisis. Small newspapers are closing, and major ones are struggling. This hasn’t happened all at once, but like a frog in a pot, raising the temperature/economic pressure a fraction of a degree per year over the long haul has dramatic consequences. MOOCs turn education into a form of IP. The same economic pressures are going to apply.
    John Regehr echoes her predictions: University Economics and the End of Large Classes
    if large lecture classes are replaced by MOOCs, universities face a revenue collapse that will force a major restructuring. This could result, for example, in people like me being asked to cover a larger fraction of our salaries from grants. That is, the university could save money by covering only 50% or 25% of our salaries instead of the current standard arrangement where it pays 75%. Another possibility is that many professors are let go while others face much larger teaching loads.
  3. Daniel Collins, a community college professor in New York, posted a fierce online critique of Sebastian Thrum's Statistics 101.
    the course is amazingly, shockingly awful. It is poorly structured; it evidences an almost complete lack of planning for the lectures; it routinely fails to properly define or use standard terms or notation; it necessitates occasional massive gaps where “magic” happens; and it results in nonstandard computations that would not be accepted in normal statistical work. In surveying the course, some nights I personally got seriously depressed at the notion that this might be standard fare for the college lectures encountered by most students during their academic careers.
    In response, Thrum promises to improve the course
    this article points out a number of shortcomings that warrant improvements that a few students have also raised and we have noted for iteration. Some are the result of my attempt to experiment with open questions (challenge students before I provide an answer), my dedication to get rid of overly formal definitions, and my desire to place optional challenges into an otherwise basic course. But I agree with the author of this article -- and our many supporters who have voiced similar things -- that the resulting course can be improved in more than one way.

It's a year of change for higher education. We were lucky enough to be able to send our children to college, but it's not clear to me how much longer that glorious period will last. Even close co-workers of mine whose children are just a few years younger than mine will be struggling mightily to enable them to attend, with college costs (in California at least) going up 20% or more every year for the last 10 years, and with no end in sight.

The online learning activity reminds me very much of early automation efforts in many areas: the first attempts are "amazingly, shockingly awful"; people nonetheless see the possibilities; iterations and improvements result; those who retain the buggy whips predict disaster and the collapse of civilization; the newer, better approaches win out.

As Thrum himself observes, what will make or break these online learning environments is their success in building thriving, evolving communities:

I believe that Udacity owes all of our students the hardest and finest work in making amazing classes. We are very grateful for any feedback that we receive. These are the early days of online education, and sometimes our experimentation gets in the way of a coherent class.

And what is to happen of the classic university experience, whatever that might be?

I don't know, but I agree with Hennessy: the tsunami is here; things will change.

Gawker outs VA

I'm no big fan of Gawker, the lead property in Nick Denton's stable of weird-yet-successful online web-zines.

But this weekend Gawker is running with a pretty big story, a public outing of a man they claim is the "in real life" identity behind the massive year-long controversy over disturbing and barely-legal content on several of the forums at reddit, perhaps the most popular site on the Internet. The story is here: Unmasking Reddit’s Violentacrez, The Biggest Troll on the Web.

To its credit, the Gawker story does more than just name names; it tries to help explain the deeper story of what is going on here:

This is how Violentacrez, Reddit's creepiest user, also became its most powerful. Sure, he was responsible for the absolute worst stuff on Reddit, and by extension, some of the worst stuff on the internet. But Violentacrez was also seen to be, as Chris Slowe put it to me, "a trustworthy and a positive member of the community." He moderated more than 400 subreddits and had many high-profile friends, amassed over many years. His stable at times included hundreds of popular mainstream subreddits, like Funny and WTF, that reach audiences of millions. Violentacrez further solidified his reach by becoming a mentor to other moderators. He created the first FAQ for Reddit's rather unintuitive moderator interface. He also helmed a number of subreddits dedicated to providing guidance and camaraderie for other moderators, including the essential modhelp.

Similarly to 4chan, a site I wrote about last year, reddit at times seems to find the exploration of the boundaries of free speech to be its entire raison d'etre:

many of Violentacrez's most offensive subreddits were created just to enrage other Reddit users. At this they were very effective. What happened was, some do-gooder would stumble upon one of his offensive subreddits and expose it to the rest of Reddit in an outraged post. Then thousands more would vote the thing to the front page of Reddit. Cries to censor it would sound out, to be almost inevitably beaten back by cries of "free speech!" The idea of free speech is sacred to many Reddit users, a product of the free-wheeling online message board culture from which Reddit springs. If you criticize someone else for posting something you don't like, you are a whiny fascist.

However, Adrian Chen, the Gawker columnist writing the article, doesn't see things the same way:

Under Reddit logic, outing Violentacrez is worse than anonymously posting creepshots of innocent women, because doing so would undermine Reddit's role as a safe place for people to anonymously post creepshots of innocent women.

I am OK with that.

The web is full of trolls, the web is full of barely-legal content, and the web is full of controversy, so you might wonder why you'd bother paying attention to this story.

One reason is a bit hard to explain, but Choire Sicha (who was one of the original creators of Gawker, but who has gone on to do other things) tries to explain: A Handy Test for Reddit Users: Are You on the Internet Right Now?:

Sometimes it's hard to tell if you are on the Internet or not. For example you are almost always typing into a box on a series of screens on your computer. Because of this, there are whole sections of the Internet that are pretty sure they are not on the Internet, because, they are just boxes, right? You could be typing into anything, who knows if it's public. This was true about LiveJournal for a long time. When you would link to a posting on LiveJournal, back in the day, you would get outraged emails about invasion of privacy. Because in their minds, they were just typing in their diary. That happened to be readable by others. Any others. On the Internet. Sometimes the youngs on Tumblr think they are not on the Internet, and also some members of Reddit think they are not on the Internet! And so there is a to-do.

Sicha's essay is partly meant as satire, but it's also partly meant to try to help you understand why people, particularly people much younger than me who have grown up in a new world, are genuinely confused:

in their minds, they were just typing in their diary.

It's horrible that these parts of the Internet exist, and that people behave awfully on the Internet. And it would be nice to just close my eyes and cover my ears and go "La-La-La" and not pay any attention.

But, as I wrote a year ago, this stuff is real, and it is important, and I think Gawker is at least to be commended for forcing this discussion to continue to occur, and making people continue to think about, discuss, and debate topics such as trolling, privacy, and anonymity on the Internet.

But when you read the articles by Chen and Sicha, you don't need to follow all the links.

Friday, October 12, 2012

Harborside Health

The New York Times covers the battle over Oakland's Harborside Health Center: Oakland Sues U.S. to Prevent Closing of Marijuana Dispensary.

The Department of Justice has focused its efforts on those dispensaries near schools, parks and playgrounds. Harborside’s two dispensaries have no such neighbors, but in July, United States Attorney Melinda Haag issued a forfeiture notice on the 7,000-square-foot Oakland property leased to Harborside since 2006 and on a 6,000-square-foot property occupied by the group’s San Jose dispensary.

In a statement, Ms. Haag called Harborside an example of a marijuana “superstore.” She went on to say that the larger the dispensary, “the greater the likelihood that there will be abuse of the state’s medical marijuana laws, and marijuana in the hands of individuals who do not have a demonstrated medical need.”

Tuesday, October 9, 2012

Leo Messi piece on ESPN

ESPN's "Outside The Lines" is running a totally epic multi-media piece on the greatest athlete of my lifetime, the astonishing Leo Messi: Lionel Messi, Here & Gone.

The garage area looked familiar, then I remembered. I saw a video of Messi playing soccer here with his young nieces and nephews, dodging and feigning, moving the tiny ball with his feet, the kids unable to take it away. The look on his face is the same as when he plays in front of millions of fans. There's something innocent about him with a ball. His friends laugh about how Messi seems somehow less than himself without one. Leguizamon told us a story. The last time Messi came to visit, they hung out in a backyard, and they watched him, uneasy, antsy. Finally, without even realizing he was doing it, he pulled a lemon from a tree and juggled it mindlessly with his feet, whole again.

It's so hard to comprehend the life of someone like Messi: his extraordinary ability recognized as a child, taken away to live a very special and different life.

Something about the pain of Messi losing his childhood seems to make him always be looking for it — or even still living it — whether he's got a ball at his feet in a packed stadium, or visiting the town where that childhood was lost. I read a interview not long ago with Bruce Springsteen where he said he spent years driving past the house where he grew up, night after night, and a psychologist told him he was trying to go back in time to change the things that happened in that house.

I would love to be able to play like Messi, to be able to enjoy the game like he does, but I am so happy that I never had to live like him.

Social coding and Atlassian Stash

If you've been wondering what all the fuss is about "social coding", or why GitHub is so hot hot hot nowadays, here's an interesting article from one of GitHub's competitors, Atlassian, about their new social coding product, Stash: Stash 1.3: Enterprise Git Gets Social with Pull Requests.

In Stash, pull requests provide an easy way for developers to review changes on a branch, discuss those changes with one another, make modifications and merge the branch back into master or the main development branch. For those familiar with centralized version control systems (like Subversion), think “pre-commit reviews.” But pull requests have a key evolutionary advantage: instead of working alone on your local repository, your changes will be shared on an isolated branch in Stash where others can review the code and contribute changes.

I'm a huge fan of code review and of the Continuous Integration methodology described by Martin Fowler here: Continuous Integration

Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.

I've seen Continuous Integration work, and I believe in it. And I continue to believe that one of the biggest benefits of Continuous Integration is in the area of communication. As Fowler puts it:

Continuous Integration is all about communication, so you want to ensure that everyone can easily see the state of the system and the changes that have been made to it.

The big challenge for DVCS systems, in my opinion, is how they address this need, and I'm interested to see systems like Atlassian's Stash providing different approaches.

Sunday, October 7, 2012

Gone Girl: a very short review

Am I the last person in the world to read Gillian Flynn's sensational Gone Girl?

Certainly not; I see that it's still #4 on the best-seller charts. And, Donna hasn't read it yet, and perhaps she won't.

If you haven't read it yet, and are planning to, here's two thoughts:

  1. Read it!
  2. I'll try not to spoil it for you.

Gone Girl is a superbly-crafted, heart-racing roller-coaster ride. It's everything you've heard about it, and then some. It must be one of the oddest books to pin down, though: it's somehow simultaneously a psychological horror story and a romance, populated by two of the most lovable, vile monsters you may ever encounter.

It's important not to say too much, since, as with any thriller, you want to experience the twists and turns and surprises for yourself.

But I do want to comment on a few other things that caught my attention.

Firstly, this is a book of the moment. Flynn takes her unbelievable story and sets it solidly, firmly in the realm of reality by drinking deeply from the fountain of current events. The time is now, and what a precise now it is: the mall has closed, everyone has lost their jobs, all the houses are abandoned and foreclosed, and what have we done: we've discovered how to live our lives on the Internet:

It seemed to me that there was nothing new to be discovered ever again. Our society was utterly, ruinously derivative (although the word derivative as a criticism is itself derivative). We were the first human beings who would never see anything for the first time. We stare at the wonders of the world, dull-eyed, underwhelmed. Mona Lisa, the Pyramids, the Empire State Building. Jungle animals on attack, ancient icebergs collapsing, volcanoes erupting. I can't recall a single amazing thing I have seen firsthand that I didn't immediately reference to a movie or TV show. A fucking commercial. You know the awful singsong of the blase: Seeeen it. I've literally seen it all, and the worst thing, the thing that makes me want to blow my brains out, is: The secondhand experience is always better. The image is crisper, the view is keener, the camera angle and the soundtrack manipulate my emotions in a way reality can't anymore. I don't know that we are actually human at this point, those of us who are like most of us, who grew up with TV and movies and now the Internet. If we are betrayed, we know the words to say. If we want to play the stud or the smart-ass or the fool, we know the words to say. We are all working from the same dog-eared script.

This passage stands alone as one of the best eviscerations of our modern soul-less, soporific life that you'll read this year, but what's best about it, what elevates it above simply a demonstration of technique in another work-a-day thriller, is how positively hypnotic, evocative, and poetic it is. Perhaps it works best if you read it aloud ("ANC-ient ICE-bergs co-LLAPS-ing"), for then you will start to feel the meter and the rhythm: "those of us who are like most of us", "the image is crisper, the view is keener", "the stud or the smart-ass or the fool".

So, secondly, this is "just" a summer thriller, but it isn't pulp. Flynn is a writer of extraordinary skill; her work is elegant and multi-layered; her technique is sophisticated and powerful. It's odd, in a way, that she chooses to tell a story like this, with such skill; reflecting on the book days later, I found myself wondering why she chose to dwell in the gutter when her abilities let her soar with the eagles. I understand that this isn't her first book; I haven't read the others, but I do wonder where she will go next. Will she be another Stephen King, an artist of the highest level who simply chooses topics and subject matter that aren't usually associated with grace and talent?

Lastly, as a life-long game player, I enjoyed seeing a book so full of games. Of course the entire book, as with any psychological thriller, is a single extended mind-game, but Gone Girl is just jam-packed with games of all sorts. The treasure/scavenger hunts, with their riddles and nuances and puzzles, are only the most obvious example, so here are a few more:

  • At the bar:
    We were thinking of introducing a board game night, even though most of our customers were too old to be nostalgic for our Hungy Hungry Hippos, our Game of Life with its plastic cars to be filled with tiny plastic pinhead spouses
  • While wandering New York City:
    a path that took me past the giant LEGO block of the United Nations headquarters, the flags of myriad countries fluttering in the wind. A kid would like this, I thought. All the different colors, the busy memory game of matching each flag to its country. There's Finland, and there's New Zealand.
  • One of the lead characters is named Go!
Practically every page has an example: there are riddles and puzzles and double-meanings, some of them quite intricate (e.g., the "little brown house", which is in fact blue, but which Nick sees as brown because he imagines it as the home of Mr. Brown).

Now, gentle reader, I do want to caution you: this is a horror story. There are sections, multiple sections, that are painful to read. There is language as coarse as you will ever read. So don't jump into this book lightly.

But if you were thinking about reading Gone Girl, I'll add my voice: you won't regret it.

Saturday, October 6, 2012

Computers and Chess

I've been thinking about computers and chess again recently.

There's been a little flurry of negative news about computers in chess in the past few months. Notably, ESPN's Grantland website featured a major article about cheating in chess: Rooked: The Evolution of Cheating in Chess. The article focuses on a major scandal in scholastic chess at Thomas Jefferson High School for Science and Technology, but expands to consider the broader question of how the sport of competitive chess must evolve to address computer cheating:

So folks at the USCF are now trying to determine exactly how Smiley cheated, and how often. The answers could affect not only the severity of his punishment, but the future of computers in competitive chess.

It does seem that there has been a bit of an epidemic of bad behavior in the chess world recently. Earlier this summer, the World Chess Federation handed down its decision in the computer-assisted cheating scandal in France: Three Players Suspended for Conspiring to Cheat.

The cheating occurred during the Paris Open, the Bienne Open and the Chess Olympiad, all in 2010, and was intended to benefit SĂ©bastien Feller, a grandmaster. The other two players being punished, Arnaud Hauchard, a grandmaster and the French national team captain, and Cyril Marzolo, an international master, were part of a conspiracy to help Feller win, the federation announced in its decision on July 30.
The report from the FIDE ethics committee on the investigation is long and detailed, and in the end decisive, but it concerns itself strictly with the incident at hand and not with any of the larger issues about whether the sport should address these issues with procedural changes or other actions, though it concedes that the incident is of the highest concern:
The violation committed by Mr. SĂ©bastien FELLER, Mr. Arnaud HAUCHARD and Mr. Cyril MARZOLO is particularly serious: it was committed during the most important FIDE competition, the Chess Olympiad, by members of a national representative, using sophisticated methods, to gain remunerative advantages.
Note that FIDE handed down the maximum penalty specified in the FIDE Code of Ethics.

But the news isn't all negative, or at least, not so severely negative. Consider, for example, the interesting revelation in Nate Silver's new The Signal and the Noise: Why So Many Predictions Fail-but Some Don't that the crucial tipping point in the Kasparov vs. Deep Blue match of 1997 may have actually involved a bug, not a brilliancy, by the computer: Did a Computer Bug Help Deep Blue Beat Kasparov?.

Either at the end of the first game or the beginning of the second, depending on who’s telling the story, the computer made a sacrifice that seemed to hint at its long-term strategy.

Kasparov and many others thought the move was too sophisticated for a computer, suggesting there had been some sort of human intervention during the game. “It was an incredibly refined move, of defending while ahead to cut out any hint of countermoves,” grandmaster Yasser Seirawan told Wired in 2001, “and it sent Garry into a tizzy.”

Note that, just 15 years ago, the discussion was still all around whether humans were intervening to assist the computers, while now everyone understands that the computers need no help from us.

But I prefer not to think about computers and chess in this way; I don't like to see it as a contest between machine and man, some sort of John-Henry-vs.-the-Steam-Engine of our time.

Rather, I prefer to continue dreaming of the glories that lie ahead, as we continue to learn to use these tools to help ourselves become better thinkers.

Consider, for example, this wonderful story about the recent performance of the U.S. team at the World Chess Olympiad: Chess Grandmasters Take Lessons From … a Windows Machine?.

Elite players often consult a database of more than 5 million games to study an opponent’s style, analyze past games and plan their moves accordingly, much like a football team will review game film. The software the U.S. squad used to prepare for the Olympiad, which concluded earlier this month, is the Houdini Pro chess engine, and you can fire it up on any Windows machine.

The team used a pretty hefty Windows machine. But it was a Windows machine. Fifteen years ago, IBM needed a supercomputer to defeat world chess champion Gary Kasparov. But nowadays, you can push the boundaries of top chess with a run-of-the-mill computer server.

To prepare for the Olympics of chess, the U.S. team called upon the High Performance Computing Center at Texas Tech University, a school known as a chess powerhouse. Such is its stature that U.S. chess team member and grandmaster Alexander Onischuk is headed there to coach the school’s team. So you can see why the school decided to lend some computational heft to the team’s training...

The result? Nakamura's epic win over Kramnik at the Olympiad will surely be on the short list for the top games of the year.

But closer to home, it is softer and subtler aspects of computers and chess that I find most pleasing. For instance, my son, who has always been fascinated by chess but has never seriously taken it up, has become a lover of the Red Hot Pawn website, where he plays game after game with other chess lovers around the world.

These, then, are the sorts of "Computers and Chess" stories I hope to read more of: people using computers to play chess against other people; people using computers to practice their chess skills; people using computers to study and enjoy the superb works of art that are chess games played at the highest level.

Friday, October 5, 2012

A translation guide for the culturally impaired

Amazing what a difference 30 years can make.

For those youngsters among you who are baffled by the popular culture of the 1980's, The Morning News offers help: For the Price of a Dime.

A jukebox was a sort of open-source, do-it-yourself, coin-operated DJ-in-a-chrome-box found in restaurants and bars during that decade. Unlike with iPods, once the music began to play, everyone in the room was forced to listen to it. “Don’t put another dime in the jukebox/I don’t wanna hear that song no more,” the girl band sings. Whatever “that song” was, it reminded the Flirts of a lover best forgotten.

Russell Coutts goes airborne!

What better way to kick off Fleet Week 2012 than this great video of Russell Coutts capsizing the Team Oracle AC-45: The World’s Best Sailors Try Taming the World’s Meanest Sailboats

To distract you from the AC-45's, this year's Parade of Ships includes the USS Makin Island, the USS Spruance, and the USS Preble.

Wednesday, October 3, 2012

Perforce Git Fusion

At work, we're quite excited because we're launching a new product!

Whereas standard Perforce, like Subversion, is classified as a Centralized Version Control System, and other products, like Git, Mercurial, and Bazaar, are classified as Decentralized Version Control Systems, our new product is not so easy to categorize.

We feel that it blends the two approaches, providing benefits from both.

It's more than just Perforce; it's more than just Git; it's Perforce Git Fusion.

Check it out!

Tuesday, October 2, 2012

SHA-3 is Keccak

NIST have today announced the selection of the new SHA-3 Secure Hash Algorithm: NIST Selects Winner of Secure Hash Algorithm (SHA-3) Competition.

The winning algorithm, Keccak (pronounced “catch-ack”), was created by Guido Bertoni, Joan Daemen and Gilles Van Assche of STMicroelectronics and MichaĂ«l Peeters of NXP Semiconductors. The team’s entry beat out 63 other submissions that NIST received after its open call for candidate algorithms in 2007, when it was thought that SHA-2, the standard secure hash algorithm, might be threatened. Keccak will now become NIST’s SHA-3 hash algorithm.

A wealth of information is available at the Keccak home page.

Keccak is a sponge construction function. Sponge functions get their name from their alternating phases of absorbing and squeezing:

The sponge construction then proceeds in two phases: the absorbing phase followed by the squeezing phase.
  • In the absorbing phase, the r-bit input message blocks are XORed into the first r bits of the state, interleaved with applications of the function f . When all message blocks are processed, the sponge construction switches to the squeezing phase.
  • In the squeezing phase, the first r bits of the state are returned as output blocks, inter- leaved with applications of the function f . The number of output blocks is chosen at will by the user.

Keccak comes with a thorough Implementation Overview document, as well as a pseudo-code Specifications Summary.

The selection of Keccak is the culmination of nearly 5 years of extensive, world-wide competition and coordination; the NIST web site has all the details.

Bruce Schneier, whose Skein function was one of the finalists, pronounced himself quite pleased by the results, even though he also contends that the SHA-2 variants are still quite solid.

Monday, October 1, 2012

San Francisco Measure F

The San Jose Mercury News has an interesting article: Hetch Hetchy controversy: Could Yosemite's 'second valley' be restored?.

As always here in California, water is so precious that the battles over it have enormous consequences:

The group Restore Hetch Hetchy, the Sierra Club and other supporters of draining the reservoir say most of its 360,000 acre-feet of water storage can be replaced through more water conservation, recycling and storing water in other reservoirs, such as the massive Don Pedro Reservoir nearby. But critics, such as U.S. Sen. Dianne Feinstein and San Francisco Mayor Ed Lee, who calls the idea "insane," say water is a precious commodity in California and that no city should take the risk of giving up a reliable water supply. Silicon Valley business leaders also say that because two-thirds of Hetch Hetchy water customers live outside San Francisco, in places like Palo Alto and north San Jose, they should get to vote, too.