Random links for August 1st

With all the big issues going on nowadays, a lot of this seems pretty trivial. Still, I find things like this interesting, so I put them on my blog.

• 'We need more': Fight against Ebola virus thin on the ground

  So there must be a cast of thousands in there, deploying equipment, medications and vaccines, and dispensing advice, right?

  Wrong.

• How we treat Ebola

  When Ebola haemorrhagic fever broke out recently in Guinea, West Africa, MSF set up three specialised treatment centres in the worst-hit areas. Ebola is so infectious -- and so deadly -- that patients need to be treated in isolation by staff wearing special protective clothing. Emergency coordinator Henry Gray and logistician Pascal Piguet, both just back from Guinea, explain why, with Ebola, every little detail counts.

• NATO's Underground Roman Super-Quarry

  There is an underground Roman-era quarry in The Netherlands that, when you exit, you will find that you have crossed an invisible international border somewhere down there in the darkness, and that you are now stepping out into Belgium; or perhaps it's the other way around, that there is an underground Roman-era quarry in Belgium that, when you exit, you will find that you have crossed an invisible international border somewhere down there in the darkness, and that you are now stepping out into The Netherlands.

• Life on the Subsurface: An Interview with Penelope Boston

  Boston has worked with the NASA Innovative Advanced Concepts program (NIAC) to develop protocols for both human extraterrestrial cave habitation and for subterranean life-detection missions on Mars, life which she believes is highly likely to exist.

• Living up to Your (Business) Ideals

  If you want to live up to your business ideals, you have to take the time to authentically identify your values, the things you care about. You also have to commit to the ongoing tending and cultivation of those values in your organization. It is not a “set it and forget it” scenario.

• Being Profitable

  Maybe time is the most important factor for you. How much can everyone work? How much does everyone want to work? How much must you then charge for that time to end up with salaries you can be content with?

• git Flight Rules

  Flight Rules are the hard-earned body of knowledge recorded in manuals that list, step-by-step, what to do if X occurs, and why. Essentially, they are extremely detailed, scenario-specific standard operating procedures.

• Hybrid Logical Clocks

  Physical Time (PT) leverages on physical clocks at nodes that are synchronized using the Network Time Protocol (NTP). PT also has several drawbacks. Firstly, in a geographically distributed system obtaining precise clock synchronization is very hard; there will unavoidably be uncertainty intervals. Secondly, PT has several kinks such as leap seconds and non-monotonic updates. And, when the uncertainty intervals are overlapping, PT cannot order events and you end up with inconsistent snapshots as the one shown below.

• A Quick Introduction to CoreOS

  CoreOS, in case you haven’t heard of it, is a highly streamlined Linux distribution designed with containers, massive server deployments, and distributed systems/applications in mind.

• Touch events on the pathfinding pages

  For my pathfinding pages I wanted to support "painting" on the map to make or erase walls. When you change the map, the pathfinding algorithm updates the paths.

• Tom Brady is the loneliest quarterback on the planet

  I thought, "the three-time Super Bowl winner and one of his wide receivers trying to high-five and missing each other's hands? That's pretty funny!" Oh no. What is funnier still is Brady trying to high-five one or more of his teammates and the other players totally ignoring him. What's even funnier than that? This has happened over and over again.

Desert news

Since I happened to be in the desert last week, I've been somewhat sensitive to desert news.

So I found these interesting:

• Sahara Dust Produces Massive Bahama Carbon Sink

  This post was inspired by a terrific new and important paper that speaks to the role of Saharan dust, the iron and other mineral micronutrients it carries to the ocean, and how this results in a new explanation of the power and potency of ocean photosynthesis in regulating global CO2. Once again we find that the living ocean is not merely a chemical test tube but rather a complex living ecology. But sometimes the chemistry shines through as well or I should say the bio-chemistry.

  In short the authors show that when Sahara dust arrives in the Bahamas cyano-bacteria, what we used to call blue-green algae, bloom. As they bloom their photosynthesis removes CO2 from the water making the pH locally rise, alleviating ocean acidification. That blooming rise of ocean pH to a slightly more alkaline state results in what the Bahamanian’s have long called “Ocean Whitings” where the ocean becomes white like milk.

• Pioneering environmental project taking root in the sand

  A one hectare site outside Doha, Qatar, will soon host the SFP AS Pilot Plant. It will contain a unique combination of promising environmental technologies carefully integrated in a system to maximize beneficial synergies. A wide specter of leading experts and technology developers are taking part in the effort to realize the project. A cornerstone of the pilot is greenhouses utilizing seawater to provide cool and humid growing conditions for vegetables. The greenhouses will also produce freshwater themselves. The greenhouses will be coupled with a state of the art parabolic trough solar collector with a thermal desalination unit supported by PV-technology. The pilot will also allow for cultivation of algae in a system of photobioreactors and open pond cultivation systems.

  An important part of the pilot is to demonstrate the potential for cultivating desert land and making it green. Outdoor vertical evaporators will create sheltered and humid environments for cultivation of plants. Additionally, the pilot will contain outdoor hydroponic raceways for cultivation of halophytes – plants tolerant of irrigation with salty water. The facilities in the pilot plant will be supported by on-site laboratories, scientists and professional growers.

I have this feeling that the Sahara Forest Project team meant "wide spectrum", not "wide specter", yes? Or maybe they wanted to say "broad coalition"?

Anyway, both pages and their supporting material are fascinating...

Backpacking 2014: Matlock Lake, John Muir Wilderness

It was that time of year again, so I packed up the pack, loaded up the car, and headed off with the gang.

We spent the night in Bishop so we could get an early start, and by 9:00 AM we were at the Onion Valley Trailhead.

Our destination was Matlock Lake, in the John Muir Wilderness Area.

The John Muir Wilderness Area is the most spectacular of all the central Sierra Nevada wilderness areas, and is also the location of the world famous John Muir Trail, surely the most amazing backpacking trail in the lower 48 states. Our trip didn't actually take us on the John Muir Trail, but we came very close to it, and there were many hikers on our trail headed to and from the JMT/PCT (in this area they are the same trail).

Although our hike was short, the elevation gain and overall altitude was substantial: the Onion Valley trailhead is at 9,200 feet, and our campsite was at 10,600 feet; with the various ups and downs of the trail we found that Deb's FitBit registered an astonishing 180 staircases at the completion of the first day's hike.

So we were good and tired at the end of the first day!

We were lucky enough to have clear skies, fair weather, and a new moon, giving us near perfect star gazing and an active evening debate about whether or not we were seeing Iridium Flashers.

On the second day we took a cross-country scramble to nearby Bench Lake, a gorgeous hidden lake which is about 300 feet above Matlock Lake. Bench Lake was beautiful and secluded, and the views were marvelous.

On our third day, we took a trip up and over Kearsarge Pass. At 11,800 feet, this pass was the highest I've been on foot in many years, perhaps decades. Sitting at the pass is an astonishing spectacle, as you can see, simultaneously, more than 60 miles to the east, across the Owens Valley and beyond, and more than 25 miles to the west, down through Kings Canyon National Park and beyond.

The trail to the pass is well-maintained and pleasant (it has to be, as it is a major mule train pack trail), but the conditions at the pass are not suited for long periods of relaxation; as another hiker at the pass commented, "above 10,000 feet in the Sierras there are only two situations that apply: either the sun is out and you are too hot, or there are clouds and you are too cold."

Indeed it is true.

Some of our party extended the hike by dropping down to visit the Kearsarge Lakes and then returning via the pass, while I made it a shorter day by returning straight to Matlock Lake where I had time for an afternoon dip in the lake.

The lakes in this are are at the forefront of a concerted campain to save the mountain yellow-legged frog:

Mountain yellow-legged frogs in the California Sierra Nevada are disappearing at an alarming rate, primarily due to a virulent fungus. UCSB scientists Cheryl Briggs and Roland Knapp are racing to understand why some populations of frogs succumb and others survive, with the aim of not only saving the frogs but also gaining knowledge of how and why organisms develop resistance to virulent pathogen attack.

Of course, your perspective on this may depend on where you are standing: Feds to list frogs as ‘endangered’

A final decision on the critical habitat proposal is expected to be made early next year, but the proposals were met with opposition in the Eastern Sierra, where residents said there is a fear that the designations will close off backcountry access (or at least appear to) and negatively impact the local, tourist-based economy. Some of those fears were validated during the public comment process when it was brought to light that the California Department of Fish and Wildlife had been removing trout from backcountry lakes for several years in an effort to protect the frogs and prevent an endangered species or critical habitat listing.

With fishing season kicking off today, anglers seeking solitude in the remote reaches of the Sierra Nevada are being advised that several lakes that were once thriving fisheries have been cleared of all trout to protect the frogs, which are eaten by the fish in their tadpole form.

Sure enough, there was not a fish to be found in the lakes we visited. As the Inyo Register observed:

In the Independence area, the DFW has removed trout from Bench, Matlock and Slim lake but the higher-elevation waters continue to produce trout. "We have no plans to remove fish from any of those."

It's not clear which higher-elevation waters they mean.

But we definitely saw lots of tadpoles and frogs, so that part of the program is certainly working!

And there is plenty of other life to see in the woods. With campfire restrictions nearly universal in California this year, we were limited to enjoying the wilderness around our camp stove, but I think this is a good thing, for Dead Trees Are Anything But Dead.

After three beautiful days in the wilderness, it was time to get back to coding, so we broke camp and had a pleasant walk down the hill to our cars.

I think it was a good thing we left as we did, for between the massive storm that came racing up from the south and the forest fire near El Portal, we were looking back over our shoulders at storm clouds and smoky skies during our drive home.

Although we might have wished for a bit more solitude, overall this was a near perfect trip for us: the weather was perfect, the scenery was glorious, everything went just as you would hope a backpacking trip would go.

If you're looking for a great place to go backpacking, and you haven't yet tried the Kearsarge Pass trail out of the Onion Valley Trailhead, you should put it on your list.

Transactional replicated databases using consensus algorithms

A trendy thing nowadays is to build a transactional replicated database without using two phase commit, but using a different type of consensus algorithm.

The gold standard in this area is Google's F1 database, which is built on top of their Spanner infrastructure, which in turn is built on their production-quality implementation of the Paxos consensus algorithm.

Now other, similar systems are starting to emerge, demonstrating that one way to get attention in the world is to hire some Google engineers who worked on-or-near Spanner and/or F1 and/or Paxos, and build something yourself.

• Man Busts Out of Google, Rebuilds Top-Secret Query Machine

  Under development for the past two years, Impala is a means of instantly analyzing the massive amounts of data stored in Hadoop, and it’s based on a sweeping Google database known as F1. Google only revealed F1 this past May, with a presentation delivered at a conference in Arizona, and it has yet to release a full paper describing the technology. Two years ago, Cloudera hired away one of the main Google engineers behind the project, a database guru named Marcel Kornacker.

• In Search of an Understandable Consensus Algorithm (Extended Version)

  Raft is a consensus algorithm for managing a replicated log. It produces a result equivalent to (multi-)Paxos, and it is as efficient as Paxos, but its structure is different from Paxos; this makes Raft more understandable than Paxos and also provides a better foundation for building practical systems.

• Introducing Ark: A Consensus Algorithm For TokuMX and MongoDB

  Ark is an implementation of a consensus algorithm (also known as elections) similar to Paxos and Raft that we are working on to handle replica set elections and failovers in TokuMX. It has many similarities to Raft, but also has some big differences.

• Ark: A Real-World Consensus Implementation

  Ark was designed from first principles, improving on the election algorithm used by TokuMX, to fix deficiencies in MongoDB’s consensus algorithms that can cause data loss. It ultimately has many similar- ities with Raft, but diverges in a few ways, mainly to support other features like chained replication and unacknowledged writes.

• Out in the Open: Ex-Googlers Building Cloud Software That’s Almost Impossible to Take Down

  But if anyone is up for the challenge of rebuilding Spanner—one of the most impressive systems in the history of computing—it’s the CockroachDB team. Many of them were engineers at Google, though none of them worked on Spanner.

• Cockroach: A Scalable, Geo-Replicated, Transactional Datastore

  Cockroach is a distributed key/value datastore which supports ACID transactional semantics and versioned values as first-class features. The primary design goal is global consistency and survivability, hence the name. Cockroach aims to tolerate disk, machine, rack, and even datacenter failures with minimal latency disruption and no manual intervention. Cockroach nodes are symmetric; a design goal is one binary with minimal configuration and no required auxiliary services.

• Cockroach

  Single mutations to ranges are mediated via an instance of a distributed consensus algorithm to ensure consistency. We've chosen to use the Raft consensus algorithm. All consensus state is stored in RocksDB.

Just so there's no confusion, let me be clear about one thing:

I am not building a consensus algorithm.

But I enjoy reading about consensus algorithms!

Walter Munk's wave experiment

The NPR website is carrying a nifty article about Walter Munk: The Most Astonishing Wave-Tracking Experiment Ever.

Yes, I'm asking a wave to tell me where it was born. Can you do that? Crazily enough, you can. Waves do have birthplaces. Once upon a time, one of the world's greatest oceanographers asked this very question.

Munk's experiment was not easy to carry out:

From a beach you can't see an old set of swells go by. They aren't that noticeable. Walter and his team had highly sensitive measuring devices that could spot swells that were very subtle, rising for a mile or two, then subsiding, with the peak being only a tenth of a millimeter high.

But what a fascinating result:

The swells they were tracking, when they reached Yakutat, Alaska, had indeed traveled halfway around the world. Working the data backward, Walter figured that the storm that had generated those swells had taken place two weeks earlier, in a remote patch of ocean near a bunch of snowy volcanic islands — Heard Island and the McDonald Islands, about 2,500 miles southwest of Perth, Australia.

Neat article, and neat to learn about Professor Munk, who I hadn't known of previously.

I wonder if he'd enjoy a visit to see the University of Edinburgh's FloWave simulator?.

What an innocuous headline...

7 safe securities that yield more than 4%

Eveans and his team analyze more than 7,000 securities worldwide but only buy names that offer payouts no less than double the yield of the overall stock market — as well as reasonable valuation and competitive advantages that will keep earnings growing over time.

Sounds like a pleasant article to read, no?

Well, it turns out that the companies that they are recommending you invest in are:

• Cigarette companies (Philip Morris)
• Oil companies (Vanguard Resources, Williams Partners)
• Leveraged buyout specialists (KKR)

I guess the good news is that they didn't include any arms dealers or pesticide manufacturers.

git clone vs fork

Two words that you'll often hear people say when discussing git are "fork" and "clone".

They are similar; they are related; they are not interchangeable.

The clone operation is built into git: git-clone - Clone a repository into a new directory.

Forking, on the other hand, is an operation which is used by a certain git workflow, made popular by GitHub, called the Fork and Pull Workflow:

The fork & pull model lets anyone fork an existing repository and push changes to their personal fork without requiring access be granted to the source repository. The changes must then be pulled into the source repository by the project maintainer. This model reduces the amount of friction for new contributors and is popular with open source projects because it allows people to work independently without upfront coordination.

The difference between forking and cloning is really a difference in intent and purpose:

• The forked repository is mostly static. It exists in order to allow you to publish work for code review purposes. You don't do active development in your forked repository (in fact, you can't; because it doesn't exist on your computer, it exists on GitHub's server in the cloud).
• The cloned repository is your active repo. It is where you do all your work. But other people generally don't have access to your personal cloned repo, because it's on your laptop. So that's why you have the forked repo, so you can push changes to it for others to see and review

This picture from StackOverflow helps a lot: What is the difference between origin and upstream in github.

In this workflow, you both fork and clone: first you fork the repo that you are interested in, so that you have a separate repo that is clearly associated with your GitHub account.

Then, you clone that repo, and do your work. When and if you wish, you may push to your forked repo.

One thing that's sort of interesting is that you never directly update your forked repo from the original ("upstream") repo after the original "fork" operation. Subsequent to that, updates to your forked repo are indirect: you pull from upstream into your cloned repo, to bring it up to date, then (if you wish), you push those changes into your forked repo.

Some additional references:

• Fork A Repo

  When a repository is cloned, it has a default remote called origin that points to your fork on GitHub, not the original repository it was forked from. To keep track of the original repository, you need to add another remote named upstream.

• Stash 2.4: Forking in the Enterprise

  In Stash, clicking the ‘Fork’ button on a repository creates a copy that is tracked by Stash and modified independently of the original repository, insulating code from the original repository to unwanted changes or errors.

• Git Branching and Forking in the Enterprise: Why Fork?

  In recent DVCS terminology a fork is a remote, server-side copy of a repository, distinct from the original. A clone is not a fork; a clone is a local copy of some remote repository.

• Clone vs. Fork

  if you want to make changes to any of its cookbooks, you will need to fork the repository, which creates an editable copy of the entire repository (including all of its branches and commits) in your own source control management (e.g. GitHub) account. Later, if you want to contribute back to the original project, you can make a pull request to the owner of the original cookbook repository so that your submitted changes can be merged into the main branch.