The island that almost wasn't

KQED's Bay Curious podcast series did a nice episode on the dredging of the canal that eventually formed the Oakland-Alameda estuary.

In the late 1800s, Oakland was coming into its own politically and economically, developing its own banks, businesses and shipping companies.

“That grows and grows so that Oakland, by the early 20th century, is really thumbing its nose at San Francisco,” Walker says.

In response to local lobbying, Congressmen worked to bring in millions of federal dollars to pay the Army Corps of Engineers to improve the harbor.

Read it here, or listen online.

ZooKeeper ZOO_SEQUENCE ZNONODE

How's that for putting the search terms right there in the post title?

I wrote a little ZooKeeper program, very simple, basically just the lock recipe out of the ZooKeeper examples. You know, the one where you create a sequenced child node inside the parent container node, and your child node represents your position in the line of tasks waiting to acquire the lock.

My program seemed to be working quite well, I was pleased. But once in a long while it would fail, with an error message stating that it got a "no node" error when trying to create a sequenced child node.

Impossible! I said, for I knew that I had successfully created such a node just a few moments earlier, and I definitely hadn't run any of my own code which deleted the lock container node. So why wasn't it there?

After a while, I found a line in my ZooKeeper leader's log:

[ContainerManagerTask:o.a.z.s.ContainerManager@135] - Attempting to delete candidate container: /path/to/lock/node

That led me to the ContainerManager documentation, which in turn reminded me to check this note in the ZooKeeper documentation, which described my mistake precisely:

Container Nodes
Added in 3.5.3

ZooKeeper has the notion of container znodes. Container znodes are special purpose znodes useful for recipes such as leader, lock, etc. When the last child of a container is deleted, the container becomes a candidate to be deleted by the server at some point in the future.

Given this property, you should be prepared to get KeeperException.NoNodeException when creating children inside of container znodes. i.e. when creating child znodes inside of container znodes always check for KeeperException.NoNodeException and recreate the container znode when it occurs.

On my system, ContainerManager seems to make this check about once a minute, which meant that every so often, after enough minutes and enough use of my program, ContainerManager would delete the lock node just when I was about to try to create a new sequenced child node.

Voila!

I love this quote about the xz compression utilities event

From Did One Guy Just Stop a Huge Cyberattack? :

In the cybersecurity world, a database engineer inadvertently finding a backdoor in a core Linux feature is a little like a bakery worker who smells a freshly baked loaf of bread, senses something is off and correctly deduces that someone has tampered with the entire global yeast supply. It’s the kind of intuition that requires years of experience and obsessive attention to detail, plus a healthy dose of luck.

Ah yes, database engineers.

Yes, that's a profession that definitely involves "years of experience and obsessive attention to detail, plus a healthy dose of luck". Spot on!

There are still a lot of mysteries left about what exactly happened with xz, and this article is just a high-level summary.

But this database engineer loved that quote!

Why does housing cost so much?

Here's a very interesting article on the underlying costs of building a single family home. Well-researched, clearly-written, lots of detail about why it costs to much to build a new house.

Because of the enormous costs of housing, it's worth understanding where, specifically, those costs come from, and what sort of interventions would be needed to reduce these costs. Discussions of housing policy often focus on issues of zoning, regulation, and other supply restrictions which manifest as increased land prices, but for most American housing, the largest cost comes from building the physical structure itself. However, in dense urban areas — the places where building new housing is arguably most important — this changes, and high land prices driven by regulatory restrictions become the dominant factor.

I really enjoyed the entire article, lots of fascinating observations and lots of additional links to chase for those who want to learn more. So I wanted to share it.