1491: A very short review

Charles Mann would certainly be the first to admit that 1491: New Revelations of the Americas Before Columbus is now getting "old".

It was published 15 years ago, after all, and even though Mann has revised it a tad, that's a significant period of time, and the revelations are certainly no longer new.

With a (slightly) deeper interpretation, however, one of Mann's major points is that books such of these are of necessity always old, even the moment after they are written:

Meanwhile, new disciplines and new technologies were creating new ways to examine the past. Demography, climatology, epidemiology, economics, botany, and palynology (pollen analysis); molecular and evolutionary biology; carbon-14 dating, ice-core sampling, satellite photograph, and soil assays; genetic microsatellite analysis and virtual 3-D fly-throughs -- a torrent of novel perspectives and techniques cascaded into use. And when these were employed, the idea that the only human occupants of one-third of the earth's surface had changed little for thousands of years began to seem implausible. To be sure, some researches have vigorously attacked the new findings as wild exaggerations. ("We have simply replaced the old myth [of untouched wilderness] with a new one," scoffed geographer Thomas Vale, " the myth of the humanized landscape.") But after several decades of discovery and debate, a new picture of the Americas and their original inhabitants is emerging.

When I started reading 1491, I had no idea what genetic microsatellite analysis was, and I had to look it up (this is a pretty approachable overview), but 1491 isn't really a book about how anthropologists, historians, and other similar disciplines operate.

Rather, it's (at least) two other books:

• It's a lively and entertaining look at what we currently know about what the Americas were when the enormous waves of immigration from Renaissance Europe began in the late 1400's
• But it's also an attempt to help you, the reader, become a better consumer of the information you receive about how we got to this point, how the world is changing, and what it might mean for what you think you know.

Much of 1491 flies by, Mann leads us at a frenetic pace through much memorable history:

Dazzled as he was, Cortes was also aware that with a single command Motecuhzoma could order his army "to obliterate all memory of us." The Spaniards counteracted this thread by inventing a pretext to seize the tlatoani in his own palace, making him first their captive and then their puppet.

In both Europe and Mesoamerica kings ruled by the dispensation of the heavens. The Mexica reacted to the sacrilegious abduction of their leader with the same baffled horror with which Europeans later reacted to Cromwell's execution of Charles I in 1649. Not wanting to act in a way that could result in Motecuhzoma's death, the Mexica took seven months to mount a counterattack. Fearing the worst, the debased tlatoani made a begging public appearance on behalf of the Spanish. He soon died, either murdered by the Spaniards (according to Mexica accounts) or slain by his own countrymen (as Spanish chronicles tell it). Soon after came the long-delayed assault. Under the leadership of a vigorous new tlatoani, Cuitlahuac, the Indians force the invaders into narrow alleys where horses were of little advantage. Under a pitiless hail of spears, darts, and arrows, Cortes and his men retreated down the long causeways that linked the island city to the mainland. In a single brutal night the Mexica utterly vanquished Cortes, killing three-quarters of his men. Although the Alliance destroyed causeways in front of the Spaniards, the remnants of the invaders were able to cross the gaps because they were so choked with the dead that the men could walk on the bodies of their countrymen. Because the Mexica did not view the goal of warfare as wiping out enemies to the last man, they did not hunt down the last Spaniards. A costly mistake: Cortes was among the escapees.

A man of unfathomable determination, Cortes never thought of giving up. He persuaded several other vassal states to join his anti-Alliance alliance with Tlaxcala. Negotiating furiously, he assembled a force of as many as 200,000 men and built thirteen big ships in an audacious plan to assault Tenochtitlan from the water.

...

When Cortes and his Indian allies finally attacked, the Mexica resisted so fiercely despite their weakness that the siege has often been described as the costliest battle in history -- casualty estimates range up to 100,000.

Yet, I suspect that Mann began his book fascinated by the first topic, and set out to write that "first book," but along the way he became even more fascinated by how challenging it is to avoid thinking that we now know everything there is to know about the past, and so he ended up writing that "second book" as well, trying to open our eyes to just how little we know, and just how alert we should be to the possibility that what we think we know, we do not in fact know at all:

At first he did nothing about his observation. Historical demography was not supposed to be his field. Six years later, in 1959, he surveyed more archives in Hermosilla and found the same disparity. By this point he had almost finished his doctorate at Cornell and had been selected for Holmberg's project. The choice was almost haphazard: Dobyns had never been to Peru.

Peru, Dobyns learned, was one of the world's cultural wellsprings, a place as important to the human saga as the Fertile Crescent. Yet the area's significance had been scarcely appreciated outside the Andes, partly because the Spaniards so thoroughly ravaged Inka culture, and partly because the Inka themselves, wanting to puff up their own importance, had actively concealed the glories of the cultures before them. Incredibly, the first full history of the fall of the Inka empire did not appear until more than three hundred years after the events in chronicled: William H Prescott's History of the Conquest of Peru, published in 1847.

Read 1491 because it's wildly entertaining and endlessly engrossing.

But remember 1491 because it makes you a better reader, and a better thinker, in general.

One way or another, however, read 1491.

Updates from The Ocean Cleanup Project

Boyan Slat and the team at The Ocean Cleanup Project are working along, tinkering and learning.

Here's their latest report: Into the Twilight Zone

First, they remind us where they were a year ago:

Our first attempt at doing so was deployed last year: System 001, also known as Wilson. After months of testing, we took Wilson back to port in the first days of this year after it suffered a fatigue fracture. This was not ideal, but both the diagnosis and solution came quite easily.

And then, they bring us up to date on where they are now:

The more complicated challenge was the system’s inability to retain plastic; instead of consistently going faster than the plastic, it alternated between going faster and going slower than the plastic. This meant plastic would float into the system, as planned, but then float out again.

...

We launched System 001/B in late June, which was followed by a six-week testing campaign to test slowing down the system using a parachute anchor and test speeding up the system using large inflatable buoys.

...

the winning concept is the slow-down approach, in which we use a parachute anchor to slow down the system as much as possible, allowing the natural winds and waves to push the plastic into the system.

...

there’s always a twist in each episode; well, here’s ours: the plastic is currently able to cross over the cork line into The Twilight Zone. While it is technically still within the boundaries of the system, there is no screen underneath the floater pipe, so we cannot consider this plastic caught because it is not securely retained in front of the screen.

...

we will now be using three rows of 32 cm floats stacked on top of each other, creating a total height of about half a meter.

It's a wonderful article, with great diagrams and deeper explanations throughout.

This is incremental engineering at its best: start with something; it does some things properly but fails in other ways; test, improve, repeat.

I'm looking forward to more great updates!

Scrappers

I was completely enthralled by Scrappers: The Big Business of Scavenging in PostIndustrial America.

Hypnarowski later told me that his company, New Enterprise Stone and Lime, also owned some of the land where Bethlehem Steel once existed. The steel mills had been scrapped long ago, but there were still nuggets to be had — or “buttons” to be precise. Buttons are essentially giant metal boulders that weigh as much as 20 tons. When the mills were still operating, iron ore was melted and poured into great big ladles, at which point the less desirable slag would form at the bottom. This slag was then dumped onto Lake Erie’s shoreline, where it hardened and formed buttons. Together, Hypnarowski and Levin worked to salvage these buttons from the lakeside. They had, it seemed, thought of every conceivable way to mine big scrap. Over time, scrappers have remade Buffalo’s landscape. The city has survived, in part, by devouring itself.

Back when steel mills first closed, Lou Jean Fleron, an emeritus professor at Cornell’s school of industrial and labor relations, ran a series of educational programs for the workers who had been laid off. She got close with the families that became destitute. It was a very hard time, she recalled, and whenever she visited Buffalo’s waterfront, her eyes inevitably drifted toward the derelict mills. “Oh, God, it was like a ghost town — like a skeleton — a big, massive black skeleton,” she recalled. Then the demolition crews and the scrappers arrived to do their work. Now when Fleron goes down to the waterfront, she sees young families with their children having birthday parties. The scene is almost pastoral.

“It was important to take it all down,” Fleron told me. “It does make some of the pain go away.”

The hero of the story is the incredibly hard-working Adrian Paisley:

Paisley typically takes scrap from his piles and moves it inside his garage, where he processes it. This is where Paisley makes his money, by extracting the most valuable nuggets. The air-conditioner that he found, for example, was promising because it contained copper tubing, copper wiring and an ACR (an aluminum-copper radiator). The scrapyard might pay him only $4 to $6 for the air-conditioner in its current form, but if he processed it and removed the copper, he might earn three times as much. For this reason, Paisley spends much of his day surgically removing the most valuable metals. He even removes each screw and sells them together in bulk. Scrapyards are willing to pay a premium for scrap like this because it saves them the trouble of having to process it themselves.

It was great to read of Paisley's dream of one day no longer having to do this back-breaking work:

It was all crystal clear in his mind: “I want to see the fog hovering across the ground on a nice cool fall morning. And I don’t want to hear nothing but the birds, and the insects chirping. I want to stand there, man, and drink my coffee and look at the fog. Peaceful.

Amen, Mr. Paisley.

This. Also, that.

School has started, here in the Bay Area.

Naturally, I have the sniffles, a scratchy throat, and a sinus headache.

Must. Sleep.

But, in the meantime, there's so much to read!

• What is Haberman?

  I’d never heard of “Haberman” before. The name of the neighborhood that people who live here would recognize is Maspeth (which you can see up-and-to-the-right of Haberman). Is Haberman even a real neighborhood? Why did Google put this giant Haberman label on the map?

• The Pin Is Mightier: Why it’s so satisfying to find—and make—fake locations in Google Maps.

  It’s chaos that Google appears hard-pressed to stop. In a June 2019 blog post, the company says it took down 3 million fake business profiles in 2018, around 85 percent of which were flagged by internal systems, and 90 percent of which were removed before users could see them. Google did not provide statistics on how many new businesses get added a year, or how many listings appear on Google Maps, but given that the service includes data from 220 countries, 3 million listings is likely a drop in the bucket. A Google spokesperson says the company has a team dedicated to Maps fraud, and has “strict policies in place” to detect fraud through “manual and automated systems,” but declined to reveal further details “so as not to tip off spammer or others with bad intent.”

• Algorithms, by Jeff Erickson

  This web page contains a free electronic version of my self-published textbook Algorithms, along with other lecture notes I have written for various theoretical computer science classes at the University of Illinois, Urbana-Champaign since 1998.

• Highlights from Git 2.23

  The open source Git project just released Git 2.23 with features and bug fixes from over 77 contributors, 26 of them new. Here’s our look at some of the most exciting features and changes introduced since Git 2.22.

• Hiring is Broken: What Do Developers SayAbout Technical Interviews?

  Posters report that these interviews cause unnecessary anxiety and frustration, requiring them to learn arbitrary, implicit, and obscure norms. The findings from our study inform inclusive hiring guidelines for technical interviews, such as collaborative problem-solving sessions.

• Tech Interview Handbook: Carefully curated content to help you ace your next technical interview

  The Tech Interview Handbook contains carefully curated content to help you ace your next technical interview with a focus on algorithms. While there are a ton of interview resources on the internet, the best ones are either not free, or they do not cover the complete interview process, usually only focusing on algorithms.

• How to Build Good Software

  Some core operating principles that can dramatically improve the chances of success:

  1. Start as simple as possible;
  2. Seek out problems and iterate; and
  3. Hire the best engineers you can.

  While there are many subtler factors to consider, these principles form a foundation that lets you get started building good software.

• Every productivity thought I've ever had, as concisely as possible

  I combed through several years of my private notes and through everything I published on productivity before and tried to summarize all of it in this post.

• They Get Fired All the Time. And They Have No Idea Why.

  Through weeks of intensive research, a singular truth emerged. People with Asperger’s syndrome, the term still commonly used for one of the most well-known forms of autism spectrum disorder, bring serious advantages to the financial markets: extreme focus, a facility with numbers, a willingness to consider unpopular opinions, a strong sense of logic, and an intense belief in fairness and justice. But, like other autistic employees, they often feel alienated from their managers, colleagues, and clients. Sometimes they simply get fired.

• A Walk In Hong Kong

  All that prelude is to say, coming in to the Hong Kong protests from a less developed country like the United States is disorienting. If you have never visited one of the Zeroth World cities of Asia, like Taipei or Singapore, it can be hard to convey their mix of high density, mazelike design, utterly reliable public services, and high social cohesion, any more than it was possible for me or my parents to imagine a real American city, no matter how many movies we saw. And then to have to write about protests on top of it!

• WeWTF

  In frothy markets, it's easy to enter into a consensual hallucination, with investors and markets, that you’re creating value. And it’s easy to wallpaper over the shortcomings of the business with a bull market's halcyon: cheap capital. WeWork has brought new meaning to the word wallpaper.

Weird ... what happened to the Forest Service website?

I can't get to any link for the United States Forest Service websites anymore.

All the various ranger stations, forest service campgrounds, etc., used to be at fs.usda.gov website addresses, but now all the web browsers just say:

fs.usda.gov’s server IP address could not be found.

Somebody messed up a DNS record somewhere and thousands of web pages vanished?

For example, go here and try clicking on any of the Geo-enabled PDFs in the "Topographic Maps" section.

Up, up, and away

Yesterday they welcomed buses back to the TransBay Bus Terminal, aka Salesforce Park.

Here's a nice story in Wired with lots of pictures and fascinating details: How 'Microcracks' Undermined San Francisco's New Bus Terminal

the construction process involved cutting away 8 inches of the flange right where it needed to be strongest. “The problem was the geometry of the weld access hole,” Frank says. “It has this corner on it, and it acts as a stress concentrator.”

The holes weren’t circular—they were rectangular, with rounded edges. And those corners, probably cut with a plasma cutter, acquired “microcracks” just a few hundredths of an inch deep. The investigators know they were cut with something hot, because the surfaces of the cracks were coated with a colored deposit, an oxide that could only have resulted from exposure to high heat. “You can actually see it,” Vecchio says. “It’s a very deep red, as opposed to what regular rusting of steel looks like, which is going to be more orange in color.”

Much of this story has been told before, but the Wired article is well-written and has some great pictures from the forensic studies of the failed girder.

Some bits are quite new to me:

The ability of something—steel, in this case—to resist fracture after it cracks is called “fracture toughness.” It’s measured with what’s called a Charpy impact test, basically a very precise banging on the metal until it breaks. According to spec, the steel in the Transbay Terminal was supposed to absorb 20 foot-pounds of energy before it fractured at room temperature. It did, but testing by LPI showed lower toughness deeper inside the steel. That’s where the pop-in cracks formed

And this new information appears to suggest an explanation for one of the most tantalizing questions: why did the beams fail in one part of the structure, but not in another very similar part:

That may also explain why the girders over Fremont Street cracked, but the ones over First Street did not. “The difference was the sequence of construction,” Engelhardt says. “On First Street, the welds were made first, and the holes were made after. On Fremont Street, the holes were made first. That turned out to be the decisive difference.”

This morning, the buses were again running in the terminal.

Let's hope things continue to work well now.

Stuff I'm reading, mid-summer edition

Here you go, stuff you might or might not find interesting, too.

• DMV Strike Team Final Report

  the DMV is facing historic and unprecedented surges in demand. This increased demand has been primarily driven by the following:

  • Stringent federal identification requirements to obtain a REAL ID, which require an in-person visit to a DMV field office.
  • Motor Voter opt-out requirements, which have greatly increased the workload of DMV staff.

  The Strike Team focused its efforts on improving the processes surrounding REAL ID, with the expectation that increased efficiencies in REAL ID transactions would then translate to other areas of the DMV.

  It is clear that changes are essential if the DMV is to meet its most immediate challenge: successfully meeting increased demand for REAL ID driver licenses before the October 1, 2020 federal deadline

• The Best Refactoring You've Never Heard Of

  We did a four step process to go from the recursive to the iterative version. First, we made it CPS so these functions appeared. And then we defunctionalized those functions. So now we have these two mutually recursive functions passing around this continuation object. We inlined one to the other. Now it's tail recursive; it becomes a loop. But, of course, the inlining is just kind of moving things around, so we can do the tail-recursion elimination. The big insight that made this all possible, the real workhorse of this transformation, was to defunctionalize the continuation!

• Cryptographic Attacks: A Guide for the Perplexed

  Over the years, the landscape of cryptographic attacks has become a kudzu plant of flashy logos, formula-dense whitepapers and a general gloomy feeling that everything is broken. But in truth, many of the attacks revolve around the same few unifying principles, and many of the interminable pages of formulas have a bottom line that doesn’t require a PhD to understand.

• Operating a Large, Distributed System in a Reliable Way: Practices I Learned

  This post is the collection of the practices I've found useful to reliably operate a large system at Uber, while working here. My experience is not unique - people working on similar sized systems go through a similar journey. I've talked with engineers at Google, Facebook, and Netflix, who shared similar experiences and solutions. Many of the ideas and processes listed here should apply to systems of similar scale, regardless of running on own data centers (like Uber mostly does) or on the cloud (where Uber sometimes scales to). However, the practices might be an overkill for smaller or less mission-critical systems.

• Distributed Locks are Dead; Long Live Distributed Locks!

  For the impatient reader, here are the takeaways of this blog post:

  • FencedLock is a linearizable distributed implementation of the java.util.concurrent.locks.Lock interface with well-defined execution and failure semantics. It can be used for both coarse-grained and fine-grained locking.
  • FencedLock replicates its state over a group of Hazelcast members via the Raft consensus algorithm. It is not vulnerable to split-brain problems.
  • FencedLock tracks liveness of lock holders via a session mechanism that works in a unified manner for both Hazelcast servers and clients.
  • FencedLock allows 3rd-party systems to participate in the locking protocol and achieve mutual exclusion for the side-effects performed on them. This is the “fenced” part of the story.
  • FencedLock is battle-tested with an extensive Jepsen test suite. We have been testing its non-reentrant and reentrant behavior, as well as the monotonicity of the fencing tokens. To the best of our knowledge, FencedLock is the first open source distributed lock implementation that is tested with such a comprehensive approach.

• Making Containers More Isolated: An Overview of Sandboxed Container Technologies

  The main difference between a virtual machine (VM) and a container is that the VM is a hardware-level virtualization and a container is a OS-level virtualization. VM hypervisor emulates a hardware environment for each VM, where the container runtime emulates an operating system for each container. VMs share the host’s physical hardware and containers share both the hardware and the host’s OS kernel. Because containers share more resources from the host, their usages of storage, memory, and CPU cycles are all much more efficient than a VM. However, the downside of more sharing is the weaker trust boundary between the containers and the host.

• Testing the CP Subsystem with Jepsen

  Running a Jepsen test on a distributed database is like sneaking up on Superman with kryptonite while he is trying to overcome his biggest challenge. Jepsen subjects the database to various system failures while running a test case and checks whether the database is able to maintain its consistency promises. It can create chaos in many ways: make a single node or multiple nodes crash or hiccup, partition the network, or even make clocks go crazy.

• BPF Performance Tools

  This is the official site for the book BPF Performance Tools: Linux System and Application Observability, published by Addison Wesley (2019). This book can help you get the most out of your systems and applications, helping you improve performance, reduce costs, and solve software issues. Here I'll describe the book, link to related content, and list errata.

• Why the dockless scooter industry is going after a repossessor and a bike shop owner

  On July 1st, the City of San Diego implemented new regulations to address the scooter complaints. The regulations will require scooter companies to obtain insurance policies, free the city from all legal liability, cap speeds on the boardwalk, and obtain permits for every scooter in circulation. It’s still too early to tell whether the new regulations will make a difference.

  “We are aware that people are still riding on sidewalks, we are aware that people are colliding into people and then taking off,” San Diego Police Department Lt. Shawn Takeuchi says.

• The Near Impossible 20-Year Journey to Translate 'Fire Emblem: Thracia 776'

  Take one common problem with fan localizations: the sheer efficiency of Japanese. You can say a lot more with fewer characters in Japanese than English. Naturally, the game’s dialogue boxes were specifically programmed with Japanese in mind, not English. When a localizer drops the dialogue translated into English, there A) might not be space for the dialogue to fit, and B) if you go over the programmed character limit, the game could crash.

• One giant ... lie? Why so many people still think the moon landings were faked

  t took 400,000 Nasa employees and contractors to put Neil Armstrong and Buzz Aldrin on the moon in 1969 – but only one man to spread the idea that it was all a hoax. His name was Bill Kaysing.

• The 100 Best Movies of the Decade

  Cinema is in a constant state of flux, but it’s never mutated faster or more restlessly than it has over the last 10 years. And while the decade will no doubt be remembered for the paradigm shifts precipitated by streaming and monolithic superhero movies, hindsight makes it clear that the definition of film itself is exponentially wider now than it was a decade ago. Places. Products. Mirrors. Windows. Reflections of who we are. Visions of who we want to be. A way of capturing reality. A way of changing it. If the most vital work of the 2010s has made one thing clear, it’s that movies have never been more things to more people than they are today. And our week-long celebration list of the Best Films of the 2010s has us more excited than ever about what they might be to you tomorrow.

• Billie Eilish and the Triumph of the Weird

  Eilish has conquered the music world in part by doing everything she’s not supposed to. Her music is darker and weirder than that of most teen pop stars, with a gothy, punkish, vaguely sinister edge and nary a hint of bubblegum. For her core teen-girl fan base, she’s like the cool senior in art class who dresses and acts the way they wish they could: stylish, outrageous, maybe a little dangerous. (As her hit single “Bad Guy” puts it, “I’m the bad type, make-your-mama-sad type. . . might-seduce-your-dad type.” You get the sense that she’d love to be a “Parents Beware” segment on the 11:00 news.) Her vibe is both semi-nihilist and joyously defiant, a perfect soundtrack for a generation facing a half-dozen existential threats before first period. But she’s also playful, mischievous, vulnerable, alienated, melancholy — in other words, a teen.

• The Ham of Fate

  What he honed in his Brussels years is the practice of political journalism (and then of politics itself) as Monty Python sketch. He invented a version of the EU as a gigantic Ministry of Silly Walks, in which crazed bureaucrats with huge budgets develop ever more pointlessly complicated gaits. (In the original sketch, the British bureaucrats are trying to keep up with “Le Marché Commun,” the Common Market.) Johnson’s Brussels is a warren of bureaucratic redoubts in which lurk a Ministry of Dangerous Balloons, a Ministry of Tiny Condoms, and a Ministry of Flavorless Crisps. In this theater of the absurd, it never matters whether the stories are true; what matters is that they are ludicrous enough to fly under the radar of credibility and hit the sweet spot where preexisting prejudices are confirmed.