The Wave Watcher's Companion: a very short review

Gavin Pretor-Pinney's The Wave Watcher's Companion: Ocean Waves, Stadium Waves, and All the Rest of Life's Undulations is not an old book, but it's not a brand-new book, either.

It's one of those odd little books that appeal to somebody like me; I had it on my list for a while, and eventually it arrived!

Waves takes as its starting point the familiar: you are sitting on the beach, watching the waves crash against the shoreline, when it occurs to you that you don't really understand what that phrase actually means: what is truly going on when "the waves crash against the shoreline".

Pretor-Pinney, being a curious sort, digs into it, and follow it where it leads him, and even though the book is at times a bit meandering and filled with a bit of unnecessary miscellany (did "Mad Jack" Churchill's World War II escapades really deserve inclusion?), the result is a thoroughly enjoyable romp through all sorts of waves:

• Ocean waves
• Sound waves
• Radio waves
• Shock waves
• Tidal waves
• Earthquake waves
• And, finally, surfers enjoying waves breaking upon the shore

With lots of other waves along the way.

Among the parts that really stood out to me are:

• a section in which Pretor-Pinney talks about how a Japanese mathematician used wave-theoretic equations to explore the behavior of traffic jams on crowded roadways;
• a part about how carefully-controlled shock waves can be used in kidney-stone therapy;
• and a silly-but-amusing diversion into the controversy about who invented "the wave" at sports stadiums (was it a Coca-Cola ad that aired during the 1986 World Cup competition in Mexico, a group of devoted University of Washington Huskies fans in the fall of 1981, or perhaps a fanatic follower of the Oakland Athletics nick-named "Krazy George" who beat a drum in the stands during the Yankees-Athletics playoff series of 1981?).

If you're into popular science books, this is a very nice one, and Pretor-Pinney is very skilled at knowing how to make you smarter without making you feel exhausted; when the Japanese mathematician explains that

"A stop-and-go wave is a cluster solution of a non-energy-conserved dissipative system"

Pretor-Pinney has the good sense to realize that there's more work to do here to help you get the point, and spends another couple pages breaking that down into bits and pieces, showing you how you already know enough to understand what is being said, and leaves you willing to keep plugging along, unafraid to, later in the book, tackle more challenging topics such as Maxell's equations, or the difference between the P-wave and S-wave delivered by earthquakes, and the tough distinctions between which sort of waves represent energy moving through a medium, versus those which appear wave-like, but in fact are wholly different, such as crowd waves, or the behavior of sand dunes in the desert.

I'm passing Waves along to the next person; I enjoyed it, and now it's time for somebody else to.

Tangled Vines: a very short review

One of my holiday gifts this year was Francis Dinkelspiel's Tangled Vines: Greed, Murder, Obsession, and an Arsonist in the Vineyards of California

Which is, really, about a 2.5 hour read.

So it's perfectly reasonable to give this a really, really, really short review.

Who knew that a single warehouse in boring old Vallejo, California could possibly hold 400 million dollars worth of wine?

Who knew that a warehouse holding 400 million dollars worth of wine would not have a state-of-the-art sprinkler system?

Who knew that wine auctions, wine clubs, and fine wine in general, were such a magnet for money and greed, and such an opportunity for fraud and deceit?

Well, as regards that last question: I did.

So there.

Anyway, Tangled Vines manages to do a perfectly respectable job of telling a "true crime" story about a wine fraudster who committed one of the biggest acts of arson in the last hundred years in order to (unsuccessfully) cover up his fraud.

And, interleaved, it does a much nicer job of re-telling the 175-year-old store of winemaking in California, starting (more or less) with the Franciscan Missions, making sacramental wine for their services, and continuing on through the decades and centuries, with hiccups like Prohibition along the way.

If you've ever been to Rancho Cucamonga, you'll enjoy this book.

But most people on the planet have never thought much about Rancho Cucamonga, and I'm not sure I can really blame them.

Anyway, I enjoyed Tangled Vines, but I'm not suggesting you go out of your way to acquire it.

But if it should cross your desk (say, via your local Public Library), you'll probably enjoy it, too.

Nicely put

John Scalzi takes a moment, and a deep breath, for a considered exploration of some famous words spoken by Dr. King, but actually written 150 years ago by Theodore Parker, in a marvelous essay: The New Year and the Bend of the Arc

We are human. We are not perfect. We will not arrive at a perfect justice, any more than we will achieve a perfect union. But just as we work toward a more perfect union, so too we bend the arc toward justice, knowing the closer we get, the better we and our lives are, as individuals, as communities, as a nation and as a world. This is a life’s work, not just work for a moment, or day, or year. You won’t see the final result. There isn’t one. It doesn’t mean the work doesn’t matter. It matters.

The January deluge continues

Check THIS out:

Shasta and Oroville BOTH at 125% of historical average!

Don Pedro is slamming too, already at 90% of capacity (and it's still mid-January!!).

Trinity and Folsom are looking fine, and even New Melones, scraping the bottom just 4 months ago, is up to two thirds of its historical average (we get our East Bay Municipal Utility District supplies from New Melones, btw).

Meanwhile, the 36-hour forecast is for 2-4 FEET of additional snow in the core region of the central Sierras (roughly, the headwaters basin of the Kings River).

The main weather concern now turns to yet another incoming storm which has the potential to be the strongest storm to impact our area so far this season. WV imagery is indicating a moist baroclinic band stretching from 130w out to 170w which is taking aim at central California. GOES high density winds are indicating that this feature is being pulled up by a strong upper low centered near 47n/143w and being pushed eastward straight toward central California. This deep moisture fetch is progged to reach our area by this evening and continue through at least midday Monday. During this time frame periods of moderate to heavy precipitation are expected across our area along with increased winds in favored areas. As the result of the anticipated heavy snowfall and strong winds in the mountains, a Winter Storm Warning has been issued for the southern Sierra Nevada where 2 to 4 feet of additional snowfall are expected over the higher elevations. Snow levels will be between 4500 and 5000 feet on satruday evening when this system arrives and rise to 7500 to 8000 feet by Sunday afternoon in a modified tropcial airmass. As the large offshore upper low approaches the norcal coast Sunday night, cooler air will push into California and snow levels will lower to 3500 to 4000 feet by Monday. Over the Kern County mountains a strong pressure gradient will result in very strong winds on Sunday with gusts up to 90 mph possible in the Grapevine area. The precipitation over the passes will turn to snow on Sunday night as the cooler air pushes in.

Please be careful if you're one of those people trying to drive the Grapevine this weekend. When the road freezes and the wind blows, it's one of the more dangerous roads in the country.

But I bet the Shasta pack is loving this.

Rain, rain, snow, rain. Oh, how sweet it is.

This and that

My challenge is simple: to master the nuances of my new job, all I have to do is to be comfortable with every page of this amazing 500 page masterwork: Is Parallel Programming Hard, And, If So, What Can You Do About It?. Happily, McKenney is a wonderful writer, but where will I find the time?

In the meantime, there's also a few other things that crept in while I was sleeping:

• Who is Anna-Senpai, the Mirai Worm Author?

  The story you’re reading now is the result of hundreds of hours of research. At times, I was desperately seeking the missing link between seemingly unrelated people and events; sometimes I was inundated with huge amounts of information — much of it intentionally false or misleading — and left to search for kernels of truth hidden among the dross. If you’ve ever wondered why it seems that so few Internet criminals are brought to justice, I can tell you that the sheer amount of persistence and investigative resources required to piece together who’s done what to whom (and why) in the online era is tremendous.

• Cloudflare’s Transparency Report for Second Half 2016 and an Additional Disclosure for 2013

  Because of the gag order, I had to sit in silence, implicitly confirming the point in the mind of the staffer. At the time, I knew for a certainty that the FBI’s interpretation of the statute diverged from hers (and presumably that of her boss).

  Cloudflare fought this battle for four years even after the request for customer information had been dismissed. In addition to protecting our customers’ information, we want to remain a vigorous participation in public policy discussions about our services and public law enforcement efforts. The gag rule did not allow that.

  Now that this gag order has been lifted, Cloudflare is able to publish a more accurate transparency report to its customers and constituents. For us, this is not the end of the story, but the beginning of a more robust, fact-informed debate.

• Operation Grand Mars: a comprehensive profile of Carbanak activity in 2016/17

  Trustwave has tracked Carbanak activity in the latter half of 2016 and found them to be targeting hospitality and retail victims in Europe and North America, specifically targeting their internal corporate secrets and protected payment card data. Trustwave published a blog earlier in the year within initial findings but has now released the complete malicious campaign profile in a 45-page Advanced Threat Report. The blog post below is a summary of the malicious profile we have developed for this actor. The complete report on this activity can be found here: https://www2.trustwave.com/Operation-Grand-Mars.html.

• Babai Strikes Back

  We were watching history. From the talk I tweeted the new news though Bill Cook, also in the audience, beat me to the punch. Babai went on to describe the issue, an error in the analysis of the running time in the recursion, and the fix, basically a way to avoid that recursive step, but I can't do it justice here. At the end he proclaimed "So it is quasipolynomial time again". And so it was.

• On exp(exp(sqrt(log n))) algorithms.

  Improving quantitative parameters such as running time or approximation factor is very useful as intermediate challenge problems that force us to create new ideas, but ultimately the important contribution of a theoretical work is the ideas it introduces and not the actual numbers.

• No One Questioned This Hedge Fund’s Madoff-Like Returns

  But until Murray Huberfeld, who founded Platinum with Nordlicht, was caught up in a New York City municipal-corruption probe in June, no one at the fund had been charged with wrongdoing. Within weeks of Huberfeld’s arrest, federal agents raided Platinum’s midtown Manhattan office. On Dec. 19, Nordlicht and six others were arrested in what the government called a $1 billion fraud. Nordlicht and Huberfeld have pleaded not guilty, and Platinum’s main fund is being wound down after filing for bankruptcy. Montieth Illingworth, a spokesman for Platinum, declined to comment.

• Why Are So Many Bee Trucks Tipping Over?

  All of this is to say that I have little more insight, except that you definitely shouldn’t trust anyone! Except for maybe bees, considering they continue to provide us avocados and almonds and blueberries and all the foods that we’ve pushed to the edge of extinction, including the bees themselves, because we’re reckless as hell.

• China’s WTO Entry, 15 Years On

  The state’s hand was clear, but not in ways that were obviously forbidden by the WTO. Or at least not in ways that have been successfully challenged in the WTO. Firms’ investment decisions aren’t technically government procurement if the investment is for the provision of a commercial service, and the state’s guidance isn’t always written down. Yet even today the preferences provided for local firms in strategic sectors, like medical equipment, aren’t exactly a secret that China tries all that hard to hide.

  ...

  The initial China shock overlaps with the dollar shock. WTO accession made producing in China for the global market attractive, but did not made China into a great market for manufacturers looking to sell globally produced goods to China. Successful WTO challenges to individual Chinese practices haven’t changed the overall pattern—China’s imports of manufactures for its own use have slid steadily relative to China’s GDP after WTO accession. The more-limited-than-expected gains for manufacturers looking to sell to China though didn’t lead (until now, when the China shock is arguably starting to fade) to a serious reconsideration of the basic gains from China’s asymmetric integration, in part because U.S. and European firms captured many of the initial gains of China’s export success. And some “within the rules” remedies weren’t used as aggressively as they could have been to challenge China’s currency management and other discriminatory practices during the years immediately after China joined the WTO.

• Cooperation at the Tigris‎: U.S. and Iraqi Efforts to Maintain the Mosul Dam

  U.S. and Iraqi scientists estimate that a breach of the Mosul Dam, one of the largest pieces of water infrastructure in the Middle East, could have a devastating impact for over a million Iraqis along the Tigris River, from Mosul itself all the way down to Baghdad. Its failure could also be a threat to U.S. personnel and our Coalition members working alongside Iraqis and drastically reshape the nature of the fight against ISIL in northern Iraq. Today, thanks to close U.S.-Iraqi partnership, however, Mosul Dam is on a path toward greater stability.

• CS Responder Trans-Oceanic Cable Layer

  Laying fiber optic cables with repeaters along the ocean floor raises super-interesting technical challenges. I recently visited the CS Responder, a trans-ocean cable-laying ship

• The greatest chess game ever played

  Garry Kasparov, who is one of the top chess players ever, said that his 1999 match against Veselin Topalov was the greatest game of chess he ever played. In this video, MatoJelic goes through the game, move by move.

Up, up and away!

Monday I'll report to work, bright and early, to a new office, in Salesforce West, the 6th tallest building in San Francisco.

Although this won't be the first time I've worked in a high-rise, I've generally avoided such office settings over the years.

But that's getting hard to do, at least in San Francisco, where an explosion of new development surrounds the Transbay Terminal development area, just one block away from my new office building.

A lot of this burst of development is due to changes to the zoning laws around downtown San Francisco, as well as considerable improvements in construction techniques.

Already built and operational are:

• 50 Frement (Salesforce West)
• 350 Mission (Salesforce East)
• the (sinking) Millenium Tower

But that's just the beginning, for here come:

• 181 Fremont, already topped out I believe
• 50 Front Street, the Oceanwide Center
• Park Tower

And, of course, the Big Daddy of them all, the Salesforce Tower, where the concrete was poured for the top floor just last week.

The new tower is just across the street, so I'll be keeping a close eye on it.

And, the transit center itself seems like it will be pretty nice, with a beautiful (I hope) blocks-long public park atop the terminal, that should be very fine on sunny afternoons.

But the city is definitely a warren of canyons. You'd never believe you were just a few blocks from some of the most beautiful scenery in the world (the Golden Gate, the Bay, Alcatraz and Angel islands, etc.); you'd think you were somewhere around midtown-Manhattan, instead.

Which, I suppose, is sort of the point.

But here I am, and up they go.

Stuff I'm reading, MLK Day edition

My, how time races by. I've been quite busy and not getting enough time to read.

Is there ever enough time to read?

• CIDR 2017

  The biennial Conference on Innovative Data Systems Research (CIDR) is a systems-oriented conference, complementary in its mission to the mainstream database conferences like SIGMOD and VLDB, emphasizing the systems architecture perspective. CIDR gathers researchers and practitioners from both academia and industry to discuss the latest innovative and visionary ideas in the field.

• Optimizing Space Amplification in RocksDB

  RocksDB is an embedded, high-performance, persistent keyvalue storage engine developed at Facebook. Much of our current focus in developing and configuring RocksDB is to give priority to resource efficiency instead of giving priority to the more standard performance metrics, such as response time latency and throughput, as long as the latter remain acceptable. In particular, we optimize space efficiency while ensuring read and write latencies meet service-level requirements for the intended workloads. This choice is motivated by the fact that storage space is most often the primary bottleneck when using Flash SSDs under typical production workloads at Facebook. RocksDB uses log-structured merge trees to obtain significant space efficiency and better write throughput while achieving acceptable read performance.

• How and why the leap second affected Cloudflare DNS

  The root cause of the bug that affected our DNS service was the belief that time cannot go backwards. In our case, some code assumed that the difference between two times would always be, at worst, zero.

• The Road to 2 Million Websocket Connections in Phoenix

  2 million is a figure we are pleased with. However, we did not quite max out the machine and we have not yet made any effort toward reducing the memory usage of each socket handler. In addition, there are more benchmarks we will be performing. This particular set of benchmarks was set exclusively around the number of simultaneous open sockets. A chat room with 2 million users is awesome, especially when the messages are broadcast so quickly. This is not a typical use case though.

• Adaptive logging: optimizing logging and recovery costs in distributed in-memory databases

  This is a paper about the trade-offs between transaction throughput and database recovery time. Intuitively for example, you can do a little more work on each transaction (lowering throughput) in order to reduce the time it takes to recover in the event of failure. Recovery is based on information in logs, classically an ARIES-style write-ahead log, that records the values of data items.

  In the case of in-memory databases, you can also go the other way, and do a little less work when creating the logs (recording information for use in recovery) at the expense of longer recovery times, but gaining higher throughput. We can simplify recovery on the assumption that there is no need to undo the effects of uncommitted transactions – these existed solely in-memory and had not yet been persisted to disk.

• Millions of Queries per Second: PostgreSQL and MySQL’s Peaceful Battle at Today’s Demanding Workloads

  The idea behind this research is to provide an honest comparison for the two popular RDBMSs. Sveta and Alexander wanted to test the most recent versions of both MySQL and PostgreSQL with the same tool, under the same challenging workloads and using the same configuration parameters (where possible). However, because both PostgreSQL and MySQL ecosystems evolved independently, with standard testing tools (pgbench and SysBench) used for each database, it wasn’t an easy journey.

• Bitpacking and Compression of Sparse Datasets

  It turns out that gzipping after bitpacking yields a 1000x compression. Even on its highest compression settings, gzip was leaving a 8x compression on the table when applied to the raw data. It turns out that if you know the structure of your own data, you can very easily do much, much better than a generic compression algorithm. -- on both speed and compression.

• The Real Reason Your City Has No Money

  All of the programs and incentives put in place by the federal and state governments to induce higher levels of growth by building more infrastructure has made the city of Lafayette functionally insolvent. Lafayette has collectively made more promises than it can keep and it's not even close. If they operated on accrual accounting -- where you account for your long term liabilities -- instead of a cash basis -- where you don't -- they would have been bankrupt decades ago. This is a pattern we see in every city we've examined. It is a byproduct of the American pattern of development we adopted everywhere after World War II.

• Software Copyright Litigation After Oracle v. Google

  Oracle America has factored into at least four cases so far. One of these cases settled, one is on appeal, and the other two likely will be appealed in the near future. The latter two cases also involve patent claims, so appeals will be heard by the CAFC. (The CAFC has nearly exclusive appellate jurisdiction over cases with patent claims.) One can assume that the plaintiffs added the patent claims to ensure CAFC jurisdiction.

• This is Fine: Engineering War Stories (and What We Learned) in 2016

  In the past an engineer would be tasked with a project, crawl into a dark hole, and come out days, weeks or months later clutching their precious code. Sometimes this worked out really well but other times it was disastrous.

• The Inside Story of BitTorrent’s Bizarre Collapse

  sometimes technologies are not products. And they’re not companies. They’re just damn good technologies.

• “Side Hustle” as a Sign of the Apocalypse

  And WTF has happened to our culture when we just take it as fact that everyone needs to have multiple jobs and work as a cab driver and rent out every square inch of space in their apartment and be a task rabbit gopher who waits in line for tickets when they’re not walking dogs or temping and we all just chalk it up to “progress”??? In the old days, this meant your life was falling apart. Now it just means you’re part of “the sharing economy.”

• The Chemistry Behind Your Home’s Water Supply

  We take for granted the water that comes out of the taps in our home when we turn them on – but a lot of work goes into getting it there. Chemistry, too, has a hand in making sure that the water is safe to drink. Here, we take a look at the water treatment process, and in particular the chemicals used to get clean drinking water to your tap.

And the water from the mountains flowed into the bay

Despite days of rain, ‘high-tide groupies’ seek out king tides

Many among the curious onlookers snapped pictures, splashed in the water, or just stared at the curious phenomenon around 10:15 a.m., when the water crested in front of Pier 14.

It was day-two of a three day stretch of the unusually high water that swelled around the Bay Area and spilled onto low-lying coastal areas, prompting the National Weather Service to issue a coastal flood advisory.

And the atmospheric river flowed into the reservoirs

20 inches of rain, 12 feet of snow finally end 5-year drought in N. California

Stations up and down the Sierra mountain chain reported twice the amount of normal rain and snow for this time of year after snowstorms doubled the vital snowpack there that provides the state with much of its year-round water supply.

U.S. Drought Monitor: California

A plethora of Pacific storms and moisture slammed into California and most of the West, dumping copious amounts of precipitation on the northern two-thirds of the state and Sierra Nevada. This very wet week maintained the great start to the Water Year (since Oct. 1) across the West where NRCS SNOTEL basin average precipitation was above or much above normal at nearly every major basin while basin average snow water content was at or above normal in most Western basins. With more than a foot of precipitation falling on the Sierra Nevada (locally 20.7 inches at Strawberry Valley, CA), most major reservoirs were at or above its Jan. 10 historical average, USGS monitored streams were at near or record high flows, Jan. 10 state snow water content was at 135%, and the Northern Sierra 8-station, San Joaquin 5-station, and Tulare Basin 6-station precipitation indices topped their wettest previous year as of Jan. 10.

A plethora of storms. Imagine that!

Even though my ferry was canceled one day, and there's definitely water everywhere, it's a merry, merry sight.

The atmospheric river ... became an actual river

Sacramento Weir Opened For First Time In More Than 10 Years

Water officials opened the weir around 4:30 a.m. Tuesday. It’s all an effort to alleviate pressure on the Sacramento River levees.

How the Sacramento Weir works

While the system’s five other weirs overflow automatically once the river reaches a certain elevation, the 1,920-foot-long Sacramento Weir consists of 48 gates that must be opened with a long hooked pole. The gates are opened when the river level at the I Street Bridge reaches 29.87 feet. The weir, built in 1916, was last opened in December 2005.

Meanwhile, closer to home, ferries were canceled due to wind and waves, BART is running a partial schedule because downed trees have blocked tracks, planes are hours late or canceled entirely.

It's pandemonium.

Happily, the forecasters promise at least a few days of relief in sight.

Farewell to the Calaveras Pioneer Cabin tree

It was a big storm: Historic Pioneer Cabin Tree toppled in California storm

The historic Pioneer Cabin Tree, a former "drive-through" giant sequoia in Calaveras Big Trees State Park in Calaveras County, was felled in California's weekend storms.

The tree was hollowed out in the 1880s to allow tourists to pass through it, and even allowed cars, but more recently has hosted only hikers. It was part of the "Big Trees Trail" at the state park.

Jim Allday of Arnold is a volunteer at the park who was working there Sunday. He said the tree went down about 2 p.m. and "shattered" on impact.

I walked that trail in 2014, and if you ever get to the Sierras, Calaveras Big Trees State Park is a wonderful place.

But we humans should never have done that to that tree.

CHKDSK to the rescue!

Last Thursday evening, the 2 TB system disk on my primary Windows machine decided to fail, developing a number of unrecoverable media errors.

It wasn't immediately obvious that it was a hard disk failure: the machine would boot, run for a while (2 minutes? 5 minutes?) then it would lock up and stop running and eventually shut down.

I was able to bring up a document or two, but it wouldn't stay up long enough for me to bring up the Windows Event Log or any other sort of diagnostic tool.

Eventually I realized that the most likely cause was a media error on the system disk, and recognized that this wouldn't be a trivial task to recover from. Luckily, I have some other machines in the house that I can use, so I wasn't totally dead in the water.

Typically, when you develop bad blocks and other media errors on your hard disk, particularly to the point that the device itself can no longer quietly re-map those bad blocks to other sectors on the disk, you're looking at total data loss.

Which was unfortunate, because I didn't actually have a backup of this disk. Yes, I know: oops.

So over the weekend, I began trying to understand what was really wrong with the machine and what could be saved. I booted into the BIOS, and ran the various self-tests and diagnostics that come with this particular Dell machine. They confirmed that (a) the system disk was returning I/O errors, and (b) the rest of the computer seemed to be fine, from a hardware point of view.

It turns out that I had a second internal hard disk already installed in this computer, so I (with the power down of course) disconnected the main system disk from power and SATA, and popped my Windows 10 DVD into the DVD drive and installed Windows on the other hard disk.

Yay! Now I at least had a functioning computer again.

I had already placed an order for a replacement storage device, but I was intrigued by the fact that the machine had still been able to boot from the damaged system disk on Thursday: perhaps there was some faint hope that I could recover my data?

So I carefully re-connected the damaged disk to power and SATA, having made sure in the BIOS that it would NOT be the boot disk, and started back up again.

Then, from an Administrator CMD.EXE window, I fired up:

CHKDSK /F D:

And I went off, to take the dog for a walk.

And watched some football.

And my son came over and we talked about his life and how things are going.

And, eventually, after about 3 hours, the CHKDISK completed!

I was astonished; I hadn't really expected that.

So I plugged one of my external USB backup drives into the machine, and opened up the damaged drive in Windows Explorer, and lo and behold!

I was able to see files!

As quickly as I could, with my fingers crossed and my toes crossed, I selected all the most important files (4 years of digital pictures, various documents I'd written, some backup copies of data from my wife's machine, and so forth), and copied them all to the external drive.

Successfully.

Anyway, I'm mostly back up and running, although I've got a lot of software to re-install on this machine.

It's interesting that much of the stuff that I care about, is no longer the sort of problem that it would have been, even just 5 years ago: my email is all in the cloud, my Steam saved games are all in the cloud.

And, of course, my eight years of writings about my life and interests are in the cloud, as well: on this blog!

Other than some various files that I don't particularly want to save in the cloud (tax documents, etc.), the files that would have caused me the most trouble to lose were my VirtualBox images. I haven't yet had time to verify whether I successfully recovered those, but I'm hopeful.

In a way, completely starting over with a computer like this is a bit like moving house: it makes you think about what you really need to keep, and what you can throw away. There's a saying that, after you move into your new house, if should happen that there are some moving boxes that you haven't opened, and it's been 6 months, well, you can just throw those boxes away.

I'm not really that person; I find it very hard to throw things away.

But I understand the motivation behind the sentiment. Declutter!

Meanwhile, I'm sort of doing the same thing with my office environment. At my last job, I spent my entire 7 years in the same physical cubicle, never even moving once. That's really rare, at any company, and especially in the chaotic world of Silicon Valley high technology. Seven years was far and away the longest amount of time that I had ever spent in the same cubicle in my entire 35 year career; I think the second longest time I ever spent in a single cubicle is much closer to 2 years than to 7.

So, over those 7 years, I of course accumulated a vast amount of junk. I tried to throw most of it away, of course disposing of the company materials that had languished on my disk and bookcase and in filing cabinets: pile after pile after pile of documents full of hard, anguished work, which I was convinced I might need to refer back to, "sometime".

Even so, I still brought home more than a dozen shopping bags. Many were full of books, my professional library that I've accumulated over the years. But there were pictures, children's art, coffee cups, backup eyeglasses, spare clothing and jackets and umbrellas and whatnot, and lots of odds and ends that didn't fit into any easy category but I couldn't just immediately throw away.

So here I am, entering the new year. I have piles of stuff in the garage, which I'll have to sort through, and decide which things to take to the new office. And I have piles of files on my recovery disk, which I'll have to sort through, and decide which things to retain on my new computer.

Check back with me in 6 months; I wonder if I'll have even opened them all, much less decided what to keep and what to (finally) throw away.

Change

Nobody is ever comfortable with change.

But, as John Lennon said,

Life is what happens to you while you're busy making other plans.

I confess I have the bad habit, when stressful news forces its way into my life, of closing my eyes and covering my ears and trying to pretend it isn't real. I fool myself; I think that if I simply re-double my efforts, refuse to acknowledge, be stubborn, that I can will away the change.

But change happens, and it's happening to me.

Sadly, over the holidays, I lost my "other mother" (I could never bring myself to call Kaye my mother-in-law; I've always hated that term).

Kaye had been ill for years, but she never let it conquer her. Every time I was with her she was a treasure: showering us with love, looking to the future, never complaining. Even as her cruel, cruel illness robbed her of mobility, trapping her in a tinier and tinier box, she was never bitter or harsh, but gentle and gracious as always.

It can be no easy thing for a mother to see her own child grow up, but when I showed up and took her daughter out of her life and into my own, moving us far, far, (too far) away, she never lashed out, nor showed any regret; instead, she welcomed me into her own life and gave me her love as if I was her own.

And, I suppose, at that point, and due to that, and from then on, I was.

If I could seek only one thing, it would be to incorporate her uncrushable spirit into my own life.

Here are the words it's too late to say now, because I can say them but you cannot hear them, but I say them anyway; let the world hear them.

Thank you, Kaye.

Thank you for having such a beautiful and wonderful daughter; thank you for letting me take her into my life; thank you for choosing to love me like you loved her; thank you for being brave; thank you for passing all your wonderful qualities on to your daughter; thank you for everything.

I meant to tell you all this; life happened while I was making those plans.

Meanwhile, other changes continue to happen. In a brutally short period of time we found out that we lost three neighbors: one next-door, one across the street, and one just a short ways away.

We weren't related to any of these folks, but we'd lived next door to them for 15 years and they were all wonderful, wonderful people as well: Harry, the Stanford football player turned mining engineer, Barry, the hotel manager who moved here from Michigan and had an unbelievable sense of humor, and Barbara the wonderfully-kind best friend of our close friend and neighbor Mark.

It's all a dirty, rotten shame, and none of it makes any sense.

But life goes on, and change continues. Just to keep me on my toes, there's still more change in my life: after seven wonderful years at my job, I got one of those "too good to refuse" offers, so I made the terribly hard decision to step away from a job I thought I'd never leave, and take the next step in my professional career.

I'll have more to say about that, some other time.

For now, I'm just dealing with change.

The gang

Perforce Server Development, January, 2017 (plus a couple alumni):

My math may be a bit off, but I think that's slightly more than 400 years of professional software development experience, all in one place.

Pretty great.

Atmospheric river ahead?

This forecast hasn't changed much in the last 36 hours.

It's still 72 hours away, but check this out:

By Friday night, another abrupt change in the forecast pattern is expected as deeper moisture begins to advect inland from the eastern Pacific. Light rain remains on track to spread inland late in the night into Saturday morning. During the day Saturday, PWAT values are forecast to increase as an atmospheric river (AR) sets aim at the central California coast. By Saturday night into Sunday, PWAT values are forecast to approach 1.50" and will likely result in widespread moderate to locally heavy rainfall. The AR is forecast to move very little from Saturday night through Sunday and will likely lead to 24 to 36 hours of steady rainfall across the region. At this time, it appears the focus for the heaviest precipitation will be south of San Francisco along the coastal ranges which have the potential to pick up more than 12.00" of rainfall through the weekend. Given the impressive amounts of precipitation recorded in the Santa Cruz and Santa Lucia Mountains from this past system (Tuesday into Wednesday morning) cannot rule out even greater rainfall totals exceeding 15.00" as the upcoming AR event has the potential to be much stronger. Meanwhile, many urban locations can expect anywhere from 2.00" to 5.00" of precipitation through the weekend. In addition, breezy to gusty winds are forecast from Saturday night into Sunday as surface low pressure approaches the coast and pushes inland to the north. This south/southeast flow will also help to enhance precipitation rates along the coastal ranges. Be sure to stay up to date with the latest information regarding this upcoming weekend`s storm system as we will continue to refine our forecast in the coming days.

5 inches of rainfall in urban areas in 3 days?

15 inches of rainfall in the mountains in 3 days?

Remember, Oakland CA averages only 24 inches of rainfall for the entire YEAR, so getting 20 percent of the yearly total in just 3 days is, what, astounding?

Check back here in a week for the outcome, unless I'm paddling around in a boat somewhere...

Topic matching and bitmaps

Some very old ideas remain new. Inverted bitmaps are a wonderful example.

In a very nice recent article, Tyler Treat surveys various search structures employed by message queueing systems to solve the Fast Topic Matching problem: Fast Topic Matching

The naive hashmap solution is generally a poor choice due to its prohibitively expensive lookup time. Inverted bitmaps offer a better solution. The standard implementation is reasonable if the search space is finite, small, and known a priori, especially if read latency is critical. The space-optimized version is a better choice for scalability, offering a good balance between read and write performance while keeping a small memory footprint. The trie is an even better choice, providing lower latency than the optimized inverted bitmap and consuming less memory. It’s particularly good if the subscription tree is sparse and topics are not known a priori. Lastly, the concurrent subscription trie is the best option if there is high concurrency and throughput matters. It offers similar performance to the trie but scales better. The only downside is an increase in implementation complexity.

Treat's survey is clear and well-illustrated, and I'm really pleased that he worked hard to evaluate the choices across many different metrics: not just search time and update time, but also memory usage, concurrency, etc.

But I was particularly pleased that he included Bitmap indexes in the survey, for they are a fascinating data structure, and they don't get the coverage they deserve, particularly for work that is now 45 years old.

Although the original Model 204 implementation was hand-coded in IBM 370 Assembly Language, there are now widely-available bitmap indexing libraries, including Concise bitmaps, EWAH bitmaps, and Roaring bitmaps.

One (small) disappointment I have with Treat's great article is that he doesn't really investigate these libraries, instead simply noting that

While inverted bitmaps allow for efficient lookups, they are quite wasteful for sparse sets, even when using highly compressed bitmaps like Roaring bitmaps.

While I have no reason to disbelieve him, it would have been great to see him explain that conclusion more deeply.

At any rate, since Model 204's bitmap indexes were what got me into the software field, some 35 years ago, it's great to see them still finding favor and fairly widespread usage, decades later.

Rogue One: a very short review

At their best, the Star Wars movies deliver story-telling in a very old-fashioned way, with emphasis on story, character, and emotion.

Rogue One: A Star Wars Story is an ideal demonstration of this.

It's a Star Wars movie, so of course there are gadgets and Sci-Fi crutches (androids! faster-than-light-travel! spaceships galore!)

But those elements are just sort of the scenery for a classic story of a group of ragamuffin wanderers who come together in a time of need and figure out how to work together as a team.

Moreover, there's something to the Star Wars style that works particularly well in Rogue One: our heroes are underdogs, scavengers and makers-do, swept up in a crisis not of their own making, trying to make the best decisions they can in situations of great pressure and uncertainty.

You know: life, as it's led.

Oh, and one last thing: if you go, be prepared to shed a tear or two.