It feels like it's been forever since we heard these words...

... Pacific Northwest Storm Parade to Bring More Rain, Wind and Mountain Snow Through the Weekend

A classic November-December setup featuring a powerful jet stream stretching from eastern Asia across the Pacific for 5,000 miles to the Pacific Northwest is acting as the conductor for this storm parade. The persistent pipeline of moisture is being supplied by what meteorologists sometimes refer to as an atmospheric river. In this case, the plume of moisture impacting the Northwest extends all the way from the western Pacific Ocean near the Philippines.

Will events play out as predicted?

New database systems Red Book!

Well, this is exciting: Readings in Database Systems, 5th Edition.

Readings in Database Systems (commonly known as the "Red Book") has offered readers an opinionated take on both classic and cutting-edge research in the field of data management since 1988. Here, we present the Fifth Edition of the Red Book — the first in over ten years.

I had the original edition, got it in 1989 if memory serves. Chewed it to death.

The arrival of Transaction Processing: Concepts and Techniques somewhat obsoleted the Red Book, but I still have wonderful memories of the original edition.

Now I have something new to chew!

fsync and Virtual Machines

Oh, the wonders of low-level systems software.

The documentation for fsync() says:

fsync() transfers ("flushes") all modified in-core data of (i.e., modified buffer cache pages for) the file referred to by the file descriptor fd to the disk device (or other permanent storage device) so that all changed information can be retrieved even after the system crashed or was rebooted.

But what if you are running in a Virtual Machine?

An interesting article says:

Hypervisors outperforming the host machine is the most interesting to me. The results of this test clearly show that the hypervisors must be lying about synced writes for performance. This corroborates what I’ve seen with Packer as well, where if the virtual machine is not cleanly shut down, committed writes are lost. fsync() in a virtual machine does not mean that the data was written on the host, only that is is committed within the hypervisor.

And yet, another interesting article says:

ESX(i) does not cache guest OS writes. This gives a VM the same crash consistency as a physical machine: i.e. a write that was issued by the guest OS and acknowledged as successful by the hypervisor is guaranteed to be on disk at the time of acknowledgement. In other words, there is no write cache on ESX to talk about, and so disabling it is moot. So that’s one thing out of our way.

Is this more than a "he says, she says" thing? That is, is there a more definitive resolution of this question somewhere?

Note, btw, that the core behavior of Linux's fsync() command is not even clear, putting aside Virtual Machines:

By durable, I mean that fsync() should actually commit writes to physical stable storage, not just the disk write cache when that is enabled. Databases and guest VMs needs this, or an equivalent feature, if they aren't to face occasional corruption after power failure and perhaps some crashes.

I'm sure there are places where this information is definitively and clearly documented.

I'm just not sure where those places are.

Time passes

I signed up for Windows 10 on my computer back in, what, June?

Around the end of July, the Inter-webs were full of people discussing their experiences.

Windows 10 wasn't ready for my computer.

In late September, I fussed with it some.

I dug into the Windows Update history screens, and there were strange event codes and unfamiliar messages.

Researching them with various search engines lead to lots of people with similar strange messages, and odd suggestions to do things like "clear your download directory, maybe you had a corrupted download."

I tried a few of those strange suggestions, but basically forgot about things, and determined that the most probable outcome was that this computer was only going to run Windows 8.1 (which is fine), and maybe in the future I might try running Windows 10.

Then, Friday, the computer seemed to wake up, and announced that it thought that Windows 10 was "coming".

And, today, it asked me if I wanted to start the download.

So, we'll see.

Maybe Windows 10 is in my future.

Maybe not.

And (is it an omen), just as I go to try to post this, Blogger is down.

For the first time in I-don't-know-how-long.

Time passes.

It's not just a game, ...

... it's GOTY 2015: The Witcher Takes it, But Should it Have Been Fallout 4?

There is something about The Witcher that is so eerie, spooky and dramatic. I can’t quite put my finger on it. Whether it’s playing as a Van Helsing-esque mutant, or the monsters themselves that you are hired to hunt. Whatever the driving force might be, there is something truly special about CD Projekt Red’s Witcher 3. Again, as far as games go, I had a fair many gripes with the title, but racking up a total play time of over 400 hours, it’s safe to say I got my money’s worth. The other thing is that The Witcher wasn’t, like Fallout, merely substance. The Witcher was art. The beauty, the immersion, it was all so well done that it felt less like a game than a film. It was truly an amazing experience. Fallout 4 is a fantastic game, and an unusually charming one, for the amount of bugs that players face each time they boot it up.

Stuff I'm reading, early December edition

Wow! Go away on vacation for a week and then have a busy week at work and my, what a lot of stuff accumulates in the in-box...

• The Moral Character of Cryptographic Work

  Cryptography rearranges power: it configures who can do what, from what. This makes cryptography an inherently political tool, and it confers on the field an intrinsically moral dimension.

• A Little More on the Graph Isomorphism Algorithm

  The key ideas of the algorithm for GI are really classic ones from design of algorithms. The genius is getting them all to work together. The ideas break into two types: those that are general methods from computer science and those that are special to the GI problem.

• The best books of 2015

• RTS AI: Problems and Techniques

  Real-Time Strategy (RTS) games is a sub-genre of strategy games where players need to build an economy (gathering resources and building a base) and military power (training units and researching technologies) in order to defeat their opponents (destroying their army and base). Artificial Intelligence problems related to RTS games deal with the behavior of an artificial player. This consists among others to learn how to play, to have an understanding about the game and its environment, to predict and infer game situations from a context and sparse information.

• DRAM’s Damning Defects—and How They Cripple Computers

  In computing systems built on such huge scales, even low-probability failures take place relatively frequently. If an individual computer can be expected to crash, say, three times a year, in a data center with 10,000 computers, there will be nearly 100 crashes a day.

  Our group at the University of Toronto has been investigating ways to prevent that. We started with the simple premise that before we could hope to make these computers work more reliably, we needed to fully understand how real systems fail. While it didn’t surprise us that DRAM errors are a big part of the problem, exactly how those memory chips were malfunctioning proved a great surprise.

• Challenges of Memory Management on Modern NUMA System

  Modern NUMA systems are quite different from the old ones, so we must revisit our assumptions about them and rethink how to build NUMA-aware operating systems. This article evaluates performance characteristics of a representative modern NUMA system, describes NUMA-specific features in Linux, and presents a memory-management algorithm that delivers substantially reduced memory-access times and better performance.

• SGX Hardware: A first look

  Without much fanfare, Intel has released Software Guard Extensions (SGX) in Skylake.

• Experimental Security Analysis of a Modern Automobile

  Of course, it’s possible that the software on those cars could have been updated back at dealerships in the intervening years – but that wouldn’t address all of the issues in the paper, and evidence suggests plenty of vulnerabilities still exist.

• Experimental Security Analysis of a Modern Automobile

  Over a range of experiments, both in the lab and in road tests, we demonstrate the ability to adversarially control a wide range of automotive functions and completely ignore driver input — including disabling the brakes, selectively braking individual wheels on demand, stopping the engine, and so on.

• Fast and Vulnerable: A Story of Telematic Failures

  In this paper we examine a popular aftermarket telematics control unit (TCU) which connects to a vehicle via the standard OBD-II port. We show that these devices can be discovered, targeted, and compromised by a remote attacker and we demonstrate that such a compromise allows arbitrary remote control of the vehicle.

• How Change Happens: Consultation draft

  How Change Happens draws on many first-hand examples from the global experience of Oxfam, one of the world’s largest social justice NGOs, as well as Duncan Green’s 35 years of studying and working on international development issues. It tests ideas and sets out the latest thinking on what works to achieve progressive change.

• Linux Performance Analysis in 60,000 Milliseconds

  You login to a Linux server with a performance issue: what do you check in the first minute?

• Fixing the #1 Problem in Computer Security: A Data-Driven Defense

  The implementation weaknesses described in this white paper are common to most organizations, and point to limitations in traditional modeling of and response to threats to computer security. Most of the problems occur due to ranking risk inappropriately, poor communications, and uncoordinated, slow, ineffectual responses.

• Why Johnny Still, Still Can’t Encrypt: Evaluating the Usability of a Modern PGP Client

  Our results shown that more than a decade and a half after Why Johnny Can’t Encrypt, modern PGP tools are still unusable for the masses. We finish with a discussion of pain points encountered using Mailvelope, and discuss what might be done to address them in future PGP systems.

• Prudent Engineering Practice for Cryptographic Protocols

  The principles are neither necessary nor sufficient for correctness. They are however helpful, in that adherence to them would have avoided a considerable number of published errors.

• Topics in High-Performance Messaging

  Successful deployment of a messaging system requires background information that is not easily available; most of what we know, we had to learn in the school of hard knocks. To save others a knock or two, we have collected here the essential background information and commentary on some of the issues involved in successful deployments.

• GD-Wheel: A Cost-Aware Replacement Policy for Key-Value Stores

  Currently , these key-value stores use either LRU or an LRU approximation as the replacement policy for choosing a key-value pair to be evicted from the store. However, if the cost of recomputing cached values varies a lot, like in the RUBiS and TPC-W benchmarks, then none of these replacement policies are the best choice. Instead, it can be advantageous to take the cost of recomputation into consideration.

• Apache Kafka, Purgatory, and Hierarchical Timing Wheels

  Apache Kafka has a data structure called the "request purgatory". The purgatory holds any request that hasn't yet met its criteria to succeed but also hasn't yet resulted in an error. The problem is “How can we efficiently keep track of tens of thousands of requests that are being asynchronously satisfied by other activity in the cluster?”

• SoK: Eternal War in Memory

  We systematize the current knowledge about various protection techniques by setting up a general model for memory corruption attacks. Using this model we show what policies can stop which attacks. The model identifies weaknesses of currently deployed techniques, as well as other proposed protections enforcing stricter policies.

• Optimizing Hash-Array Mapped Tries for Fast and Lean Immutable JVM Collections

  In this paper we reduce memory overhead and runtime performance overhead from the implementations of immutable collections on the Java Virtual Machine (JVM).

• What's Worked in Computer Science

  In 1999, Butler Lampson gave a talk about the past and future of “computer systems research”. Here are his opinions from 1999 on “what worked”.

• Good Leaders are game changers: Raft & Paxos

  In this blog post, we will briefly show the similarities and differences between Paxos and Raft. Firstly, we will describe what a consensus algorithm is. Secondly, we will describe how to build a replication solution using instances of a consensus algorithm. Then we will describe how leaders are elected in both algorithms and some safety and liveness properties.

• The story of one latency spike

  A customer reported an unusual problem with our CloudFlare CDN: our servers were responding to some HTTP requests slowly. Extremely slowly. 30 seconds slowly. This happened very rarely and wasn't easily reproducible. To make things worse all our usual monitoring hadn't caught the problem. At the application layer everything was fine: our NGINX servers were not reporting any long running requests.

• Daily Report: Google Gets Serious About Competing in Cloud With Amazon Web Services

  One of the great mysteries of the tech industry in recent years has been the seeming disinterest of Google, which is now called Alphabet, in competing with Amazon Web Services for corporate customers.

• A Cabinet of Infocom Curiousities

  If you’re coming into this relatively new, or even if you need a little brush-up, let me state: Steve Meretzky has earned the title of “Game God” several times over, having been at the center of the early zenith of computer games in the 1980s and persisting, even thriving, in the years since. He continues to work in the industry, still doing game design, 35 years since he started out as a tester at what would become Infocom.

  But more than that – besides writing a large amount of game classics in the Interactive Fiction realm, he also was an incredibly good historian and archivist, saving everything.

• It Was Never Going to Work, So Let’s Have Some Tea

  "The master's tools will never destroy the master's house"

• Rainbow Six Siege Review: This Thing Is Disturbingly Real

  Ubisoft’s latest tactical shooter, Tom Clancy’s Rainbow Six Siege, adopts a striking bent towards a unique brand of pseudo-realism. Siege evokes a perverse version of the uncanny valley. It mixes the over-the-top, arcade-style renditions of violence games often lean towards with the gut-wrenching reality that we are, in fact, remarkably fragile.

• How Fallout 4 Mastermind Todd Howard Builds His Epic Dream Worlds

  “I think if someone has a gaming obsession, Ultima became mine,” says Howard. “I would say no other series ingrained itself in how I want to make games or what I want them to be more than Ultima did.”

• Why Ball Tracking Works for Tennis and Cricket but Not Soccer or Basketball

  Most ball tracking systems rely on two different approaches. The first looks to follow the movement of the ball in three dimensions and then predicts various likely trajectories in the future. This “tree” of possible trajectories can then be pruned as more ball-tracking data becomes available.

  The advantage of this approach is that the laws of physics are built in to the trajectory predictions so unphysical solutions can be avoided. However, it is hugely sensitive to the quality of the ball tracking data and so tends to fail when the ball is occluded or when players interact with the ball in unpredictable ways.

  Another method is to track the players and note when they are in possession of the ball. The movement of the ball is then assumed to follow the player and when possession transfers from one player to another. The advantage here is that the system does not get so confused by rapid or unpredictable passes—indeed, this approach works well in basketball, where dribbling and occlusion can make life difficult for ball trackers. However, without physics-based constraints on the motion of the ball, these systems can produce inaccurate tracks.

• How To Make Millions Of Hoverboards (Almost) Overnight

  Shenzhen is also, and only very recently, the hoverboard manufacturing capital of the world. In the smoke and asphalt of Bao An, a sprawling industrial flatland roughly the size of Philadelphia that serves as one of the city’s main manufacturing districts, hundreds of factories churn out much of the world’s supply of the boards, which are then shipped, rebranded, and sold around the globe.

And a question for the weekend: how many hours have you spent on your Hoverboard? (And no, I don't mean these hoverboards.)

It's not just a game, ...

Open-World Games Are Changing the Way We Play

It’s a slow cartography, maps of imaginary spaces growing in my head, inch by inch.

I feel like this might be the way well crafted open worlds are supposed to be experienced—not as gluttonous binges or narrowly focused rampages, but as long-term occupancies. I’ve found that these games exist more vividly in my mind as I embrace this style of gameplay. They grow in my imagination as they occupy more and more space in my memory. Instead of rushing through them or viewing them as content generators, I abide in them.