Saturday, December 27, 2014

Mazes, rooms, and pathways

You can tell it's the holiday season, because I've been playing games and thinking about games, a lot.

So I thoroughly enjoyed Robert Nystrom's recent essay: Rooms and Mazes: A Procedural Dungeon Generator.

Nystrom takes us through a variety of different approaches to generating dungeons for adventure games, because it turns out there are a lot of things to consider.

For example:

You could make a roguelike with perfect dungeons, and many simple roguelikes do that because generators for those are easier to design and implement.

But I find them less fun to play. When you hit a dead end (which is often), you have to do a lot of backtracking to get to a new area to explore. You can’t circle around to avoid certain enemies, or sneak out a back passage.

Nystrom proceeds to explore two basic approaches to dungeon generation:

  • Start with a maze. Trim back the maze to leave holes in the space, then turn those holes into rooms.
  • Start with a bunch of randomly-placed rooms. Flood-fill the spaces between the rooms with mazes, then connect things together

As he puts it:

Where Buck and Karcero start with the maze and then add the rooms, mine does things in the opposite order. First, it places a bunch of random rooms. Then, it iterates over every tile in the dungeon. When it finds a solid one where an open area could be, it starts running a maze generator at that point.

The best part of the article is the illustrations. Nystrom fills the article with live demonstrations of various techniques, which bring the code to life and make it easy to understand the alternatives and how they impact the resulting generated dungeon.

If you've ever tried your hand at generating a dungeon (and what game-playing computer programmer hasn't?!), I think you'll really enjoy reading Rooms and Mazes.

Who knows? It may even get you excited enough to start writing that game you've been putting off...

Wasteland 2: a very short review

I've been whiling away many an hour recently playing Wasteland 2.

There's no doubt about it: this is a very fun game.

You and your party find yourselves in a post-apocalyptic wasteland, loosely modeled on the territory and scenery of southern Arizona, in a chaotic world full of mystery and adventure.

You travel about from place to place, take on quests, unearth clues, unravel mysteries, defeat bad guys, and generally have yourself a wild-and-wooly rip-roaring good time.

The making of the game is rather interesting, as revealed by this Wired article: How One Guy Got Kickstarters to Give Their Profits to Other Campaigns

Fargo went on to launch other games, including the now celebrated Fallout series, and even became a game publisher himself, as founder of Interplay Entertainment. But Wasteland was his baby, and by the early 2000s, he wanted to create a sequel to the game that put him on the map, called Wasteland 2.

The problem was, the studios wanted nothing to do with it. “I got nowhere for another decade,” Fargo says.

Then, in early 2012, Fargo stumbled upon a still up-and-coming company called Kickstarter that let creators raise funding from their friends and fans.

But you don't have to care about how the game is made; it's more fun just to play the game itself.

Now, where was I? Oh, yes, back to trying to negotiate with the Servants of the Mushroom Cloud...

Wednesday, December 24, 2014

Christmas Eve link clearance

Get 'em while they're hot!

  • On the Sony Pictures Security Breach
    Reports say Sony Pictures had trouble attracting and recruiting security talent, which isn’t too surprising for a company known for its disdainful attitude toward technology. Being on the wrong side of issues like SOPA/PIPA couldn’t have helped—what technologist would want to work for a company that is trying to break the Internet?
  • Did North Korea Really Attack Sony?
    The agency's evidence is tenuous, and I have a hard time believing it. But I also have trouble believing that the US government would make the accusation this formally if officials didn't believe it.
  • The Case for N. Korea’s Role in Sony Hack
    The “silent” part of the moniker is a reference to the stubborn fact that little is known about the hackers themselves. Unlike hacker groups in other countries where it is common to find miscreants with multiple profiles on social networks and hacker forums that can be used to build a more complete profile of the attackers — the North Koreans heavily restrict the use of Internet communications, even for their cyber warriors.
  • What Does "Responsibility" Mean for Attribution?
    Using the Spectrum of State Responsibility, in my assessment, the US government's statements include a range of possibilities, from State-encouraged to State-integrated.
  • Stupid Costly Patent Nuclear War By Microsoft & Apple Against Android Averted
    In short, this is basically Google and Cisco (with some help from a few others) licensing these patents to stop the majority of the lawsuits -- while also making sure that others can pay in as well should they feel threatened. Of course, Microsoft, Apple and the others still have control over the really good patents they kept for themselves, rather than give to Rockstar. And the whole thing does nothing for innovation other than shift around some money.
  • Why String Theory Still Offers Hope We Can Unify Physics
    Most gratifying, the mathematics revealed that one of these notes had properties precisely matching those of the “graviton,” a hypothetical particle that, according to quantum physics, should carry the force of gravity from one location to another. With this, the worldwide community of theoretical physicists looked up from their calculations. For the first time, gravity and quantum mechanics were playing by the same rules. At least in theory.
  • Cerebros and the art of drug smuggling
    This is not the work of a pickaxe army of drug war foot soldiers. These are multi-million-dollar underground networks, created covertly with professional machinery under the guidance of top-end engineers or architects who have been pulled—willing or not—to the dark side.
  • The Future: A Cat Litter Box and DRM
    I did some Googling, and I found that the “Smart” in SmartCartridge is that it has an RFID chip inside of it to keep track of how much solution it has, and once it runs out, well, you can't refill it. I honestly did not believe this and tore one of the cartridges apart, and there it was, looking back at me, a tiny chip holding up it’s little metal finger.

    Seriously CatGenie, you added fairly sophisticated DRM to a litter box? I’m a tad hurt you spent my money on building in a restriction instead of figuring out how to avoid constantly cooking poop.

  • Drobo vs. QNAP vs. Synology vs. G-Drive and MORE! Which Storage Should You Choose?
    To me, the ideal size for a NAS device is at least 4 drives. You can get them with as few as two if you’re really never going to use much storage, and if you are going to do that I’ve got a recommendation for you in a minute, but with 4 or more you can have plenty of space without sacrificing redundancy. And I’ve got three different options for you here. The QNAP TS-470 Pro, The Synology 1513+, and the Drobo 5N.
  • The Top Metrics You Need to Track to Improve Operational Performance
    At PagerDuty we’ve thought hard about what you should monitor and why from a systems perspective, but what about monitoring data on your operations performance? We’d like to share some specific metrics and guidelines that help teams measure and improve their operational performance.
  • Interesting papers from NIPS 2014
    NIPS is the premier conference on Deep Learning. Given the accelerating state of the art, it’s interesting to see what is new.
  • Testing TokuDB's Group Commit Algorithm Improvement
    As part of our effort to verify the new Binary Log Group Commit functionality introduced in TokuDB 7.5.4 for Percona Server, we wanted to demonstrate the substantial increase in throughput scaling but also show the bottleneck caused by the skewed interaction between the binary log group commit algorithm in MySQL 5.6 and the transaction commit mechanism used in TokuDB 7.5.3 for Percona Server.
  • Are We Consistent Yet?
    Traditional systems provide strong consistency, where clients can immediately view updates. Some distributed systems relax their consistency model to allow greater availability or better performance. Eventual consistency manifests itself to clients as stale views of data.
  • The Softsel Hot List for the week of December 22, 1986
    Back in the days before Internet-based software distribution, heck back even before the Internet existed in a form resembling what it is today, one of the most important ways of keeping track of the consumer computing industry was to subscribe to the Softsel Hot List, a weekly poster of the top sellers in various categories. Here is the Softsel Hot List for the week of December 22, 1986
  • Our local fill
    The original wetland that became Lake Merritt was known as San Antonio Slough. From Oakland’s earliest days, the locals kept trying to “reclaim” it by turning it into dry land, just as they did all around the bay. The whole waterfront is reclaimed land. The basic technique was to haul dirt and rock and rubbish down to the water, shove it in and tamp it down. In Gold Rush San Francisco they’d use abandoned ships for fill, but Oakland’s founding fathers had advanced beyond such crude strategems.

For my Christmas Eve meal, my grand-daughter made me hand-made crab-and-shrimp sushi.

Life is good.

Friday, December 19, 2014

Some links for understanding CVE-2014-9390

This is a particularly interesting CVE for me, because it's "right up my alley", as they say.

  • Git 1.8.5.6, 1.9.5, 2.0.5, 2.1.4 and 2.2.1 and thanking friends in Mercurial land
    Git maintains various meta-information for its repository in files in .git/ directory located at the root of the working tree. The system does not allow a file in that directory (e.g. .git/config) to be committed in the history of the project, or checked out to the working tree from the project. Otherwise, an unsuspecting user can run git pull from an innocuous-looking-but-malicious repository and have the meta-information in her repository overwritten, or executable hooks installed by the owner of that repository she pulled from (i.e. an attacker).
  • [ANNOUNCE] Git v2.2.1 (and updates to older maintenance tracks)
    Fixes since v2.2
    ----------------
    
     * We used to allow committing a path ".Git/config" with Git that is
       running on a case sensitive filesystem, but an attempt to check out
       such a path with Git that runs on a case insensitive filesystem
       would have clobbered ".git/config", which is definitely not what
       the user would have expected.  Git now prevents you from tracking
       a path with ".Git" (in any case combination) as a path component.
    
     * On Windows, certain path components that are different from ".git"
       are mapped to ".git", e.g. "git~1/config" is treated as if it were
       ".git/config".  HFS+ has a similar issue, where certain unicode
       codepoints are ignored, e.g. ".g\u200cit/config" is treated as if
       it were ".git/config".  Pathnames with these potential issues are
       rejected on the affected systems.  Git on systems that are not
       affected by this issue (e.g. Linux) can also be configured to
       reject them to ensure cross platform interoperability of the hosted
       projects.
    
     * "git fsck" notices a tree object that records such a path that can
       be confused with ".git", and with receive.fsckObjects configuration
       set to true, an attempt to "git push" such a tree object will be
       rejected.  Such a path may not be a problem on some filesystems
       but in order to protect those on HFS+ and on case insensitive
       filesystems, this check is enabled on all platforms.
    
    A big "thanks!" for bringing this issue to us goes to our friends in
    the Mercurial land, namely, Matt Mackall and Augie Fackler.
    
  • 3f1509809e728b70ea7912e4e1b40f22965e45ee
    fsck: complain about NTFS ".git" aliases in trees 
    read-cache: optionally disallow NTFS .git variants 
    path: add is_ntfs_dotgit() helper 
    fsck: complain about HFS+ ".git" aliases in trees 
    read-cache: optionally disallow HFS+ .git variants 
    utf8: add is_hfs_dotgit() helper 
    fsck: notice .git case-insensitively 
    t1450: refactor ".", "..", and ".git" fsck tests 
    verify_dotfile(): reject .git case-insensitively 
    read-tree: add tests for confusing paths like ".." and ".git" 
    unpack-trees: propagate errors adding entries to the index
    
  • Git vulnerability with .git\config
    Normally the git client avoids ever overwriting that file. Even if you commit a .git\config file and push it to a shared repo, no one else’s Git client will check it out into their private repos. However, a bug was discovered where various permutations of the .git folder name (e.g. mixed case, gIT, GiT, etc, Windows filename shortening .git~123, Ignorable Unicode codepoints .g\u200cit\config, etc) were not caught be the Git client’s filtering logic. As such, if someone pushed a malicious config file with one of these permutations, other people’s Git clients would check them out, overwriting their personal config file and hijacking their Git commands. This affects, at least, Windows NTFS and Mac OS X HFS+ filesystems, both of which are case insensitive filesystems.
  • Vulnerability announced: update your Git clients
    Repositories hosted on github.com cannot contain any of the malicious trees that trigger the vulnerability because we now verify and block these trees on push. We have also completed an automated scan of all existing content on github.com to look for malicious content that might have been pushed to our site before this vulnerability was discovered. This work is an extension of the data-quality checks we have always performed on repositories pushed to our servers to protect our users against malformed or malicious Git data.

Some resources for trying to understand the SPE hack

If you're trying to make sense of it all, here are some fairly well-reasoned essays, from a variety of smart people:

  • North Korea and the attack on Sony Pictures Entertainment
    In HPSR Security Briefing Episode 16, we profiled North Korea’s cyber threat landscape, highlighting the regime’s known capabilities and deficiencies in cyberspace. In that report, we noted North Korea’s initial response to The Interview. Based on our previous research of North Korean cyber capabilities, it is difficult to discern whether the regime acted alone. It is plausible that the actors responsible for this attack relied on the assistance of an insider.
  • FBI: North Korea to Blame for Sony Hack
    While the United States government seems convinced by technical analysis and intelligence sources that the North Koreans were behind the attack, skeptics could be forgiven for having doubts about this conclusion. It is interesting to note that the attackers initially made no mention of The Interview, and instead demanded payment from Sony to forestall the release of sensitive corporate data. It wasn’t until well after the news media pounced on the idea that the attack was in apparent retribution for The Interview that we saw the attackers begin to mention the Sony movie.
  • Lessons from the Sony Hack
    This could be any of us. We have no choice but to entrust companies with our intimate conversations: on email, on Facebook, by text and so on. We have no choice but to entrust the retailers that we use with our financial details. And we have little choice but to use cloud services such as iCloud and Google Docs.
  • Did the DPRK Hack Sony?
    Drawing positive conclusions from the public evidence is incorrect. The NSA and the CIA may (or may not) have many other details they'll never disclose. The much-ballyhooed language setting, for example, is completely useless.
  • Why it's possible North Korea was behind the Sony hack
    NSA analysts might start with what’s known as “flow records,” which are very similar to Netflow. Each of these records details a communication between computers and the ports they used to initiate said communication. It might read something like this: "Computer A, port X sent K bytes of data to computer B, port Y between T0 and T1". Although flow records are the simplest data collected by the NSA’s wiretaps, it’s a powerful tool for tracking online activity.
  • What is happening?
    It's not about a movie or even Sony, at all. When you build a nuclear program, you have to explode at least one warhead so that other countries see that you can do it. The same is true with Cyber.
  • goodbye horses
    I suppose the more nuanced point Dave is trying to make, is not so much that this is an Act of War, or even that it should be. But more that this Sony thing is not about Sony, it's about the US necessarily having to prepare for a future where it will have to politically and publicly address and respond to a game that up until very recently, was only played in the shadows.

Hacker School publications are quite impressive

Somehow I stumbled across the inaugural issue of the Hacker School journal: Code Words: A quarterly publication about programming from Hacker School.

I haven't had time to read the entire issue yet, but so far I'm really pleased.

You know that you're really reading a publication targeted at programmers when the first paragraph of the first article is as follows:

Floating point is a giant mess. There are known best practices for most issues that come up in everyday use (e.g., using Kahan summation or adding stochastic noise to reduce aggregate numerical error), but there are still corner cases. Different libraries and implementations give different results because there’s no generally accepted standard.

"Numerical error"? "Kahan summation"? "Stochastic noise"?

Oh, yes, yes, YES! Bring it on!

And I also really enjoyed An introduction to functional programming, with its straight-forward, plain-english approach:

When people talk about functional programming, they mention a dizzying number of “functional” characteristics. They mention immutable data, first class functions and tail call optimisation. These are language features that aid functional programming. They mention mapping, reducing, pipelining, recursing, currying and the use of higher order functions. These are programming techniques used to write functional code. They mention parallelization, lazy evaluation and determinism. These are advantageous properties of functional programs.

Ignore all that. Functional code is characterised by one thing: the absence of side effects. It doesn’t rely on data outside the current function, and it doesn’t change data that exists outside the current function. Every other “functional” thing can be derived from this property. Use it as a guide rope as you learn.

They seem to be off to a great start, so I'm hoping that Hacker School continue to find success, and continue to publish more issues of their journal.

Meanwhile, reading the journal led me to the Hacker School blog, which seems pretty great too; I particularly like their "Read Along" feature, in which they select a research topic each week, post a pointer to the paper with some introductory thoughts, and encourage the audience to read along with the paper and contribute their own reactions.

If you're looking for some good reading to strengthen and extend your programming skills, give the Hacker School publications a try!

I suppose I should pay more attention to my investments...

Frankly, I don't monitor my retirement accounts anywhere near as closely as I should.

Who has the time?

But I do try to look at them every quarter or so, to think about how they are doing and what I should change, if anything.

So I signed on and looked at my IRA, and was reading through the various positions.

And I noticed information for a company I'd never heard of!

How did a company I'd never heard of appear in my account? Did I fumble-finger some trade, months ago? Was I hacked? The "purchase history" for the new company showed three apparently legitimate trades, dating back to 2011.

Then I did some web surfing, and arrived at: Kimberly-Clark Announces Details for Completion of Kimberly-Clark Health Care Spin-Off

Kimberly-Clark Corporation (NYSE: KMB) today announced the record date, distribution date and distribution ratio for the previously announced tax-free spin-off of its health care business. The spin-off will form the new publicly traded company, Halyard Health, Inc. Kimberly-Clark also increased its 2014 share repurchase program to take into account expected proceeds as a result of the spin-off.

Kimberly-Clark shareholders will receive one share of Halyard Health common stock for every eight shares of Kimberly-Clark common stock held as of the close of trading on Oct. 23, 2014, the record date for the spin-off.

And, sure enough, the dates of the three trades are precisely the three occasions on which I purchased Kimberly-Clark stock in my IRA.

The computers had quietly taken care of it all.

But, really, I ought to pay more attention to my retirement accounts.

Maybe next year, he says...

Wednesday, December 17, 2014

In which I try to learn more about things I don't understand.

Building on this post of a month ago, and also closely related to Behind the Beautiful Forevers, today I came across a fascinating article on Quartz: The secret to the Uber economy is wealth inequality

These luxuries are not new. I took advantage of them long before Uber became a verb, before the world saw the first iPhone in 2007, even before the first submarine fibre-optic cable landed on our shores in 1997. In my hometown of Mumbai, we have had many of these conveniences for at least as long as we have had landlines—and some even earlier than that.

It did not take technology to spur the on-demand economy. It took masses of poor people.

It's a fascinating article, and not too long.

Well worth reading.

Tuesday, December 16, 2014

Those who do not remember history...

... or something like that.

Anyway, BIG STORM UPDATE: California Drought Situation Improves says:

The much anticipated big storm that hit California on December 10-12 lived up to its expectations with torrential rains, heavy snow in the Sierra, and high winds (although these were not as powerful as forecast). A small tornado (EF-0) even touched down in South Los Angeles Friday morning, December 12th. Rainfall in the San Francisco Bay Area was especially impressive with San Jose picking up 3.23” on December 11th, its 3rd greatest calendar day rainfall on record (POR since 1893). Venado, in Sonoma County, received 9.44” in 24 hours. What was most welcome, however, was that the heavy rain also drenched the southern portions of California including the Central Valley, erasing the seasonal precipitation deficits that were still in place just a week ago.

but also notes that:

Although neither of these changes are any guarantee that the winter will continue to be a wet one, they are at least a good sign. The RRR (‘Ridiculously Resilient Ridge’) that has been the hallmark of drought is nowhere in sight (for the time being). Of course, this is the way the season of 2012-2013 began when a very wet November and December came to an end by January when the RRR set up and didn’t let go for the following two years.

Oh, yes, we remember that well.

The story mentions Venado, a place I've never been. Mill Creek Road looks like it would be a fun drive on a nice summer day. I'll have to try that sometime...

Meanwhile, though, the state is still bone dry, so: let it snow, let it snow, let it snow.

Tuesday, December 9, 2014

Markov chain humor

On the heels of last year's git man page generator, we now have:

  • The Doom that Came to Puppet
    Posts generated by a Markov chain trained on the Puppet documentation and the assorted works of H. P. Lovecraft. Created by @branan using barrucadu/markov. Inspired by King James Programming.

Very, very geeky.

But quite humorous (assuming you know a little bit about Lovecraft and a little bit about Puppet).

Here's a nice example:

“As our guarded messages stated, we rested at midnight after our day of terror and bafflement—but not without a tentative plan for one or more specific features”

Monday, December 8, 2014

Indeed, you can't simply call for a Tow Truck at this point...

The team at Wired have been doing a good job covering some of the events in the current Volvo Ocean Race:

  • Grueling 39K-Mile Yacht Race Tests the Sanity of Cramped Crews
    The 2014-15 edition of the race, which takes place every three years, will cover 39,000 miles, hit six continents, and run from October to June. This is the first time it will be a “one-design” race: All entrants must use a specially designed boat—the $6 million Volvo 65—with the same exact specifications. The new carbon-fiber boats, designed by Farr Yacht Design in Annapolis, Md. specifically for the next two Volvo Ocean Races and assembled in different spots around the world, are strong and sturdy.
  • Why the World’s Best Sailors Still Hit Reefs in Open Water Races
    But even with precautions and top equipment, you don’t always know what’s around you. The charts that record things like reefs can be off by up to five miles at some points, says Will Oxley, the navigator for Team Alvimedica. The archipelago where Vestas Wind ran aground (16°32’00.0″S 59°32’00.0″E) isn’t the proverbial uncharted territory, but it’s not particularly well charted, either. Because it’s not on a shipping route, rarely hosts races, and is nearly uninhabited, there’s been little need for detailed mapping. “This is quite new territory,” Oxley says.
  • How to Repair a $6M Racing Yacht You Sailed Into a Reef
    The good news is that putting in fresh carbon fiber “is relatively easily done,” Hogoboom says. The companies that built the boats still have the molds for all the parts (the current design will be used for the 2017-2018 race), and they could start making replacements while the damaged boat is dragged back. Then it’s just a question of cutting out the damaged areas and bonding on the new bits. Nonetheless, it would require a big commitment: Four companies, each in a different country, contributed parts of the boat, and they would all probably need to provide replacement parts.

Saturday, December 6, 2014

Early December link clearance

If the rain is keeping you indoors, here's some stuff to read.

  • The Case for Slow Programming
    Venture-backed software development here in the San Francisco Bay area is on a fever-pitch fast-track. Money dynamics puts unnatural demands on a process that would be best left to the natural circadian rhythms of design evolution. Fast is not always better. In fact, slower sometimes actually means faster – when all is said and done.
  • The Myth of the Interchangeable Programmer: Can't We Just Offshore Him?
    The problem here is that the SMFs incorporate a number of flawed assumptions. The first of these assumptions is that programmers are fungible. The SMFs assume all programmers will contribute roughly the same amount to a project and that all programmers are interchangeable.
  • Immigration is about talent, not costs
    So you can be sure that all things being equal we would prefer to hire local talent for a position. But all engineers are not created equal, and great engineers are scarce worldwide, so when we find great engineers abroad we make them compelling offers and try to hire them.
  • Google Test Automation Conference
    All of the GTAC 2014 video recordings and slides are publicly available. You can watch them from the GTAC 2014 YouTube playlist or browse the talks below
  • Never Send a Human to do a Machine’s Job: How Facebook uses bots to manage tests
    Developers aren't the jerks, bots are!

    Filing tasks, sending email on failure isn't enough.

    All tests need owners, if we fail to blame a revision.

  • From Test Driven Development to Behavioral Driven Design
    Behavioral-driven development was introduced by Dan North with issues he continually came across in test-driven development. (2) He suggested that instead of simply writing tests, developers should think of specifying behaviors, which is how the users want the feature to behave. In BDD, you should always start with the features that are most important to the users. Through collaboration and continual feedback, the practice of knowing what is most important becomes clearer.
  • Too Big to Fail
    "The system is slow" is a poor bug report—in fact, it is useless. However, it is the one most often uttered in relation to distributed systems. Typically the first thing that users of the system notice is that the response time has increased and that the results they get from the system take far longer than normal. A distributed system needs to express, in some way, its local and remote service times so that the systems operators, such as the devops or systems administration teams, can track down the problem. Hot spots can be found through the periodic logging of the service request arrival and completion on each host. Such logging needs to be lightweight and not directed to a single host, which is a common mistake. When your system gets busy and the logging output starts taking out the servers, that's bad. Recording system level metrics, including CPU, memory and network utilization will also help in tracking down problems, as will the recording of network errors. If the underlying communications medium becomes overloaded, this may not show up on a single host, but will result in a distributed set of errors, with a small number at each node, which lead to chaotic effects over the whole system. Visibility leads to debuggability; you cannot have the latter without the former.
  • Exactly-Once Delivery May Not Be What You Want
    I think there are two lessons here for people building distributed systems. One is that end-to-end system semantics matter much more than the semantics of an individual building block, and sometimes what seems like a very desirable semantic for a building block may end up making the end-to-end problem harder. The other is that simple, practical, solutions like unique IDs can make really hard problems much easier, and allow us to build and ship real systems that work in predictable ways.
  • Aliens in the Valley: The complete history of Reddit, the Internet's front page
    This is the story of how a bootstrapped startup with a funny name and no initial ties to the tech scene outlasted better-funded competitors, survived founder drama, endured tensions with its parent company and later navigated life as a standalone business — all in order to build the front page of the Internet.
  • PostgreSQL’s Powerful New Join Type: LATERAL
    a LATERAL join is like a SQL foreach loop, in which PostgreSQL will iterate over each row in a result set and evaluate a subquery using that row as a parameter.
  • Steve Hetzler's "Touch Rate" Metric
    As an example of the use of these graphs, Steve analyzed the idea of MAID (Massive Array of Idle Drives). He used HGST MegaScale DC 4000.B SATA drives, and assumed that at any time 10% of them would be spun-up and the rest would be in standby. With random accesses to data objects, 9 out of 10 of them will encounter a 15sec spin-up delay, which sets the response time limit. Fully powering-down the drives as Facebook's cold storage does would save more power but increase the spin-up time to 20s.
  • Java for Everything
    If you and I have a contest to write a simple blogging system and you’re using (say) Python, you’ll have something interesting in 30 minutes using pickling and whatnot, and it’ll take me two days to build something with MySQL. Many language choices are based on trivial contests like these. But after two weeks of development, when we both have to add a feature, mine will take at most as long as yours, and I won’t be spending any time figuring out how to get my system to handle so many users, or tracking down why some obscure if clause breaks because you misspelled the name of a function, or figuring out what the heck this request parameter contains.
  • The Unreasonable Effectiveness of C
    C is the total package. It is the only language that's highly productive, extremely fast, has great tooling everywhere, a large community, a highly professional culture, and is truly honest about its tradeoffs.

    Other languages can get you to a working state faster, but in the long run, when performance and reliability are important, C will save you time and headaches.

  • The Last Homesteads of Wonder Valley, California
    Arid, undeveloped, and sparsely beautiful, Wonder Valley was deemed “useless” by the federal government, and so was portioned out for mostly recreational use at $10 to $20 an acre. The sole requirement was that the applicant "proved up" the land with a small house.
  • NPR’s Book Concierge: Our Guide To 2014’s Great Reads
    Use the filters below to explore some 250 titles NPR staff and critics loved this year.

Friday, December 5, 2014

Great article on benchmarking SSDs

Check out this great article: The SSD Endurance Experiment: Two freaking petabytes: The survivors soldier on to another really big number

The authors have been running a benchmark on a bunch of SSD drives for more than a year!

They're looking at reliability, manageability, and performance.

We benchmarked all the SSDs before we began our endurance experiment, and we've gathered more performance data after every 100TB of writes since. It's important to note that these tests are far from exhaustive. Our in-depth SSD reviews are a much better resource for comparative performance data. What we're looking for here is how each SSD's benchmark scores change as the writes add up.

Yes, you read that right: "every 100 TB".

They are forcing petabytes of data through their SSDs, and observing how they fare.

And they fare, in fact, quite well:

The results of our experiment do, however, point to some more general conclusions about SSDs as a whole. Although only two drives made it to 2PB, all six wrote hundreds of terabytes without issue, vastly exceeding their official endurance specifications. More importantly, the drives all survived far more writes than most users are likely to generate. Typical consumers shouldn't worry about exceeding the endurance of modern SSDs.

The whole article is pretty interesting.

The end of TNR

For about 15 years, from about 1985 to about 2000, I was not only a reliable subscriber but a devoted reader of The New Republic

I loved pretty much everything about the magazine: its style, its selection of topics, its ability to attract the best writers.

Most of all, I loved its fearlessness; it always seemed to me to be willing to ask the hardest questions and not to shy away from the answers.

I'm sure that I changed, but sometime around the turn of the millenium The New Republic changed even more. I drifted away, reluctantly, and finally stopped reading it entirely.

Now comes news of the final collapse, as covered in Slate: What We Lost With the Loss of the New Republic

I sort of found my own voice, as an adult, by reading TNR. Even though it's nothing like it once was, I'll miss it.

Wednesday, December 3, 2014

Post-L-Triptophan reading list

OMG there's so much to read.

It's amazing how much accumulates when you take a week off...

  • The QA Mindset
    My concern is that the absence of QA is the absence of a champion for aspects of software development that everyone agrees are important, but often no one is willing to own. Unit tests, automation, test plans, bug tracking, and quality metrics. The results of which give QA a unique perspective. Traditionally, they are known as the folks who break things, who find bugs, but QA’s role is far more important. It’s not that QA can discover what is wrong, they intimately understand what is right and they unfailingly strive to push the product in that direction.

    I believe these are humans you want in the building.

  • Gangnam Style Video Overflows YouTube Counter
    If you hover your mouse over the counter, it spins like a slot machine; if you hold the mouse there long enough it will show a negative number. But the negative number is not what I expected. Is there a bug in the Easter egg?
  • The real and complete story - Does Windows defragment your SSD?
    The short answer is, yes, Windows does sometimes defragment SSDs, yes, it's important to intelligently and appropriately defrag SSDs, and yes, Windows is smart about how it treats your SSD.
  • Future of Popular Coding Tool In Doubt After Public Split
    The split highlights the tensions that often exist between the corporate sponsor of an open source project and the many other coders and businesses who use it and help build it. Docker, the company behind a new approach to cloud computing that has exploded in popularity in the past year and half, is in a similar boat, with some community members complaining that the parent company has strayed from its original mission and one outfit going so far as to create a new rival for the project.
  • Two Farmers and Common Knowledge
    They needed some way to make their cycling workers reliable again, and that meant finding a technical solution to the problem.
  • Uber's Secret Weapon: The World
    But the international aspect is if anything even more important. This might not be obvious to people in San Francisco, who are spoiled with dozens of hopeful and well-funded startups, many of which are doing much the same thing that Uber is aspiring to. But leave the Bay Area, and the fears and frustrations of trying to get a cab start getting magnified – especially when you’re in a foreign country. The value of Uber is only partially in the service it provides; increasingly, it’s also in the global ubiquity of that service.
  • Introduction and Initial Plans
    Welcome to Quantum OS! We are working on developing an operating system based upon Linux which conforms to Google’s Material Design guidelines. The focus will be on creating a stable and easy-to-use operating system with a heavy emphasis on well-thought-out design.
  • God's Lonely Programmer
    TempleOS is more than an exercise in retro computing, or a hobbyist’s space for programming close to the bare metal. It’s the brainchild—perhaps the life’s work—of 44-year-old Terry Davis, the founder and sole employee of Trivial Solutions. For more than a decade Davis has worked on it; today, TempleOS is 121,176 lines of code
  • The Art of Insight in Science and Engineering
    To master complexity, we can organize it or discard it. The Art of Insight in Science and Engineering first teaches the tools for organizing complexity, then distinguishes the two paths for discarding complexity: with and without loss of information. Questions and problems throughout the text help readers master and apply these groups of tools. Armed with this three-part toolchest, and without complicated mathematics, readers can estimate the flight range of birds and planes and the strength of chemical bonds, understand the physics of pianos and xylophones, and explain why skies are blue and sunsets are red.
  • Out the Window
    A group of researchers that were immensely valuable according to Microsoft’s own metric just a couple of months before were thrown out to the hands of Microsoft’s competitors that were more than happy to oblige. Similarly, previously valued research projects were carelessly lost (quite possibly to be picked up by others). Excellence as defined by Microsoft did not protect you, impact did not protect you
  • Towards a more perfect link underline
    Overall, the underline is much thicker than Wichary’s ideal and sits too close to the text for my comfort. But most damningly to my mind, the underline does not change weight to adapt to the weight of the font, leading to unfortunate mismatches
  • Why Renewable Energy (Alone) Won't Full Solve the Problem
    I argue that all employees should be limited only by their ability rather than an absence of resources or an inability to argue convincingly for more. This is one of the most important yet least discussed advantages of cloud computing: taking away artificial resource limitations in support light-weight experimentation and rapid innovation. Making individual engineers and teams responsible to deliver more value for more resources consumed makes it possible encourage experimentation without fear that costs will rise without sufficient value being produced. And, because cloud computing is so inexpensive and comes without a long term commitment, a single engineer to do a trial run of a 1,000 core analysis to improve supply chain logistics without appreciable financial risk. If it works, keep doing it and reap the economic gain. If it doesn’t work, little was spent and it may have been a failed experiment but it was an inexpensive failed experiment. Economic systems are very powerful at driving innovation.
  • Best at Everything? It’s Closer to True
    It makes sense considering that the quarterback, Peter Williams, is studying, actually, to be a rocket scientist. Or that the senior linebacker Cameron Wagar, a mechanical engineering major, last week endured a review of a semester-long group project on Monday, a biology report due Wednesday and a biology exam Friday, just before the team bus left for Bangor, Me., where the undefeated Engineers faced Husson University in the first round of the N.C.A.A. Division III playoffs.
  • Some of these Things are not like the others
    What’s more important is to consider all the discrete moving parts, the small pieces loosely joined, and how they fit together: networked systems, objects, and actors, interacting with one another at a distance, over APIs, legislature, process. These are all interfaces for the city. They’re not all digital technology, either – they’re frequently transitions between state, the digital communicating with the human, or the physical, or the abstract, and vice versa.

    Cities are made of many different things, in many states, and digital technology’s role shouldn’t just be to bridge between all those states and the digital – but also to link things of one state to another.

  • How browsers get to know you in milliseconds
    When the user clicks on a link or enters a URL, the server returns a page with ad tags that contain JavaScript to invoke an auction with a particular exchange. The browser then issues a request to an exchange such as AppNexus, or to a supply-side platform (SSP) that then sends it to the exchange.
  • Nifty tech delivers ineffective crap at incredible speed!
    Hold on a minute. Online display ads are terribly ineffective, despite all the bleeding-edge technology being thrown at them?

    Close. But not despite. Because.

  • Unpacking privacy
    The problem is that if the medium is targetable, then the best strategy for an individual site is to do targeting, even if (because of the signaling value of its content) the site would do better in a system where no user could be targeted. When we stop thinking about privacy as a big, complicated, hard concept, and try to break out some kind of Minimum Viable Privacy, just enough to protect that "car intender" from site to site tracking, then ways out of the race to the bottom start to present themselves.
  • Consider Your Perspective: A Chart – Gold vs U.S. Debt
    There was a period where the two seemed to move together, and there was a much longer period where they did not. One can selectively choose start and endpoints that torture the data in an effort to further a narrative or confirm a bias, but when one does that, one should not be shocked when one’s conclusions turn out to be incorrect.
  • The Ship Breakers
    The ships are driven right up onto shoreline lots set aside for ship breaking, then attacked by hammer and blowtorch until all usable material has been stripped away to be sold or recycled.
  • Under London
    I'm a sucker for images of the human form stranded amidst the shadows of massive, dimensionally abstract spatial environments, so I thought I'd post these purely as eye candy.
  • Yerba Buena Island
    aids to navigation became more important and the unique octagonal lighthouse and the fog signal, which are still operational on the southern end of the island, were completed in 1875. The beautiful house just above them was constructed for the lighthouse keeper and now serves as the home of a Coast Guard admiral.
  • The Missing Men
    Why didn’t a Rolling Stone writer talk to the alleged perpetrators of a gang rape at the University of Virginia?
  • Ill Doctrine
    ill Doctrine is a video blog hosted by Jay Smooth, founder of New York's longest running hip-hop radio show, WBAI's Underground Railroad.

Tuesday, December 2, 2014

Behind the Beautiful Forevers: a very short review

As I mentioned the other day, I had some time to read recently, so I caught up on a few books that had been sitting on my Kindle for a while.

One, in particular, was Katherine Boo's Behind the Beautiful Forevers: Life, Death, and Hope in a Mumbai Undercity.

Beautiful Forevers is simply an astonishing book.

I thought I knew a little bit about India. Though I've never been there, I've worked closely with Indian colleagues (both India-based and U.S.-based) on many occasions, I've known both family and co-workers who have traveled extensively in India, and I've read about India over many years.

But Beautiful Forevers is a shocking eye-opener.

It's non-fiction that reads like the latest crime thriller you pick up from the newsstand rack.

It's investigative journalism of the first order, digging down below the surface and helping you truly understand what is happening and why.

And it's the heart-breaking, emotional, compelling story of some of the most interesting people you'll never meet.

The story that Boo tells is horrible, tragic, and shattering.

But it is also important, and timely, and necessary.

If you get a chance, take the time to read Beautiful Forevers.

The Martian: a very short review

I happened to do some traveling recently and had some time to read.

One of the books I read was Andy Weir's The Martian: A Novel.

The Martian is a book about survival.

On Mars.

There are many great books about survival, both fiction and non-fiction, and the topic is very popular in movies, as well (Gravity is a recent example; Castaway a slightly older one).

The Martian's great appeal in this genre is that, well, it occurs on Mars.

And the author is fascinated by the concept of extraterrestial exploration, and has done his research, and thus writes with both passion and substance.

It doesn't hurt that Weir clearly has an abundance of writing talent, and can spin a tale.

The book just races along. I'll often dawdle in a book, as I have so many other distractions available. But The Martian is a compelling thriller and I sped through it in barely 10 hours.

I hope Weir continues to write, and I hope his subsequent efforts are as good as his debut.