Sunday, October 26, 2014

And some software engineering things, too

Because, you know, that's just who I am.

  • FIT : Failure Injection Testing
    Simulating failure starts when the FIT service pushes failure simulation metadata to Zuul. Requests matching the failure scope at Zuul are decorated with failure. This may be an added delay to a service call, or failure in reaching the persistence layer. Each injection point touched checks the request context to determine if there is a failure for that specific component. If found, the injection point simulates that failure appropriately. Below is an outline of a simulated failure, demonstrating some of the inflection points in which failure can be injected.
  • Ice Cream and Distributed Systems
    Mary, Mom and Dad sat down and tried to figure out how to all agree on the problem with the fewest number of messages. Mary invented a simple scheme: when I asked her if I could have some ice cream, she messaged both my mom and dad and ask for their opinion, while asking that they didn't change their opinion until hearing back from her. If they both agreed, she'd go ahead and let them know she was going to serve dessert. If either said no, she let them know that the bowl would remain empty. The protocol, which they called two-phase commit after the frozen and liquid phases of ice cream, took four messages to complete.
  • Cuckoo Filters
    If you're going to use multiple choice hashing schemes, though, you should think about using cuckoo hashing. The ability to move keys around means you should get better space utilization; for example, even with 2 choices, if your buckets can hold 4 items, cuckoo hashing can get you about 95% space utilization. The problem with cuckoo hashing in this setting is that, for a Bloom filter, you want to just keep fingerprints of keys, not the keys themselves. So, when you want to move the key, how do you figure out where to move it to -- you no longer have the key to hash?
  • Instant Loading for Main Memory Databases
    While hardware limitations for fast loading have disappeared, current approaches for main memory databases fail to saturate the now available wire speeds of tens of Gbit / s. With Instant Loading, we contribute a novel CSV loading approach that allows scalable bulk loading at wire speed. This is achieved by optimizing all phases of loading for modern super-scalar multi-core CPUs.
  • Message Systems in Programming: Callbacks, Events, Pub Sub, Promises, and Streams
    Messaging systems are used to communicate in larger code bases by helping decouple classes that need to know about changes or happenings in certain areas of the code . One of Object Oriented Programming‘s core concepts is encapsulation. How you decide to allow objects to talk to each other has pro’s and con’s for each method and it’s good to know your options as you can use many together in effective hybrid approaches.

    This article will cover the 5 common ones you’ll often encounter.

  • Amazon Kinesis and Apache Storm: Building a Real-Time Sliding-Window Dashboard over Streaming Data
    In this whitepaper, we propose a reference architecture for ingesting, analyzing, and processing vast amounts of clickstream data generated at very high rates in a smart and cost-efficient way using Amazon Kinesis with Apache Storm. We also explore the use of Amazon ElastiCache (Redis) as an in-memory data store for aggregated counters and use of its Pub/Sub facility to publish the counters on a simple dashboard.
  • Avoiding the tragedy of the anticommons
    In his white paper for the Bio-Commons, RĂ¼diger Trojok writes about a significantly more ambitious vision for open biology: a bio-commons that holds biological intellectual property in trust for the good of all. He also articulates the tragedy of the anticommons, the nightmarish opposite of a bio-commons in which progress is difficult or impossible because “ambiguous and competing intellectual property claims…deter sharing and weaken investment incentives.” Each individual piece of intellectual property is carefully groomed and preserved, but it’s impossible to combine the elements; it’s like a jigsaw puzzle, in which every piece is locked in a separate safe.
  • Which Online Discussion Archetype Are You?
    What Mike created is a brilliant deconstruction of the various archetypes you'll encounter in any long running discussion group
  • 10 Tricks to Appear Smart During Meetings
    Opinions and data and milestones are being thrown around and you don’t know your CTA from your OTA. This is a great point to go, “Guys, guys, guys, can we take a step back here?” Everyone will turn their heads toward you, amazed at your ability to silence the fray. Follow it up with a quick, “What problem are we really trying to solve?” and, boom! You’ve bought yourself another hour of looking smart.
  • 15 Tricks to Appear Smart in Emails
    Whenever something good happens, always be the first to respond and always reply all. This will make you seem like a highly engaged team player.

Completely non-software-engineering things I'm reading

It rained a little bit yesterday. Nothing like it's been raining in Oregon and Washington, but maybe it's a start. I can see from the chart that Lake Shasta is still falling, but yesterday, for the first day in a long time, inflow exceeded outflow.

  • The Astonishing Story of the Federal Reserve on 9-11
    I had planned to spend this week on the thrilling topic of the discount window. It was plain old curiosity that took me to the internet to find out what the Federal Reserve did on 9-11. As it turns out, it was not an easy story to unravel and between late Sunday night when I first started reading and Tuesday night when I started writing I read several hundred pages of reports as well as the tiny amount of media reporting available. Here’s the thing I didn’t know and I’ll bet you a wheelbarrow of carrots you didn’t either, on 9-11 and the days which immediately followed, a relatively small number of people did some genuinely, physically heroic things in order to keep the economy from going off the rails and none of them were named Alan Greenspan.
  • “Mount Thoreau” and the naming of things in the wilderness.
    And across Piute Canyon from it there stands another big peak, unnamed. On the maps it’s marked 12,691. If named after Thoreau, the two peaks would then form a gateway, like Scylla and Charybdis, through which hundreds of hikers would pass every year. Peak 12,691 is somewhat lower than Mount Emerson, but much more gnarly and interesting; the two peaks have much the same relationship that Emerson and Thoreau had, not just in size and aspect but in position, being close to each other but separated by a huge gulf of air. It was just like that in Concord.
  • Expert Critique of Burmese Cat Project
    However, keeping a colony of 40 cats is a vastly different proposition from keeping two or three cats in a home environment. With such a large colony, it is vitally important from a health perspective that cats are kept in a fresh, breezy environment at all times. I indicated that the solution would be to build an enclosure that surrounded Heritage House from water level to tree top and a shade cloth roof to provide some shade and protection from the rain.
  • Starship Size Comparison Chart
    Scale: 1 pixel = 10 meters
  • How Rebounds Work
    Much has been made about the player-tracking revolution in the NBA and how it will advance the state of basketball analytics. This is truly a brave new world; to date, a vast majority of the energy spent researching advancements has been aimed at developing richer characterizations of player performance and constructing newfangled scouting reports. That makes sense, but basketball is bigger than any one player or team, and it’s also important to realize that the same data set that tells us Chandler Parsons and Jimmy Butler ran a lot, or Patty Mills runs the fastest, also holds incredible information about how basketball works. This goes beyond properly evaluating individuals; we are on our way to being able to map basketball itself. This work will eventually help coaches, players, and press more elegantly understand ball movement, defensive positioning, offensive architecture, and, yes, rebounding.
  • What A Former Olympian And NFL Player Can Teach Us About Advertising And Marketing
    I’ve seen firsthand in football and business how victims can bring down the morale of an entire team. It’s impossible to build anything with a victim mentality.
  • FORGET VIDEO GAMES: Here's What It's Like To Put On A Costume And Go Live-Action Role Playing
    Live-action role-playing (or LARPing) was born on the fringes of American pop culture, a descendant of much-maligned hobbies like Dungeons and Dragons and other table games.

    In LARPing, players spend their weekend dressing up in costumes, adopting elaborate personae, and inhabiting a complex imagined world.

I don't have a Halloween costume this year. Maybe I'll go as Programmer of a Certain Age.

Saturday, October 25, 2014

From the Repertoire

I've been thoroughly enjoying my second taste of the online music classes developed by the Curtis Institute of Music.

Some time ago, I followed Jonathan Biss's delightful Exploring Beethoven’s Piano Sonatas.

This fall, I've been taking Jonathan Coopersmith's superb From the Repertoire: Western Music History through Performance.

I like the way that this class moves through a selection of different music from different time periods and schools, so that the class is always varied and never dull.

I also really like that Coopersmith's class is illustrated by performances by the Curtis Institute students themselves, which makes the music feel much more alive than watching some much older video-taped presentation, great though those older performances may be.

The classes are certainly aimed at a much more serious student of music than I am, but they are at a level where even a casual listener such as myself can enjoy them and learn.

Meanwhile, since I do know next to nothing about music theory, I'm happy to have stumbled upon Toby Rush's wonderful Music Theory for Musicians and Normal People. Rush's presentation style is delightful, the poster format works very well (for me, at least), and the individual lessons are presented in small digestable amounts, which fits my stupidly busy schedule.

I still don't understand why Coursera insists on operating these classes on a fixed schedule; it seems like a student such as myself, who has the time only to watch the videos, read the background materials, and listen to the performances, should be able to start such a class at any time. The computers don't care, after all; they have no notion of what day or month or year it is.

But no, currently you can only take Coopersmith's class, not Biss's class nor Steinhardt's.

Oh well, some mysteries are not to be solved, and I have more of Coopersmith to listen to now.

Sunday, October 19, 2014

A day at ARK 2000

We had the opportunity to spend a glorious day at ARK 2000, which is one of the facilities of a rather unusual organization called the Performing Animal Welfare Society.

Through the generosity of friends, we found ourselves with a pair of tickets to one of PAWS's annual fund-raisers, the "Elephant Grape Stomp." This event is sort of an open house to visit the sanctuary, which is located in the Sierra Nevada foothills, about 2 hours from our house.

During the event, we were able to visit three parts of the sanctuary:

  • The cats and bears area, which holds Siberian Tigers, African Lions, and American Black Bears, as well as at least one leopard (who was feeling unsocial so we didn't see her).
  • Bull Mountain, where PAWS has a facility for two male Asian Elephants (held separately, but adjacently)
  • The Asian and African Elephant compound, where about 10 female elephants are living in two separate areas.

At all three locations, booths were set up with information, local restaurants were serving delicious food, and local wineries (from the thriving Murphy's wine region) were pouring scrumptious Sierra Nevada wines.

Visiting ARK 2000 is sort of an unusual experience.

It is not a zoo, and the animals are not there to entertain you.

And it's not a breeding facility; they aren't trying to produce more of these animals here.

I would say it's more like a senior citizen facility for animals who have been taken from rather difficult circumstances and given a dramatically more humane situation in which to live out their lives.

Still, it was very nice and peaceful. The weather was superb, and we had all the time we wanted to stand quietly and watch the animals as they relaxed, contentedly, in their space.

Several of the staff were on hand, including the primary elephant keeper and the primary bear keeper, to answer questions and help explain what we were seeing and why.

And some of the sights were indeed unusual, such as the three custom transport containers that they use to move the elephants long distances (most recently used to bring three elephants from Toronto to California). This is not the sort of item you can get at your local hardware store!

For example, keeping bull elephants is rather different than keeping female elephants. The extraordinary strength and aggressive tendencies of the bull elephants means that they must be located in a particular situation, with a pen of fantastic strength. In some of the pictures, you can see, I think, the difference in the containment fences for the male elephant as opposed to those for the females. (Of course, the females are plenty strong enough; apparently they like to uproot the oak trees just for fun, and so the facility has built massive protective cages around some of the trees to try to keep the ladies from clearing them out entirely.)

I think the highlight, for me, were the 4 Siberian Tigers, absolutely majestic animals, who were all together in one pen and were particularly active, bounding around their space, playing together, alertly aware of everything and everyone that was around them.

There's lots of information about PAWS on their website. It's not obvious what is going to come of the organization now that its founder, Pat Derby, has passed on. Still, from all evidence they are still going strong, and hopefully they will find a new generation to continue their excellent work.

Thursday, October 16, 2014

He had me at "the Largest Ship in the World"

Don't miss Alastair Philip Wiper's photo-journalism essay about the building of the new Maersk Triple-E container vessels: Building the Largest Ship In the World, South Korea

The Daewoo Shipbuilding and Marine Engineering (DSME) shipyard in South Korea is the second largest shipbuilder in the world and one of the “Big Three” shipyards of South Korea, along with the Hyundai and Samsung shipyards. The shipyard, about an hour from Busan in the south of the country, employs about 46,000 people, and could reasonably be described as the worlds biggest Legoland. Smiling workers cycle around the huge shipyard as massive, abstractly over proportioned chunks of ships are craned around and set into place: the Triple E is just one small part of the output of the shipyard, as around 100 other vessels including oil rigs are in various stages of completion at the any time.

Wednesday, October 15, 2014

Stuff I'm reading, mid-October edition

There was wind last night, but no rain.

Rain to the north, they say.

But not here.

  • Harvest and Yield: Not A Natural Cure for Tradeoff Confusion
    Yield is the availability metric that most practitioners end up working with, and it's worth noting that its different from CAP's A. The authors don't define it formally, but treat it as a long-term probability of response rather than the probability of a response conditioned on there being a failure. That's a good common-sense definition, and one that fits well with the way that most practitioners think about availability.
  • Apple's "Warrant-Proof" Encryption
    Code is often buggy and insecure; the more code a system has, the less likely it is to be secure. This is an argument that has been made many times in this very context, ranging from debates over the Clipper Chip and key escrow in the 1990s to a recent paper by myself, Matt, Susan Landau, and Sandy Clark. The number of failures in such systems has been considerable; while it is certainly possible to write more secure code, there's no reason to think that Apple has done so here. (There's a brand-new report of a serious security hole in iOS.) Writing secure code is hard. The existence of the back door, then, enables certain crimes: computer crimes. Add to that the fact that the new version of iOS will include payment mechanisms and we see the risk of financial crimes as well.
  • Keyless SSL: The Nitty Gritty Technical Details
    Extending the TLS handshake in this way required changes to the NGINX server and OpenSSL to make the private key operation both remote and non-blocking (so NGINX can continue with other requests while waiting for the key server). Both the NGINX/OpenSSL changes, the protocol between the CloudFlare’s server, and the key server were audited by iSEC Partners and Matasano Security. They found the security of Keyless SSL equivalent to on-premise SSL. Keyless SSL has also been studied by academic researchers from both provable security and performance angles.
  • Intel® SGX for Dummies (Intel® SGX Design Objectives)
    At its root, Intel® SGX is a set of new CPU instructions that can be used by applications to set aside private regions of code and data. But looking at the technology upward from the instructions is analogous to trying to describe an animal by examining its DNA chain. In this short post I will try to uplevel things a bit by outlining the objectives that guided the design of Intel® SGX and provide some more detail on two of the objectives.
  • Ads Don't Work That Way
    The key differentiating factor between the two mechanisms (inception and imprinting) is how conspicuous the ad needs to be. Insofar as an ad works by inception, its effect takes place entirely between the ad and an individual viewer; the ad doesn't need to be conspicuous at all. On the other hand, for an ad to work by cultural imprinting, it needs to be placed in a conspicuous location, where viewers will see it and know that others are seeing it too.
  • The ultimate weapon against GamerGate time-wasters: a 1960s chat bot that wastes their time
    Alan Turing proposed that an artificial intelligence qualified as a capable of thought if a human subject, in conversation with it and another human, cannot tell them apart; the strange thing about the Eliza Twitter bot is it doesn't come across as any more like a machine than those who keep repeating their points over and over and over, ad nauseum. It's difficult to decide who's failed the Turing test here.
  • Gabriel Knight’s Creator Releases Incredible 20th Anniversary Remake
    Staring at the remake version brings all those old memories of DOS mouse drivers and command prompts flooding back. Gazing at protagonist Gabriel Knight’s dazzling, polychromatic bookstore (your base of operations in New Orleans as the game begins) is like seeing the mental interpolation your brain made of the original pixelated wash beautifully, if weirdly, reified.
  • Bridge Troll
    I know this sounds a bit crazy, but trust me, there’s a troll up there! He or she, it’s tough to tell the gender of trolls, is approximately two feet tall, made of steel, and perched atop the southern end of the transverse concrete beam where the eastern cable makes contact with the road deck. The troll cannot be seen by car or from the bike path next to the bridge—you need to be underneath the bridge, on a boat to actually see the bridge troll.
  • Don’t Mourn the Passing of the New York Times Chess Column
    If those who know enough about the game to understand the diagrams in a newspaper chess column can access thousands of times more information, free and instantly, than a weekly column could possibly provide, then why run one at all? The answer is that most weekly newspaper chess columns don’t need to exist and won’t in the near future. The one exception: when there’s an excellent writer and chess professional at the helm, someone like Robert Byrne.
  • Serbia vs. Albania in Belgrade brings their troubled history to the fore
    But even if football takes the headlines, there is still the sense that Tuesday night might be an opportunity missed. On October 22, Albanian Prime Minister Edi Rama will visit Belgrade to discuss bilateral relations with his Serbian counterpart, Aleksandar Vukic. No Albanian leader has visited Belgrade since Enver Hoxha in 1946.

    It is significant, and maybe it brings a glimmer of hope that a repeat of Tuesday's fixture might one day be all about the game instead. Having a harmonious football match to oil the conversation would have done little harm, but the anticipation of that noticeable absense inside Partizan Stadium stands as a reminder that sport does not always have the power to untangle wider complexities.

  • In Transition
    We picked 10 of the most progressive skaters to choose one location each and film a full part.
  • Things I Won't Work With: Dioxygen Difluoride
    The paper goes on to react FOOF with everything else you wouldn't react it with: ammonia ("vigorous", this at 100K), water ice (explosion, natch), chlorine ("violent explosion", so he added it more slowly the second time), red phosphorus (not good), bromine fluoride, chlorine trifluoride (say what?), perchloryl fluoride (!), tetrafluorohydrazine (how on Earth. . .), and on, and on. If the paper weren't laid out in complete grammatical sentences and published in JACS, you'd swear it was the work of a violent lunatic. I ran out of vulgar expletives after the second page. A. G. Streng, folks, absolutely takes the corrosive exploding cake, and I have to tip my asbestos-lined titanium hat to him.

Sunday, October 12, 2014

Link Clearance

Man it was hot today. Doesn't fall mean that it starts to cool down?

  • The Physics of Doing an Ollie on a Skateboard, or, the Science of Why I Can’t Skate
    So here’s a thought – maybe I can use physics to learn how to do an ollie. Here’s the plan. I’m going to open up the above video of skateboarder Adam Shomsky doing an ollie, filmed in glorious 1000 frames-per-second slow motion, and analyze it in the open source physics video analysis tool Tracker.
  • 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI '14).
    As part of our commitment to open access, the proceedings from the Symposium are now free and openly accessible via the technical sessions Web page.
  • The Horror of a 'Secure Golden Key'
    A “golden key” is just another, more pleasant, word for a backdoor—something that allows people access to your data without going through you directly. This backdoor would, by design, allow Apple and Google to view your password-protected files if they received a subpoena or some other government directive. You'd pick your own password for when you needed your data, but the companies would also get one, of their choosing. With it, they could open any of your docs: your photos, your messages, your diary, whatever.
  • Malware needs to know if it's in the Matrix
    A presentation from UCSM's professor Giovanni Vigna (who runs the Center for CyberSecurity and Seclab), he's seeing more and more malware that keeps its head down on new infection sites, cautiously probing the operating system to try and determine if it's running on a real computer or if it's a head in a jar, deploying all kinds of tricks to get there.
  • 44 engineering management lessons
    30. Most conflict happens because people don’t feel heard. Sit down with each person and ask them how they feel. Listen carefully. Then ask again. And again. Then summarize what they said back to them. Most of the time that will solve the problem.
  • Unlocked 10Gbps TX wirespeed smallest packet single core
    The single core 14.8Mpps performance number is an artificial benchmark performed with pktgen, which besides spinning the same packet (skb), now also notifies the NIC hardware after populating it's TX ring buffer with a "burst" of packets.
  • Redis cluster, no longer vaporware.
    The consistency model is the famous “eventual consistency” model. Basically if nodes get desynchronized because of partitions, it is guaranteed that when the partition heals, all the nodes serving a given key will agree about its value.

    However the merge strategy is “last failover wins”, so writes received during network partitions can be lost. A common example is what happens if a master is partitioned into a minority partition with clients trying to write to it. If when the partition heals, in the majority side of the partition a slave was promoted to replace this master, the writes received by the old master are lost.

  • Using Git Hunks
    Many of the git subcommands can be passed --patch or -p for short. When used with git add, we can compose a commit with exactly the changes we want, instead of just adding whole files. Once you hit enter, you get an interactive prompt where you're presented with a diff and a set of options.
  • Slasher Ghost, and Other Developments in Proof of Stake
    The fundamental problem that consensus protocols try to solve is that of creating a mechanism for growing a blockchain over time in a decentralized way that cannot easily be subverted by attackers. If a blockchain does not use a consensus protocol to regulate block creation, and simply allows anyone to add a block at any time, then an attacker or botnet with very many IP addresses could flood the network with blocks, and particularly they can use their power to perform double-spend attacks – sending a payment for a product, waiting for the payment to be confirmed in the blockchain, and then starting their own “fork” of the blockchain, substituting the payment that they made earlier with a payment to a different account controlled by themselves, and growing it longer than the original so everyone accepts this new blockchain without the payment as truth.
  • Economies of Scale in Peer-to-Peer Networks
    I've been working on P2P technology for more than 16 years, and although I believe it can be very useful in some specific cases, I'm far less enthusiastic about its potential to take over the Internet.

    Below the fold I look at some of the fundamental problems standing in the way of a P2P revolution, and in particular at the issue of economies of scale.

  • A Scalability Roadmap
    You might be surprised that old blocks aren’t needed to validate new transactions. Pieter Wuille re-architected Bitcoin Core a few releases ago so that all of the data needed to validate transactions is kept in a “UTXO” (unspent transaction output) database. The amount of historical data needed that absolutely must be stored depends on the plausible depth of a blockchain reorganization. The longest reorganization ever experienced on the main network was 24 blocks during the infamous March 11, 2013 chain fork.
  • Why the Trolls Will Always Win
    But here’s the key: it turned out he wasn’t outraged about my work. His rage was because, in his mind, my work didn’t deserve the attention. Spoiler alert: “deserve” and “attention” are at the heart.