A peculiar behavior of the Windows "echo" command

Here's an odd little behavior. On your Windows computer, try this:

echo bryan > c:\temp\bryan.txt

(You should probably cut-and-paste the command to get it exactly, as that turns out to be important for this example.)

Now bring up the file c:\temp\bryan.txt in your favorite editor.

Meanwhile, try this command, too:

echo bryan> c:\temp\bryan2.txt

And bring up the file c:\temp\bryan2.txt in your favorite editor.

Do you see the difference?

Yes, the two commands look very similar, and the two files they produce look very similar, but they are slightly different.

Do you see it yet? It's pretty subtle ...

OK, here's the answer: in the first case, there is a trailing space character in the resulting file, but in the second case there is not! That is, the first command results in a single-line file with a line that is six characters long: 'b', 'r', 'y', 'a', 'n', ' ', while the second command results in a single-line file with a line that is five characters long: 'b', 'r', 'y', 'a', 'n'.

You'll get similar behavior when you use the "pipe" symbol ('|') to construct a command pipeline; that is,

echo bryan | some-program-that-reads-stdin

is slightly different than

echo bryan| some-program-that-reads-stdin

because in the first case the program will read a 6-character line from stdin, while in the second case the program will read a 5-character line from stdin.

I'm sure this is probably documented somewhere, but it was a surprise to me so I thought it would be worth a short note.

Epic SQL statement

Today's issue of The Daily WTF has a wonderful example of an epic SQL statement.

Of course, nobody sits down and writes something like this from scratch. A bit of code like this evolves, over time, incrementally, as developers add to it.

And, as many of the commenters note, this statement actually packs an enormous amount of functionality into a very compact form. It is dense.

There's something like 20 tables mentioned in the FROM clause; I bet this monster is fun to run through a modern query optimizer!

In a previous job, I built and maintained a Continuous Integration system which operated a fleet of automated build robots that performed build and test tasks and provided tools to analyze and interpret the results. The core logic of the system was as follows:

1. Design the underlying database schema as carefully as you can


2. Express the primary operations of the system as database queries; pack as much intelligence as possible into the query itself


3. Provide a thin layer of execution (in the build bots) and visualization (in the web UI) logic around the underlying database system; let the basic database structure show through (see point #1)

Many of our queries were rather complex; the most complex query was the one which scheduled the waiting jobs to the available bots, matching priorities and capabilities according to the system's rules. The query was hell to write, but once we got it built, the rest of the system just flowed.

This afternoon's score is: Proxoid 0, PDANet 1

We were trying to figure out how to tether our old faithful backup notebook computer (an ancient Compaq Presario) to the net using our current smartphone, an LG Ally.

After a bit of hunting and pecking we gave Proxoid a try. From the home page:

Proxoid is a ProxyServer on Android. It allow users of not rooted phones to use their phone's internet access from a computer.

It seemed reasonable, but we couldn't get it to work; the driver installation never completed, and we couldn't get Windows XP to comprehend the phone that was connected via the USB cable.

So we gave PDANet a try. From the PDANet home page:

PdaNet does NOT require root access or hacking your phone's firmware in order to work. It is just a regular Android application that works on all Android phones as-is. Your phone can connect to the data service, WiFi, or even through VPN and PdaNet will share the connection with your computer.

The PDANet installer whirled and hummed, and it, too, was having trouble getting the right drivers installed.

HOWEVER, the PDANet installer gave us a clue:

Sometimes if USB debugging is checked but there is still no "Android ADB Inteface" listed when phone is attached or if there is a driver error, your system may need the official USB driver for some reason. If that is the case, install one of the following drivers on your computer first:

Driver for LG Android Phones

Once we got that special LG driver installed, the rest of the PDANet installation proceeded without complaint, and a few minutes later we were online, tethered via the phone!

Quite possibly, the installation of the LG driver would have enabled Proxoid to work, but since the PDANet software seemed to be working and diagnosed the problem well and clearly, we've decided to stick with that for now.

Smart phones are a fast-moving technology; I suspect that, in a year, we'll be setting up a new computer with a new smartphone, and we'll be amazed how much things have changed, and how easy they've become. Still, the PDANet installation process was overall pretty reasonable, so I'm suitably impressed.

The best steam engine on the planet

I find this technique baffling. Why would you want to go to all this trouble and complexity when you could just propagate true virtual machines? It all seems so last century, one of those incredible feats of virtuosity that makes your jaw drop in admiration, then shake your head and think: Why? Why? Why?

If anything, I think that I am still setting up too few virtual machines, not too many. With disk space running at about $100/TB, and memory at somewhere around $15/GB, any desktop machine that you set up this year for serious development work should have at least 5 TB of disk and 16 GB of memory, plenty of horsepower to support 50-100 installed virtual machine images, with at least 2-3 live and active at any given time.

And it's about 30 seconds to suspend one VM and revive another.

Well, anyway, it's always interesting to read about alternate techniques, and I hadn't known about debootstrap before, nor about rinse, and my experience with chroot mostly involved security considerations, not alternate operating system configurations.

So I learned, and I'm smarter, and I'm grateful for that, but thank you very much I think I'll stick with my virtual machines for now.

SandForce SSD report from CES

On his AnandTech web site, Anand Lal Shimpi has a report from CES about his impressions of the latest hardware samples of the OCZ Vertex Pro 3 solid state drive technology:

OCZ let me run some of my own Iometer tests on the drives to verify the claims. Surprisingly enough, the Vertex 3 Pro looks like it’s really as fast as OCZ and SandForce are claiming. When running highly compressible data (pseudo random in Iometer) at low queue depths, I get 518MB/s sequential write speed and nearly 500MB/s for sequential read speed.

In addition to standalone drives, OCZ was also showing a packaging of 4 devices into a RAID-0 configuration on a single card.

A few months back, AnandTech had a very nice in-depth analysis of the underlying technology and architecture of these devices:

SandForce’s controller gets around the inherent problems with writing to NAND by simply writing less. Using real time compression and data deduplication algorithms, the SF controllers store a representation of your data and not the actual data itself. The reduced data stored on the drive is also encrypted and stored redundantly across the NAND to guarantee against dataloss from page level or block level failures. Both of these features are made possible by the fact that there’s simply less data to manage.

These are really, really, really fast devices:

Now the shocker. Thanks to 6Gbps and ONFI 2/Toggle support, the SF-2000 will support up to 500MB/s sequential read and write speeds. On an 8 channel device that’s actually only 62.5MB/s per channel but the combined bandwidth is just ridiculous for a single drive. At full speed you could copy 1GB of data from a SF-2000 drive to another SF-2000 drive in 2 seconds. If SandForce can actually deliver this sort of performance I will be blown away.

On my own moderately-high-end hardware nowadays, I tend to see about 50 Mb/s from my (traditional technology) storage devices, and on our fast internal lab machines we see double that. So these new devices will be approximately 5-10 times faster than the storage devices we're using now!

Looks like the age of SSD devices is upon us.

Netflix engineering and the Chaos Monkey

Here's a great excerpt from an interesting "lessons learned" article on the Netflix technical blog: 5 Lessons We’ve Learned Using AWS:

One of the first systems our engineers built in AWS is called the Chaos Monkey. The Chaos Monkey’s job is to randomly kill instances and services within our architecture. If we aren’t constantly testing our ability to succeed despite failure, then it isn’t likely to work when it matters most – in the event of an unexpected outage.

I love the idea of the Chaos Monkey!

Testing error recovery is hard. It's often hard to provoke errors. It's fairly straightforward to provoke errors that are caused by bad input, so you should always have a thorough suite of tests which tries lots of invalid invocations of your code: syntax errors, parameters out of range, missing values for required fields, invalid combinations of requests, etc.

It's harder to provoke errors that are caused by other conditions: resource shortages, disk or network I/O errors, etc. In a number of my tests I provoke these errors using surrogate mechanisms:

• To simulate I/O-related problems I tamper with file or directory permissions, or I remove or rename files and directories that the program wants to access


• To simulate network problems I shutdown one or the other end of a conversation, or I use invalid network addresses

There are more sophisticated tools for doing this, for example check out Holodeck.

And you can also modify your application so that it is easier to test; quite commonly this involves modifications which allow testers to force the software through error conditions. This is often called "testability"; here's a pointer to a recent testing conference -- note how many of the talks are focused on various aspects of testability. Currently, much of the focus in the testing world is on the notion of "mock" objects, and indeed they can be very powerful and worth building into your test harness. Here's an interesting recent example: Mocking the File System to Improve Testability.

You can also use randomization, and stress: here at my day job we have something we call the Submitatron, which is a tiny little script that simply loops around, generating arbitrary data and sending it to the server. Similar techniques, which focus more on randomization, are often referred to as "fuzz testing".

But the most important thing is to think about testing, think about errors, think about failures, and try it out!

So, big kudos to the Netflix team and their Chaos Monkey!

Postcards from the bleeding edge

The results of some cross-firing neurons this morning:

First, from a short article in the ACM Queue regarding the performance of the NFS networking protocols across trans-oceanic links, Bound by the Speed of Light:

Unfortunately, the speed of light gets involved when you start creating networks over global distances. It's typical for a transpacific network link to have a 120-ms round-trip time.

...

For every mile between the client and the server, a message cannot get to the server and back to the client in less than 10 microseconds, because light travels one mile in 5.4 microseconds in a vacuum. In a fiber-optic network, or in a copper cable, the signal travels considerably slower. If your server is 1,000 miles from your client, then the best round-trip time you could possibly achieve is 10 milliseconds.

Second, from a long article in the New York Times Business section, The New Speed of Money, Reshaping Markets:

In August, Spread Networks of Ridgeland, Miss., completed an 825-mile fiber optic network connecting the South Loop of Chicago to Cartaret, N.J., cutting a swath across central Pennsylvania and reducing the round-trip trading time between Chicago and New York by three milliseconds, to 13.33 milliseconds.

Then there are the international projects. Fractions of a second are regularly being shaved off of the busy Frankfurt-to-London route. And in October, a company called Hibernia Atlantic announced plans for a new fiber-optic link beneath the Atlantic from Halifax, Nova Scotia, to Somerset, England that will be able to send shares from London to New York and back in 60 milliseconds.

If the best possible time for a 1000 mile round trip is indeed 10 milliseconds, then achieving 13.3 milliseconds is superb progress.

But even 60 milliseconds to make a round-trip between London and New York can feel like an eternity, as the New York Times article points out:

Almost each week, it seems, one exchange or another claims a new record: Nasdaq, for example, says its time for an average order “round trip” is 98 microseconds — a mind-numbing speed equal to 98 millionths of a second.

These are different uses of the term "round-trip", of course, but the basic conclusion holds.

Unfortunately, both articles concentrate mostly on the mechanical details of order processing speed, pointing out why financial engineering companies tend to operate within a few miles of the major financial centers: New York, Chicago, London, Frankfort, etc., while I was hoping that the New York Times article would focus on the more interesting question (in my opinion), which has to do with the fairness and open-ness of modern electronic exchanges, and concerns questions that I think are still open from last spring's "Flash Crash":

But some analysts fear that some aspects of the flash crash may portend dangers greater than mere mechanical failure. They say some wild swings in prices may suggest that a small group of high-frequency traders could manipulate the market. Since May, there have been regular mini-flash crashes in individual stocks for which, some say, there are still no satisfactory explanations. Some experts say these drops in individual stocks could herald a future cataclysm.

For example, the NYT mentions the on-going debate about when information becomes visible on these exchanges:

Most of the exchanges have already eliminated a controversial electronic trading technique known as flash orders, which allow traders’ computers to peek at other investors’ orders a tiny fraction of a second before they are sent to the wider marketplace. Direct Edge, however, still offers a version of this service.

Nothing definite to report here, I guess: computers get faster, the world gets smaller, we continue to try to understand how to build fair and equitable markets. Just more postcards from the bleeding edge.