Friday, January 20, 2017

This and that

My challenge is simple: to master the nuances of my new job, all I have to do is to be comfortable with every page of this amazing 500 page masterwork: Is Parallel Programming Hard, And, If So, What Can You Do About It?. Happily, McKenney is a wonderful writer, but where will I find the time?

In the meantime, there's also a few other things that crept in while I was sleeping:

  • Who is Anna-Senpai, the Mirai Worm Author?
    The story you’re reading now is the result of hundreds of hours of research. At times, I was desperately seeking the missing link between seemingly unrelated people and events; sometimes I was inundated with huge amounts of information — much of it intentionally false or misleading — and left to search for kernels of truth hidden among the dross. If you’ve ever wondered why it seems that so few Internet criminals are brought to justice, I can tell you that the sheer amount of persistence and investigative resources required to piece together who’s done what to whom (and why) in the online era is tremendous.
  • Cloudflare’s Transparency Report for Second Half 2016 and an Additional Disclosure for 2013
    Because of the gag order, I had to sit in silence, implicitly confirming the point in the mind of the staffer. At the time, I knew for a certainty that the FBI’s interpretation of the statute diverged from hers (and presumably that of her boss).

    Cloudflare fought this battle for four years even after the request for customer information had been dismissed. In addition to protecting our customers’ information, we want to remain a vigorous participation in public policy discussions about our services and public law enforcement efforts. The gag rule did not allow that.

    Now that this gag order has been lifted, Cloudflare is able to publish a more accurate transparency report to its customers and constituents. For us, this is not the end of the story, but the beginning of a more robust, fact-informed debate.

  • Operation Grand Mars: a comprehensive profile of Carbanak activity in 2016/17
    Trustwave has tracked Carbanak activity in the latter half of 2016 and found them to be targeting hospitality and retail victims in Europe and North America, specifically targeting their internal corporate secrets and protected payment card data. Trustwave published a blog earlier in the year within initial findings but has now released the complete malicious campaign profile in a 45-page Advanced Threat Report. The blog post below is a summary of the malicious profile we have developed for this actor. The complete report on this activity can be found here:
  • Babai Strikes Back
    We were watching history. From the talk I tweeted the new news though Bill Cook, also in the audience, beat me to the punch. Babai went on to describe the issue, an error in the analysis of the running time in the recursion, and the fix, basically a way to avoid that recursive step, but I can't do it justice here. At the end he proclaimed "So it is quasipolynomial time again". And so it was.
  • On exp(exp(sqrt(log n))) algorithms.
    Improving quantitative parameters such as running time or approximation factor is very useful as intermediate challenge problems that force us to create new ideas, but ultimately the important contribution of a theoretical work is the ideas it introduces and not the actual numbers.
  • No One Questioned This Hedge Fund’s Madoff-Like Returns
    But until Murray Huberfeld, who founded Platinum with Nordlicht, was caught up in a New York City municipal-corruption probe in June, no one at the fund had been charged with wrongdoing. Within weeks of Huberfeld’s arrest, federal agents raided Platinum’s midtown Manhattan office. On Dec. 19, Nordlicht and six others were arrested in what the government called a $1 billion fraud. Nordlicht and Huberfeld have pleaded not guilty, and Platinum’s main fund is being wound down after filing for bankruptcy. Montieth Illingworth, a spokesman for Platinum, declined to comment.
  • Why Are So Many Bee Trucks Tipping Over?
    All of this is to say that I have little more insight, except that you definitely shouldn’t trust anyone! Except for maybe bees, considering they continue to provide us avocados and almonds and blueberries and all the foods that we’ve pushed to the edge of extinction, including the bees themselves, because we’re reckless as hell.
  • China’s WTO Entry, 15 Years On
    The state’s hand was clear, but not in ways that were obviously forbidden by the WTO. Or at least not in ways that have been successfully challenged in the WTO. Firms’ investment decisions aren’t technically government procurement if the investment is for the provision of a commercial service, and the state’s guidance isn’t always written down. Yet even today the preferences provided for local firms in strategic sectors, like medical equipment, aren’t exactly a secret that China tries all that hard to hide.


    The initial China shock overlaps with the dollar shock. WTO accession made producing in China for the global market attractive, but did not made China into a great market for manufacturers looking to sell globally produced goods to China. Successful WTO challenges to individual Chinese practices haven’t changed the overall pattern—China’s imports of manufactures for its own use have slid steadily relative to China’s GDP after WTO accession. The more-limited-than-expected gains for manufacturers looking to sell to China though didn’t lead (until now, when the China shock is arguably starting to fade) to a serious reconsideration of the basic gains from China’s asymmetric integration, in part because U.S. and European firms captured many of the initial gains of China’s export success. And some “within the rules” remedies weren’t used as aggressively as they could have been to challenge China’s currency management and other discriminatory practices during the years immediately after China joined the WTO.

  • Cooperation at the Tigris‎: U.S. and Iraqi Efforts to Maintain the Mosul Dam
    U.S. and Iraqi scientists estimate that a breach of the Mosul Dam, one of the largest pieces of water infrastructure in the Middle East, could have a devastating impact for over a million Iraqis along the Tigris River, from Mosul itself all the way down to Baghdad. Its failure could also be a threat to U.S. personnel and our Coalition members working alongside Iraqis and drastically reshape the nature of the fight against ISIL in northern Iraq. Today, thanks to close U.S.-Iraqi partnership, however, Mosul Dam is on a path toward greater stability.
  • CS Responder Trans-Oceanic Cable Layer
    Laying fiber optic cables with repeaters along the ocean floor raises super-interesting technical challenges. I recently visited the CS Responder, a trans-ocean cable-laying ship
  • The greatest chess game ever played
    Garry Kasparov, who is one of the top chess players ever, said that his 1999 match against Veselin Topalov was the greatest game of chess he ever played. In this video, MatoJelic goes through the game, move by move.

Thursday, January 19, 2017

Up, up and away!

Monday I'll report to work, bright and early, to a new office, in Salesforce West, the 6th tallest building in San Francisco.

Although this won't be the first time I've worked in a high-rise, I've generally avoided such office settings over the years.

But that's getting hard to do, at least in San Francisco, where an explosion of new development surrounds the Transbay Terminal development area, just one block away from my new office building.

A lot of this burst of development is due to changes to the zoning laws around downtown San Francisco, as well as considerable improvements in construction techniques.

Already built and operational are:

But that's just the beginning, for here come:

And, of course, the Big Daddy of them all, the Salesforce Tower, where the concrete was poured for the top floor just last week.

The new tower is just across the street, so I'll be keeping a close eye on it.

And, the transit center itself seems like it will be pretty nice, with a beautiful (I hope) blocks-long public park atop the terminal, that should be very fine on sunny afternoons.

But the city is definitely a warren of canyons. You'd never believe you were just a few blocks from some of the most beautiful scenery in the world (the Golden Gate, the Bay, Alcatraz and Angel islands, etc.); you'd think you were somewhere around midtown-Manhattan, instead.

Which, I suppose, is sort of the point.

But here I am, and up they go.

Sunday, January 15, 2017

Stuff I'm reading, MLK Day edition

My, how time races by. I've been quite busy and not getting enough time to read.

Is there ever enough time to read?

  • CIDR 2017
    The biennial Conference on Innovative Data Systems Research (CIDR) is a systems-oriented conference, complementary in its mission to the mainstream database conferences like SIGMOD and VLDB, emphasizing the systems architecture perspective. CIDR gathers researchers and practitioners from both academia and industry to discuss the latest innovative and visionary ideas in the field.
  • Optimizing Space Amplification in RocksDB
    RocksDB is an embedded, high-performance, persistent keyvalue storage engine developed at Facebook. Much of our current focus in developing and configuring RocksDB is to give priority to resource efficiency instead of giving priority to the more standard performance metrics, such as response time latency and throughput, as long as the latter remain acceptable. In particular, we optimize space efficiency while ensuring read and write latencies meet service-level requirements for the intended workloads. This choice is motivated by the fact that storage space is most often the primary bottleneck when using Flash SSDs under typical production workloads at Facebook. RocksDB uses log-structured merge trees to obtain significant space efficiency and better write throughput while achieving acceptable read performance.
  • How and why the leap second affected Cloudflare DNS
    The root cause of the bug that affected our DNS service was the belief that time cannot go backwards. In our case, some code assumed that the difference between two times would always be, at worst, zero.
  • The Road to 2 Million Websocket Connections in Phoenix
    2 million is a figure we are pleased with. However, we did not quite max out the machine and we have not yet made any effort toward reducing the memory usage of each socket handler. In addition, there are more benchmarks we will be performing. This particular set of benchmarks was set exclusively around the number of simultaneous open sockets. A chat room with 2 million users is awesome, especially when the messages are broadcast so quickly. This is not a typical use case though.
  • Adaptive logging: optimizing logging and recovery costs in distributed in-memory databases
    This is a paper about the trade-offs between transaction throughput and database recovery time. Intuitively for example, you can do a little more work on each transaction (lowering throughput) in order to reduce the time it takes to recover in the event of failure. Recovery is based on information in logs, classically an ARIES-style write-ahead log, that records the values of data items.

    In the case of in-memory databases, you can also go the other way, and do a little less work when creating the logs (recording information for use in recovery) at the expense of longer recovery times, but gaining higher throughput. We can simplify recovery on the assumption that there is no need to undo the effects of uncommitted transactions – these existed solely in-memory and had not yet been persisted to disk.

  • Millions of Queries per Second: PostgreSQL and MySQL’s Peaceful Battle at Today’s Demanding Workloads
    The idea behind this research is to provide an honest comparison for the two popular RDBMSs. Sveta and Alexander wanted to test the most recent versions of both MySQL and PostgreSQL with the same tool, under the same challenging workloads and using the same configuration parameters (where possible). However, because both PostgreSQL and MySQL ecosystems evolved independently, with standard testing tools (pgbench and SysBench) used for each database, it wasn’t an easy journey.
  • Bitpacking and Compression of Sparse Datasets
    It turns out that gzipping after bitpacking yields a 1000x compression. Even on its highest compression settings, gzip was leaving a 8x compression on the table when applied to the raw data. It turns out that if you know the structure of your own data, you can very easily do much, much better than a generic compression algorithm. -- on both speed and compression.
  • The Real Reason Your City Has No Money
    All of the programs and incentives put in place by the federal and state governments to induce higher levels of growth by building more infrastructure has made the city of Lafayette functionally insolvent. Lafayette has collectively made more promises than it can keep and it's not even close. If they operated on accrual accounting -- where you account for your long term liabilities -- instead of a cash basis -- where you don't -- they would have been bankrupt decades ago. This is a pattern we see in every city we've examined. It is a byproduct of the American pattern of development we adopted everywhere after World War II.
  • Software Copyright Litigation After Oracle v. Google
    Oracle America has factored into at least four cases so far. One of these cases settled, one is on appeal, and the other two likely will be appealed in the near future. The latter two cases also involve patent claims, so appeals will be heard by the CAFC. (The CAFC has nearly exclusive appellate jurisdiction over cases with patent claims.) One can assume that the plaintiffs added the patent claims to ensure CAFC jurisdiction.
  • This is Fine: Engineering War Stories (and What We Learned) in 2016
    In the past an engineer would be tasked with a project, crawl into a dark hole, and come out days, weeks or months later clutching their precious code. Sometimes this worked out really well but other times it was disastrous.
  • The Inside Story of BitTorrent’s Bizarre Collapse
    sometimes technologies are not products. And they’re not companies. They’re just damn good technologies.
  • “Side Hustle” as a Sign of the Apocalypse
    And WTF has happened to our culture when we just take it as fact that everyone needs to have multiple jobs and work as a cab driver and rent out every square inch of space in their apartment and be a task rabbit gopher who waits in line for tickets when they’re not walking dogs or temping and we all just chalk it up to “progress”??? In the old days, this meant your life was falling apart. Now it just means you’re part of “the sharing economy.”
  • The Chemistry Behind Your Home’s Water Supply
    We take for granted the water that comes out of the taps in our home when we turn them on – but a lot of work goes into getting it there. Chemistry, too, has a hand in making sure that the water is safe to drink. Here, we take a look at the water treatment process, and in particular the chemicals used to get clean drinking water to your tap.

Saturday, January 14, 2017

And the water from the mountains flowed into the bay

Despite days of rain, ‘high-tide groupies’ seek out king tides

Many among the curious onlookers snapped pictures, splashed in the water, or just stared at the curious phenomenon around 10:15 a.m., when the water crested in front of Pier 14.

It was day-two of a three day stretch of the unusually high water that swelled around the Bay Area and spilled onto low-lying coastal areas, prompting the National Weather Service to issue a coastal flood advisory.

Thursday, January 12, 2017

And the atmospheric river flowed into the reservoirs

20 inches of rain, 12 feet of snow finally end 5-year drought in N. California

Stations up and down the Sierra mountain chain reported twice the amount of normal rain and snow for this time of year after snowstorms doubled the vital snowpack there that provides the state with much of its year-round water supply.

U.S. Drought Monitor: California

A plethora of Pacific storms and moisture slammed into California and most of the West, dumping copious amounts of precipitation on the northern two-thirds of the state and Sierra Nevada. This very wet week maintained the great start to the Water Year (since Oct. 1) across the West where NRCS SNOTEL basin average precipitation was above or much above normal at nearly every major basin while basin average snow water content was at or above normal in most Western basins. With more than a foot of precipitation falling on the Sierra Nevada (locally 20.7 inches at Strawberry Valley, CA), most major reservoirs were at or above its Jan. 10 historical average, USGS monitored streams were at near or record high flows, Jan. 10 state snow water content was at 135%, and the Northern Sierra 8-station, San Joaquin 5-station, and Tulare Basin 6-station precipitation indices topped their wettest previous year as of Jan. 10.

A plethora of storms. Imagine that!

Even though my ferry was canceled one day, and there's definitely water everywhere, it's a merry, merry sight.

Tuesday, January 10, 2017

The atmospheric river ... became an actual river

Sacramento Weir Opened For First Time In More Than 10 Years

Water officials opened the weir around 4:30 a.m. Tuesday. It’s all an effort to alleviate pressure on the Sacramento River levees.

How the Sacramento Weir works

While the system’s five other weirs overflow automatically once the river reaches a certain elevation, the 1,920-foot-long Sacramento Weir consists of 48 gates that must be opened with a long hooked pole. The gates are opened when the river level at the I Street Bridge reaches 29.87 feet. The weir, built in 1916, was last opened in December 2005.

Meanwhile, closer to home, ferries were canceled due to wind and waves, BART is running a partial schedule because downed trees have blocked tracks, planes are hours late or canceled entirely.

It's pandemonium.

Happily, the forecasters promise at least a few days of relief in sight.

Monday, January 9, 2017

Farewell to the Calaveras Pioneer Cabin tree

It was a big storm: Historic Pioneer Cabin Tree toppled in California storm

The historic Pioneer Cabin Tree, a former "drive-through" giant sequoia in Calaveras Big Trees State Park in Calaveras County, was felled in California's weekend storms.

The tree was hollowed out in the 1880s to allow tourists to pass through it, and even allowed cars, but more recently has hosted only hikers. It was part of the "Big Trees Trail" at the state park.

Jim Allday of Arnold is a volunteer at the park who was working there Sunday. He said the tree went down about 2 p.m. and "shattered" on impact.

I walked that trail in 2014, and if you ever get to the Sierras, Calaveras Big Trees State Park is a wonderful place.

But we humans should never have done that to that tree.