Saturday, October 22, 2016

You can't become a Full Stack Developer all at once.

Patience! One step at a time

I spend a fair amount of time hanging out on places like Stack Overflow helping people with simple questions about things I know a lot about (e.g., Apache Derby).

One thing I notice is that people often try to learn a lot of new things all at once.

I suspect many of these people may be college students, as situations like this seem to pop up regularly at certain times of the year, which I imagine is around the time when Introduction to Databases classes around the globe are all getting to the point where they are assigning their first homework assignments.

I think there's also a lot of hysteria in the professional software development community recently about the notion of the Full Stack Developer.

Everybody wants to be a Full Stack Developer, because then they can get a job at Facebook/Google/Twitter/Uber/Dropbox/etc.!

Anyway, a typical Derby question on Stack Overflow might be something like:


I'm new to Java and am trying to learn by writing a simple app.

I've written a JSP web app and I'm trying to make it so that when I fill out a form and click this button, my Java code will use JDBC to store my record in the database.

But it isn't working.

... giant dump of blocks of various code pasted into question ...

Please help.

Let's look at what's GOOD here:

  • The questioner is self-aware enough to realize that they're trying to learn some new technologies,
  • and they've decided to approach that by learning-by-doing: that is, they're trying to write some actual programs to help themselves learn

All of that is wonderful!

But, you can't learn too much at once.

You just can't.

SQL is a deep technology; it's more than 30 years old.

Java is a deep technology; it's more than 20 years old.

The JDBC API is enormous and complex; it's at least 15 years old.

And web app development technologies such as JSP are almost as old as Java itself.

So don't try to do all of that at once!

I suspect some of this arises due to poor instruction in many universities. The state of the art in programming education appears to remain dismal, decades into the discipline.

However, I think that it also just reflects unrealistic expectations by people who feel like they should be able to master deep and complex subjects immediately.

After succeeding in primary education, and in secondary education, and finding that they can learn many, many things quite easily, young learners meet the world of computer systems and unexpectedly flail and flounder.

One of the things that's wonderful about computers is that there's essentially no limit to how complex a computer program can be: if you can imagine it, you can (try to) write a computer program to do it.

That's why computer programming is a career that can bring you valuable and rewarding employment for 30, 40, even 50 years of your life, which is a truly wonderful aspect of our modern life that doesn't get celebrated anywhere near enough.

But, it's also why you Just Can't Expect To Learn It All Instantly. There's simply too much.

And it will only get more complex, over time.

So what might be a successful approach?

First, get a nice simple standalone SQL product like SQLite or H2 or Derby. Look through that tool's docs and tutorials until you're comfortable starting it up, and shutting it down.

Then add a simple interactive SQL editor like SQuirrel, and learn how to write SQL. You don't have to learn ALL of SQL, but you need to learn some basics: SELECT, INSERT, UPDATE, DELETE. CREATE TABLE. The concept of indexes, and views. Referential constraints. Table design and at least the concepts of normalization.

Then have a look at the JDBC API, and write some JDBC programs of your own.

OK, NOW you are ready to try writing a JSP web app which makes JDBC calls to issue SQL statements against a database system of your choice.

At least at this point all you're really trying to learn is the JSP and web technology.

And hopefully, the result will be that the young learner:

  • Has some success, and doesn't despair, and doesn't throw the computer out the window and decide to switch their major to, say, Diesel Mechanics (not that there's anything fundamentally wrong with Diesel Mechanics, but the world needs more successful, well-educated computer programmers badly!)
  • Gets enough of an introduction to the various sub-specialties of the computing world that they can make an educated and informed choice about what appeals to them, what they have a knack for, and what they'd like to learn more about.

Very, very, very few of us are going to actually end up being Full Stack Engineers, after all, and there are a lot of good reasons to not even try.

OK, odd rant over (for now).

Wednesday, October 19, 2016

VLDB 2016

So I finally got around to looking at the full program from last month's 42nd International Conference on Very Large Data Bases (VLDB) 2016

What a massive conference this has become!

Anyway, a few things that particularly appealed to me:

  • Multi-Version Range Concurrency Control in Deuteronomy
    In this paper, we enhance our multi-version timestamp order technique to handle range concurrency and prevent phantoms.
  • Not for the Timid: On the Impact of Aggressive Over-booking in the Cloud
    In this paper, we examine policies that inherently tune the system’s idle sensitivity. Increased sensitivity to idleness leads to aggressive over-booking while the converse leads to conservative reclamation and lower utilization levels. Aggressive over-booking also incurs a “reserve” capacity cost (for when we suddenly “owe” capacity to previously idle databases.) We answer these key questions in this paper: (1) how to find a “good” resource reclamation policy for a given DBaaS cluster of users; and (2) how to forecast the needed near-term reserve capacity.
  • Incremental Computation of Common Windowed Holistic Aggregates
    This paper provides the first in-depth study of how to efficiently implement the three most common holistic windowed aggregates (count distinct, mode and quantile) by reusing the aggregate state between consecutive frames.
  • Aerospike: Architecture of a Real-Time Operational DBMS
    In this paper, we describe the solutions developed to address key technical challenges encountered while building a distributed database system that can smoothly handle demanding real-time workloads and provide a high level of fault tolerance. Specifically, we describe schemes for the efficient clustering and data partitioning for the automatic scale out of processing across multiple nodes and for optimizing the usage of CPUs, DRAM, SSDs and networks to efficiently scale up performance on one node.
  • Comdb2 Bloomberg’s Highly Available Relational Database System
    Comdb2 is a distributed database system designed for geographical replication and high availability. In contrast with the latest trends in this field, Comdb2 o↵ers full transactional support, a standard relational model, and the expressivity of SQL. Moreover, the system allows for rich stored procedures using a dialect of Lua. Comdb2 implements a serializable system in which reads from any node always return current values. Comdb2 provides transparent High Availability through built-in service discovery and sophisticated retry logic embedded in the standard API.
  • How Good Are Query Optimizers, Really?
    Finding a good join order is crucial for query performance. In this paper, we introduce the Join Order Benchmark (JOB) and experimentally revisit the main components in the classic query optimizer architecture using a complex, real-world data set and realistic multi-join queries. We investigate the quality of industrial-strength cardinality estimators and find that all estimators routinely produce large errors. We further show that while estimates are essential for finding a good join order, query performance is unsatisfactory if the query engine relies too heavily on these estimates. Using an- other set of experiments that measure the impact of the cost model, we find that it has much less influence on query performance than the cardinality estimates.
  • The End of Slow Networks: It’s Time for a Redesign
    The next generation of high-performance networks with re- mote direct memory access (RDMA) capabilities requires a fundamental rethinking of the design of distributed in-memory DBMSs. These systems are commonly built under the assumption that the network is the primary bottleneck and should be avoided at all costs, but this assumption no longer holds.
  • Accelerating Analytics with Dynamic In-Memory Expressions
    With Oracle Database 12.2, Database In-Memory is further enhanced to accelerate analytic processing through a novel lightweight mechanism known as Dynamic In-Memory Expressions (DIMEs). The DIME mechanism automatically detects frequently occurring expressions in a query workload, and then creates highly optimized, transactionally consistent, in-memory columnar representations of these expression results.
  • Distributed Data Deduplication
    In this paper, we show how to further speed up data deduplication by leveraging parallelism in a shared-nothing computing envi- ronment. Our main contribution is a distribution strategy, called Dis-Dedup, that minimizes the maximum workload across all worker nodes and provides strong theoretical guarantees.

I'm not lacking for fascinating things to read!

Sunday, October 16, 2016

The Orenda: a very short review

I devoured Joseph Boyden's The Orenda, as it devoured me.

The Orenda, to be frank, is often not a pleasant book to read. It is fiction, but set very firmly in a tragic historical context: the utter collapse of the Huron Nation in the late 1600's.

The Orenda is a story of tragedy, devastation, and despair, yet somehow it is also a story of humanity, beauty, devotion, and affection.

The author uses an unusual narrative structure: the book is told entirely in the first person, but it alternates, chapter by chapter, among the voices of three separate co-narrators:

  1. Snow Falls, an Iroquois princess
  2. Bird, a Huron war chief
  3. and Christophe, a Jesuit missionary

In and out of the overall story we go, sometimes letting a single narrator carry the story for long stretches, at other times seeing the same event from two or sometimes even three completely different perspectives.

Boyden's genius is that he completely inhabits each of these characters, allowing us, in turn, to live these events ourselves, witnessing the horror, yet simultaneously comprehending how it was and how it came to be.

If you read this book (and you should!), let me caution you to prepare yourself for a long and dark ride: The Orenda starts out bleak, descends into tragedy, and concludes in a maelstrom of horror.

Yet, somehow, it is so exquisitely drawn that it is simultaneously full of aching beauty.

Let me try to illustrate with a short excerpt:

Each day as we struggle against the current, I watch the men turn leaner, more focused, more silent. From first light until night threatens we push up this wide, black river with birch and maple and poplar thick on the banks. So many good places for my father's brothers to ambush these canoes. I hope they've brought a hundred men, two hundred men. Enough to kill Bird and all of his war-bearers. The country here is beautiful. The rocks run right down into the water that's dark as the darkest night, and when the men stop to rest, I lie upon those rocks and let their heat soak into me. A wind from the east has brought good skies, and this kind wind blows away the flies and mosquitoes. These might be the most beautiful days of sun I've ever known after the rain of last week stopped. This is the perfect time, and the prettiest of country, in which to witness my father's brothers kill these enemies.

Boyden doesn't overwhelm you with technique and style. He understands that these people, and their stories, are all that he needs, and he (remarkably!) disappears from the book, so that soon you feel that Bird, Snow Falls, and Christophe are real people, whose real recollections were simply captured by some well-placed tape recorder, and then faithfully transcribed.

It's an astonishing achievement, and an astonishing book.

Saturday, October 15, 2016

Thursday, October 13, 2016

Unicorn strangling

Loved this Bloomberg article: How Not to Strangle Your Unicorn.

Way back in the mid-1980's, we used to sit around in the office and chuckle at computer companies whose strategy was clearly:

OK, we may lose money on each sale.

But we'll make it up in volume!

OK, it's an old joke, certainly it goes much further back than the 1980's.

But isn't it delightful to read this:

“We kept getting new customers, but we started to see that the profit margins were zero to negative,” said Howard Morgan, an investor at First Round Capital who was on Fab’s board. “The only way the model made money was if people bought multiple items. If you could lose money on each customer, you’re not going to get money out of the whole, and it took us too long to understand that’s what was happening.”

Or this:

Investors swarmed. Goldberg's ideas and charisma helped the e-commerce site raise at least $325 million in venture capital from such firms as Andreessen Horowitz and China's Tencent Holdings, fueling expansion in Europe and plans to tackle Asia.

I mean, these are guys who run multi-billion dollar investment funds, not just some dumb schmuck coders who sit around in the office critiquing stupid business plans that can't ever work.


Wednesday, October 12, 2016

Planes, trains, and automobiles

BART is so full that they are paying people to stay away.

The freeways are so clogged that rush hour begins before 5:00 AM and lasts until after 9:00 PM.

What's left?

Well, there are the ferries.

As well, there are new trains, that run TO the ferries.

The Bay Area has always had ferries, but it's also always seemed rather under-served by them.

So it's interesting to see some recent progress in this area:

  • With Crowds Flocking to Ferries, Agency Adds New Runs
    “The ferry service, like so many transit agencies in the region, is carrying more and more people,” said Ernest Sanchez, marketing manager for San Francisco Bay Ferry. “The demand on the system is substantial and is beyond what we expected to happen, so what we’re trying to do is take our 11 boats and get them assigned in the most affective manner.”

    Bay Ferry has also redirected routes to create a direct 4 p.m. trip from San Francisco’s Pier 41 to Alameda, then Oakland.

    But the changes will only make a small dent in demand that shows no sign of waning.

    “The numbers are going to be small,” Sanchez said, adding the new runs amount to a temporary fix while the system awaits longer-term expansions, like two new 400-passenger boats expected to hit the Bay next winter.

  • Alameda breaks ground on $49 million ferry operations center
    Ferry service representatives joined local and state officials Thursday for the groundbreaking of the Water Emergency Transportation Authority’s new $49 million operations and maintenance center.

    The event, which featured wine and live music under a tent at the former Alameda Naval Air Station, also was a tribute to Ron Cowan, the Alameda developer who helped spearhead the creation of the ferry system.

    Construction of the center, which will be named after Cowan and will be located at Ferry Point Road and West Hornet Avenue, has already started with underwater work and is expected to be completed in 2018.


    A regional public transit agency, the Water Emergency Transportation Authority operates ferries on San Francisco Bay under the “San Francisco Bay Ferry” brand.

    Originally known as the San Francisco Bay Area Water Transit Authority, it was established in 1999 and has 12 boats on four routes that visit eight terminals. The goal is to have 44 vessels on 12 routes with 16 terminals by 2035, according to WETA officials.

  • Water taxi service approved for SF-Berkeley runs
    Tideline is one of two private water taxis to get the OK to run scheduled trips across the bay. The other is Prop, which plans to have service to Berkeley, Emeryville, San Francisco and Redwood City in early January.

    “We have an untapped waterway,” said Prop CEO James Jaber. “Unlike the bigger lines, we can offer cities an inexpensive look at public ferries without a 10-year, multimillion-dollar commitment on their part.”

    Unlike the publicly subsidized ferries, which have boats that hold up to 400 passengers, the smaller taxis hold about 40 commuters or less.

  • Sonoma-Marin Area Rail Transit: FAQ
    If SMART doesn’t go to San Francisco, isn’t it a “train to nowhere?” SMART will connect to Larkspur, across the street from the Larkspur Ferry Terminal, providing access to San Francisco. It’s important to recognize, however, that most of the traffic on Highway 101 in the North Bay is not bound for San Francisco.

    Commuting patterns in the 21st Century are much different from those of 50 years ago. The vast majority of North Bay commuters on Highway 101 are going to jobs in Marin and Sonoma Counties, and the number of commuters to San Francisco is shrinking. The Metropolitan Transportation Commission projects 130,000 new jobs along the Highway 101 corridor in Marin and Sonoma between 2000 and 2025 – none of which will require a North Bay commuter to cross the Golden Gate Bridge. Far from being a “train to nowhere,” SMART is a train to where the jobs are, and where the people are.

  • San Rafael, Novato grapple with SMART quiet zones
    Kevin Fitzgerald, a Federal Railroad Administration official, said SMART is going through a checklist as it gets ready for service.

    “The start date could be Dec. 1, or it could be next year,” he said.

  • Sonoma-Marin Area Rail Transit grinds through tech challenges
    So far, SMART has spent roughly $460 million, according to Gamlen, and purchased 14 cars. Operated as two-car sets, the system will have seven sets. The doors and cars were designed in Japan. BART systems cost about $120 million per mile. Light rails costs about $50 million a mile. “We are running about $10 million a mile,” he said, excluding any costs for acquiring right-of-way.

    There are four passing sidings, where dual-track sections allow northbound and southbound trains to pass. The longest stretches about two miles; the track so far runs 43 miles. Dispatchers, operating somewhat like air-traffic controllers, line routes and can direct trains onto passing sidings as needed. If schedule changes occurred, a train could be held on a passing siding for another train to go by.

    An estimated run time will be just over an hour from Sonoma County’s airport to San Rafael. “We have yet to run a train from end to end at planned speed,” Gamlen said. “All this stuff is still in test mode.”

Well, you know what they say:

"A bad day on the water is better than a good day in the office"
Go ferries!

Monday, October 10, 2016

Yes, Virginia, there is a world outside my cubicle...

Yesterday afternoon at 3:00 we were sitting on the beach at the east end of Tenaya Lake.

At 8,000 feet, it was 73 degrees, with a beautiful clear blue sky.

With binoculars, I was following the progress of two different pairs of climbers as they ascended the granite face across the lake from us.

It was wonderful.