It's more than just networking hardware that mystifies software guys like me. Take, for example, the incredibly sophisticated memory caches that are found on modern CPUs.
I recently came across a superb paper by Paul McKenney of IBM's Linux Technology Center: Memory Barriers: a Hardware View for Software Hackers.
McKenney's paper sets out to teach software types like myself just enough about the design and implementation of cache coherency algorithms in modern multi-core processors so that we can write software that is both correct and efficient.
The hardware designers cannot help directly here, since the CPUs have no idea which variables are related, let alone how they might be related. Therefore, the hardware designers provide memory-barrier instructions to allow the software to tell the CPU about such relations. The program fragment must be updated to contain the memory barrier
The paper is clear and straightforward, with great diagrams and examples. I particularly enjoy the "quick quiz" format, in which McKenney intersperses review questions in the text, and provides answers at the end of the paper so you can periodically stop, take the quiz, check your answer, and keep going.
If you've ever wondered about the MESI protocol, or about store buffers, or about writeback, snooping, and invalidation, or how the DEC Alpha processor impacted the Linux kernel memory-ordering primitives, don't hesitate, dig right in!
This is the sort of paper that every system software engineer should read, understand, and periodically re-read. Thank you IBM and Paul McKenney for making it available!