On Performance and Scaling

Finally, here's what you've been waiting for...

Performance is about getting to the bone with a butcher's knife to get that last bit of flesh (sure, it takes me great effort to say that as a vegetarian...so savor and ruminate this key point). It is about getting the last drop of milch from that udder. Think instruction counts. Think about the shortest path between two points. Think about latency and real-time systems.

Scalability, interestingly, lands you on a different planet. If you had 256 cores underneath, can you get 256,00% utilization? If this is of principal interest to you first then don't worry too much about performance as defined above. Use coarse grained locks, look at the wait time of locks (how long do you have to wait to get inside mutually exclusive code), and worry about pruning your critical sections (mutex protected code) later. Scalability is about throughput, mainly. Performance, is roughly, about latency. Eventually we will combine both and have some real fun.

To get the jargon right - we will use the term performance for a single thread of execution, and scalability for the "parallel performance" of multiple threads of execution.

Designing MT code from scratch can be a bitch if you haven't trained your mind to do so. In my company, PixBlitz Studios, we have some vision engineers (not hard-core programmers but good scientists and mathematicians) solve the algorithm correctly using simple single-threaded logic. Once the algorithm has been nailed, it is psuedo-coded and then parallelized. Sure, there are trade-offs in taking this route, the positive one being that the vision engineers can produce some kick-ass algorithms without worrying about the systems side of life. All algorithms are not naturally parallelizable. Knowledge of this fact can greatly reduce risk and agony. There are cases, however, where such a non-parallelizable problem may start looking interesting (say, to someone like my colleagues Arvind Nithrakayshap or Tudor Bosman from my past lives, or even me for that matter) especially when we know that solving it has much value. Exceptions apart, it is important to solve the problem for correctness before multi-threading it. Before you kill me...

Even in the general case of a solvable MT problem, just because you have nailed the single-threaded algorithm does not mean you can readily multi-thread it. A good approach is to start from a clean sheet of paper and write the parallel algorithm using the single-threaded algorithm in psuedo-code. A sound understanding of the algorithm and the underlying problem domain can quickly give a good sense for what parts can we written as concurrent code and what is intrinsically non-parallelizable. In the difficult case of non-parallelizable logic, it is helpful to understand the effect of sync points (points in the algorithms where some/all threads must synchronize) on the parallel logic. When writing parallel (or scalable) code the sync points play a key role in determining the scalability of the system.

You'll often hear me say - what is the effect of the fast path on the slow path? If the slow path has a systemic feedback loop into the fast path of the system you have a pathological problem to deal with and should be going out fishing instead.

Phew, was that much to digest? Let's focus on the easy case...the code is naturally parallelizable with no frickin gotchas. Should you be worrying about the execution time of a single thread of execution? Or go after the lock wait times? Put coarse locks in your code and use 256,00% CPU utilization first? Let's nail these questions first. Haha. We can then come back with the pruning shears and start shaving off instructions!!

Without being too cynical we must wait to milch the instruction count udder until we actually need to do that. In all likelihood your code is running on today's latest multi-core CPUs and GPUs; contrary to the urban legend - where the Gods of Performance will write you an eulogy for a lifetime of instruction pruning - your time is best spent on ensuring that your algorithm is parallel before going after instruction counts of a single thread of execution.

Let's leave it at this for now.

To be continued into many blogs from now on with real examples, code, etc.

The Philosopy of Performace and Scaling

For those of you who have worked with me (or currently do), it may be better to read through my non-technical blogs (business, CEO type stuff), mainly because there is nothing new here than what you may already know through our technical conversations.

We will get a lot into the meaning of the terms - Performance and Scaling. This blog is the start. When I have time on hand I'll try to compile some of this and hopefully much more into a book. I believe there are two ways to write a book: keep taking detailed notes as you do development of some hard-assed problem in computer science, or, teach a class in systems design and programming. The latter suits me better as I'm too selfish with my time when I'm writing code. Coding is very beautiful and to get distracted is to give an ounce less to the thing on hand (which is what all these blogs are all about :)).

I'm excited to see The Art of Multiprocessor Programming by Maurice Herlihy and Nir Shavit. I'll share my thoughts when I have read this book. But flipping quickly through it I see much out there where we can value add in terms of getting those great algorithms a run for the money on some of the coolest h/w coming out in industry.

What a shame seeing some good computer science talent coming out of school wanting to manage people or do business development. Embarrassing as that statement may seem given my life as a programmer today (Not!), I ask you - the brilliant CS student - what do you go to school for if you don't want to step out and give what you've learnt a road-test? If the CS degree was just a stepping stone for landing a well-paying job in a completely different domain then you are doing the right thing getting into business; rest assured, the world of building "beautiful" will not miss you. The economics of being a programmer v/s a manager or biz person are similar to that of a masonry brick layer v/s the contractor; sure, there is a scalar multiplier involved in terms of actuals $$s involved :). But some of my most successful engineer friends have done pretty damn well financially, compared to their management counterparts, and contributed immensely to academia and industry. I'm excited to say that the $$s business has never come in a significant way (haha) of my love for programming.

To conclude, performance and scaling is not a job IMHO. It is a way of life. Marrying s/w to the underlying h/w to get the most out of performance and scaling is a natural by-product of well written code. It does not require a "performance engineer," to run through your dirty laundry after you have written code, to "speed" things up. We will focus on becoming well-rounded engineers who do not have a surrogate relationship with processors, compilers, Instruction Set Architectures, machine architecture, etc. As we go higher through the s/w stack and away from silicon we will use the same building blocks that we used when close to bare metal. The problem is not very different when building a distributed system or a loosely coupled cluster than when designing for a symmetric shared-memory MP system with snoopy or directory based caches. Neither is it different when writing a compiler for a processor pipeline. The key is to distill the mathematical basis - once you have that down you are in a powerful position to build any vector. I may go years before I've discovered something fundamental that can become a part of my synchronization toolchest, thereby adding to my basis.

Concurrency and synchronization are an intrinsic part of any large real-life system. It is good to train ourselves on home turf, take baby steps, and set ourselves for bigger challenges over time.