Threads are not an architectural element

What do I mean by this, and why am I taking the energy to write about it?

Threads as I have mentioned in an earlier post are a means of solving a narrow set of problems. They are a tool, just like a database , XML/XSLT, HTML, ... are tools. They are not a way of life.

Let me tell you a sad, and most unfortunate story...

Once upon a time at a high-tech start-up, a new manager was hired to clean up a mess. This mess was caused by a naive developer who was being very clever.

This clever developer had created an "architecture" with some very clever building blocks. These building blocks were Microsoft COM components each which had two interfaces and its own thread of execution. The interfaces were very clever and "elegant". One interface was "DoIt(String)" and the other was "String ReportIt(string)".

The application built on this architecture was a large stack of these building blocks. Each block would serialize a command and send it to the next. Each call was asynchronous because each clever block had its own thread.
Function calls would rattle through these blocks and each block did some "useful" work.

Unfortunately this design suffered from the following maladies and ultimately had to be euthanized:

1. Poor, poor, poor, poor performance. Waking a thread costs time. Crossing several thread boundaries to get work done was many times slower than letting a single thread do the work.
2. Large memory foot print. Each of these components where their own dll. The COM overhead and the serialization/de-serialization code was often larger than the application code.
   
3. Impractical to debug. Even with the use of trace statements it was nearly impossible to debug. We went to heroic lengths to tag trace statements such that we could follow the logical thread of execution we never came up with a good scheme.
4. Non-deterministic behavior. Once you get beyond three or four threads in an application you lose determinism. The application becomes very sensitive to the operations of other applications on the computer. Your critical threads get swapped out at inopportune times. Timing dependent defects come and go arbitrarily.
5. Costly to modify. Since each change alters the dynamics of the system (e.g. timing is changed) entirely new classes of defects emerge with even the most innocent change.
6. sfd

When deployed this application had 40 threads of execution. We made it work by sheer will, but it nearly killed the product. Over time the application was reworked to have 2 threads. The new version took half the number of lines of code, was faster, and had more features.

I have since found other instances where developers have followed similar approaches with the same results. Threads are a tool that you to implement solutions, they have absolutely nothing to do with software architecture.

So why the facination with using threads as an architectural element? What is the allure?

Most engineers use a "divide and conqur" approach to solving problems. Big problems are recursively broken into smaller problems, and each sub-problem is then solved. The threaded component design provides a box into which you can place to solution to a sub-problem. This box has its own thread so all you have to do is present it with an input and it will notify you when it is done processing it. Its a magic box that problems go into and solutions come out of. How nice!

Here we illustrate a big problem being decomposed into smaller sub-problems.



As an engineer I can decompose my problem into a series of independent sub-problems and assign each one to a magic box. I implement the solution to a sub-problem, stick it in the box, and put it on the shelf. I then string together all of these sub-problem solutions to create my solution. Better yet, I can reuse my magic boxes in other solutions. Since each is self-contained with its own thread of execution, I can drop it into a different solution and it will just work.

This diagram shows a problem decomposition directly mapped into a set of threaded components. Unless your problem domain is boxes such an architecture will get you nowhere.



Yes we engineers have rich fantasy lives. This software architecture will absolutly not work in almost all cases.

In the end a software architecture must contain objects/components that are part of the problem domain that it is trying to address. The thread as a architectural element architecture is devoid any domain information.

I hope that I have explained the flaws of this approach. I hope that I have also explain the strange appeal of this approach. The end advise is: Don't do it!

Threads increase design time

Keeping with my regular "post every couple of months" trend here is the next installment in my discussion of threads.

In addition to pulling from my personal experience I've been doing research on threading (which is half of the reason I write this blog). I have been amazed at the chorus of other voices warning of threading problems. There are many others with credentials better than mine who are are calling out the warning. So why bother again? Well the reason is: you can't have too many people warning you not to run with scissors! Perhaps my small contribution will be the last straw to make someone reconsider their multi-threaded application design. If so, my time has been well spent.

To the topic at hand.

Threads Increase Design Time

This is pretty simple. In the same way that it is infinitely easier to toss a single ball in the air as opposed to juggling three balls, it is easier to write a single threaded application than a multi-threaded application.

When designing a multi-threaded application you have many additional elements to consider:

1. Concurrency
   
   
   
   • Each time you access memory you must carefully think through if it is possible that two threads will be executing this code segment at the same time.


2. Performance
   
   
   
   • How do you insure that your threads are sleeping most of the time?
   
   
   • How do you prevent your threads from executing at the same time which will impare your performance?
   
   
   • Does your locking mechanism require a call to the kernal? If so are you prepared for the performance hit of getting locks?
   
   
   • Are you locking too much code for too long? Are you forcing your other threads to wait unnecessarily?
   
   


3. Deadlock
   
   • If you have more than one lock can your threads deadlock?
   
   
   • What is your strategy for resolving/preventing deadlocks?



4. State
   
   • Making an asynchronous call breaks the flow of your code. You make the call and expect a response at a later time. When you make the call you must save your state so that when the call eventually returns you can continue from where you left off. This is a major design consideration that is more often implemented poorly. Organizing your modules to have a single place where ansynchrounous returns are handled helps.
   • What happens when an asynchrounous call doesn't return? Are you tracking the time of your calls? What is your policy?
     


5. Resource consumption and memory management
   
   • If one thread allocates memory, who deallocates it? Managing the lifetime of global resources (memory, file pointers, database connections) becomes especially difficult. Which thread "owns" the resource? How are you sure that all threads are done using the resource?
   
   
   • How many threads and locks are you using? These are kernel resources that are not free?


6. Design for testing
   • Current debuggers are (and have been for over a decade) ineffective at tracking problems in multi-threaded applications. How will you debug your code? What tools will you use?
   
   
   • You will make use of extensive trace statements in your code (this is the answer to the previous question). What trace library will you use? How will you deploy your trace statements such that they don't significantly change the behavior of your code? What tools will you use to analyze the many megabytes of trace output produced?

It would seem that I am painting a bleak picture of multi-threaded development. Well I am, and it is well founded. This is one of the primary reasons that J2EE, COM+, and other Enterprise frameworks make you write single threaded code. These are highly multithreaded frameworks, but they make writing applications easier by hiding this fact.

Threads impair performance

Well I'm back after a vacation and chasing kids around. So my next entry is on how threads impair performance.

The basic reason that threads impair program performance is that each thread you spawn is competing for limited resources such as CPU time, or memory, or I/O bandwidth. When you spawn a thread you are competing with yourself. The more threads, the less resources for each.

That observation doesn't sound very convincing since modern operating systems are designed to share resources in a fair manner. The difference with thread is that they consume CPU timeslices. One of the jobs of an operating system is to split the CPU between all processes that are running on the system. Rather complex algorithms have been developed to insure fairness in a wide variety of situations.

When you spawn a second thread your application has potentially twice the number of CPU needs, however it will not get twice the number of time slices. All you have accomplished is limiting the number of CPU time each of your threads get. Each successive thread just digs you into a deeper hole.

But what about multiple-CPU machines? Sorry no dice. The operating system controls access to the CPUs. There is no way to guarantee that your threads will be efficiently scheduled to take advantage of the processors. Some operating systems allow processor affinity to be specified for threads. However at best this is only a hint.

A secondary effect is the impact multiple threads have on CPU cache coherency. Each thread has its own program pointer and stack pointer. When a thread is scheduled these pointers must be restablished, which takes time. When the new thread runs it establishes a new pattern of memory access in the CPU cache which churns the one from your other thread. Cache misses are expensive.

Up until now I've made the implicit presumption that threads your threads are running free and competeing with each other. Obviously this is the worst case scenario. Let's consider a more benign situation. We have two threads one of which is performing computation and the other which is waiting on a relatively long I/O call. The computing threads job is to perform operations on the results returned by the I/O thread.

All is good right? Your application is maximizing the use of computing resources. This should be very efficient. It is not! To see why consider the following cases:

Case 1: Computing thread wakes when I/O thread has posted new results. In this case there is no overlap between the threads, so they are not in contention for the same resources. However nothing has been gained. First there is the thread context switch overhead that is now incurred. Second on a modern computer calculations take a diminishingly short amount of time compared to I/O. All you have done in this case was gain a couple of microseconds of overlap with the I/O. Overall your application will not run any faster than the I/Os that it needs to perform. If this application were to perform a great deal of computation per I/O, ane execute billions of I/O you might gain an overall benefit. More likely than not these benefits will be swamped by context switching and lock overhead.

Case 2: The computation thread is still active when the I/O thread wakes up. You are so dead. You now have two threads that fighting for CPU time. Your performance will suffer greatly.

There is a great discussion of avoiding thread optimizations here. Don't just take my word for it.

The History of threads

Threads have not always been around. The first versions of Unix only had processes. Versions of Windows prior to Windows 95 didn't even have processes. The Usenet group FAQ for comp.os.research has an excellent history of threads.

Prior to the development of threads the developer was limited to using processes. A process is a heavyweight object that includes its own memory space, code space, heap space, and stack space. This means that if you needed to do a task in the background you had two choices:

1. You could spawn an entirely new process. Since a process has its own memory space you needed to develop a means of communicating between the two processes. Some of the choices were, shared memory, signals, shared files, pipes, or sockets. All of these approaches were very complex and difficult to get right. On top of that these means of "interprocess communication" were all heavy weight.

On older Unix systems you had the problem of Zombie processes which couldn't be killed. These were caused by one process spawning another and then not cleaning up properly.

2. You could emulate a backgroud task through a message loop style design. In this case you had an infinite loop that serviced a queue of tasks. Work ot be done was added to the queue. You were really implementing your own multitasking layer on top of the OS. This is techique used by the evil Motif Widgets package as well as all versions of Microsoft Windows prior to Windows 95.

As you can see implementing a background task was a significant effort. This was not as bad as it sounds. It forced developers to think through their designs. Only the most worthy of uses for background tasks were actually implemented. Again this is good since the use of background tasks is easily an order of magnitude more difficult and complex than single threaded programming.

Work on operating systems that supported threads began in the mid-1960's. Threading packages began to show up in in mainstream Unix releases in the early 1980's.

Threads were hailed as the solution to many problems in systems development. A thread has its own stack and address pointer, but shares the memory space and heap of its parent task. Communication between threads is within the same task. Communication can be as easy as pointing to a common place in memory. (Sure it is ...)

What most folks don't understand is that all threads did was to lower the barrier to creating multi-tasking applications. It did nothing to make multi-tasking development any easier. Using multiple threads is still that order of magnitude more difficult than single threading.

I like to think of the emergence of threads like this. Imagine if a cheap, easy, and widely available means of creating weapons grade uranium were to be found. Everyone with a little bit of initiative could churn our kilograms of this material. Would this be good for the world? Would there be folks who have no business messing with this material be playing with it? Would there be some idiot trying to find a way to integrate it into a childs toy?

I would not want to go back to systems without threads, but it only made the job of software project management more difficult.

The Threads thread: The common abuse of a fine idea

As promised in the byline of this blog I am going to address a common architectural/design blunder: The misuse of threads. I have seen this so often, and have seen it impair code performance that I feel something must be said. I am not the first voice to say these things. However given the magnitude of the issue the perils cannot emphasized enough.

I will present the case for care with threads as follows;

1. The history of threads

2. Threads impair performance

3. Threads increase design/implementation/debug time

4. Threads are not an architectural element

5. The proper use of threads

A great time in Switzerland ...

I am currently in Switzerland on business in the Lake Zurich area. I had the pleasure of spending the evening in Rapperswil at the southern end of of the lake. This is a very pleasant city that is built around an old cloister.

Imagine the this scene:

It is a warm spring evening at sunset. You are sitting at an outdoor cafe in a grove of manicured chestnut trees on the shore of Lake Zurich. You look to your right and you see the sun setting over the lake. You look to your left and you see the golden evening light reflected off the snow capped Alps in the distance. In front of you you see the green hills that dive into the lake.

This makes the hassle of traveling worthwhile...

NAnt Part 2

We'll I've successfully gotten my first build to work with NAnt and it works great. It took me a week to do what a full time contractor took a month to do previously. My build does the following:

Creates support directories:
Checks the entire build tree out of source control
Recursively compiles the code. (Using the solution task).
Runs NUnit on the output to execute the unit tests.
Runs NDoc to produce all of the product documentation.
Creates and installer.
Emails the development team about the state of the build.

So what is the secret? The secret for NAnt is that its tools are directly relevant to the task of building. For example for the NUnit task you simply specify the assembly containing the unit tests and the format you want your report in, and you're done.

The other key thing about NAnt is that it unifies the calling conventions for all of the tools you use in a build. To build a product you need compilers, test harnesses, source code control commands, and so on. Normally you have a learning curve for each one of these tools. You need to go through the documentation for each and learn the syntax and calling conventions for each tool. And to be certain each tool has a different calling convention.

With NAnt all of the hard work of how to call the tools is hidden. Instead there is a unified syntax for everything. I think that is the biggest reason that it is so easy to create build environments with this tool.

So here is a application design pattern: The Unified Syntax Pattern. With this pattern you take tools/components that are all used together and wrap them in a unifying syntax. The end result is orders of magnitude ease of use improvement.

NAnt - Part 1

I've been developing a new build system for my project. Since we are developing entirely in C# under .NET I am using NAnt. NAnt is the .NET version of Java project Ant. It represents an enormous step forward in build technology. I won't go into great detail here since this is a topic that has been widely written about here and here and here.

NAnt is a step forward because it is smart. The tasks in NAnt have parameters and and actions that are relevant to performing builds. There are tasks for creating directories, deleting files, running regression (Nunit) tests, and creating NDoc documentation sets. NAnt is also extensible. New tasks can be created as C# assemblies and are seamlessly integrated into NAnt.

Contrast this to old systems such as nmake and make by comparison were dumb tools. The old systems were really just scripting environments with some intelligence about dependencies. Build systems always had to be supplemented with shell scripts or perl scripts. On top of that they had crappy syntax. A Tab was an significant piece of syntax. If you missed a tab all was lost.

Builds based on make were so difficult to create we didn't even bother with it the build system at my last company. As it turns out the the feature of tracking dependences isn't that important. Computers today are so fast that it is almost always worth it to recompile everything from the top. NAnt is like having a set of tools tailored for doing builds. More to come ...

Blogger is buggy!

Well I spent the better part of the last hour typing up my first full blog entry describing MSBuild. The title of the entry was Missed Opportunities. Well upon hitting the "Publish Post" button I was greeted with a "Server not found" error. And the content of my post was lost. So the message is: Do not rely on the Blogger interface to type in articles. I guess I'm going to have to use an external editor and then paste my content into Blogger.

This is indeed a missed opportunity for Google/Blogger. Perhaps Blogger isn't ready for prime time...

First Looks

Greetings world ...

This is my first step into the world of blogging. I am an Engineer who writes software. As such I feel it is a bit of a cop-out to write a blog using something like Blogger. Real engineers craft their own blogs natively out of Java or .NET with hosted databases. I have designed a blog engine, however I also have a bunch of children. So guess what, I'm going to choose the simple route.

I am a pragmatic engineer, who sees too much pain and suffering in project because of disconnects between true customer needs and mis-use of software technology. I plan on using this as a forum for expressing my great thoughts on engineering. My particular bugbears are abstractions and patterns. I have seen these over used and mis-used. More horror stories another day...

I need to get to bed now. Perhaps you'll see more activity here in the near future.