Thoughtworks breakfast - Emergent Design & Evolutionary Architecture

Posted on February 16th, 2010 by Stephen Hurn – 10 Comments

This morning I was the most casually dressed person at a Thoughtworks run seminar called “Emergent Design and Evolutionary Architecture”. The seminar itself was extremely well planned and executed, despite the lack of effective air conditioning. The speaker was an American named Neal Ford, who Thoughtworks had flown out to Australia to speak at these events (there are two more, which you can catch for free in Sydney and Melbourne).”

Neal has previously written a series of articles on this topic for IBM, which can be found here: http://www.ibm.com/developerworks/views/java/libraryview.jsp?search_by=evolutionary+architecture+emergent+design:

One thing that Neal said that really stood out to me so much that I wrote it down was this “Requirements gathering is a lossy compression algorithm”. This comes down to a philosophy of software engineering most famously articulated in the Fall 1992 “C++ Journal” by Jack Reeves, who said that every engineering discipline produced a document as a deliverable and with software engineering this document is the source code. What this means is that when I, as the product manager, go out on site and visit clients and gather requirements I do not produce the final design. The design is not final until the last line of source code has been written.

Neal warned of a number of mistakes that are made by many software development teams. There is this idea that software architecture needs to be thought through before coding begins. This unfortunately leads to a number of problems because of the tendency to abstract things too early in the process, which increases the “accidental complexity” of software. Accidental complexity is the complexity of a solution over and above the complexity necessary to solve the actual problem. This is often caused when a developer says or thinks “we should do this thing now because we might want it later”.

Instead of up front design, Neal talked about architecture based on extracting idiomatic patterns from source code. This is different from Design Patterns in that idiomatic patterns are solution-specific architectures, while Design Patterns are generalised problem solutions. One example that Neal gave on extracting idiomatic patterns from code involved analysing an open source Java project. He worked out which areas of the code were both highly used and highly complex by analysing cyclomatic complexity (highly complex) and afferent coupling (how much other code uses this code). Simply doing this analysis showed that the most complex and highly used parts of the code in this project did similar things, which made it obvious what had to be done to improve the code.

The final concept that I wish to discuss (this is by no means an exhaustive list of what was covered - for that, attend one of the Sydney or Melbourne meetings or read the linked articles) is that of the concept of “technical debt”. Technical debt is a metaphor used to describe the difference between perfectly clean code and the code that you have. In many ways it works like credit card debt. The faster that code is created the messier it tends to be. This messiness causes future work on the code to take longer. The extra time that it takes to develop new features in the code is the “interest payment” on the technical debt. Some level of messiness is usually ok and it is up to each individual company to decide how much technical debt they wish to incur. In some cases the only factor is speed to market, in which case it is probably best to ignore technical debt. In other cases (in the case of us) the code that we write has a very long lifespan which means that addressing technical debt is something that we wish to reduce as much as possible.

In all I was extremely impressed with the whole presentation and would recommend it to anyone who is involved in software development. This was one of the best presentations that I have ever been to on software development. If you’re in Sydney or Melbourne, make sure you get along to the next one!

This entry was posted on Tuesday, February 16th, 2010 at 12:39 pm and is filed under 03 - Product Development, Design, Development, Event Review, Life-Long Learning, Methodology, Requirements Gathering. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Comments

Paul says:

February 19, 2010 at 2:39 pm

Thanks Stephen,

Sounds like an interesting presentation.

When I get some time, I would like to check out the materials referenced on the IBM site - though it is more important that those in our Division for DDD WCW (a.k.a. Products Division) do this. Certainly when our General Manager for DDD WCW is onboard, I would expect that they would be leading this kind of ongoing learning for our growing team.

I will contact Neal Ford directly, as I am unclear on whether something was the intent of his message, or whether it was just lost in translation. This has to do with the following:

1) With reference to Software Engineering:

You reference the comment “The design is not final until the last line of source code has been written”.

This is one of the points I was trying to make back in September 2009 as one of the key reasons we needed to be more agile (by adopting Agile) with our development.

At the time, I heard internally that “I still don’t see why we need to go Agile” which illustrates that I was not as clear as I needed to be - which is why I was more direct (in Nov 2009) ,
despite the fact that this confused a few people externally and needed clarification (Feb 2010).

Thankfully, we have come a long way since then, with everyone now “on the bus” in terms of accepting Agile as the way forward - though there is still much that we actually need to implement (and our GM will help us with this).

The context of your quote is the implicit assumption that:

(a) In “design”, there are still uncertainties (which might be in terms of technical uncertainties, or budget uncertainties, or in terms of what the customer wants - or, in our case, all three).

(b) After “design”, the implicit assumption is that these have been alleviated and the solution is known - how to do it, how much it will cost, and what the customer response will be. Hence all that is left to do is “crank the handle”.

From what I have seen (internally and externally), most software projects never get to the “after design” phase. However some do, in which case Waterfall processes might suit just fine for them.

2) With reference to other branches of Engineering:

You quote Neal Ford quoting Jack Reeves as stating that “every engineering discipline produced a document as a deliverable …” - with the implication being that the design (in other disciplines) as “done” (and all uncertainties addressed) when the document is drawn up, whereas it is not in software.

I am unsure of where in the chain of chinese whispers the communication became crossed, but in my mind this is misleading, and myopic.

In my mind, this line of argument is much too similar to the line I have heard in many different industries, companies and countries whereby “that’s fine for them, but we’re different, and [INSERT METHOD] won’t work here.

To be direct, there are more similarities between software development and other streams of engineering than there are differences - and, for us to achieve our vision in terms of 20x number of clients with only 4x number of staff, we will have to be very smart about “stealing ideas shamelessly” from whatever field we can find them.

(a) That’s the point I was trying to make back in September 2009, though unsuccessfully by comparing and contrasting the “design” stage from the “manufacture” stage.

(b) That’s also the point that Scott Berkun made in chapter 1 of “Making things Happen” - drawing on examples as diverse as restaurant kitchens and hospital emergency rooms. Incidentally, being able to drag in ideas from diverse fields is requires a combination of the five core skills facilitating disruptive innovation.

(c) Finally, this was one of the main messages in a Webinar presented by Mary Poppendick a on February 3rd through the IEEE on “Advanced Agile - Below the Low-Hanging Fruit”, which Adam sat in on and will post about later. In the presentation, Mary drew on the experience of the construction of the Empire State Building to highlight how:

i. In the same paradigm, the “design” of the building was not done until the building was completed; and
ii. How some engineering practice (used almost 100 years ago in that example, and now labelled “Lean”) are just as relevant to some software practices today.

Hope this makes sense.

We have come a long way in 8 months, but have a long way still to go…

Paul

Reply
Stephen says:

February 19, 2010 at 3:46 pm

“You quote Neal Ford quoting Jack Reeves as stating that “every engineering discipline produced a document as a deliverable …” - with the implication being that the design (in other disciplines) as “done” (and all uncertainties addressed) when the document is drawn up, whereas it is not in software.”

I am unsure of what you mean by this statement. I was trying to state that in software, the final design document is the source code itself. It is the document which describes to the “builders” (compiler) where to put “all their nuts and bolts” (machine or interpreter level instructions).

In the empire state building example, the final deliverable for the engineers was not the building itself, but the plans for the completed building. Just because the plans were delivered in parallel with construction doesn’t change the fact that the plans were the engineers’ deliverable.

Perhaps I am misunderstanding your point.

Reply
Justin says:

February 19, 2010 at 6:26 pm

Good article Stephen. Sound’s like you enjoyed the seminar.

I’d be careful with how you think about technical debt. Maybe I’ve taken what you are saying the wrong way but I don’t think you should consider there being 2 different cases here (speed to market and long lived code) that need to be treated differently. It is not something that you either ignore or don’t ignore. You will always have some tech debt. The comparison with financial debt is based on the assumption that you would take on debt for short term gain with the intention of paying it off at some later point. If you never pay it off then things get worse and worse (much like financial debt).

The important thing is that you’re not trying and keep your debt at zero at anytime. There are always going to be things in your code base that you don’t like but it doesn’t mean that you should change them immediately when you notice them to keep your tech debt down. Keep track of them and pay them off by treating them like any other chunk of work (prioritise it relative to other upcoming things in terms of importance etc.).

Regarding your discussion about the document being a deliverable and what that means. It sounds a bit like you guys are discussing different kind of things so don’t get your wires crossed. I think that Stephen is talking about the act of actually writing the software (a developer concern) and Paul more about the entire process of delivering it (the concern of the whole team). The point is I think you might be arguing about slightly different things.

Reply
- Paul McArdle says:
  
  February 20, 2010 at 2:17 pm
  
  Thanks Justin,
  
  You make some good points:
  
  1) re Technical Debt
  
  With respect to Technical Debt, this is another case where we need to consciously avoid the thinking that is best summed up as “Tyranny of the OR” - from “Built to Last”.
  
  I have posted previously that the ability to reject this will be one of the important soft-skills that our GM DDD WCW will bring to the company, and which we will then have to continue embedding as part of our culture.
  
  Investing this time effectively will give us the platform for the next stages of growth.
  
  This is along the lines of what you have noted - i.e. Technical Debt is neither good nor bad - it’s what we do with it that counts.
  
  In my view, it’s just another investment in the ongoing growth of how we delight customers, and hence grow the company.
  (a) If we invest well (as we have with NEM-Watch and NEM-Review, for instance) then our incurring of Technical Debt was massively worthwhile.
  (b) In other products that have not performed so well, our investment (in all its forms) has not been as favourable. However, even this investment is not completely wasted, so long as we learn from it.
  
  As you know, we have gained major benefit from the recurrent revenues generated by products developed in the first 4-5 years (thanks for that!), even though we incurred some Technical Debt to get to where we are now. Because we can afford to now, we have begun to pay off this technical debt, starting with a re-write of NEM-Review in the past year.
  
  Hence, I agree with you that we should use caution in blanket statements like “technical debt is something that we wish to reduce as much as possible”, less they be misinterpreted as implying we need to avoid Technical Debt.
  
  For us (as with everyone else) it comes down to a cost-benefit trade-off. On that note, our ability to - as a team - understand NPV is something we need to improve, moving forward (another thing our GM DDD WCW will help us with).
  
  Similarly, new products we develop to serve an expanding range of clients won’t be perfect, either (internally, in terms of the way the code is put together - and externally, in terms of how many client needs we nail). Our challenge is to continue to iteratively improve - bootstrapping the company forward.
  
  2) re Crossed Wires
  
  You are correct in that Stephen and I had wires crossed. We clarified this offline.
  
  1) Stephen was just making the point that the design is not finished when a spec is written (I agree - and see this as one of the fallacies of a Waterfall paradigm)
  
  2) I was just making the point that this is not dissimilar to other disciplines, in many cases - and we need to learn from them (as they have been running for many years longer than the software engineering one, so have made plenty of mistakes we can learn from).
  
  Note it was more of a discussion than an argument - as you will recall, I certainly welcome different ideas (so long as they are positively promoted, and so long as we collectively continue to learn something new every week, and not get stuck in inappropriate paradigms)!
  
  Hope the snows are enjoyable!
  
  Cheers
  
  Paul
  
  Reply
  - Justin says:
    
    February 20, 2010 at 6:03 pm
    
    My apologies if I sounded like I was refereeing an argument. I didn’t mean it that way and completely agree that this sort of thing is healthy!
    
    Also while I think it’s very important to take pointers from other disciplines like you’re saying, it’s also important to realise that software has it’s own unique qualities as well. I think the big difference for me comes because very often software can be changed without the cost of it being prohibitive. Or at least that’s the perception everyone has.
    
    For example, you can’t go changing the way the foundation’s of the Empire State Building are designed when the top floor is in place (unless you’re spending a whole lot of money). Software tends to not be like that. The fact is, because it’s (perceived as) able to change, techniques that enable it to change more rapidly that minimise the cost make a lot more sense. While change still does cost (and too often people think it’s free) it is no where near as much if it’s done right.
    
    On a related point, i.e. “design is not finished when a spec is written”. I’d be thinking about why you would even need a spec in the first place. If the design is never complete until the completion of the release then that spec and the time spent on it become a big waste of money. A big formal spec document will tend to get ignored when the project is in development anyway.
    
    Reply
    - Stephen says:
      
      February 22, 2010 at 10:10 am
      
      A design spec is useful for some purposes. The primary use for a design spec is to assist the developer(s) get their head around the high level problems. It should only ever be done by the people who are going to be writing the code and only ever to assist them getting started on what they are doing. And is in many cases not really necessary.
      
      A design spec is an aid, not a holy book.
      
      Reply
      - Paul McArdle says:
        
        February 23, 2010 at 9:34 pm
        
        Regarding written specifications I think the key point here is, as Stephen says, that its purpose is “to assist the developer(s) get their head around the high level problems “.
        
        As we have been working through a business planning process for a spin-off business recently, something I said with reference to that debate is relevant here:
        
        (in my view) a short, sharp process of business planning is invaluable - however a business planning document (on its own) has very low value
        
        For more of my view, see my post “Why do we document”.
        
        The conventional view of a spec as a bible, or as a record, does not figure for me. A stapled group of User Stories, and properly commented code, would be much more effective.
        
        To avoid future confusion, I would much prefer we did not even use the term “spec”, because of the conventional view it conveys.
Emergent Design and Evolutionary Architecture in Sydney | Behind the Scenes at Global-Roam says:

February 23, 2010 at 8:59 pm

[...] by Stephen’s glowing recommendation of the Thoughtworks presentation “Emergent Design & Evolutionary Architecture”, I [...]

Reply
Paul McArdle says:

March 8, 2010 at 8:43 pm

What “Technical Debt” is, and is not

Thanks also, Justin, for pointing us (offline) to this explanation of why “A Mess is not a Technical Debt”.

I note that the comments in this article reference this YouTube clip in which Ward Cunningham discusses why he came up with the metaphor of “Technical Debt” in the first place

Makes it easier to understand than before.

One more example of how we all have much still to learn…

Reply
- Stephen Hurn says:
  
  March 9, 2010 at 9:32 am
  
  Well I would disagree with that blog post. A mess is a type of technical debt (credit card debt instead of home loan debt). Time still needs to be spent repaying it. In this case it is “bad debt” (e.g. borrowing money to pay for a big night out) and not good debt (e.g. borrowing money to finance an appreciating asset, like a business or property).
  
  What is being advocated is that businesses should only try and acquire good debt. When a better understanding of the problem is apparent after the development work has been done, that good debt may be paid off. The bad debt should never be acquired in the first place.
  
  Reply

Behind the Scenes at Global-Roam