Ben J. Christensen

I have long considered it a fallacy to call software development “engineering”. Even though the term technically works for software as “the science, discipline, art and profession of acquiring and applying technical, scientific and mathematical knowledge to design and implement structures, machines, devices, systems, and processes that safely realize a desired objective or inventions” (see Wikipedia), this is not how the term ends up being interpreted and therefore misrepresents how software development actually occurs – since engineering has become synonymous with building physical structures such as buildings, bridges, roads. Recently someone with authority on the subject expressed this opinion much better than I could have, and since he has authority on the subject and in the industry, it means something.

According to Wikipedia, “Tom DeMarco has authored over nine books and 100 papers on project management and software development” and has been intimately involved over the past 40 years in the software industry and the establishment of “structured analysis”, software project management concepts and legendary software development projects such as the first large scale telephone exchange for Bell.

He recently wrote an article titled “Software Engineering:  An Idea Whose Time Has Come and Gone?“

The article is well written and there is no value in me repeating it verbatim, so read the original.

The following quotes however are worth me repeating along with my commentary:

“Software development is inherently different from a natural science such as physics and its metrics are accordingly much less precise.”

Software development is not like building a bridge – where the materials, laws of physics and chemistry, labor times etc have all been experienced and practiced thousands of times, can be accurately predicted and correctly planned for, then managed.

“Software development is and always will be somewhat experimental. The actual software construction isn’t necessarily experimental, but its conception is. And this is where our focus ought to be. It’s where our focus always ought to have been.”

If we are writing software, it’s because we’re trying to do something that hasn’t been done before or is different in some way from what already exists, otherwise we’ll purchase the existing software. We are building a solution to a problem – to create value of some kind.

If we’re doing something that hasn’t been done before – either with new technologies, new rules, new performance criteria or whatever it is that makes it unique – then we are doing “research and development” – experimentation. It is not as if a blueprint for a house is given and the construction team comes in to build it as they’ve done 1000 other houses in the same community. Software “architecture” doesn’t work that way. There is never a replicate – because it’s not physical.

Yet business and project management would love it to be – it makes it easier to control, schedule and most critically, budget for.

Tom DeMarco discusses “metrics” in the most detail in the article, about why we spend so much time trying to control software development projects, and particularly the timeline/cost aspect.

He says:

“For the past 40 years, for example, we’ve tortured ourselves over our inability to finish a software project on time and on budget. But as I hinted earlier, this never should have been the supreme goal. The more important goal is transformation, creating software that changes the world or that transforms a company or how it does business.”

After giving an example of two projects, one creating little value and one creating significant value he states:

“This leads us to the odd conclusion that strict control is something that matters a lot on relatively useless projects and much less on useful projects. It suggests that the more you focus on control, the more likely you’re working on a project that’s striving to deliver something of relatively minor value.“

Ironically, I’ve generally seen the opposite perspective from business – that the “expensive” projects, the ones typically capable of creating the most value, are the ones where the most control of timeline and cost is desired and therefore the most damage done by “management” as opposed to allowing the necessary experimentation, design, thought, research and development to occur to create the most value out of the software.

Being criticized of being “unable to delivery on time” in software development is not something I can really take personally – when the industry at large struggles so significantly with this, largely because it has tried so desperately to liken software development to a public works engineering project such as building a bridge, a building or a road.

Software development is more like art than most seem willing to accept.

It is driven by creativity, experimentation, research, design and artistic drive.

Is it any wonder that the “transformative” software projects most often come from startups (Apple apparently being the exception with their iPhone OS and demonstrating the amazing culture of design they allow to flourish) and then as they mature become stagnant once quarterly reports drive business as opposed to the development of a product? Or that opensource, where timelines mean nothing, can build great software such as Linux, Firefox and MySQL? Look at what has happened to MySQL over the past 2 years as business “management” has replaced “experimentation” and focus on quality — the key people who built MySQL over the years have left to work on something where strict control is not the focus.

As Tom DeMarco suggests, I hope to be focusing on development where the value is worth allowing the proper time, people, experimentation and development to occur – so that something of true value can be created. If management of “costs” is the most important aspect of the project, the greatest value to be created, I don’t even want to bother. I’d rather work on a project that has the ability to be transformative, and create greater value than the cost of the effort.

An interesting article by James McGovern on recruiting IT talent.

The gist of it is that it’s very difficult to rely upon resumes, and that we as an industry seem to put up with “falsification” of skills and keep people onboard who really don’t have the skills they need to do their job.

It closes with the following statement:

“World class organizations are built on foundations where people can trust each other and don’t have to second guess the ability of the team. If you find yourself in a shop that doesn’t get it and you stay, maybe you are getting exactly what you deserve… ” – James McGovern

In a previous post I spoke of “scaling distributed development” in reference to the challenges in increasing the size of a team and the complexity of projects while working under limitations of the experience level of developers available in the remote location.

Continuing on this theme I have continued pondering what it takes to effectively do development with a remote office.

I use the term “distributed development” as opposed to “outsourcing” since the word outsourcing generally implies one company sending the work to a second company – not multiple teams of the same company in different geographic regions, but my thoughts more or less apply to both scenarios.

My thoughts are shaped greatly by my experience with a remote development team in Pakistan and myself based in the US. Therefore they may or may not be applicable to other regions such as India and China.

What I Feel is Necessary for Effective Remote Development

In order for remote development to be successful, I believe the following things are necessary:

• Senior leads “on the ground” with the team who have experience gained over years of development and most importantly in delivery of applications to customers and operating them in production.
• Frequent communication between the US and remote office with as much of it being in “real-time” as possible, not just “touching base” morning and night. Development is not a shift-work job like a factory. Continuity and comprehensive knowledge is critical.
• Human interaction through travel and face-to-face meetings. Nothing establishes team rapport like sharing a meal together. Rapport is what enables communication and difficult work to be done together in the long haul. Relations can be built remotely – but much more slowly, with much more effort and with far fewer people.
• The ratio of senior to junior must be healthy. Too few senior resources means the junior developers are not getting the mentoring and guidance they need, they make mistakes, code quality goes down, production issues arise, morale decreases. It’s my belief that a senior developer can handle no more than 5 junior developers if one expects proper mentoring, training and good code quality. This ratio may be able to change if there are solid ‘intermediate’ developers who can then have junior developers managed by them, but I have not seen this work very well. Intermediate generally means you’re not fresh (completely junior) but you still need guidance, direction and mentoring, and are therefore not ready to lead, mentor or manage others and therefore should not be setup to fail as such.
• The developers themselves must be consumers of technology products. If a developer writing an ecommerce application has never actually purchased a product online and had it shipped to their door, they have no common relationship to “how it should work” let alone “how it should feel”. Common sense is taken for granted – it is something learned through life experience.

Positives of Pakistan Remote Development

In 7 years of working with Pakistan developers, and likely seeing over 100 developers during that time, I have found the following to be true.

• Very intelligent.
• Very eager to work and grow.
• Amazing work ethic.
• English language skills (for being a non-english country) are quite good.

Of course, the extremely low cost is a final factor that can not be ignored and is the driving reason for having a team in Pakistan.

I have delivered some significant software and production systems with Pakistan development teams – myself being the only US based person. It can work and be very cost effective and rewarding.

Negatives of Pakistan Remote Development

There are definite drawbacks however to working remotely in Pakistan.

Time Zone

The time difference is 12 hours (from Mountain or Pacific US time depending on time of year) which makes real-time communication very difficult as it is limited to early morning or late night. It would be perfect for shift-work as you could do hand-offs every 12 hours – but development is not that type of work.

Cultural

Cultural difference means that “common sense” in each country is very different. Thought process is definitely different. Consumer experience with technology products is limited. This has an impact on understanding and requires much more detailed communication, requirements definitions and business reviews as nothing can be taken for granted. When it is, or something is “assumed”, the cost is wrong implementations and/or delayed timelines.

Perhaps another cultural difference and a side effect of the positive eagerness is not knowing what they don’t know. A desire to try something does not equate to knowing what or how to do it. Saying “yes” is great – except when that “yes” is not accompanied by a comprehension of what it entails. Commitments to deliverables can be rather misleading if in fact the person does not know what has been committed to or know how to deliver it, but wants to try it all the same – and therefore says yes.

Geopolitical

Let’s face it, Pakistan is not an ideal place in the world these days. It hasn’t been for years and likely won’t be for many more. My hopes and prayers are that the country will improve in its stability, infrastructure, economy, education and other matters, but today it is not a country many people desire to be in. This is a big part of why the cost is so low to hire people in the country. At the same time, it has a very significant impact on talent: people don’t stay.

Virtually anyone with a skill and the opportunity to leave will do so. It is evident to me in how many people I have seen emigrate away from Pakistan to Canada, US, Britain, Singapore and other countries over the years. It is more evident in the seemingly complete lack of “senior” talent in Pakistan.

It is virtually impossible to find and retain a “senior” level developer with 8+ years of experience, let alone 15+. Even 5+ is hard to find.

If the effort is made to train someone, after a few years as their skills and experience mature, the pull to leave the country increases and the company loses them.

I understand the pull. I work with 3 people in the US all who have immigrated from Pakistan. Incredibly smart, hard working people – very good at what they do. But they all left Pakistan instead of living their lives and careers there.

Impact on Cost and Productivity

Considering the extremely low cost of developers in Pakistan, doesn’t that outweigh the negatives?

Under certain conditions, it can work very well, but those conditions must be met or it falls apart, and in general I question the actual “low” costs.

Assuming communication and timezones are dealt with, and the cultural barriers addressed – one item can not be overcome through any amount of process, hard work or documentation. Geopolitical.

The geopolitical situation is beyond resolution for any one company or team, and it is a force that can not be ignored. To ignore it is to invite failure.

Long timelines, poor quality code, production issues and failed deliverables are all symptoms of one thing: a team that does not have the skills to deliver on what is being attempted.

What about mentoring, grooming and training? All of these are great – but if the people that are trained do not stick around longer than the training, then there is an incredible cost of training that is not recouped.

The senior developers spend their time training, not developing. Then the junior developers leave before they can become productive. The time of the senior developer is wasted as far as the company is concerned. The senior developer is seen to be “unproductive” because delivery is so slow – yet they have no control over the loss of the junior resources who were just getting good enough to work productively.

To make up for it, more junior developers are added – making the matter worse. A higher ratio of junior to senior means less mentoring and training for each developer. This means lower morale, lower skills, and poorer quality of delivery.

This catches up — bugs, production issues, delivery failures, long timelines. This then leads to morale problems and further turnover of resources, generally the ones with the most potential and skill.

Conclusion

Remote teams such as this can work very well if they are small and directly led by a senior developer lead.

Mentoring and grooming of developers in Pakistan (or other remote locations) can be done and can be successful. I’ve done it. It has worked well. It has only worked well though when the remote team has worked my hours and the team size was small — so that I had constant real-time communication. I did this for several years.

This however does not grow a business if those people don’t stick around. Nor does it scale.

Thus, I believe the actual cost of development for an outsourced development team with these traits is in fact much higher than the monthly salary of a developer in Karachi.

If a senior developer from the US is spending 75% of their time mentoring, training, grooming and managing junior developers – and those developers all leave within 2-3 years, what is the cost of that 75% of time compared to if it had been spent developing the products directly?

If a 4 month project takes 8 months because of junior talent, and 4 months of sales are lost because of this, are those 4 months of sales higher or lower than the cost of having a team of more senior talent?

To ignore these facts and not look at the actual costs is to set oneself up for failure.

No amount of documentation, process, hope or patriotism can overcome the geopolitical issues and lack of long-term talent.

So what is the solution?

The first thought is to have everything return to the US (or Europe, Canada etc, wherever the parent company resides) where developers expect to live their lives and have long careers, so talent can be groomed and retained.

Some business models just can’t afford this however. In that case, another location must be found where the following applies:

• cost of developers at least half of the US
• geopolitical situation where developers desire to stay, to live their full careers in their country
• good english communication skills
• intelligent and educated people
• preferably a culture exposed to technology driven consumerism
• preferably in a more similar timezone to the “parent company”, in this case, the US timezones
• safe and easy for North Americans and Europeans to travel to and stay for extended periods of time

I believe distributed development is here to stay, and creates significant value when done correctly. It is not an easy thing to do well though. Nor does ignoring the challenges of culture, distance, time and politics benefit the business.

I want to see a distributed team setup according to the thoughts I’ve described above and see it achieve the level of productivity and efficiency that is its potential.

I’ve spent the last day considering product and/or project requirements – in particular how to document and communicate them.

A draft of a requirements document numbering in excess of 100 pages has very thoroughly covered the technical definitions of all of the various ‘features’ – what the system needs to contain. However, it reads like a dictionary definition and does not in any way help me understand the following:

• What is the business purpose of the product?
• What is the business purpose of a particular requirement?
• What are the use cases that requirement must serve?
• Who will use the specific requirement and for what?
• How does a given requirement fit into the overall system so I know its context?
• How will the product implement the feature that address the given requirement?
• Where in the system architecture should the developers implement the given feature?

I became aware while reading that most of the people for whom this document exists would not have a much better understanding of what needs to be built after reading the document than from before.

During my consideration I listened to a podcast on “requirements engineering” that interviewed Christof Ebert.

I think the recommendations and principles are valuable regardless of what process is followed — whether they are done upfront or at the beginning of each iteration with waterfall, agile or a mixture of both.

A few notes from the podcast:

What Requirements Should Define

• What the Customer Needs Are
• What the Product Should Do
• How the Product Should Do It

Common Requirement Problems

• Missing Requirements: Incorrect effort or tactics to seek out (elicit) all of the requirements.
• Wrong Requirements: Poor capture, communication or lack of validation with stakeholders.
• Changing Requirements: Requirements are never static, they always change.

3 Levels of Requirements That Need to be Addressed

Market (External)

• what user or client expects
• the “problem space”, defining the problem and what is desired to solve it
• what will make the product successful

Product (Internal)

• what the product will do
• how the product will address the market requirements
• the “solution space”, defining the solution that will meet the user (market) requirements
• define what functionality and requirements will be made into a product

Component (Software)

• lower level details of how to implement the requirements in software
• non-functional requirements such as quality, service levels, maintainability etc
• context of requirement in overall system

What is good enough?

Reality must be paid attention to. Generally the economics of a project can not justify the “ivory tower” of solutions for all aspects of a project.

What is good enough to serve the user – in both functional and non-functional requirements?

What is of high priority?

Almost always there is more work to be done than resources, so what is of priority?

What can and will be committed to for development, with what resources and by when?

What can users live without, what must they have?

My Final Thoughts

As I prepare to attempt a revision of the requirements I’m working on, I must remember the primary reason for the documents existence: to communicate to developers, QA, analysts and business the reason for the product, what value it’s creating and what use cases it will provide functionality for.

If it doesn’t leave the reader with comprehension of what they’ll end up with at the end of development – at least at a 10,000 foot level – and is just a list of features with no context or purpose, then it’s a failure, no matter the length or level of detail.

Business demand as of late and future projects on my horizon are causing me to ponder and deeply consider the staffing requirements to deliver high quality, cost effective solutions in reasonable timelines.

Over the past 7 years I have had very hands on experience with distributed development – working from the US with over 100 people in development, QA, operations and other teams in Karachi, Pakistan. It has included two personal trips to Pakistan, thousands of hours of MSN or Skype chatting and many odd working hours in order to accommodate a 12 hour time zone difference.

All in all it has been a fruitful experience. It has definitely had its troubles, pains and inefficiencies, but products and services have been delivered cost effectively to our customers ranging from Fortune 100 on down. Applications have been built which support 10s of millions of dollars in revenue and millions of transactions per day.

In the past year however I have begun to see a different set of challenges emerge — the ability to scale.

By scale I mean 2 types of scale:

1) Scaling the size of the team.

By this I simply mean adding more people to the team in Karachi to accommodate increasing work load.

2) Scaling the size and complexity of projects, code bases and system architecture.

This refers to the ever increasing amount of code, projects and modules to manage and develop as years of projects become legacy or enter “operational support” status, as increased business success and growth demands more from its systems and feature sets become increasingly larger and as architectures require more advanced models such as SOA, distributed computing, distributed infrastructures etc to handle scale, volume and global distribution of business.

In short, recent experience is leading me to believe that a ratio of approximately 1:5 is the maximum scale that can be achieved in a distributed development model. 1 “local” senior developer to 5 “remote” junior/intermediate developers.

Successful Experience

A team of 5-10 junior/intermediate developers and QA working with leadership from 12 time zones away proved to work well when the following conditions were met:

- The Karachi team worked their nights to accommodate the business and customer schedule which was US Pacific time zone.

- Daily interaction between myself and the Karachi team accounting to several hours a day of direct real-time communication on design, planning, business, and code related subjects.

- Micro-level involvement of myself at the code and task levels involving review of virtually every line of code written, every test, every deliverable.

Three successes came out of the 2+ years of this type of model:

1) Successful and cost effective delivery of the projects.

2) Mentoring, training and grooming of junior developers into what eventually became the best developer talent pool of the company in later years.

One member of that original team in particular, after 3+ years of 10-12 hour work days, all night shift, has immigrated to the US and works as one of our key resources in the role of Application Architect. He had the raw materials, personal drive and competency, but the opportunity to progress from a fresh graduate to “senior” level skills and roles in a matter of years was obviously accelerated greatly by the virtually non-stop direct contact with a senior developer, high-profile projects and US customers.

3) Proof that the distributed development model can work (under certain conditions at least).

What a Junior Developer Needs

I have come to the conclusion that a junior or intermediate developer needs quick and constant communication with their lead — whatever we call that person (senior developer, architect, development manager etc).

The lead must be capable of providing timely and thorough responses, training, mentoring and guidance.

Nothing can replace working in the code together, explaining why decisions are made, how to refactor something, how to add a feature to an existing codebase, how to profile, tune, optimize, debug, test and deploy.

Generally a young developer has little to no experience with “production” – what it really means to build and deliver a system that can be deployed and operate 24/7 — by itself with little to no involvement from developers — or even more difficult, be handed off to an operations team for deployment and operations with no direct involvement of the development team.

The concepts of logging, configuration, performance, multi-threading, system interaction, hardware and infrastructure are all simple in dev. Until one has experienced true production, and more importantly — issues in production, it’s difficult to understand why certain design decisions are made which seem to not be that important in dev, but make a significant impact in production, or to the teams maintaining production, or to the teams maintaining the code a year later.

Raw intelligence and capability exists in many people in many locations, but the proximity and availability of skilled, experienced senior personnel is key to maturing the raw skills of the young inexperienced developers into productive “seasoned” developers

No different than how a fresh graduate from a college in the US is of limited value until several years of grooming and experience under senior guidance, a developer working remotely has the same challenges — but with some further hurdles:

- Language barriers: Educated in english does not equate to fluent or comfortable.

- Time differences: If 2 people are 12 hours apart, that means that unless they purposefully shift their schedules, they only have a short window of time early morning and late night to work together.

- Physical separation: Can’t just walk around the corner to ask a quick question and can’t easily judge the emotions of the person you’re working with.

- Cultural separation: Many things in North America and Europe are taken for granted, such as purchasing a product on Amazon.com and having it show up at your door 2 days later. If someone has never experienced this it’s much more difficult to relate to a US business requirement when they say “the user experience should be like Amazon.com”.

The concept of “common sense” is not universal – it applies to experiences gained in relation to when, where and how one has lived.

How I’ve Countered the Hurdles

Language

I have found that rarely is verbal communication a good thing, even when it appears that it’s going well.

“Yes, I understand” is not a trust worthy statement in many cases – written or verbal – but specifically in verbal with developers whose English is a 2nd or 3rd language primarily learned through school, textbooks and the internet.

Reading and writing on the other hand can generally have very high comprehension and actually be much faster.

Using something like a Skype chat (text not verbal) allows for a number of positive things:

- built in meeting notes that can be referred back to

- dialect and speed of talking do not impair comprehension, as everyone can read and re-read if necessary what has been written

- appropriate time to think (and translate thoughts to English if needed) is built in to the medium

- multiple conversations can be had in multiple windows at a time if needed

Time Difference

On the projects where I have been most successful, I eliminated the time difference barrier by having the Karachi team work US hours to align with our customers and US business.

This solved 2 critical issues:

- The team in Karachi could now participate in business and customer interaction and communicate directly with customers via phone calls, conference calls, Skype or MSN chats – instead of everything being done in emails with 12 hour intervals or routed back and forth through a single US contact

- The US team and Karachi team worked together as a true team with direct real-time communication all day. The Skype/MSN window became the office, talking almost as naturally via text as if we were next door.

Physical Separation

This is obviously the most difficult to solve, but handling the time difference by everyone working the same hours and having virtually permanent communication windows open at all times in many ways made this issue seem negligible. Yes whiteboarding is more challenging – and you all have to be fast typists, but it worked.

It did not work for all though. I had several senior US/Canadian people that didn’t work out because they couldn’t adjust to an environment where typing instead of talking face-to-face was the primary means of interaction and whiteboarding could not be done in an impromptu manner.

Culture

I have found no secret solution for this. I have had to learn to be extremely detailed in communication, take nothing for granted, think from their perspective and avoid colloquial use of the English language and all forms of sarcasm.

Challenge to Scaling Distributed Development

Growing a team from 5 to 15 or 20 developers, but still having a single senior developer leading them has not worked well.

Obviously the team of 20 is broken into smaller teams, and there are team leads.

The challenge is as follows:

If a “team lead” has 4-5 years of experience, and that experience itself is of only limited exposure to senior skillsets, then that person is being setup to fail as they do not yet have the skills, experience, maturity or understanding to do what they are being asked to do.

This person, who may be an excellent developer on a 5 person team led by 1 senior developer – when asked to lead the development of 5 others is now put in a position where they are struggling to deliver, they are not able to learn as they once did while working just as a developer with the senior developer and the 5 below them are learning even less and the quality of code and timelines of delivery fall thru the floor.

The solution on paper is simple: “Find senior developers in Karachi to lead the team in Karachi.”

In practice however this is far more difficult. Even with restricted immigration policies to US, Canada, UK and other candidate countries in recent years, senior skilled technical people seek out opportunities wherever they may be and are not generally available in Karachi by the time their skills have matured to what would be a “senior lead” or “architect”.

Finding and retaining what is considered a “lead”, “senior developer” or “application architect” in the US is not easy or cheap anywhere – and in “outsourcing” countries challenging to the point of “it’s probably not going to happen”. At least not in a way that can be counted on. When it does happen it’s great, but it’s not something one can plan for.

Process Doesn’t Replace Skill and Experience

I believe process and documentation is needed and has its place, especially as projects increase in size and complexity.

However, no amount of documentation and process can replace competence (see comments by James McGovern on the subject).

Junior developers do not become competent and capable through documentation and process – it comes through mentoring and experience, being groomed by those ahead of them and by applying their raw skills to educate and improve themselves.

I can be particularly verbose in writing when I want or need to be. I have written thousands of pages of documentation including hundreds of pages of visual wireframe mockups, use cases, requirements, workflow diagrams etc. The end result? Most of the people they were intended for don’t understand how it all fits together, how the business use cases apply, or how the user interfaces accomplish the use cases.

It has been a frustration for that to be the base — and I’ve attempted making documentation ever more detailed, having thorough review and feedback sessions and even getting to the point of spelling out exact implementation details of given requirements right down to where the code should be done in a given codebase.

Looking back over these experiences, the documentation worked very well for the few people on the team who had already worked with me for a few years and were more advanced in their skillsets and understanding. They would read the documents and question them very specifically. They would find mistakes, or complete design errors. They would find inconsistencies in use cases or workflows.

It was obvious they understood it – and contributed to improving and enabling the delivery. The other 80% of the team however had very little if anything to say.

Thus, documentation and process is important – they are tools to organize and enable large and complex systems to be built and maintained by skilled and competent people. They do not however enable just anybody with a degree in computer science to deliver.

Conclusion

Successful development depends on skilled, competent, experienced people taking ownership and delivering a project, product or system.

Ideally all people involved would be equally skilled and senior – but that’s a rare scenario that I’ve never experienced.

Thus, the mixture of senior and junior needs to be right so as to enable the senior members to still be productive in accomplishing business objectives while grooming and mentoring the next generation and increasing productivity on tasks that can be done by those with less experience with guidance.

It seems that the ratio can’t be much higher than about 1-to-5. The amount of time a senior developer must spend in mentoring, communicating, teaching, reviewing, and designing with the junior members, and then performing their own development, architecture and other roles limits the number of junior people a senior resource can effectively work with.

Regardless of location, whether a team be all physically together, distributed in cities across the US or distributed across 12 time zones like I’ve been, I believe this principle applies. Perhaps the ratio differs for different geographic placements, but in a distributed environment where one can’t walk around the corner to communicate for 5 minutes on a white-board there is a definite effort and time commitment to communication that limits the scale of a development team to a ratio of senior and junior developers that if surpassed dramatically deteriorates code quality, morale and timelines.

Good summary of a “tech lead” by James McGovern:

http://duckdown.blogspot.com/2009/06/enterprise-architecture-so-exactly-what.html

If you are in this role and want to live up to your title, then you need the following skills:

• Accountable – The buck stops here, someone who can shoulder the responsibility without pointing fingers at others.
• Adaptable – Life changes fast, if can’t change with the times, you’ll be left behind.
• Approachable – their team can ask them questions and talk to them.
• Attentive – they actually listen to their team and understand them.
• Aware – they’re aware of what is going on around them instead of sticking their head in the sand.
• Can type – No hen peck typists need apply in this industry.
• Charismatic – People have got to want to listen to you.
• Communicative – You’ve got to want to communicate with your team.
• Competent – You’ve gotta know your stuff.
• Confident – If you’re not confident in yourself, how is anyone else going to be?
• Decisive – can make decisions themselves and be accountable for them.
• Driven – You’ve gotta see the goal and go right after it.
• Focused – You’ve gotta have the staying power to keep going and reach the goal.
• Inspirational – this is the most important one for me! If you can’t inspire people, what are you doing?
• Meticulous – The truth is in the details.
• Nurturing – The test of a good teacher is when their students surpass them.
• Resourceful – You’ve gotta be able to find the answers to the things you don’t know.
• Technically minded – In this industry at least.
• Understanding – If your team don’t think you understand them, they won’t bother trying to understand you.

As a leader, it’s your delegates that help build your success. Treat them like gold, and they will do the same for you. One non-programmer that can inspire 10 “average” programmers to greatness is worth more than one great programmer who can’t inspire their team to do more than meet their job requirements…

http://martinfowler.com/bliki/FixedPrice.html

FixedPrice agile 29 July 2003

Many people belive that you can’t do a fixed price contract in an agile project. Since the whole point of an agile process is that you cannot predict the future, this isn’t an unreasonable supposition. However this doesn’t mean you can’t come up with a fixed price agile contract, what it means is that you can’t come up with a fixed price and fixed scope contract.

Usually when people say fixed price, they mean fixing price, time, and scope. This requires detailed, stable, and accurate requirements. The whole point of agile development is that it works with more fuzzy requirements. To handle this with a fixed price contract you essentially come up with a plan that says “we have $x to spend and we need a release on 1 Dec. We’ll collaborate together to come up with the best set of features to go live with on that date.”

The initial plan

The initial plan for this kind of arrangement is not really any different to a predictive project. The essential difference is that we don’t expect things to go according to plan. Instead we expect to deliver a better product that we can currently envision, because we’ll learn more about the project as the project proceeds.

Or if we find things are much tougher, we’ll find that out too. In which case we’ll modify the plan. If the customer doesn’t like the resulting plan they can cancel. While this isn’t good, they’ll usually cancel earlier than they would under a predictive project, becuase predictive plans tend to discourage change, and thus make it easier to not realize when things are going off course.

http://martinfowler.com/bliki/FixedScopeMirage.html

FixedScopeMirage agile 30 September 2004

Many companies like the idea of writing a contract that fixes scope and price because they think it lowers their risk. The mirage says that their financial obligation is fixed at the price of the deal. If they don’t get satisfactory software, then it won’t cost them.

In my independent days, I always advised clients to avoid this mirage, it works fine in theory – but practice bites a big chunk out of its strengths.

For a start focusing only on the cost is short-changing the financial issues. You get software written because it has a business value to you – one that’s greater than the cost. Otherwise why do it? If the software isn’t satisfactory the financial damage isn’t just limited to the what you paid for it. There is also the opportunity cost because you didn’t get the business value you were expecting. The cost is also higher than people think, it’s not just the money paid to contracting company, it’s also the cost of people’s time on the project. I remember seeing one project that went badly belly-up – the costs there were large, while the client tried to sue the contracting company for the money they’d already paid I suspect only lawyers saw any benefit.

The other thing that ruins the pretty mirage is well-known by contracting companies. A fixed scope contract only is fixed is the contractor really understands the requirements. But such knowledge is so rare that you can win by banking on its absence. Such contracting companies deliberately low-bid the fixed price, with the explicit plan of making a profit on change requests. Indeed some firms actually incentivize their sales and account managers based on how many change requests a project receives.

These are the reasons why I think fixed scope and price contracts are bad for the customer. It’s why we at ThoughtWorks avoid this model as much as we can. It is possible to do a FixedPrice contract in an agile manner, but it’s not wise to fix the scope.

http://martinfowler.com/bliki/ScopeLimbering.html

ScopeLimbering agile 27 October 2004

One of the basic tenets of agile development is that requirements changes aren’t just expected, they are welcomed. This poses a particular challenge when an external company, like ThoughtWorks, is doing work for client. Many clients want a FixedPrice arrangement, which is really fixing scope because they see the FixedScopeMirage. But a fixed scope contract is totally at odds with agile development, so what is a company like us to do?

Over the last year or so we’ve been pretty successful at what I call Scope Limbering – starting with a fixed scope contract but working with the client to help them understand the way agile development works to overcome the FixedScopeMirage.

To illustrate, here’s a specific example with a client that I’ll call Nebbiolo. As is often the case with our work, we came in because another delivery company had failed (you’d recognize their name, but I’ll spare their blushes). They’d spent a long time gathering requirements but got nowhere with development. So the client had detailed requirements which had cost a lot of time and money, and were feeling rather sore from the other firms inability to get things done. So they insisted on a fixed scope arrangement, based of course on those detailed requirements.

We took a look at those requirements and estimated it would take us about half a million dollars to build it. Like most people tackling a fixed scope project, we added a buffer – in this case quite a large buffer doubling it to a full million. This was still less than the original contracting company’s estimate. (We charge much higher daily rates for our people, but since we have better and more productive people, we can actually do the job for less.)

We weren’t at all shocked to discover that despite the detailed and heavily reviewed requirements, the requirements changes still came thick and fast. For each one we estimated the scope of the change and figured out how much it would cost, but didn’t charge the client for the change. Slowly but steadily the changes ate up the buffer. After about six months or so the changes had used up the buffer completely. We’d been quite open with Nebbiolo during this whole time, so they weren’t surprised when we told them that we couldn’t afford to eat the cost of the changes any more. During that time we’d collaborated closely with Nebbiolo and they’d grown to trust us. We had no problem finding more money, indeed it took another half million to cover the requirements changes that were needed before we delivered.

At the end of all this Nebbiolo agreed that the fixed scope approach was a mirage, and we would do future projects together using a more flexible charging scheme.

I think the key to this story (and we have half a dozen similar examples) is that from the beginning we sought to put the relationship between our companies on a collaborative note rather than a confrontational note. The biggest problem with the fixed scope contract is it immediately pits the client and contractor on opposite sides where they are fighting each other about whether something is a change and who should pay for the change. An agile approach hinges upon replacing that confrontation with collaboration (customer collaboration over contract negotiation).

A tripling of actuals over initial estimates isn’t unusual in our industry. Mostly I believe this isn’t because we are so bad at estimating (although we aren’t exactly stellar at it), but it’s mainly because it’s so hard to get a decent set of requirements. Many delivery companies take advantage of that by low-balling the initial bid and making profit on scope changes. But this approach sours the ongoing relationship with the client – which leads to the whole industry gaining a bad reputation. The story of Nebbiolo is one way to arrange things to keep the relationship in good shape, and I think we need more that for our whole industry.

http://martinfowler.com/bliki/SpreadingIncrementalism.html

SpreadingIncrementalism agile 5 January 2005

From time to time people question whether a particular specialty can be used incremental way: “You can’t do (security | user interface design | databases | internationalization | * ) with an agile project because this aspect has to be done up front.”

When a question like that’s put to me, I’m immediately on a sticky wicket because I’m not that knowledgeable on that specialty. Application design is something I think I can talk about, but security (for instance) isn’t – and my questioner may well be a well regarded leader in that field.

Despite my acknowledged limitations in that field, I’m not about to say that you can only use planned design in that area. What I can say is that we don’t really know whether you can do incremental design in that area. I’ve seen enough cases where people have said “you can’t use incremental design for x” only to find you can. Application design is one, database design is another. So until people try incremental design out in a serious way, I’m very reluctant to rule it out.

Part of the difficulty of assessing this question is that it’s too easy to do incremental design poorly. If you do incremental design in an uncontrolled way, you are most likely to end up with a design that is a mess – to make incremental design work you need something that makes the design converge into order. In Is Design Dead I referred to these as enabling practices. For software design I labeled testing, continuous integration, and refactoring as key enabling practices to get software design to converge and avoid software entropy. When we talk about something else, like UI design, the issue is finding what those enabling practices are. They may be similar or inspired by those in software design, or they may be something different. In database design, for instance, incremental data migration is a key enabling practice. Until you find a good set of enabling practices, incremental design is thin ice.

Despite that thinness, I do think that an incremental approach is so worthwhile, that it’s worth experimenting to find the right way. Phased up-front design approaches fail far too frequently – furthermore they do a poor job in the chance of volatile requirements – which I see as a unavoidable factor, at least in enterprise software development.

The biggest reason, however, that I favor incrementalism is the oldest one – risk management. Without trying out your designs, you are too vulnerable to things not working out the way you think they would, too vulnerable to schedule slippages late in development. These risks are reason enough to look for ways to introduce incrementalism into more aspects of software development.