I didn't have time...

 "I didn't have time to write any unit tests."

"I didn't have time to workout."

"I didn't have time to visit my parents."

"I didn't have time to read any books."

When I find myself starting to say things like these, or when I hear them from others, I have started substituting "take the time" for "have the time."  So, for example,

"No, you mean, 'I didn't take the time to workout.'"  

That one change changes everything.  It adds agency and accountability.   

I take the time, every morning, to go running or walking with my dogs.  Some days that is really hard to squeeze in.  But it comes off the top.  Like sleep, taking the trash out, brushing my teeth.  Hard days are hard days.  Skipping workouts or other activities of daily life actually only makes them harder.

When I see code with no tests and I hear the excuse "I didn't have time to write tests,"  I see lack of accountability.  I would rather have well-defined contracts and failing tests with no implementation than code with no tests.  The former is progress.  The latter just adds risk and debt.  People who work for me sometimes don't get this right away.  It is a great example of how you usually have more agency than you realize.  I expect people to take the time to do things correctly and to include the time needed in their estimates of how long things are going to take.  I know that I am not alone in this.  People often create caricatures in their minds of leaders, family members and other unseen forces driving them to squeeze out all but the most urgent activities in their work or home life.  When you ask them, "Are you sure that is really what X wants?" where "X" is the leader, spouse or other stakeholder, the answer is often surprising and in fact just asking the question can lead to a realization of agency.

So just try this:  every time you find yourself saying "I didn't have time to..."  force yourself to say out loud "I didn't take the time to..."

Leading in uncertain times

At the beginning of the pandemic, I wrote the post, "Put on your own mask first."  That was good advice then and it is good advice now.  But the foundational challenges that we are facing today require a little more than just maintaining a confident and positive attitude. 

Acknowledge uncertainty and factor it into your plans, but have a plan and stick to it

People need to know that their leaders have a plan.  They don't need to know all of the details and they don't have to understand the full context, but they do need to know that there is a plan and they need to understand the basic rationale behind it.  Nothing makes people more nervous than the feeling that their leaders don't have a coherent plan.  It is OK for the plan to change when it has to, but at any given time, there has to be a plan of record that team members can anchor themselves to.

Encourage questions and answer them honestly

One of the hardest things about leading in uncertain times is that you face this double-whammy of not having buttoned-up answers to everything but needing to get in front of the team more often and sometimes with little time to prepare.  Hiding from the team until you have a polished message or just talking at them is the absolute worst thing to do in these times.  Show up as your authentic self and keep coming back to the plan and the principles underneath it.

Show your team that they can count on you personally

In difficult times, bad leaders break.  Even good leaders make mistakes.  But their teams see them acknowledging their mistakes and doing everything possible to recover from them.  Your team needs to see you as the one who is going to lead them out of whatever kind of mess they or your business have gotten into.  They need to feel like you can solve any problem, even though in fact the way that you do that is by getting them to think about the problem in the right way.

Manage conflict proactively

Conflict is natural and can be a healthy part of team dynamics.  Like a controlled burn, however, it can have really bad effects if it is not managed.  In uncertain times, it's as though there is a ton of very dry tinder just waiting to explode on your team.  You need to be extra vigilant to control the burn in these times.

Keep up the pace

When I run with my dogs, if the pace is too slow, they get distracted and the whole thing kind of falls apart.  The same applies to teams.  Stop/start, indecision, putting things "on hold" - you can't let these things slow the pace to the point where the distractions kick in.   In uncertain times there are lots of distractions.  You need to keep the dogs barking.  Similar to managing conflict, this requires that you anticipate the stops and starts and keep things moving.

Celebrate initiative, agency and control along with success

Celebrating success is always important, but in uncertain times it really helps to link and label the evidence of initiative, agency and control that led to the success.  When people have a sense that a lot of their world is "out of control"  it really helps to show them, ideally with something that they have just completed, that in their job at least, they do have agency and control and all they have to do is take initiative.  

In uncertain times, people need more from their leaders.  You absolutely need to "put your own mask on first" and make sure you have the support of your own leaders, peers, friends and family, but you need to show up for your team - even when you don't have all of the answers and when you may be facing doubts yourself.  Just taking the hard question, celebrating a small success, or helping personally with a small problem can have a big impact.

How not to get breached (too badly)

The other day, someone asked me if I had ever experienced a major security breach. They were shocked when I responded that I had not. That is because I have been a CTO for the past 18 years, including stints at some bleeding edge SaaS companies that were constantly under attack. I have managed many security incidents, but none that resulted in a major breach. While of course it is possible and even likely that dumb luck has played a role in my success, I think that the following things have certainly helped. Each of the imperatives below have been important to me. I have never achieved perfection in any of them, but I have never stopped pushing.  The imperatives are written from the standpoint of a CTO; but anyone who cares about security can use them to help protect their company.

1. Know your risks and be honest and transparent about them

You should always have a top n list of key risks that you are worried about, and n should be less than or equal to 10.  You should have a weekly conversation with your Security officer that reviews each risk, mitigation plans, compensating controls and any help that the security team needs managing it.  It is critical that these conversations be open, honest, comprehensive and engaged. If you can't understand some of the technical security content, you need to study it and keep asking questions until you understand it.  You need to build a culture on your team that does not sugar coat or hide risks and your response to learning about them needs to be consistently positive, supportive and action-oriented.  You need to never appear to be annoyed by risks or angry at those who report them.  You also need to review critical risks at least quarterly with your partners on the executive team.  These reviews need to be open, interactive, transparent and engaging.  Your objective is not to show that you have things under control, to justify investment or to cover your ass.  Your objective is to proactively cover their ass.  If you do a good job clearly representing risks, mitigation plans and compensating controls, you will get funding and leadership support as a side effect. 

2. Focus on actual risk

Don't let compliance, vendors, talking heads or your own cool ideas distract you from systematically reducing risk.  If you do a good job identifying and managing risk, you will achieve compliance as a side effect.  Attackers don't care how "buttoned up" your security program looks or how beautifully illustrated your risk registry is.  What matters is where you are weak and what you have done to compensate for your weaknesses.  Note that this is exactly the same thing that auditors care about.  Try to put every possible cycle into things that significantly reduce actual risk.  Compensating controls can be ugly and low tech, but they can save your ass.  I am certain that some of the ugliest and most cumbersome shims that I have put in while working on permanent solutions have saved me and my companies from great harm.  

3. Pull on every thread

When you have evidence of an attack or vulnerability, make sure that you investigate everything fully. Don't just celebrate having thwarted or annoyed the attackers into leaving. You need to have people whose job it is to investigate security incidents and you have to give them the time, tools and access to do their jobs.  Ask probing questions and don't assume that your initial analysis of an anomaly is correct.  I wrote a few years back about fully solving problems.  That same thinking applies here - especially the part about "widening bugs."

4. Make security@ a welcoming and helpful place

Respond kindly and helpfully to all security-related questions or concerns from employees.  Don't put the burden on the reporter to establish that an issue is worth investigating.  You want them to come back, not to feel stupid or ignore things.  You also want them to learn.  The most important single thing that any company can do to reduce security risk is to develop or acquire high-quality security awareness training and make sure that everyone takes it.  If your company produces any kind of software (including for internal use), make sure to also acquire - and require - high-quality secure application development training.  It's very important that all training, consulting, reviews and standards promoted by your security team be of the highest quality and backed by friendly people who understand your business and are willing to patiently answer questions and help people understand security concepts.

5. Stay current

Attack types and vectors change all of the time. Many of these changes have no impact on your environment, but you need to make sure you see and assess impact of new threats as they emerge. Here again, you need to have people whose job includes monitoring security advisories and assessing their impact on you.  You need to make sure that you understand clearly what the advisories and your team are telling you.  That means you need to devote a significant amount of your professional development time to keeping up with the changing threat landscape.  As a CTO, you have a unique vantage point that virtually nobody else in the company has.  It is critical that as new threats emerge, you personally think through their implications across your technology estate.  Just as you and your team need to stay current on the threat landscape, so your systems need to remain current in terms of patch levels.  If you do not have fully automated, short-cycle patch deployment capability, this needs to be put in place.  Even if it is in place, it needs to be actively maintained and continuously exercised and you need to eliminate the things that can't be patched.

6. Do real diligence on suppliers

Some companies have well-established supplier risk management capabilities.  Even in those companies, however, sometimes the level of technical security diligence is not where it needs to be.  You need to step back from the questionnaires and boilerplate RFP responses and think clearly about where the material risks are with suppliers and dive deeply into what controls you and they have in place to mitigate them. Again, your vantage point as CTO is critical here.  You can't count on somebody else's checklist to see the full picture.  You need to direct your best spidey-sense toward where vendors may be weak and adaptively probe to make sure that you have discovered all of the risks.  Be especially careful with low-cost vendors.  Finally, make sure that you regularly review vendors' security posture.  If a vendor is acquired or the product or service you are using is slated for end of life, that should trigger a special review and contingency plan.

7. Use architecture as a weapon

Everyone knows that it's way less effective to try to add security to a naively implemented system.  On the opposite side of this are systems that deliver security "natively," exposing services and features in a way that is hostile to attackers.  Zero trust and end-to-end encryption are examples of this.  They are baked into the architecture and inherently hostile to attackers.  Really effective, dynamic and fast credential management and revocation is another great weapon.  The same service that allows applications to quickly get keys and secure communications can help you quickly respond to an attack or vulnerability.  Constantly push for the architecture win-wins that make your systems both more robust and more secure.

8. Constantly question privilege

Make least privilege an organizing principle.  Start with your own.  Do you really need production access?  You are a fat target.  Don't have anything valuable on your laptop and don't let your account be a valuable attack vector.   The same applies to everyone in your company and every process running in your environment.  The less they can do, the less valuable they are as attack vectors.  Constantly ask why individuals or processes need the access that they have and constantly prune.  Just like patching, if your environment does not have the automation, test environments or other infrastructure required to stop individuals or batch jobs from having to have weakly limited production access, you need to fix that.  If offboarding is not bulletproof when it comes to system access, make it so.  Now.  This sometimes looks daunting or even impossible to fix.  Trust me, it never is.

9. Don't acquire, decrypt, transmit or store data that you don't need

I often make the joke that the most secure and highly available system is the null system - the system that does nothing.  To deliver business value, systems need to access and process data.  But many can fully achieve their objectives with a much lighter touch on data.  Here I am reminded of an army adage about conserving energy quoted by Colin Powell in his awesome leadership book:  "Don’t run if you can walk; don’t stand up if you can sit down; don’t sit down if you can lie down; and don’t stay awake if you can go to sleep."  Here is my version for data processing:  Don't receive if you can do without; don't decrypt if you don't need to see;  don't store if you don't need to persist;  don't transmit if can omit.  The wonderful thing about modern distributed architectures is that they don't force you to create massive centralized honey pots of raw data.  So don't.  

10. Don't assume that any zone, subnet, vm or other subsystem can be hardened

I saved the best one for last.  It never ceases to amaze me how coming on 40 years now after the famous Gage/McNeally pronouncement that "the network is the computer," people keep thinking that they can somehow wall off little islands of safety and security.  You need to disabuse yourself and your team of that archaic fantasy.  You need to constantly assume that the bad guys are inside "your" network and focus on limiting what they can do once inside.  Like multi-factor authentication, strong crypto keys and end-to-end encryption, network controls are good architectural weapons, but they are just one weapon.  Like the others, they make your estate a more hostile place for attackers and slow them down; but they need the others behind them.

Some of these imperatives might seem to limit or constrain what you can do with your products or how fast you can move.  They absolutely don't have to.  For example, number 9 does not say you can't store any data. It just says you should limit what you store.  And guess what, if you do the hard thinking in system design to limit things to what you actually need, you can move faster and do more.  Once you get hard core about 8, you will also pick up speed and value because the only way to do it is to automate.  The win-wins in 7 really are all over the place and once you get that thinking baked in, first in your own mind and then in your team and entire company, you will get benefits way beyond just being a harder target.

Put on your own mask first

Difficult times are the ultimate test of leadership. These are the times when lifelong bonds can be established or connections can be lost. Deep human connections are all ultimately based on faith. Faith that through them strength, security and growth will come. Leaders need to ground that faith in confidence, consistency and an unfailing positive attitude.  

At this moment, many leaders have been put in the impossible position of having to show confidence when the foundations of their businesses, their markets and the global economy are being challenged. It doesn’t work to try to hide things or to spin things or to simply put on a happy face. What people need from their leaders today is authentic confidence. That is what leaders have to reach deep down to find right now. The great ones always find it and their connections are strengthened as a result.

Selling the truth

This month's Significance magazine includes a jarring article about fake news.  As one would expect in Significance, there is interesting empirical data in the article.  What I found most interesting was the following quote attributed to Dorothy Byrne, a British broadcast journalism leader:

"You can't just feed people a load of facts...we are social animals, we relate to other people, so we have to always have a mixture of telling people's human stories while at the same time giving context to those stories and giving the real facts." 

Just presenting and objectively supporting debunking factual evidence is not sufficient.  We need to acknowledge that just as emotional triggers are key to spreading fake news, so they need to be considered in repairing the damage.  I saw a great example of that in yesterday's Wall Street Journal.  An article, titled "Video Contradicts Kelly's Criticism of Congresswoman," sets out to debunk the fake news story promulgated by the Trump administration claiming that Florida Rep. Frederica Wilson had touted her personal efforts in getting funding for an FBI building in her district while not acknowledging the slain FBI agents for whom the building was named.  The Journal article could have stopped at the factual assertions that she had not been elected when the funding was approved and that a video of the speech she gave includes her acknowledging the agents.  But it goes on to provide emotive context, describing the Congresswoman's lifelong focus on issues affecting low-income families and her personal connection with Army Sgt. La David Johnson, the Green Beret whose passing ultimately led to her confrontation with the Trump administration.  The details on how she had known Sgt. Johnson's family for generations and that he himself had participated in a mentoring program that she founded provided context for the facts.  The emotive picture painted by the original fake news claim and the administration's name-calling "all hat, no cattle" was replaced with the image of a caring human being.  In that light, it's easier to believe the truth - that Rep. Wilson was gracious and respectful of the fallen agents and their families just as she was of Sgt. Johnson and his family.

The lesson learned here is that in debunking fake news, "factual outrage" is not enough - we need to focus on selling the truth as the more emotionally satisfying position.  As the Significance article points out, people are drawn to simple explanations and beliefs that fit with what they want to be true.  So to repair the damage of fake news, we have to not just show people that their beliefs are inconsistent with reality - we need to provide them with another, emotionally acceptable reality that is closer to the truth.

I Pledge Allegiance...

Last week,  I heard a segment on NPR about patriotic rituals such as saying the Pledge of Allegiance or standing for the National Anthem.  One statement by a young person was hard for me.  She said she could not recite the Pledge of Allegiance because saying that our republic has "liberty and justice for all" is false.  I get that.  And I certainly respect anyone's right to participate or not participate in saying the Pledge.  What bothers me is that "the republic, for which it stands" is an idea and that idea really does mean liberty and justice for all.  We have never had - and no real nation ever will have - perfect liberty and justice. What we pledge allegiance to is the idea that such a nation can exist.  I bet that 150+ years ago when Abraham Lincoln made his long-remembered remarks at Gettysburg, he knew well that this idea would never be perfectly realized.  He knew that what he himself had done to preserve it was not perfect.  But he really did believe in the idea.  This idea is the source of everything that has ever been good about the United States and everything that will ever be good about us, our children or our children's children.  We cannot abandon this idea because we have not lived up to it - even collectively.  Even if we see endemic and systemic injustice and prejudice, we have to see that as not who we are.  And we need our children to see that.  We don't need to make them say the Pledge or even stand when others do, but we do need them to have faith that this idea really can "long endure" and that there really can be "a new birth of freedom" in the United States.  

When someone who works for you recommends a book, read it!

People who work for you have a great perspective on what you need to learn.  Here are some great examples:

Death by Meeting - in which I learned that my team meetings were, ...um, "suboptimal."   Thanks, Bob!

The Goal - in which I learned that there was a better way to think about process optimization.  Thanks, Kevin!

The Phoenix Project - in which I am learning that I did not fully understand the consequences of the previous book.  Thanks, Scott!

Fully solving problems

Bryan Pendleton's great post, "Anatomy of a bug fix" suggests some basic principles that apply to all kinds of problem resolution.  The attributes that he calls out as separating "great developers" from the not-so-great apply in lots of other contexts, distinguishing the people who you really want to have on your team from those who you can't really count on.   Bryan's conclusion:

I often say that one of the differences between a good software engineer and a great one is in how they handle bug fixes. A good engineer will fix a bug, but they won't go that extra mile:

• They won't narrow the reproduction script to the minimal case
• They won't invest the time to clearly and crisply state the code flaw
• They won't widen the bug, looking for other symptoms that the bug might have caused, and other code paths that might arrive at the problematic code
• They won't search the problem database, looking for bug reports with different symptoms, but the same underlying cause.

The scenario above applies whenever there is a problem to be resolved.  I once led a great team responsible for resolving operational problems at a bank.  The "great ones" on that team always performed analogs of all of the things Bryan mentions above.  They always got to a very precise problem statement and recipe to reproduce and (sometimes painfully) a really exhaustive explication of impacts (what Bryan calls "widening the bug") as well as understanding of the relation between the current problem and any others that may have been related.

I have seen this separation of talent in lots of business domains - operations, engineering, finance, marketing - even business strategy and development.  The great ones don't stop until they have a really satisfying understanding of exactly what an anomaly is telling them.  The not-so-great are happy just seeing it go away.

The difference is not really "dedication" or "diligence" per se - i.e.,  its not just that the great ones "do all the work" while the not-so-great are lazier.  The great ones are driven by the desire to understand and to avoid needless rework later.  They tend to be less "interrupt-driven" and may actually appear to be less responsive or "dedicated" in some cases.  They solve problems all the way because they can't stop thinking about them until they have really mastered them.  I always look for this quality when I am hiring people. 

Core Leadership

We get the word "integrity" from the same root that gives us "integer" or "whole number."  To have integrity is first and foremost to be one thing.  Kant built his entire theory of knowledge on the premise that experience has to make sense - the world has to be one thing in this sense.   We have to be able to say "I think..." before every perception that we have about the world.

The core of all effective leadership really comes down to this.  It all has to make sense.  Everyone has to be able to start with "I think...".  Not "Mr. X says..."  Not "policy is.."  Not "I was told..." but "I think..."

For this to work, leaders have to be firmly grounded in a shared vision and they have to be committed to maintaining integrity in the sense above.  Values, principles, objectives, strategies, communications, performance evaluations, policies, processes, commitments all have to be constantly integrated.  Leaders who force themselves to be able to say "I think..." before a comprehensive view of all of these things can lead from the core.  Just as it is painful to do the exercises to strengthen your physical core, so it can be painful to maintain core leadership strength in this sense.  It is very easy to get "out of shape" by neglecting core values, objectives, strategy and execution alignment.  But without a strong core, none of the most important leadership attributes - authenticity, inspiration, strategic vision, followership, transformational impact - are possible.

Leaders who "skip the abs work" can get some things done and, depending on their good fortune and / or cleverness, some achieve material success.  But no one remembers them.  No great change is ever led by them.  No great leaders are ever developed by them.  Leading durable transformational change and developing great leaders requires core strength.

So how do you develop core strength?  A great mentor and an already established values-based vision and strategy can help get you started, but you always end up having to do the work to build your own core yourself.  Here are some little exercises that can help.  There is nothing particularly deep here and there are lots of variations on these practices.  The point is to regularly and critically focus on core integrity.

Look-back sit-ups. Starting once a week and working up to once a day, look back on all of the decisions, communications and interactions that you had and explain how it is possible that one person did all of these things.  I guarantee that if you are really observant and critical, you will find lots of little inconsistencies - things that in retrospect you can't say, "I think..." in front of.  For each of these, you have two choices: either come up with an alternative course of action that, had you done it, would have made sense; or modify whatever aspects of your vision, strategy or values it is inconsistent with (or more precisely, resolve yourself to conceive and align the necessary changes with your team, your peers and your leadership).  Done honestly, this is painful.  Think of each example as a little integrity sit-up.  Here are a couple of concrete examples.

1. Suppose that last week you negotiated an extension to a service contract. In exchange for a healthy rate reduction, you doubled the term length and added minimums to the contract. This will help achieve your annual opex reduction goal; but your agreed upon strategy is to ensure supplier flexibility and aggressively manage demand in the area covered by the contract. Your decision basically said near-term opex reduction was more important than flexibility or demand management. Either your strategy was wrong or your decision was wrong. To be one person, you need to either acknowledge the mistake or harmonize the decision with the strategy.
2. Last week you agreed with your leader and peers in a semi-annual performance ratings alignment meeting that one of your direct reports was not fully meeting expectations in some key areas.  You agreed to deliver the "needs improvement" message in these areas in his performance appraisal and to adjust his overall rating downward.  You did change the rating and some of the verbiage in the assessment; but when you delivered the review and he challenged the overall rating, you were swayed by his arguments and in the end you admitted that you had been told to adjust the rating downward.   Here either you failed to consider everything when agreeing to the rating adjustment or you were overly influenced by the feedback.

In some cases, the look-back exercise can and should lead you to take some remediating actions; but that is not the point of the exercise.  The point is to do a little "root cause analysis" of what caused the integrity breakdown.  In the first example, it may have been extreme near-term financial pressure causing things to get out of focus, or possibly just lack of clarity in the relative importance of the different factors in the strategy.  In the second example, the feedback may have pushed some "hot buttons" causing you to temporarily lose some core strength.  The key is to face these integrity gaps directly and honestly by yourself.  First think clearly and honestly about what went wrong and why.  Then think about how to "fix things." 

Virtual 360 crunchies.  Again starting once a week and working up to daily, imagine you are specific person on your team, in your company or a partner (alternate among randomly chosen people from these groups) and respond to the question, "What is most important to X?"  where X is you.  Don't just repeat goals or big initiative names or repeat your own communications.  Actually try to imagine what it would be like being the selected person and what they really think is important to you and how that relates to what they do on a day to day basis.  Think about how they would say it in their own words, not yours.  If you can't do it, or what naturally comes out is far from what you see as your core,  you have two options.  Either you have a communication problem - i.e. there is no way this person can have a clear understanding of what is important to you because you have failed to communicate it - or you don't make sense from their vantage point.  In the first case, you need to work on communication and in the second, you need to patch whatever holes exist in your vision, strategy or values that make you incomprehensible to this person.  Here are some examples.

1. You have recently been promoted from a marketing leadership position to leading an entire business unit, including sales and operations.  You have communicated your growth strategy, which is heavily top-line focused.  You are also facing significant cost containment pressure.  You know that learning about operations is critical to your success and you have been asking a lot of questions of your operations leadership team, focusing on quickly identifying some areas where you can cut costs.  Your broad-scale communications have made high level references to "operations excellence" and "empowerment," but you have not provided any details on your plans for operations.  Now suppose your randomly selected person for virtual 360 is a first level manager in operations.  It is hard to imagine that she would not say that what you care about is top line revenue growth and cutting costs and what she does on a day-to-day basis is "not important to you."
2. You are a well-respected leader of a software engineering team.  There are 10 products in your portfolio.  You have developed a robust set of goals for the organization, cascaded effectively through the team.  There are 5 top-level goals, each of which has been broken down by each of the teams in your group.  You know every product and team inside out and you regularly do "deep dives" at the product level.  Your communications are detailed, often focusing on knowledge-sharing across the group and encouraging collaboration among teams.  Imagine that the virtual 360 candidate is an engineer working on one of the products.  He might say something like, "Let me go look at the goals statement.  I know the features we are working on are important because she mentioned them in the deep dive last week."

The examples above show opposite extremes.  In the first case, the leader has a big gap in communicated - possibly even conceived - vision and values.  In the second case, the team has lost sight of the forest through the trees.  Both require not just communication, but critical assessment of core leadership.  In the first case, there is a hole.  In the second case, the core appears to be disintegrating.  The basic problem is the same.  It all has to make sense to all stakeholders all the time.

I have never met a leader who did not have small or large "core integrity" problems to deal with from time to time.  The great ones recognize them quickly and get whatever help they need to build and maintain a strong core.

Scaling DevOps

In a great InfoQ talk, John Allspaw (Flickr, Etsy) presents a compelling argument for really aggressively breaking down the barriers between IT development and operations.  The talk presents lots of simple examples that anyone who has ever managed development and/or operations can relate to.  It also shows great tech leadership attitude, IMO.

One thing that Allspaw mentions off hand is that the organizational scalability of the model is still being proved out.  It is ironic that the secret behind manageable massive scalability in some of the largest web sites may be small team size.  When the whole tech team consists of 20 people, it is not hard to get them all in a room and develop and maintain a completely shared vision.

There are two questions that I have been thinking about related to this.  First, how do you scale it organizationally - i.e., how do you make it work in large organizations with multiple applications?  Secondly,  while I think the basic principles of DevOps are great, how do you manage the collision with ITIL and other traditional management processes and structures?  I find myself more than a little amused by this post that basically points out the lack of any kind of blueprint or methodology worked out beyond "break down the silos" and do the obvious in terms of config management, continuous integration and "infrastructure as software."

The scaling problem here is the same problem faced by "new management" principles such as the transition from bureaucracy to dynamic linking advocated by Steve Denning.   Part of what is needed is the management equivalent of inversion of control - some kind of "execution framework" enabling small, collaborating but decentralized teams to achieve their goals without either having a static master plan or detailed knowledge of everything everyone else is doing [1].  The other is a scalable approach to prioritization, planning and performance measurement.  Again without a static top-down process, we need a way to develop plans and measures that maintain direct and strong connection between each small team with the end customer and product. 

The second question is even harder.  Allspaw acknowledges for example that trying to throw out traditional change management altogether is a bad idea, even with the greatest, most well-coordinated team.  For some changes, you need to, as he puts it "get out the stack of ITIL books" and follow a more traditional process.  The problem here is both what to aim for and how to engineer the transformation, especially when the point of departure is a large group with traditional processes and detached, bureaucratic management.

---

[1] For another way to see the analogy here, have a look at Robert C Martin's,  Dependency Inversion Principle.  Look at the small, self-directed teams as the "lower level modules," senior management as "higher level modules" and the "abstractions" as the higher level goals and values of the enterprise.

Engineering emergent behavior

Complex adaptive systems seem to be all the rage now.  This month's Harvard Business Review has several articles on managing complexity.   It has become fashionable to use computational complexity to explain failure or lack of engineering control in diverse areas, ranging from derivatives pricing to strong AI.

All of this is fun and interesting, though not exactly new.  It was new in the 1950's when Lorenz demonstrated the impossibility of predicting the weather.  Or in the 1970's when the "chaos cabal" began developing the ideas of self-organization and emergent behavior in nonlinear dynamical systems.

So what does all of this mean to us today?  In particular, how exactly are policy-makers and leaders supposed to "embrace complexity?"  The HBR articles make some reasonable practical recommendations that I am not going to repeat.  What I am interested in thinking about here is the general question of what it means to try to engineer or control emergent behavior and how the systems that we inhabit need to change to support meaningful attempts at this.

Unexpected outcomes are to be expected.
Emergent behavior is by definition unpredictable.  Broad patterns can be anticipated and to some extent engineered; but new and different behaviors need to be understood and exploited, rather than attempting to "minimize variance" or ensure mirco-level predictability.  In a recent article on open source product evolution, Tarus Balog talks about one way to cultivate what might be called emergent value by means of what he calls the "practice effect" - release early, release often and allow the community to work with and extend the product.  What this comes down to is presenting the interacting agents that make up a complex adaptive system with opportunities for productive evolution rather than trying to push completely predetermined outcomes through the system.  Commercial companies trying to leverage open source are learning the art of how to do this.  Open source community leaders are learning lessons that are broadly applicable from this experience as well.

Extreme sensitivity to initial conditions means small changes can create big effects.
Lorenz famously named this the "butterfly effect."  This is the essence of why long-term behavior of chaotic dynamical systems is not practically predictable and it is what makes managing complex adaptive systems extremely difficult.  The best strategy for dealing with this again comes from open source - move things along in what Stefano Mazzocchi dubbed "small, reversible steps."  At the Apache Software Foundation, most projects follow a process that we call "commit, then review."  Committers make changes to the source code and then the community reviews the changes.  "Patches" (where Apache got its name) get applied visibly in small increments and the community provides feedback.  If the feedback is not good, the patches get rolled back.  Big bang commits of very large amounts of code or sweeping changes are rare.  Proceeding via small, reversible steps and observing effects while reversibility is still possible is a great strategy for dealing with complex adaptive systems when this is practical.

There is no kernel.
The traditional systems engineering model starts with a "kernel" and builds manageable complexity on top of it.  Linux and the TCP/IP protocol stack are great examples.  Starting with a stable and solid foundation, we build specialized and advanced capabilities on top.  Similarly, governments, policies and organizations can be looked at this way, with a stable, centralized core forming the basis for bureaucratically articulated extension points.  Complex adaptive systems have no kernels.  Their core dynamics are driven by networks of relationships that evolve dynamically over time.  That means that to effectively drive change in these systems, you need to focus not on top-down, centralized programs but decentralized connection-oriented interventions.  See this article for an interesting analysis of a practical example in education reform.  As a side note, check out this fascinating analysis of how TCP/IP has itself evolved (or resisted evolution).

Broad and concurrent search for solutions.
Engineering well-conditioned, deterministic systems always comes down to posing and solving optimization problems sequentially.  Public policy and strategic planning in the command and control model of corporate governance has traditionally mimicked this approach.  Form a task force or planning team to analyze a problem.  Develop a plan.  Mobilize the organization to execute.  If things go awry, revisit the plan and try something else.  In computer science terms, this could broadly be described as depth-first search.  Given the weak and only extremely local engineering control available in complex adaptive systems, this approach tends to fail miserably.  What is needed is massively concurrent, breadth-first search.  Instead of a centralized, top-down approach to engineering emergent behavior, a loosely federated approach distributed across interaction points gets to better solutions faster.  Google works this way both literally as a search technology and by some accounts as an organization as well.

Courage and humility

...my dear hope is that political leaders will have the courage and the humility as well to overcome political sensitivity and concerns and doctrines, which are perfectly legitimate, for the sake of the entire country and for the sake of the global economy.   

IMF Managing Director Christine Lagarde in a PBS News Hour interview,  on the U.S. debt ceiling crisis.

Courage and humility.  Without these qualities, leaders simply can't be counted on.  Ms. Lagarde later says,

Political courage is required. And I'm not - I'm not suggesting that it is lacking, but it has to be demonstrated in moments of crisis. I was last week in Brussels, and there was a moment of courage and solidarity amongst the European leaders, members of the eurozone area. It comes in times in crisis. And when it does, it's quite extraordinary.

Well, we are all dearly hoping to see this real soon now from our US political leaders. 

The combination of these two qualities is what makes leaders who can be counted on.  To be humble, to acknowledge mistakes, to genuinely listen, to take accountability - all of these things require courage and all are necessary to gain confidence, followership and support in difficult times.  A great leader once said, "the role of a leader is to define reality and to give hope."  In difficult times, the "defining reality" part often requires accepting responsibility, acknowledging mistakes and sometimes being clear about uncertainty and personal shortcomings.  It also means asking for help and being open and humble in accepting it.  What some leaders don't understand is that showing this kind of courage and humility in defining reality provides a solid foundation for hope and when you give people that kind of hope, amazing things can happen.  

Ms. Lagarde's gentle but firm admonishment to the US political leadership really applies to all of us.  We have all enjoyed the "exorbitant privilege" that she (quoting Giscard d'Estaing) ascribes to our currency.  That "privilege" was earned by generations of Americans who built homes, families, careers, businesses and institutions showing these key qualities of leadership.  We need to get back to that.  Now.  Not just in politics, not just in government, but all of us, every day.

Those of you who are parents, teachers or leaders, think about the example that you are setting.  Think about what behaviors you are rewarding and who you are choosing to promote.  Will your children/students/proteges respond with courage and humility or CYA, obfuscation, finger-pointing or "positioning" when real problems arise?  We can only reverse this slide into weak, self-centered, myopic leadership by calling it out when we see it and stopping the cycle of rewarding it.

Thank you, Ms. Lagarde for calling us out!

Saving the phenomenon of OSS

I just reread Dan Pink's Drive, a book that I recommend highly for anyone who leads people or is interested in how human motivation works.  The basic premise of the book is that there is a growing gap between "what science knows and what business does" to motivate people.  He calls the traditional setup that still dominates the mainstream corporate world "Motivation 2.0" and makes a compelling case that this throwback to the industrial age is in need of a major version bump - Motivation 3.0.    Drawing on a substantial body of behavioral science research, Pink demonstrates that the "If-then" reward structure that dominates how we try to motivate people today is not well-suited to the post-industrial workplace where "work" is no longer primarily made up of boring, repetitive tasks.  Instead, he argues that intrinsic rewards are far more effective in today's world.  He calls out three primary drivers of intrinsic motivation: mastery, autonomy and purpose.

What I found very interesting, and at the same time a little troubling, after reading this book, is how well it saves the phenomenon of open source contribution.  Pink mentions OSS as an example of the beginnings of  Motivation 3.0.  At least looking at my own involvement in OSS, the drivers above do a pretty good job explaining how it is that I am willing to spend so much time volunteering.  Much better than what other "anthropologists" have come up with, which generally devolves to Motivation 2.0-speak (we do it because it will enable us to make more money somehow).  Working with great developers on hard problems can provide a sense of mastery that is hard to attain otherwise.  Autonomy is also key.  We "scratch itches" that we have as - and when - we have them.  And finally, there is a sense of contributing to something larger than ourselves or our code - a purpose.

The troubling bit is that we stand at the crossroads now in open source.  We got where we are as a result of really forward-thinking community engagement - being on the vanguard of Pink's Motivation 3.0.   People are now, in large numbers, being paid to work on OSS projects and commercial software companies are "open sourcing" their products faster than established OSS communities can absorb them.   Will we be able to maintain the powerful intrinsic reward structure that has gotten us to this point and use our elevated position to drive positive change in the Motivation 2.0-dominated establishment?  Or will the "OSS phenomenon" gradually vanish as we collectively "grow up?"

With all hats that I may wear, I will be trying to lead the change toward environments that provide the kinds of intrinsic rewards that I have enjoyed in OSS and tried my best to bring to the corporate world.