tag:blogger.com,1999:blog-45286692739926731852024-03-21T17:43:58.059-07:00Walks in the neigborhoodPhil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.comBlogger29125tag:blogger.com,1999:blog-4528669273992673185.post-54636394934499637372024-01-23T09:03:00.000-08:002024-01-23T09:03:34.832-08:00How to read<p>When I was first starting to read research papers in mathematics, I got some great advice from one of my professors. He said, "Always have paper and pencil with you when you read a paper. Read a little bit and then try to write the next part yourself. Look at what is written in the paper as a sequence of hints. That's all you are going to get. You need to fill in the details yourself and if you can't do that, you have not understood the paper." Over the years, I have realized that while research mathematics is kind of an extreme case, the same actually applies to any challenging text. So here is "the method":</p><p></p><ol style="text-align: left;"><li>Read a sentence or paragraph or however much you need to get an idea.</li><li>Write or say to yourself what you think is going to come next.</li><li>Start from the beginning and read through the next chunk. Compare your continuation to what actually came next. </li><li>Go to 2</li></ol><div>You end up re-reading the whole piece many times this way. For long things, use major breaks like chapters or whatever to limit the look back.</div><div><br /></div><div>If you are trying to really learn the material, you can do the whole process repeatedly. In that case, the checking in step 3 should start to show less and less divergence, mostly just style or sequencing. You need to be careful though not to devolve into memorization. You want to actually come up with the ideas that come next, not the words.</div><div><br /></div><div>I do this kind of thing when I read hard material of any kind - not rigidly and sometimes changing chunks around. If I go slowly enough, unless the material is really over my head or I am lacking needed background or something, I always end up feeling like I have had the ideas that the author was trying to convey. That usually means that I can start to apply the ideas myself.</div><div><br /></div><p></p>Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com0tag:blogger.com,1999:blog-4528669273992673185.post-85911318083851997842024-01-14T12:26:00.000-08:002024-01-14T12:26:57.803-08:00Why I love mathematics<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBTAxCF7DuVnrGwlZCGIa06KL-JrD9y7P5q7kjqYpQNxV95c5RbYSTQri9ScPEs1aNoiAMdEOZwCg2VsUgQ69VohoC1WO3UwLkhFyH_nzioyxrmJb-TXPuq8oRMH5wgLdvqqZkykKdrCUgYa3vN8IDekwDJ2psHWuGqYvwgg0li0drz_UhNgXvYVm5M9c/s264/mandas.jpeg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img alt="Millie and Al's https://www.flickr.com/photos/27480193@N05/" border="0" data-original-height="191" data-original-width="264" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBTAxCF7DuVnrGwlZCGIa06KL-JrD9y7P5q7kjqYpQNxV95c5RbYSTQri9ScPEs1aNoiAMdEOZwCg2VsUgQ69VohoC1WO3UwLkhFyH_nzioyxrmJb-TXPuq8oRMH5wgLdvqqZkykKdrCUgYa3vN8IDekwDJ2psHWuGqYvwgg0li0drz_UhNgXvYVm5M9c/s16000/mandas.jpeg" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div>I love mathematics because it never says one thing and does something else. If it ever seems to do that, it is always because I am missing some idea. I never stay mad at mathematics.</div><div><p></p><p>I love mathematics because it is always there, waiting for me. It will always be there even if I don't jump on it <i>right now</i>. It worn't run away or turn into some not fun thing. When I go back over mathematics that I haven't looked at in a while, it's like going back to the old neighborhood and having that warm and happy feeling you get when nothing has changed. </p><p>I love mathematics because it loves me. Mathematics has infinite patience for me. I can be arbitrarily stupid for arbitrarily long. Mathematics keeps the light on for me.</p><p>I love mathematics because it surprises me and makes me think differently all the time. I feel like Aeneas in the world of mathematics, constantly meeting <i>monstra mirable dictu</i>, but without the carnage.</p></div>Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com0tag:blogger.com,1999:blog-4528669273992673185.post-12933317668258537212023-09-15T17:46:00.001-07:002023-09-17T19:39:48.305-07:00How not to get breached (too badly)<div class="separator" style="clear: both; text-align: left;">The other day, someone asked me if I had ever experienced a major security breach. They were shocked when I responded that I had not. That is because I have been a CTO for the past 18 years, including stints at some bleeding edge SaaS companies that were constantly under attack. I have managed many security incidents, but none that resulted in a major breach. While of course it is possible and even likely that dumb luck has played a role in my success, I think that the following things have certainly helped. Each of the imperatives below have been important to me. I have never achieved perfection in any of them, but I have never stopped pushing. The imperatives are written from the standpoint of a CTO; but anyone who cares about security can use them to help protect their company.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><h3>1. Know your risks and be honest and transparent about them</h3><div>You should always have a top <i>n</i> list of key risks that you are worried about, and <i>n</i> should be less than or equal to 10. You should have a weekly conversation with your Security officer that reviews each risk, mitigation plans, compensating controls and any help that the security team needs managing it. It is critical that these conversations be open, honest, comprehensive and <i>engaged</i>. If you can't understand some of the technical security content, you need to study it and keep asking questions until you understand it. You need to build a culture on your team that does not sugar coat or hide risks and your response to learning about them needs to be consistently positive, supportive and action-oriented. You need to never appear to be annoyed by risks or angry at those who report them. You also need to review critical risks at least quarterly with your partners on the executive team. These reviews need to be open, interactive, transparent and engaging. Your objective is not to show that you have things under control, to justify investment or to cover your ass. Your objective is to proactively cover <i>their ass</i>. If you do a good job clearly representing risks, mitigation plans and compensating controls, you will get funding and leadership support as a side effect. </div><div><br /><h3 style="text-align: left;">2. Focus on actual risk</h3><div>Don't let compliance, vendors, talking heads or your own cool ideas distract you from systematically reducing risk. If you do a good job identifying and managing risk, you will achieve compliance as a side effect. Attackers don't care how "buttoned up" your security program looks or how beautifully illustrated your risk registry is. What matters is where you are weak and what you have done to compensate for your weaknesses. Note that this is exactly the same thing that auditors care about. Try to put every possible cycle into things that significantly reduce actual risk. Compensating controls can be ugly and low tech, but they can <i style="font-weight: bold;">save your ass. </i>I am certain that some of the ugliest and most cumbersome shims that I have put in while working on permanent solutions have saved me and my companies from great harm. </div><div><br /><h3 style="text-align: left;">3. Pull on every thread</h3>When you have evidence of an attack or vulnerability, make sure that you investigate everything fully. Don't just celebrate having thwarted or annoyed the attackers into leaving. You need to have people whose job it is to investigate security incidents and you have to give them the time, tools and access to do their jobs. Ask probing questions and don't assume that your initial analysis of an anomaly is correct. I wrote a few years back about <a href="https://psteitz.blogspot.com/2013/11/fully-solving-problems.html" target="_blank">fully solving problems</a>. That same thinking applies here - especially the part about "widening bugs."<br /><br /><h3 style="text-align: left;">4. Make security@ a welcoming and helpful place</h3></div><div>Respond kindly and helpfully to all security-related questions or concerns from employees. Don't put the burden on the reporter to establish that an issue is worth investigating. You want them to come back, not to feel stupid or ignore things. You also want them to learn. The most important single thing that any company can do to reduce security risk is to develop or acquire high-quality security awareness training and make sure that everyone takes it. If your company produces any kind of software (including for internal use), make sure to also acquire - and require - high-quality secure application development training. It's very important that all training, consulting, reviews and standards promoted by your security team be of the highest quality and backed by friendly people who understand your business and are willing to patiently answer questions and help people understand security concepts.</div><div><br /><h3 style="text-align: left;">5. Stay current</h3>Attack types and vectors change all of the time. Many of these changes have no impact on your environment, but you need to make sure you see and assess impact of new threats as they emerge. Here again, you need to have people whose job includes monitoring security advisories and assessing their impact on you. You need to make sure that you understand clearly what the advisories and your team are telling you. That means you need to devote a significant amount of your professional development time to keeping up with the changing threat landscape. As a CTO, you have a unique vantage point that virtually nobody else in the company has. It is critical that as new threats emerge, you personally think through their implications across your technology estate. Just as you and your team need to stay current on the threat landscape, so your systems need to remain current in terms of patch levels. If you do not have fully automated, short-cycle patch deployment capability, this needs to be put in place. Even if it is in place, it needs to be actively maintained and continuously exercised and you need to eliminate the things that can't be patched.</div><div><br /><h3 style="text-align: left;">6. Do real diligence on suppliers</h3>Some companies have well-established supplier risk management capabilities. Even in those companies, however, sometimes the level of technical security diligence is not where it needs to be. You need to step back from the questionnaires and boilerplate RFP responses and think clearly about where the material risks are with suppliers and dive deeply into what controls you and they have in place to mitigate them. Again, your vantage point as CTO is critical here. You can't count on somebody else's checklist to see the full picture. You need to direct your best spidey-sense toward where vendors may be weak and adaptively probe to make sure that you have discovered all of the risks. Be especially careful with low-cost vendors. Finally, make sure that you regularly review vendors' security posture. If a vendor is acquired or the product or service you are using is slated for end of life, that should trigger a special review and contingency plan.</div><div><br /><h3 style="text-align: left;">7. Use architecture as a weapon</h3></div><div>Everyone knows that it's way less effective to try to add security to a naively implemented system. On the opposite side of this are systems that deliver security "natively," exposing services and features in a way that is hostile to attackers. Zero trust and end-to-end encryption are examples of this. They are baked into the architecture and inherently hostile to attackers. Really effective, dynamic and fast credential management and revocation is another great weapon. The same service that allows applications to quickly get keys and secure communications can help you quickly respond to an attack or vulnerability. Constantly push for the architecture win-wins that make your systems both more robust and more secure.</div><div><br /><h3 style="text-align: left;"><b>8. Constantly question privilege</b></h3><div>Make least privilege an organizing principle. Start with your own. Do you really need production access? You are a fat target. Don't have anything valuable on your laptop and don't let your account be a valuable attack vector. The same applies to everyone in your company and every process running in your environment. The less they can do, the less valuable they are as attack vectors. Constantly ask why individuals or processes need the access that they have and constantly prune. Just like patching, if your environment does not have the automation, test environments or other infrastructure required to stop individuals or batch jobs from having to have weakly limited production access, you need to fix that. If offboarding is not bulletproof when it comes to system access, make it so. Now. This sometimes looks daunting or even impossible to fix. Trust me, it never is.</div><br /><h3 style="text-align: left;">9. Don't acquire, decrypt, transmit or store data that you don't need</h3></div><div>I often make the joke that the most secure and highly available system is the null system - the system that does nothing. To deliver business value, systems need to access and process data. But many can fully achieve their objectives with a much lighter touch on data. Here I am reminded of an army adage about conserving energy quoted by Colin Powell in his <a href="https://www.npr.org/2012/05/22/153296714/it-worked-for-me-life-lessons-from-colin-powell" target="_blank">awesome leadership book</a>: "Don’t run if you can walk; don’t stand up if you can sit down; don’t sit down if you can lie down; and don’t stay awake if you can go to sleep." Here is my version for data processing: Don't receive if you can do without; don't decrypt if you don't need to see; don't store if you don't need to persist; don't transmit if can omit. The wonderful thing about modern distributed architectures is that they don't force you to create massive centralized honey pots of raw data. So don't. </div><div><br /></div><div><h3 style="text-align: left;">10. Don't assume that any zone, subnet, vm or other subsystem can be hardened</h3></div><div>I saved the best one for last. It never ceases to amaze me how coming on 40 years now after the famous Gage/McNeally pronouncement that "the network is the computer," people keep thinking that they can somehow wall off little islands of safety and security. You need to disabuse yourself and your team of that archaic fantasy. You need to constantly assume that the bad guys are inside "your" network and focus on limiting what they can do once inside. Like multi-factor authentication, strong crypto keys and end-to-end encryption, network controls are good architectural weapons, but they are just one weapon. Like the others, they make your estate a more hostile place for attackers and slow them down; but they need the others behind them.</div></div><div><br /></div><div>Some of these imperatives might seem to limit or constrain what you can do with your products or how fast you can move. They absolutely don't have to. For example, number 9 does not say you can't store any data. It just says you should limit what you store. And guess what, if you do the hard thinking in system design to limit things to what you actually need, you can move faster and <i>do more. </i>Once you get hard core about 8, you will also pick up speed and value because the only way to do it is to automate. The win-wins in 7 really are all over the place and once you get that thinking baked in, first in your own mind and then in your team and entire company, you will get benefits way beyond just being a harder target.</div>Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com0tag:blogger.com,1999:blog-4528669273992673185.post-67252601775664673012020-11-28T16:59:00.003-08:002021-09-14T14:37:07.130-07:00Governance and Humility<div dir="ltr" style="text-align: left;" trbidi="on">I recently listened to Sen. Ben Sasse read his own challenging book, <a href="https://us.macmillan.com/books/9781250193681" target="_blank">Them</a>. In a discussion of the origins of American democracy, he mentions the following quote from James Madison:<br />
<blockquote class="tr_bq">
If men were angels, no government would be necessary. If angels were to govern men, neither external nor internal controls on government would be necessary. In framing a government which is to be administered by men over men, the great difficulty lies in this: you must first enable the government to control the governed; and in the next place oblige it to control itself.</blockquote>
The core idea that the human fallibilities that make governance necessary can exist in the governing as well as the governed applies to a lot of other settings too.<br /><br />
In corporate settings, corporate boards provide oversight to executives who lead teams. Great teams require little directive governance. Great leaders provide vision and perspective, clear away obstacles and make sure that the right conversations happen at the right times with the right people involved. Key conversations involve them personally and their own ideas and behaviors are improved as a result. Sometimes directive leadership is needed or decisions have to be made that can't practically involve the team, but when that happens, great leaders listen carefully and actively solicit opinions of others. And when they do stupid things, healthy organizations gently but firmly move them back onto a better path.<br />
<br />
The basic idea that we are better off when we allow and encourage others to help us overcome our individual fallibilities is common to all of these settings and is at the core of Madison's vision, which Sasse is rightly challenging us to re-embrace.<br />
<br />
To do that, we have to restore the humility and consideration for others that in our better moments has characterized American culture and political dialogue. Sasse illustrates this with another powerful quote, this one from Judge Learned Hand, given in a speech called "The Spirit of Liberty" in 1944:<br />
<blockquote class="tr_bq">
What do we mean when we say that first of all we seek liberty? I often wonder whether we do not rest our hopes too much upon constitutions, upon laws and upon courts. These are false hopes; believe me, these are false hopes. Liberty lies in the hearts of men and women; when it dies there, no constitution, no law, no court can save it; no constitution, no law, no court can even do much to help it...The spirit of liberty is the spirit which is not too sure that it is right; the spirit of liberty is the spirit which seeks to understand the minds of other men and women; the spirit of liberty is the spirit which weighs their interests alongside its own without bias; the spirit of liberty remembers that not even a sparrow falls to earth unheeded; the spirit of liberty is the spirit of Him who, near two thousand years ago, taught mankind that lesson it has never learned, but has never quite forgotten; that there may be a kingdom where the least shall be heard and considered side by side with the greatest.</blockquote>
Hand and Sasse both ground their commitment to humility and service in their religious views, but the idea of liberty, which both see as core to the idea of America, includes a deep commitment to religious freedom and tolerance for religious views different from their own.<br />
<br />
Both also understand that governance cannot bring about the change in perspective needed to right sick organizations or societies. That change has to happen in the hearts and minds of people. Political leaders like Sasse can make a difference by setting good examples and I applaud him for writing the book and regularly speaking on what we need to do as a nation to heal; but to "make America great again" we have to all individually become humble again and start really listening to one another.<br />
<br />
And that greatness will not be the greatness of a nation with borders and parochial interests, but the greatness of an idea - the idea that all human beings are created equal with inalienable rights. That idea extends beyond any national borders and whatever greatness America ever had lies in our commitment to that idea.<br />
<div>
<br /></div>
</div>
Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.comtag:blogger.com,1999:blog-4528669273992673185.post-5293668689150668552020-10-11T16:22:00.002-07:002020-10-11T16:22:29.316-07:00A little dusting vs. getting buried - why variance matters in ranking risks<p><span style="font-family: arial;">Some simple risk management frameworks recommend stack ranking risks based on expected loss should they materialize. The simple formula $L(r) = P(r) E(C(r))$ is sometimes used, where $P(r)$ is the probability that a risk event of type $r$ materializes and $E(C(r))$ is an estimate of the cost associated with such an event. So for example, if the cost of a data breach at a company is estimated to be $\$1m$ and the estimated probability of this happening is $.01$, then $L$ is in this case estimated to be $\$10k$. Given estimates of $P(r)$ and $E(C(r))$ for the different risks faced by a company or individual, a natural way to rank them is by $L$ values. The problem with this approach is that it fails to take into account the variance in $C$ and it also puts too much emphasis on the point estimate of $L$.</span></p><p><span style="font-family: arial;">Consider the following example. Suppose that a company has one risk $r$ with $P(r) = 0.01$ and $E(C(r)) = \$1m$ and another risk $s$ with $P(s) = 0.01$ and $E(C(s))= \$800k$. Suppose further that a realized loss of more than $\$1.2m$ is catastrophic for the company, meaning losses of this amount or greater cannot be absorbed. If $C$ has no variance for both $r$ and $s$, neither is a catastrophic threat and it makes sense to prioritize mitigating $r$ over $s$.</span></p><p><span style="font-family: arial;">The problem is that in real world situations, $C$ always has variance and when what you really care about is guarding against large losses, that variance changes the equation. In the example above, suppose that $C(r)$ is normally distributed with mean $1m$ and standard deviation $100k$ but $C(s)$ has a uniform distribution with minimum $0$ and maximum $1.6m$. This means that the values of $C(r)$ should follow a bell-shaped curve clustering around the mean $E(C(r)) = 1m$ but the values of $C(s)$ are randomly spread across the interval $[0,1.6m]$ with no range of values any more likely than any other.</span></p><p><span style="font-family: arial;">The probability that $r$ results in a catastrophic loss under these assumptions is $P(r) P(C(r) > 1.2m) = .01 \times P(N(1, 0.1) > 1.2)$ where $N(\cdot,\cdot)$ is the normal distribution. Now $P(N(1, 0.1) > 1.2)$ is approximately $0.023$, so that means the probability that $r$ unmitigated will result in a catastrophic loss is approximately $0.01 \times 0.023 = 0.00023$.</span></p><p><span style="font-family: arial;">For $s$, $P(C(s) > 1.2) = (1.6 - 1.2) / 1.6 = 0.25$, so the probability that $s$ results in a catastrophic loss is $0.01 \times 0.25 = 0.0025$, which is <i>more than ten times higher</i>. So if what the company wants to do is to minimize the probability of catastrophic loss, $s$ is actually a much more important risk to mitigate. </span></p><p><span style="font-family: arial;">The practical problem is that in general the distributions of the cost functions are unknown as are their expected values. But just asking the question of how bad a loss can be and how likely a tail event is can lead to better risk prioritization decisions. The example above is mathematically extreme. The uniform distribution is just one big tail. But it does illustrate what happens when there is a lot of probability mass in the bad part of the cost distribution.</span></p><p><span style="font-family: arial;">I recently ran across <a href="https://www.collaborativefund.com/blog/the-three-sides-of-risk/" target="_blank">this tragic, but beautiful post</a> that illustrates the main point here very well. The author puts it simply,</span></p><ul style="text-align: left;"><p><span style="font-family: arial;">"There are three distinct sides of risk </span></p><blockquote><li><span style="font-family: arial;">The odds you will get hit</span></li><li><span style="font-family: arial;">The average consequences of getting hit</span></li><li><span style="font-family: arial;">The tail-end consequences of getting hit" </span></li></blockquote></ul><span style="font-family: arial;">The odds of getting hit are $P(r)$, the average consequences are $E(C(r))$ and the tail-end consequences of getting hit are the upper tail of the distribution of $C(r)$.</span>Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com0tag:blogger.com,1999:blog-4528669273992673185.post-58761224286868474282020-08-31T16:32:00.002-07:002020-09-05T09:59:17.201-07:00Five things I look for in Product Managers<p class="MsoNormal" style="font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span lang="EN" style="font-family: arial;">SaaS companies live or die by the quality of their product managers. Great ones really power the future of their companies. Not-so-great ones waste precious time and investment resources.</span></p><p class="MsoNormal" style="font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span lang="EN" style="font-family: arial;"><br /></span></p><p class="MsoNormal" style="font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span lang="EN" style="font-family: arial;">It’s very hard to tell from a resume, however, who is going to be a great PM. Great product managers can work for not-so-great companies and vice versa. And what people with this job title actually do varies wildly from company to company.<o:p></o:p></span></p><p class="MsoNormal" style="font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span lang="EN" style="font-family: arial;"> </span></p><p class="MsoNormal" style="font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"><span style="font-family: arial;">I have been involved in hiring product managers for the past 10+ years. I have hired some great ones and I have made some mistakes. Over the years, I have learned that there five key things that really make a difference:</span><span face=""><o:p></o:p></span></span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"> </span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><b><span face="" lang="EN">1. Ability to think like someone else<o:p></o:p></span></b></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"> </span></p><p class="MsoNormal" style="font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"><span style="font-family: arial;">Product managers are trained to identify user types or personas. These include information about users’ day-to-day experience, mindset and needs. It’s important to be able to do this, but to be a really great PM, you have to be able to take the next step and actually <i>think</i> like customers, and internal stakeholders as well.</span><span face=""><o:p></o:p></span></span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"> </span></p><p class="MsoNormal" style="font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"><span style="font-family: arial;">I want to see that they can actually put themselves in the position of a user, of a salesperson, a support agent, or other product stakeholder in their company. That they can get into their heads, almost as if they were actors in a play.</span><span face=""><o:p></o:p></span></span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"> </span></p><p class="MsoNormal" style="font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"><span style="font-family: arial;">The best product managers have a tremendous amount of empathy for other people. They can forget about what they think they know about how to use, sell, or support their product and put themselves in the mind of someone who lacks all of that context and has a very different set of day-to-day concerns than they do.</span><span face=""><o:p></o:p></span></span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"> </span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN">During an interview, I test for this by asking the candidate about how users and other stakeholders thought about the products and product development ideas that they worked on. I ask them what surprised them and what they learned from it. It’s not hard to tell who has this ability because those who do exercise it regularly and those who don’t find it hard to come up with examples.<o:p></o:p></span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"> </span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><b><span face="" lang="EN">2. Determination<o:p></o:p></span></b></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"> </span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN">The most effective product managers are creative and innovative thinkers with positive mindsets. They do not give up. They see the positive outcome as expected and they are surprised by failure. And they see every partial failure as success in need of small refinements - even when there are gaping holes to fill.<o:p></o:p></span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"> </span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN">A lot of today’s most popular products should have died a dozen deaths, but somehow they made it. They made it because their PMs would not give up.<o:p></o:p></span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"> </span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN">Often in an interview, I will give the candidate an example problem of a now-famous payments product that had so many obstacles to overcome in its original conception that people laughed at the idea.<o:p></o:p></span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"> </span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN">I ask the candidate to imagine what it might have been like in the early days of that product, and what problems may have arisen and how they would have tackled them. Their answers show me both their analytical ability and their ability to suspend disbelief and persevere.<o:p></o:p></span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"> </span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><b><span face="" lang="EN">3. Cultivator of great ideas<o:p></o:p></span></b></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"> </span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN">In most of my product manager interviews, I ask the candidate to tell me some of the top ideas in which they’ve been involved.<o:p></o:p></span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"> </span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN">What I’m looking for here isn’t a detailed explanation of what the person has come up with themselves. Instead, I want to hear about the ideas and products that they helped develop. The best product managers have an uncanny ability to be part of conversations that lead to great ideas. To use a sports analogy, I am less interested in how many points they score individually. Instead, I’m more interested in how much their team scores when they are in the game.<o:p></o:p></span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"> </span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN">To test for this, I ask them to describe the process for developing the top ideas that they mention. I probe to learn about how others were involved and how they helped make sure diverse perspectives and expertise were brought in. If all I hear is “I” and simple logic (talking about done ideas), I know I do not have a great PM in front of me. Similarly, if all I hear is “we” and I don’t pick up any original contribution from the person I am talking to, I know I am not dealing with a potential game changer.<o:p></o:p></span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"> </span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><b><span face="" lang="EN">4. Organizing force<o:p></o:p></span></b></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"> </span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN">Great product managers are organizing forces. Note that this is not the same as "being organized." It means that they help define problems in a way that they can be solved with simple solutions and they help develop designs, requirements, and plans so that the teams they are part of can <i>be</i> organized. How product development work is organized and how team members understand the structure of the work is critical. Great PMs can make a night and day difference to the performance of their teams just due to the fact that they help them structure their work better.<o:p></o:p></span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"> </span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN">To test this in an interview, I ask candidates how they would handle a complex product development challenge from my past experience. I ask them to come up with options for how to structure and sequence work, what information they would need to choose among them, and how different stakeholders would understand the approaches.<o:p></o:p></span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"> </span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><b><span face="" lang="EN">5. Inspiring<o:p></o:p></span></b></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"> </span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN">One of the biggest challenges faced by product managers is how to inspire people who aren’t their direct reports to do things in support of product development efforts. Product managers work across organizations, from engineering to marketing, support, finance, and beyond. It is imperative that they are able to connect and motivate people cross-functionally.<o:p></o:p></span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"> </span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN">In an interview, I will ask them to share past challenges they had with people who had different priorities, and how they were able to motivate the team to get things done. I look for the signs of earned authority - the ability to build followership and commitment to shared goals.<o:p></o:p></span></p><p class="MsoNormal" style="font-family: arial, sans-serif; font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="" lang="EN"><br /></span></p><p class="MsoNormal" style="font-size: 11pt; line-height: 16.8667px; margin: 0in;"><span face="">Finding all of these qualities in one person is rare, but I have found that if I probe into each of them I can get a nice well-rounded picture of where a product manager is in their development. And those who are strong in each of them are a joy to work with and great assets to their teams.</span></p>Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com1tag:blogger.com,1999:blog-4528669273992673185.post-44457011493367945272020-03-29T18:56:00.001-07:002020-03-29T18:57:37.568-07:00Put on your own mask first<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; color: rgba(0, 0, 0, 0.9); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; cursor: text; font-family: -apple-system, system-ui, system-ui, "Segoe UI", Roboto, "Helvetica Neue", "Fira Sans", Ubuntu, Oxygen, "Oxygen Sans", Cantarell, "Droid Sans", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Lucida Grande", Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.5; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
Difficult times are the ultimate test of leadership. These are the times when lifelong bonds can be established or connections can be lost. Deep human connections are all ultimately based on faith. Faith that through them strength, security and growth will come. Leaders need to ground that faith in confidence, consistency and an unfailing positive attitude. </div>
<div style="background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; color: rgba(0, 0, 0, 0.9); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; cursor: text; font-family: -apple-system, system-ui, system-ui, "Segoe UI", Roboto, "Helvetica Neue", "Fira Sans", Ubuntu, Oxygen, "Oxygen Sans", Cantarell, "Droid Sans", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Lucida Grande", Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.5; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
At this moment, many leaders have been put in the impossible position of having to show confidence when the foundations of their businesses, their markets and the global economy are being challenged. It doesn’t work to try to hide things or to spin things or to simply put on a happy face. What people need from their leaders today is <b>authentic</b> confidence. That is what leaders have to reach deep down to find right now. The great ones always find it and their connections are strengthened as a result.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCNWU0e72bWmJI4XRSC-c4ny5FI4peH0uf4UFS2lOCIdFLebxjycGw7rdmTzXjEWuq2bY5UduVpxmCLs8Oy-gOLsoVDhWzPotkgYxodnvBavc6mBOv0gvVI_rNz6vR3Wib6HNV-Ryllaw/s1600/burghers.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="530" data-original-width="601" height="282" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCNWU0e72bWmJI4XRSC-c4ny5FI4peH0uf4UFS2lOCIdFLebxjycGw7rdmTzXjEWuq2bY5UduVpxmCLs8Oy-gOLsoVDhWzPotkgYxodnvBavc6mBOv0gvVI_rNz6vR3Wib6HNV-Ryllaw/s320/burghers.jpg" width="320" /></a></div>
<div style="background: rgb(255, 255, 255); border: 0px; box-sizing: inherit; color: rgba(0, 0, 0, 0.9); counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; cursor: text; font-family: -apple-system, system-ui, system-ui, "Segoe UI", Roboto, "Helvetica Neue", "Fira Sans", Ubuntu, Oxygen, "Oxygen Sans", Cantarell, "Droid Sans", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Lucida Grande", Helvetica, Arial, sans-serif; font-size: 18px; line-height: 1.5; padding: 0px; vertical-align: baseline; white-space: pre-wrap;">
<br /></div>
</div>
Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com0tag:blogger.com,1999:blog-4528669273992673185.post-52634963645189669862018-11-17T08:11:00.000-08:002018-11-17T08:33:21.797-08:00Hipparchus 1.4 Released<div dir="ltr" style="text-align: left;" trbidi="on">
We just cut another <a href="https://www.hipparchus.org/" target="_blank">Hipparchus</a> release, which should now be available on Maven Central. Hipparchus is a library of lightweight, self-contained mathematics and statistics components addressing the most common problems not available in the Java programming language. <br />
<br />
New features and enhancements include:<br />
<br />
<ul style="text-align: left;">
<li>Bilinear interpolation for 2D grids</li>
<li>Field version of sinCos</li>
<li>Support for complex ordinary differential equations (both primary and secondary equations)</li>
<li>Accessors for state transition matrix, Jacobian matrix H</li>
<li>Innovation covariance matrix and Kalman gain in Kalman filters</li>
<li>Three-dimensional field arrays</li>
<li>Several fixes in the Euclidean 3D and spherical 2D partitioning </li>
</ul>
<br />
<div>
The release is available on the<a href="https://www.hipparchus.org/downloads.html" target="_blank"> Hipparchus download page</a>. The release jars are available from maven central with groupId <b>org.hipparchus</b> and artifactId by library component. For example, the ODE jar is here:</div>
<div>
<div>
<br /></div>
<br />
<div class="section" style="background-color: white; color: #333333; font-family: "helvetica neue", helvetica, arial, sans-serif; font-size: 14px;">
<div class="source">
<pre class="xml" style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgba(0, 0, 0, 0.15); font-family: monaco, menlo, consolas, "courier new", monospace; font-size: 13px; line-height: 20px; margin-bottom: 10px; overflow-wrap: break-word; padding: 9.5px; white-space: pre-wrap; word-break: break-all;"><span class="tag"><<span class="title" style="color: black; font-weight: bold;">dependency</span>></span>
<span class="tag"><<span class="title" style="color: black; font-weight: bold;">groupId</span>></span>org.hipparchus<span class="tag"></<span class="title" style="color: black; font-weight: bold;">groupId</span>></span>
<span class="tag"><<span class="title" style="color: black; font-weight: bold;">artifactId</span>></span>hipparchus-ode<span class="tag"></<span class="title" style="color: black; font-weight: bold;">artifactId</span>></span>
<span class="tag"><<span class="title" style="color: black; font-weight: bold;">version</span>></span>1.4<span class="tag"></<span class="title" style="color: black; font-weight: bold;">version</span>></span>
<span class="tag"><<span class="title" style="color: black; font-weight: bold;">type</span>></span>pom<span class="tag"></<span class="title" style="color: black; font-weight: bold;">type</span>></span>
<span class="tag"></<span class="title" style="color: black; font-weight: bold;">dependency</span>></span></pre>
<div>
<span class="tag"><br /></span></div>
<div>
<span class="tag">Bug reports, patches, suggestions for improvement are always welcome. Subscribe to our <a href="https://www.hipparchus.org/mail-lists.html" target="_blank">mailing lists </a>and dive into our <a href="https://github.com/Hipparchus-Math/hipparchus" target="_blank">source code</a> to get involved.</span></div>
</div>
</div>
<div>
<div class="section" style="background-color: white; color: #333333; font-family: "helvetica neue", helvetica, arial, sans-serif; font-size: 14px;">
</div>
</div>
</div>
</div>
Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com0tag:blogger.com,1999:blog-4528669273992673185.post-54573799723840679332018-06-11T17:17:00.000-07:002018-06-11T17:17:18.420-07:00Read the proofs!<div dir="ltr" style="text-align: left;" trbidi="on">
Now and then one of my friends asks me for recommendations for books to use to re-engage with mathematics. I am always happy to help and I try to come up with recommendations that will be "goldilocks hard" but also interesting and fun for the person who is asking. I also selfishly think a little about what it will be like for me to talk about the book's content with my friend.<br />
<br />
I look for books that motivate the material well, but also actually prove things and I always make a point of telling my friends that they have to read and understand the proofs, ideally well enough that they can reproduce them. Today I asked myself why I always make such a big deal of this. Here's what I came up with:<br />
<br />
<ol style="text-align: left;">
<li>Mathematics is a progression of ideas, expressed in arguments that prove results. The results are often useful in solving problems, but the arguments are really where the mathematics is. If you understand the arguments, you will be able to do problems - often in many different ways.</li>
<li>Proofs show exactly what is needed to get results. They show you, therefore, exactly where the limits are in applying the results. When you understand a proof, you know exactly what you have to work with.</li>
<li>Proofs are rigorous and often hard. If your goal is to be able to apply mathematical concepts and hone your mathematical intuition, working through hard proofs is much better training than just practicing algorithms. </li>
<li>What you can prove in mathematics is what you know. Results that you can apply, but don't understand the proofs for, are what Plato would call "true opinions," not knowledge. Not grounded in proof, your understanding of algorithms and their valid application might "fly away" at any time (See Plato's <i>Meno</i> 97-98, for a delightful account of this).</li>
<li>Go back to 1. The beautiful thing about studying mathematics "for fun" is that you get to actually do it. Under time / grade / getting applications done pressure, the mathematics ends up taking a back seat to the applications. It's like fast-forwarding the show and watching the commercials. Coming back now, you can actually watch the show.</li>
</ol>
<div>
So read the proofs!</div>
<div>
<br /></div>
</div>
Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com0tag:blogger.com,1999:blog-4528669273992673185.post-16913445284504538442018-05-08T17:07:00.000-07:002018-05-08T17:07:55.951-07:00Hipparchus 1.3 Released<div dir="ltr" style="text-align: left;" trbidi="on">
Today we cut another release of our little Java mathematics library. <a href="https://www.hipparchus.org/index.html" target="_blank">Hipparchus</a> is based on a fork of Apache Commons Math.<br />
<br />
The 1.3 release includes some nice new features - most notably a redesigned Kalman filter implementation capable of handling nonlinear processes, along with quite a few additional enhancements and bug fixes. <br />
<br />
<ul style="text-align: left;">
<li>Support was added for complex eigenvalue decomposition.</li>
<li>A solver for continuous time algebraic Riccati equations was added.</li>
<li>Secondary equations can now update the derivatives of the primary equation in ODE.</li>
<li>A matrix decomposer was added to allow configuration of decomposition thresholds independently of the matrix.</li>
<li>FastMath was extended to cover the new Java 9 methods so it can still serve as a drop-in replacement for java.lang.Math.</li>
</ul>
<br />
<ul style="text-align: left;"></ul>
<br />
<div>
I had fun fixing some bugs in the stat package and reviewing the Kalman filter design and implementation. One of the bug fixes, for the Mann-Whitney test implementation, included implementation of exact computation of the sampling distribution of the Mann-Whitney U statistic. The algorithm for this is nice and simple, but it does not handle the case where there are ties in the data. I have started working on generalizing the algorithm (to some extent, the definition, actually) to support this. Comments, patches welcome!</div>
<div>
<br /></div>
<div>
The release is available on the<a href="https://www.hipparchus.org/downloads.html" target="_blank"> Hipparchus download page</a>. The release jars are available from maven central with these coordinates:</div>
<div>
<br /></div>
<br />
<div class="section" style="background-color: white; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px;">
<div class="source">
<pre class="xml" style="background-color: whitesmoke; border-radius: 4px; border: 1px solid rgba(0, 0, 0, 0.15); font-family: Monaco, Menlo, Consolas, "Courier New", monospace; font-size: 13px; line-height: 20px; margin-bottom: 10px; padding: 9.5px; white-space: pre-wrap; word-break: break-all; word-wrap: break-word;"><span class="tag"><<span class="title" style="color: black; font-weight: bold;">dependency</span>></span>
<span class="tag"><<span class="title" style="color: black; font-weight: bold;">groupId</span>></span>org.hipparchus<span class="tag"></<span class="title" style="color: black; font-weight: bold;">groupId</span>></span>
<span class="tag"><<span class="title" style="color: black; font-weight: bold;">artifactId</span>></span>hipparchus-aggregator<span class="tag"></<span class="title" style="color: black; font-weight: bold;">artifactId</span>></span>
<span class="tag"><<span class="title" style="color: black; font-weight: bold;">version</span>></span>1.3<span class="tag"></<span class="title" style="color: black; font-weight: bold;">version</span>></span>
<span class="tag"><<span class="title" style="color: black; font-weight: bold;">type</span>></span>pom<span class="tag"></<span class="title" style="color: black; font-weight: bold;">type</span>></span>
<span class="tag"></<span class="title" style="color: black; font-weight: bold;">dependency</span>></span></pre>
</div>
</div>
<div>
<div class="section" style="background-color: white; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px;">
</div>
</div>
<br />
<div>
<br /></div>
<div>
<br /></div>
</div>
Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com0tag:blogger.com,1999:blog-4528669273992673185.post-50210716809932550382017-10-22T10:31:00.000-07:002017-10-22T10:31:04.503-07:00Selling the truth<div dir="ltr" style="text-align: left;" trbidi="on">
This month's <i><a href="https://www.significancemagazine.com/" target="_blank">Significance</a></i> magazine includes a jarring <a href="http://onlinelibrary.wiley.com/doi/10.1111/j.1740-9713.2017.01066.x/abstract" target="_blank">article about fake news</a>. As one would expect in <i>Significance</i>, there is interesting empirical data in the article. What I found most interesting was the following quote attributed to Dorothy Byrne, a British broadcast journalism leader:<br />
<blockquote class="tr_bq">
"You can't just feed people a load of facts...we are social animals, we relate to other people, so we have to always have a mixture of telling people's human stories while at the same time giving context to those stories and giving the real facts." </blockquote>
Just presenting and objectively supporting debunking factual evidence is not sufficient. We need to acknowledge that just as emotional triggers are key to spreading fake news, so they need to be considered in repairing the damage. I saw a great example of that in yesterday's Wall Street Journal. An article, titled "<a href="https://www.wsj.com/articles/video-contradicts-kellys-criticism-of-congresswoman-1508554197" target="_blank">Video Contradicts Kelly's Criticism of Congresswoman</a>," sets out to debunk the fake news story promulgated by the Trump administration claiming that Florida Rep. Frederica Wilson had touted her personal efforts in getting funding for an FBI building in her district while not acknowledging the slain FBI agents for whom the building was named. The Journal article could have stopped at the factual assertions that she had not been elected when the funding was approved and that a video of the speech she gave includes her acknowledging the agents. But it goes on to provide emotive context, describing the Congresswoman's lifelong focus on issues affecting low-income families and her personal connection with Army Sgt. La David Johnson, the Green Beret whose passing ultimately led to her confrontation with the Trump administration. The details on how she had known Sgt. Johnson's family for generations and that he himself had participated in a mentoring program that she founded provided context for the facts. The emotive picture painted by the original fake news claim and the administration's name-calling "all hat, no cattle" was replaced with the image of a caring human being. In that light, it's easier to believe the truth - that Rep. Wilson was gracious and respectful of the fallen agents and their families just as she was of Sgt. Johnson and his family.<br />
<br />
The lesson learned here is that in debunking fake news, "factual outrage" is not enough - we need to focus on <i>selling the truth</i> as the more emotionally satisfying position. As the <i>Significance</i> article points out, people are drawn to simple explanations and beliefs that fit with what they <i>want </i>to be true. So to repair the damage of fake news, we have to not just show people that their beliefs are inconsistent with reality - we need to provide them with another, emotionally acceptable reality that is closer to the truth.<br />
<br />
<br /></div>
Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com0tag:blogger.com,1999:blog-4528669273992673185.post-61935628775069298142017-07-30T16:14:00.001-07:002017-07-30T18:58:56.533-07:00Five things I look for in Software Engineers<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: "times" , "times new roman" , serif;">I have been interviewing a lot of software engineers recently, as I am leading a new team and looking to expand it. That has led me to reflect a little on what I am actually looking for. The following five qualities have been shared by all of the really good, fun-to-work-with developers who I have had the pleasure to work with.</span><br />
<div>
<span style="font-family: "times" , "times new roman" , serif;"><br /></span></div>
<div>
<b><span style="font-family: "times" , "times new roman" , serif;">1. Technical mastery</span></b></div>
<div>
<span style="font-family: "times" , "times new roman" , serif;">Really good developers fully understand what they are doing. This might sound funny, but unfortunately, it is all too common for people to get things to work by cutting and pasting examples or fumbling through a quasi-random hacking process to arrive at code that "works" without actually understanding how or why (or in fact even if) it works. There is nothing wrong with experimentation and leveraging experience - and working code - of others. When really good developers do that, though, they always find their way to full understanding of the technologies and techniques that they are using. When I interview developers, I always ask them to explain exactly how the solutions that they developed work. I can usually tell very quickly if I am talking to an individual who masters the technology that they use. I would much rather have a developer with strong mastery of a small set of technologies than someone whose resume is full of advanced technologies that they don't understand.</span></div>
<div>
<span style="font-family: "times" , "times new roman" , serif;"><br /></span></div>
<div>
<b><span style="font-family: "times" , "times new roman" , serif;">2. Simple mindedness</span></b></div>
<div style="text-align: left;">
<span style="font-family: "times" , "times new roman" , serif;">In <a href="http://www.cs.utexas.edu/users/EWD/transcriptions/EWD03xx/EWD340.html" target="_blank">The Humble Programmer</a>, <span style="background-color: white; text-align: -webkit-center; text-indent: 32px;">Edsger W. Dijkstra said "</span><span style="background-color: white; color: #222222;">The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague." Really good developers have a wonderful tendency to keep things as simple as possible, as long as possible. Fancy constructs, excessive OO complexity, needless external dependencies and exotic algorithms never find their way into their code. If there is a simple way to do something, that is what they do. Reading the code of a simple-minded developer is like reading a mathematical paper written by a great mathematician. If the content is straightforward, the progression is 100% predictable. You can stop in the middle and scribble out what should come next and then see that is what comes next. When you get to a difficult part where you have to think, you are happy to see that the author found something so simple that you should have thought of it.</span></span></div>
<div style="text-align: left;">
<span style="background-color: white; color: #222222; font-family: sans-serif; font-size: 14px;"><br /></span></div>
<div style="text-align: left;">
<span style="background-color: white; color: #222222; font-family: "times" , "times new roman" , serif;"><b>3. Organizing</b></span></div>
<div style="text-align: left;">
<span style="font-family: "times" , "times new roman" , serif;"><span style="color: #222222; font-family: sans-serif;"><span style="background-color: white;"><span style="font-family: "times" , "times new roman" , serif;">Another one of my favorite Dijkstra quotes is that </span><span style="font-family: "times" , "times new roman" , serif;">t</span></span></span><span style="background-color: white; color: #222222; font-family: "times" , "times new roman" , serif;">he art of programming is the art of organizing complexity. Great developers are organizing forces. Note that this is not the same as "being organized." It means that they help <i>define</i> problems in a way that they can be solved with simple solutions and they help get contracts, interface boundaries and tests in place so that the teams they are part of can <i>be</i> organized. The scope of what developers can organize naturally grows as they progress in their careers; but they need to have the drive and ability to be "organizers" from the beginning. Developers that have to be told how to think about problems are net drags on the teams they are part of. Good ones are key contributors to their teams arrival at nicely organized approaches to problems.</span></span></div>
<div style="text-align: left;">
<span style="background-color: white; color: #222222; font-family: "times" , "times new roman" , serif;"><br /></span></div>
<div style="text-align: left;">
<span style="background-color: white; color: #222222; font-family: "times" , "times new roman" , serif;"><b>4. Fast-learning</b></span></div>
<div style="text-align: left;">
<span style="background-color: white; color: #222222; font-family: "times" , "times new roman" , serif;">Technology changes so fast and business problems are spread across such a large surface that developers constantly need to learn new things. And these things are not just programming languages, frameworks or constructs. Developers need to learn business domain concepts, data science and AI concepts as needed, often in ridiculously short timeframes. This means that they have to be able to learn very fast. And they have to be able to do this and immediately exercise their knowledge with a high level of independence. Its great to be able to learn together and share knowledge with others, but sometimes developers need to figure things out for themselves and good ones have the ability and determination to learn what they need to learn - however hairy it gets - to solve the problems in front of them.</span></div>
<div style="text-align: left;">
<span style="font-family: "times" , "times new roman" , serif;"><br /></span></div>
<div style="text-align: left;">
<span style="background-color: white; color: #222222; font-family: "times" , "times new roman" , serif;"><b>5. Situational awareness</b></span></div>
<div style="text-align: left;">
<span style="color: #222222; font-family: sans-serif;"><span style="background-color: white; font-family: "times" , "times new roman" , serif;">Good developers ask about - and clearly understand - the execution context of the code they work on. If something needs to be thread-safe, they write it that way. They know what the performance and scalability bottlenecks in and around their code are. They know about its security context. They see enough of the larger system that their code is running in / interacting with to ensure that it will be <i>operable, </i>failing fast and loudly when it needs to fail, maintaining invariants that it needs to maintain, and providing sufficient logging / events / monitoring interfaces. And of course, all of this is validated in unit tests.</span></span></div>
<div style="text-align: left;">
<span style="color: #222222; font-family: sans-serif;"><span style="background-color: white; font-family: "times" , "times new roman" , serif;"><br /></span></span></div>
<div style="text-align: left;">
<span style="color: #222222; font-family: sans-serif;"><span style="background-color: white; font-family: "times" , "times new roman" , serif;">I know some people will say that some of what I have above - especially in 3. and 5. - can't really be expected of "SE's." These qualities, one might argue, are qualities of <i>architects. </i>Developers just need to be "feature machines" and architects can worry about how to organize the code and make sure the whole system is operable. My biggest learning in 30 years of software development is that that is the wrong way to think about it. Architecture influence and scope of vision naturally increases as developers progress in their careers; but it is part of what they do, every day, from day 1 and the better they do it, the more successful the teams they are part of can be. And the senior ones - those who might have "Architect" or "Principal" or "Staff" in their titles - need to encourage, cultivate, challenge and <i>be influenced by </i>the design thinking of SEs at all levels.</span></span></div>
<div style="text-align: left;">
<span style="color: #222222; font-family: sans-serif;"><span style="background-color: white; font-size: 14px;"><br /></span></span></div>
<div style="text-align: left;">
<span style="color: #222222; font-family: sans-serif;"><span style="background-color: white; font-size: 14px;"><br /></span></span></div>
</div>
Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com1tag:blogger.com,1999:blog-4528669273992673185.post-5829963652269510502016-09-17T14:14:00.000-07:002016-09-17T14:15:39.573-07:00I Pledge Allegiance...<div dir="ltr" style="text-align: left;" trbidi="on">
Last week, I heard a segment on NPR about patriotic rituals such as saying the Pledge of Allegiance or standing for the National Anthem. One statement by a young person was hard for me. She said she could not recite the Pledge of Allegiance because saying that our republic has "liberty and justice for all" is false. I get that. And I certainly respect anyone's right to participate or not participate in saying the Pledge. What bothers me is that "the republic, for which it stands" is an <b>idea</b> and that idea really does mean liberty and justice for all. We have never had - and no real nation ever will have - perfect liberty and justice. What we pledge allegiance to is the idea that such a nation can exist. I bet that 150+ years ago when Abraham Lincoln made his <a href="http://www.abrahamlincolnonline.org/lincoln/speeches/gettysburg.htm" target="_blank">long-remembered remarks</a> at Gettysburg, he knew well that this idea would never be perfectly realized. He knew that what he himself had done to preserve it was not perfect. But he really did believe in the idea. This idea is the source of everything that has ever been good about the United States and everything that will ever be good about us, our children or our children's children. We cannot abandon this idea because we have not lived up to it - even collectively. Even if we see endemic and systemic injustice and prejudice, we have to see that as<i> not who we are</i>. And we need our children to see that. We don't need to make them say the Pledge or even stand when others do, but we do need them to have faith that this idea really can "long endure" and that there really can be "a new birth of freedom" in the United States. <span style="font-family: "arial"; font-size: x-small;"><b><span style="font-family: "arial"; font-size: x-small;"><br /></span></b></span></div>
Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com0tag:blogger.com,1999:blog-4528669273992673185.post-45017017653840381802016-03-16T19:03:00.000-07:002016-03-19T10:30:29.849-07:00When someone who works for you recommends a book, read it!<div dir="ltr" style="text-align: left;" trbidi="on">
People who work for you have a great perspective on what you need to learn. Here are some great examples:<br />
<br />
<a href="http://www.tablegroup.com/books/dbm" target="_blank">Death by Meeting </a>- in which I learned that my team meetings were, ...um, "suboptimal." Thanks, Bob!<br />
<br />
<a href="https://www.toc-goldratt.com/en/product/The-Goal-A-Process-of-Ongoing-Improvement" target="_blank">The Goal</a> - in which I learned that there was a better way to think about process optimization. Thanks, Kevin!<br />
<br />
<a href="http://itrevolution.com/books/phoenix-project-devops-book/" target="_blank">The Phoenix Project </a>- in which I am learning that I did not fully understand the consequences of the previous book. Thanks, Scott!<br />
<br />
<br />
<br />
<br /></div>
Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com0tag:blogger.com,1999:blog-4528669273992673185.post-70446219629107188252015-11-18T07:58:00.000-08:002015-11-18T07:58:09.174-08:00I can see it from where I live<div dir="ltr" style="text-align: left;" trbidi="on">
After seeing it <a href="http://www.strategy-business.com/blog/Daniel-Pink-Required-Reading?gko=03bfe" target="_blank">recommended by Dan Pink</a>, I started reading Studs Terkel's classic, <a href="http://thenewpress.com/books/working" target="_blank">Working</a>. The following quote brought back old memories for me<br />
<blockquote class="tr_bq">
There’s not a house in this country that I haven’t built that I don’t look at every time I go by. (Laughs.) I can set here now and actually in my mind see so many that you wouldn’t believe. If there’s one stone in there crooked, I know where it’s at and I’ll never forget it. Maybe thirty years, I’ll know a place where I should have took that stone out and redone it but I didn’t. I still notice it. The people who live there might not notice it, but I notice it. I never pass that house that I don’t think of it. </blockquote>
<span style="font-family: inherit;">I was lucky to have as my first boss a man who <span style="font-family: inherit;">really valued workmanship</span>. His landscape construction and maintenance business was hard and I could see every day the pressure to cut corners. But he never did and he got very, very mad when he saw any of us doing shoddy work. </span><br />
<span style="font-family: inherit;"><br /></span>
I remember a few years later, I was working on an interstate highway construction project. My job was to break off concrete pipes and parge cement around the gaps between them and the junction boxes where they came together. I always tried to do a nice job, leaving the box looking like it had been cast as a single piece of concrete. I remember once having a hard time with one of the pipes and struggling with my coworkers to get the finish smooth. One of them said, "I can't see it from where I live." I immediately thought of my first boss, yelling at me once for justifying a sloppy joint by saying that no one would notice it because it was going to be backfilled. He said, "but <i>I</i> just noticed it and <i>you</i> saw it yourself. When you go home, you will see it again. And if you don't see it again, you haven't learned anything from me." </div>
Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com1tag:blogger.com,1999:blog-4528669273992673185.post-10095438559584456752015-11-11T16:09:00.000-08:002015-11-22T09:55:13.248-08:00A very crowded corner<div dir="ltr" style="text-align: left;" trbidi="on">
OK, time for a little math walk. Imagine that Bolzano's grocery is running a special on Weirstrass' Premium Vegan Schnitzel. People start converging on the corner in front of Bolzano's from all around. Based on counts using awesome new <b><i>really</i></b> big data technology, the local news media makes the amazing announcement that there are <i>infinitely many </i>people in the city block around Bolzano's. The subject of this walk is showing that there must be at least one location in that block where you can't move even the slightest distance without bumping into someone.<br />
<br />
To simplify things, lets smash everything down into one dimension and pretend that the city block above is the closed interval $[0, 1]$ on the real number line. Let's represent the infinite set of people as points in this interval. Now consider the subintervals $(0, .1), (.1, .2), ... (.9, 1).$ At least one of these intervals must contain infinitely many people. Suppose, for example, that the interval $(.5, .6)$ contains infinitely people. Then split that interval into 10 segments, as shown in the picture below. At least one of these has to contain infinitely many people. Suppose, again for example, that this subinterval is $(.537, .538)$.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgRjgywF4rml7al7W0KvQADqAFeAhI6miLFx7GfrGa4fFjnG13jBzAwaCD3Sqz0_RZQbOZFgzmMONL5NUY1wJWcQpv3lLzPzF8OP48pV6ltYeCTOjH4ol4otLGV1Lh5fRQY4XQqC0t2ZDc/s1600/bolzano.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="209" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgRjgywF4rml7al7W0KvQADqAFeAhI6miLFx7GfrGa4fFjnG13jBzAwaCD3Sqz0_RZQbOZFgzmMONL5NUY1wJWcQpv3lLzPzF8OP48pV6ltYeCTOjH4ol4otLGV1Lh5fRQY4XQqC0t2ZDc/s320/bolzano.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
Now consider the number .537. We know that there are infinitely many people within .001 of .537. There is nothing stopping us from continuing this process indefinitely, finding smaller and smaller subintervals with left endpoints $.5, .53, .537... $ each containing infinitely many people. Let <i>$r$ </i>be the number whose infinite decimal expansion is what we end up with when we continue this process <i>ad infinitum</i>. To make $r$ well-defined, let's say that in each case we choose the left-most subinterval that contains infinitely many people. Depending on how the people are distributed, $r$ might be boring and rational or something exotic like the decimal expansion of $\pi$. The point is that it is a well-defined real number and it has the property that no matter how small an interval you draw around it, that interval includes infinitely many people. This is true because for each $n$, the interval starting at $r$<i> </i>truncated to $n$ decimal digits and ending $1 / 10^n$ higher than that contains both $r$ and infinitely many other people by construction. In the example above, for $n = 3$, this interval starts at $.537$ and ends at $.538$.<br />
<br />
Now let's remove the simplification, one step at a time. First, let's see how the same construction works if in place of $[0, 1]$ we use any bounded interval $[a, b]$. Consider the function $f(x) = (x - a) / (b - a)$. That function maps $[a, b]$ onto $[0, 1]$. Its graph is a straight line with slope $1/(b - a)$. If $b - a$ is larger than 1, points get closer together when you do this mapping; otherwise they get further apart. But the expansion or contraction is by a constant factor, so the picture above looks exactly the same, just with different values for the interval endpoints. So if we do the construction inside $[0, 1]$ using the mapped points, then the pre-image of the point $r$ we end up with will be an accumulation point for the set in $[a, b]$.<br />
<br />
OK, now lets pick up our heads and get out of Flatland. Imagine that the square block around Bolzano's is the set of points in
the x-y plane with both x and y coordinates between 0 and 1. Divide up the square containing those points into 10 equal-sized pieces. One of those pieces has to contain infinitely many people. Suppose it is the square with bottom-left coordinates (.5, .2). Now divide that little square into 10 subsquares. Again, one of these has to contain infinitely many people. Say it is the one with lower-left coordinates (.53, .22). The picture below shows these points and a next one, say, (.537, .226). Just like the one-dimensional case, this sequence of points converges to an accumulation point (x,y) that has infinitely many people within even the smallest distance from it.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjB1f37aGxFBHT4BEr3ezTpwMSurFwHHjm1M3Snjr3yNPt4EpfJBJ2s665hV8zzj4-Hs1YK6mrkpeq3rxkuV97cLRmSFbkmMcT_dZX1Au-jYN8zFzmz-LzL-LXrsBcNOO_kDvRow_o6BjY/s1600/bolz2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="307" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjB1f37aGxFBHT4BEr3ezTpwMSurFwHHjm1M3Snjr3yNPt4EpfJBJ2s665hV8zzj4-Hs1YK6mrkpeq3rxkuV97cLRmSFbkmMcT_dZX1Au-jYN8zFzmz-LzL-LXrsBcNOO_kDvRow_o6BjY/s320/bolz2.png" width="320" /></a></div>
The ideas presented above are the core of one proof of the <a href="http://mathworld.wolfram.com/Bolzano-WeierstrassTheorem.html" target="_blank">Bolzano-Weierstrass Theorem</a>, a beautiful and very useful result in Real Analysis. The existence of the limiting values is guaranteed by the Least Upper Bound Axiom of the real numbers. <br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br /></div>
Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com0tag:blogger.com,1999:blog-4528669273992673185.post-91610176100641592022013-11-25T18:14:00.000-08:002013-11-25T18:16:27.879-08:00Fully solving problemsBryan Pendleton's great post, "<a href="http://bryanpendleton.blogspot.com/2013/11/anatomy-of-bug-fix.html">Anatomy of a bug fix</a>" suggests some basic principles that apply to all kinds of problem resolution. The attributes that he calls out as separating "great developers" from the not-so-great apply in lots of other contexts, distinguishing the people who you really want to have on your team from those who you can't really count on. Bryan's conclusion:<br />
<blockquote class="tr_bq">
I often say that one of the differences between a good software engineer
and a great one is in how they handle bug fixes. A good engineer will
fix a bug, but they won't go that extra mile:
<br />
<ul>
<li>They won't narrow the reproduction script to the minimal case</li>
<li>They won't invest the time to clearly and crisply state the code flaw</li>
<li>They won't widen the bug, looking for other symptoms that the bug
might have caused, and other code paths that might arrive at the
problematic code</li>
<li>They won't search the problem database, looking for bug reports with different symptoms, but the same underlying cause. </li>
</ul>
</blockquote>
The scenario above applies whenever there is a problem to be resolved. I once led a great team responsible for resolving operational problems at a bank. The "great ones" on that team always performed analogs of all of the things Bryan mentions above. They always got to a very precise problem statement and recipe to reproduce and (sometimes painfully) a really exhaustive explication of impacts (what Bryan calls "widening the bug") as well as understanding of the relation between the current problem and any others that may have been related.<br />
<br />
I have seen this separation of talent in lots of business domains - operations, engineering, finance, marketing - even business strategy and development. The great ones don't stop until they have a really satisfying understanding of exactly what an anomaly is telling them. The not-so-great are happy just seeing it go away.<br />
<br />
The difference is not really "dedication" or "diligence" <i>per se</i> - i.e., its not just that the great ones "do all the work" while the not-so-great are lazier. The great ones are driven by the <i>desire to understand </i>and to <i>avoid needless rework </i>later. They tend to be less "interrupt-driven" and may actually appear to be less responsive or "dedicated" in some cases. They solve problems all the way because they can't stop thinking about them until they have really mastered them. I always look for this quality when I am hiring people. Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com0tag:blogger.com,1999:blog-4528669273992673185.post-26671511468461896362013-04-14T14:54:00.000-07:002013-04-14T14:55:55.070-07:00Core LeadershipWe get the word "integrity" from the same root that gives us "integer" or "whole number." To have integrity is first and foremost <i><b>to be one thing.</b></i> Kant built his entire theory of knowledge on the premise that experience has to make sense - the world has to be <i><b>one thing</b></i> in this sense. We have to be able to say "I think..." before every perception that we have about the world.<br />
<br />
The core of all effective leadership really comes down to this. It all has to make sense. Everyone has to be able to start with "I think...". Not "Mr. X says..." Not "policy is.." Not "I was told..." but "I think..."<br />
<br />
For this to work, leaders have to be firmly grounded in a shared vision and they have to be committed to maintaining integrity in the sense above. Values, principles, objectives, strategies, communications, performance evaluations, policies, processes, commitments all have to be constantly integrated. Leaders who force themselves to be able to say "I think..." before a comprehensive view of all of these things can <i><b>lead from the core.</b></i> Just as it is painful to do the exercises to strengthen your physical core, so it can be painful to maintain core leadership strength in this sense. It is very easy to get "out of shape" by neglecting core values, objectives, strategy and execution alignment. But without a strong core, none of the most important leadership attributes - authenticity, inspiration, strategic vision, followership, transformational impact - are possible.<br />
<br />
Leaders who "skip the abs work" can get some things done and, depending on their good fortune and / or cleverness, some achieve material success. But no one remembers them. No great change is ever led by them. No great leaders are ever developed by them. Leading durable transformational change and developing great leaders requires core strength.<br />
<br />
So how do you develop core strength? A great mentor and an already established values-based vision and strategy can help get you started, but you always end up having to do the work to build your own core yourself. Here are some little exercises that can help. There is nothing particularly deep here and there are lots of variations on these practices. The point is to regularly and critically focus on core integrity.<br />
<br />
<b>Look-back sit-ups.</b> Starting once a week and working up to once a day, look back on all of the decisions, communications and interactions that you had and explain how it is possible that <i><b>one person</b></i> did all of these things. I guarantee that if you are really observant and critical, you will find lots of little inconsistencies - things that in retrospect you can't say, "I think..." in front of. For each of these, you have two choices: either come up with an alternative course of action that, had you done it, would have made sense; or modify whatever aspects of your vision, strategy or values it is inconsistent with (or more precisely, resolve yourself to conceive and align the necessary changes with your team, your peers and your leadership). Done honestly, this is painful. Think of each example as a little integrity sit-up. Here are a couple of concrete examples.<br />
<ol>
<li>Suppose that last week you negotiated an extension to a service contract. In exchange for a healthy rate reduction, you doubled the term length and added minimums to the contract. This will help achieve your annual opex reduction goal; but your agreed upon strategy is to ensure supplier flexibility and aggressively manage demand in the area covered by the contract. Your decision basically said near-term opex reduction was more important than flexibility or demand management. Either your strategy was wrong or your decision was wrong. To be <i><b>one person</b></i>, you need to either acknowledge the mistake or harmonize the decision with the strategy.</li>
<li>Last week you agreed with your leader and peers in a semi-annual performance ratings alignment meeting that one of your direct reports was not fully meeting expectations in some key areas. You agreed to deliver the "needs improvement" message in these areas in his performance appraisal and to adjust his overall rating downward. You did change the rating and some of the verbiage in the assessment; but when you delivered the review and he challenged the overall rating, you were swayed by his arguments and in the end you admitted that you had been told to adjust the rating downward. Here either you failed to consider everything when agreeing to the rating adjustment or you were overly influenced by the feedback.</li>
</ol>
In some cases, the look-back exercise can and should lead you to take some remediating actions; but that is not the point of the exercise. The point is to do a little "root cause analysis" of what caused the integrity breakdown. In the first example, it may have been extreme near-term financial pressure causing things to get out of focus, or possibly just lack of clarity in the relative importance of the different factors in the strategy. In the second example, the feedback may have pushed some "hot buttons" causing you to temporarily lose some core strength. The key is to face these integrity gaps directly and honestly by yourself. First think clearly and honestly about what went wrong and why. Then think about how to "fix things." <br />
<br />
<b>Virtual 360 crunchies.</b> Again starting once a week and working up to daily, imagine you are specific person on your team, in your company or a partner (alternate among randomly chosen people from these groups) and respond to the question, "What is most important to X?" where X is you. Don't just repeat goals or big initiative names or repeat your own communications. Actually try to imagine what it would be like being the selected person and what they really think is important to you and how that relates to what they do on a day to day basis. Think about how they would say it in their own words, not yours. If you can't do it, or what naturally comes out is far from what you see as your core, you have two options. Either you have a communication problem - i.e. there is no way this person can have a clear understanding of what is important to you because you have failed to communicate it - or you <i><b>don't make sense</b></i> from their vantage point. In the first case, you need to work on communication and in the second, you need to patch whatever holes exist in your vision, strategy or values that make you incomprehensible to this person. Here are some examples.<br />
<ol>
<li>You have recently been promoted from a marketing leadership position to leading an entire business unit, including sales and operations. You have communicated your growth strategy, which is heavily top-line focused. You are also facing significant cost containment pressure. You know that learning about operations is critical to your success and you have been asking a lot of questions of your operations leadership team, focusing on quickly identifying some areas where you can cut costs. Your broad-scale communications have made high level references to "operations excellence" and "empowerment," but you have not provided any details on your plans for operations. Now suppose your randomly selected person for virtual 360 is a first level manager in operations. It is hard to imagine that she would not say that what you care about is top line revenue growth and cutting costs and what she does on a day-to-day basis is "not important to you."</li>
<li>You are a well-respected leader of a software engineering team. There are 10 products in your portfolio. You have developed a robust set of goals for the organization, cascaded effectively through the team. There are 5 top-level goals, each of which has been broken down by each of the teams in your group. You know every product and team inside out and you regularly do "deep dives" at the product level. Your communications are detailed, often focusing on knowledge-sharing across the group and encouraging collaboration among teams. Imagine that the virtual 360 candidate is an engineer working on one of the products. He might say something like, "Let me go look at the goals statement. I know the features we are working on are important because she mentioned them in the deep dive last week."</li>
</ol>
The examples above show opposite extremes. In the first case, the leader has a big gap in communicated - possibly even <i>conceived</i> - vision and values. In the second case, the team has lost sight of the forest through the trees. Both require not just communication, but critical assessment of core leadership. In the first case, there is a hole. In the second case, the core appears to be disintegrating. The basic problem is the same. It all has to make sense to all stakeholders all the time.<br />
<br />
I have never met a leader who did not have small or large "core integrity" problems to deal with from time to time. The great ones recognize them quickly and get whatever help they need to build and maintain a strong core.<br />
<ol>
</ol>
Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com0tag:blogger.com,1999:blog-4528669273992673185.post-33040639093347602562011-11-19T15:14:00.001-08:002011-11-20T12:41:19.538-08:00Thinking Together<blockquote class="tr_bq">
"Nestor, gladly will I visit the host of the Trojans over against us, but if another will go with me I shall do so in greater confidence and comfort. When two men are together, one of them may see some opportunity which the other has not caught sight of; if a man is alone he is less full of resource, and his wit is weaker. " Homer, <i>Illiad 10</i> (Diomedes about to go out spying on the Trojans)</blockquote>
I have always loved the simplicity in the image above, which Plato paraphrased simply as "when two go together, one sees before the other." Real collaboration works this way - like Diomedes and Odysseus, we <i>think together</i>, and do more, better, faster than we could alone. For this to work, we have to be <i>thinking about the same thing</i>. Rather than just periodically sharing fully working ideas, we need to <i>think out loud</i>, sharing the not-yet-sense that eventually workable ideas come from.<br />
<br />
I remember when I was a mathematics grad student I thought I would never be capable of producing the brilliantly incomprehensible work that I saw published in professional journals. It took me forever to read papers and there appeared to be so many little miracles embedded in them that there had to be some magical wizardry involved. All of that changed when I started <i>thinking together</i> with some real mathematicians. Seeing the thought process, the not-quite-sense at the base of the ideas, the stops and starts, the workarounds, let me see that while I would never be Diomedes, I could at least have a little fun as Odysseus.<br />
<br />
The key here is to actually externalize and share the thought process underneath ideas in the making, rather than just handing them back and forth as "products." I am belaboring this point because it bears on a dynamic in the open source world that I find alternatively exciting and troubling - the rise of distributed version control systems (DVCS). Thanks to Git and GitHub, the process of contributing to open source has become much easier and the number of projects and contributors is exploding. That is the exciting part. The thing that I sometimes worry about is that the "forking is a feature" meme that can result from DVCS can take us away from thinking together and more toward "black box" exchange of completed ideas. The traditional, central VCS, central mailing list model works more or less like this:<br />
<ol>
<li>Someone has an "itch" (idea for enhancement, bug fix, idea for a change of some kind)</li>
<li>Post to the project development mailing list, talking about the idea</li>
<li>Patch, maybe attached to an issue in the project issue tracker, maybe directly committed</li>
<li>Commit diff from the central repo triggers an email to the list - everyone sees each small change</li>
<li>Idea evolves via discussion and commits, all visible to all following the project</li>
</ol>
Using DVCS, the process can work differently:<br />
<ol>
<li>Someone has an itch</li>
<li>Clone the repo, creating a local branch</li>
<li>Experiment with new ideas in the local branch, possibly publishing it so others can see and play with it</li>
<li>Submit a request for the changes in the local branch to be incorporated in the "mainline"</li>
<li>Changes are eventually integrated in a batch</li>
</ol>
Now there is no reason that the second flow can't involve the full community and be done in a <i>thinking together</i> way in small, reversible steps; but it is also easy for it to just happen in relative isolation and to end with a big batch of "black box" changes integrated in step 5. If this becomes the norm, then the advantages of OSS communities<i> thinking about the same thing</i> as they evolve the code is significantly diminished. It's also a lot less fun and interesting to review large, unmotivated pulls than to participate in the development of ideas and work collaboratively on the code expressing them. But most importantly, if the primary means of communication becomes patch sets developed by individuals or small subcommunities, and the only thinking together that the community does is after the fact review, we lose one of the biggest benefits of open source - the collaborative development of ideas by a diverse community influencing their origin and direction.<br />
<br />
The DVCS train has left the station and as I said above tools by themselves don't determine community dynamics. Its up to those of us who participate in OSS communities to keep engaging in the real collaboration that created the great software and communities-around-code that have made OSS what it is. So lets take a lesson from Diomedes. Great warrior that he was, he still thought it best to bring "the wily one" with him when he went out into new territory.Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com5tag:blogger.com,1999:blog-4528669273992673185.post-81816015952062906862011-10-28T15:38:00.000-07:002011-10-28T15:38:09.543-07:00Correlated failure in distributed systemsEvery time I start to get mad at Google for being closed and proprietary, they release something really interesting. The paper, <a href="http://research.google.com/pubs/pub36737.html">Availability in Globally Distributed Storage Systems</a> is loaded with interesting data on component failures and it presents a nice framework for analyzing failure data. The big takeaway is that naive reasoning about "HA" deployments can lead to inflated expectations about overall system availability.<br />
<br />
Suppose you have a clustered pair of servers and each is 99% available. What availability can you expect from the clustered pair? "Obviously," $99.99\%.$<br />
<br />
Unfortunately, that expectation assumes that the servers fail independently - i.e., component failures are not correlated. The Google research shows that this can be a bad assumption in practice. Referring to storage systems designed with good contemporary architecture and replication schemes, the authors say, "failing to account for correlation of node failures typically results in overestimating availability by at least two orders of magnitude." They go on to report that due to high failure correlation, things like increasing the number of replicas or decreasing individual component failure probabilities have much weaker effects when you take correlation into account.<br />
<br />
<a href="http://steveloughran.blogspot.com/2011/10/reading-availability-in-globally.html">Steve Laughran</a> does a nice job summarizing the practical implications of the specific area covered by this research. What is interesting to me is the model proposed for quantifying correlation in component failure and estimating its impact on overall system availability. Its easy to come up with scenarios that can lead to correlated component failures, e.g. servers on a rack served by a single power source that fails; switch failures; bad OS patches applied across a cluster. What is not as obvious is how to tell from component-level availability data what counts as a correlated failure and how to adjust expectations of overall system availability based on the extent of correlation.<br />
<br />
The first question you have to answer to get to a precise definition of correlation in component failure is what does it mean for two components to fail "at the same time" or equivalently what does it mean for two observed component failures to be part of a single failure event? The Google authors define a "failure burst" to be a maximal sequence of node failures, all of which start within a given window-size, $w$, of one another. They use $w=120$ seconds for their analysis, as this matches their internal polling interval and it also corresponds to an inflection point on the curve formed when they plot window size against percentage of failures that get clubbed into bursts.<br />
<br />
We can define correlation among node failures by looking at how the nodes affected by bursts are distributed. The practically relevant thing to look at is how often nodes from system architecture <i>domains</i> fail together - for example, to what extent do node failures occur together in the same rack. If failures are highly rack-concentrated, for example, having system redundancy only within-rack is a bad idea.<br />
<br />
Given a failure burst consisting of a set $N = {f_0,..., f_n}$ of failing nodes and a partition $D = {d_0, ... , d_m}$ of $N$ into domains, we will define the $D$<i>-affinity</i> of $N$ to be the probability that a random assignment of failing nodes across domains will look less concentrated than what we are observing. High $D$-affinity means correlation, low means dispersion or anti-correlation. If domains are racks, high rack-affinity means failures are concentrated within-rack.<br />
<br />
To make the above definition precise, we need a measure of domain concentration. The Google paper proposes a definition equivalent to the following. For each $i = 0, ..., m$ let $k_i$ be the number of nodes in $N$ included in $d_i$. So for example if the $d_i$ are racks, then $k_0$ is the number of nodes in rack $0$ that fail, $k_1$ counts the failures in rack $1$, etc. Then set $x = \sum_{i=0}^{m}{k_i \choose 2}$. This makes $x$ the number of "failure pairs" that can be defined by choosing pairs of failing nodes from the same domain. Clearly this is maximized when all of the failures are in the same domain (every pairing is possible) and minimized when all failing nodes are isolated in different domains. Increasing domain concentration of failures increases $x$ and disaggregating failing nodes decreases it.<br />
<br />
Now let $X$ be a random variable whose values are the values of $x$ above. For each possible value $x$ define $Pr(X = x)$ to be the likelihood that $X$ will take this value when failing nodes are randomly distributed across domains. Then for each value $x$, define $r_x = Pr(X < x) + \frac{1}{2}Pr(X = x)$. Then $r_x$ measures the likelihood that a random assignment of failing nodes to domains will result in concentration at least as large as $x$. The $\frac{1}{2}$ is to prevent the measure from being biased, as we will see below. A value of $r$ close to $1$ means that failures are highly correlated with respect to domain, while values close to $0$ indicate dispersion. With domains equal to racks and $r$ called <i>rack-affinity</i>, the Google paper reports:<br />
<blockquote class="tr_bq">
We find that, in general, larger failure bursts have higher rack affinity. All our failure bursts of more than 20 nodes have rack affinity greater than 0.7, and those of more than 40 nodes have affinity at least 0.9. It is worth noting that some bursts with high rack affinity do not affect an entire rack and are not caused by common network or power issues. This could be the case for a bad batch of components or new storage node binary or kernel, whose installation is only slightly correlated with these domains.</blockquote>
The authors point out that it can be shown that the expected value of $r$ is $.5$. To see this, let $x_0, x_1, ..., x_t$ be the values of $X$ as defined above and for each $i = 0, ..., t$ let $p_i = Pr(X = x_i)$. Then the expected value of $r$ is $$E(r) = \sum_{i=0}^{t}\left\{p_i \left(\sum_{j=0}^{i-1}p_j + \frac{1}{2}p_i\right)\right\}.$$Since $\sum p_i = 1$, we must have $(\sum p_i)^2 = 1$. Expanding this last sum and the sum for $E(r)$, it is easy to see that $E(r) = \frac{1}{2}(\sum p_i)^2$. Note that this applies to any discrete probability distribution - i.e., $r$ as above could be defined for any discrete distribution and its expectation will always be $.5$. Note also that while $r$ can take the value $0$, its maximum value is $1 - \frac{1}{2}p_t.$ For $X$ as defined above, $p_t$ is the probability that all failures are in the same domain, which is $1/B_N$ where $N$ is the total number of nodes and $B_N,$ the $Nth$ <a href="http://en.wikipedia.org/wiki/Bell_number">Bell number</a>, is the number of ways that the $N$ nodes can be partitioned. <br />
<br />
Computing the value of $r$ given counts $c_0, c_1, ..., c_m$ of failing nodes by domain is non-trivial. According to the Google authors,<br />
<blockquote class="tr_bq">
It is possible to approximate the metric using simulation of random bursts. We choose to compute the metric exactly using dynamic programming because the extra precision it provides allows us to distinguish metric values very close to 1.</blockquote>
I have not been able to figure out a straightforward way to do this computation. Maybe the Googlers will release some code to do the computation on Google Code. The only way that I can see to do it is to fully enumerate partitions over the node set, compute $x$ for each partition and build the distribution of $X$ using frequency counts. Patches welcome :)<br />
<br />
The Google paper stops short of developing a framework for using estimates of node failure correlation in end-to-end system availability modelling. That would be an interesting thing to do. Here are some simple observations that might be useful in this regard and that also illustrate some of the practical implications.<br />
<br />
<b>Correlation cuts both ways</b> - i.e., it is possible to do better than independence if a system's deployment architecture splits over domains with high failure affinity. Consider, for example, an application that requires at least one database node to be available for it to provide service. Suppose that database node failures are perfectly rack-correlated (i.e., all database node failures are concentrated on single racks). Then if the application splits database nodes over racks (i.e. has at least one node in each of two different racks) it can deliver continuous availability (assuming the database is the only thing that can fail).<br />
<br />
<b>End-to-end HA requires splitting over all domains with high failure correlation.</b> Suppose that in the example above, database node failures also show high switch affinity. Then to deliver HA at the application level, you need to ensure that in addition to having database nodes in two different racks, you also need nodes connected to at least two different switches.<br />
<br />
<b>As always, correlation does not imply causation.</b> The Google paper makes this point in a couple of places. Suppose that in our simple example all database failures are in fact due to database upgrades and the operational practice is to apply these upgrades one rack at a time. That will result in high rack affinity among failures, but the failures have nothing to do with the physical characteristics or failure modes of the racks or their supporting infrastructure. <br />
<br />
The observations above are basic and consistent with the conventional wisdom applied by operations engineers every day. In an ideal world, HA systems would be designed to split over every possible failure domain (rack, switch, power supply, OS image, data center...). This is never practical and rarely cost-effective. What is interesting is how quantitative measurements of failure correlation can be used to help estimate the benefit of splitting over failure domains. Just measuring correlation as defined above is a good start.Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com1tag:blogger.com,1999:blog-4528669273992673185.post-69477541006666895002011-10-17T18:34:00.000-07:002011-10-17T18:34:13.504-07:00Open source community economicsMuch has been written about how to make money from open source and the impact of open source on commercial software markets. The economic agents in these analyses are “open source companies,” traditional technology companies leveraging open source and individuals trying to make a living writing software. I have not found much that looks at open source communities as agents. Here are some pretty obvious things that call out the contrast between the interest of an open source community and the more traditional economic agents that engage it. Throughout, I am assuming a real open development, open source community - not a corporate fishbowl or commercial consortium.<br />
<br />
<b>Membership is the bottom line</b><br />
Companies eventually go out of business if they do not make money - i.e., if their revenues do not exceed their costs over time. Open source communities go out of business if they do not “make members” over time - i.e., if they do not succeed in attracting and retaining volunteers faster than people leave to do other things with their time. Just as commercial companies need to relentlessly focus on making sure they remain profitable, OSS communities need to constantly strive to remain interesting, attractive and welcoming. Infusing paid volunteers is one way to keep “the books balanced” but it is analogous to borrowing capital in the commercial world - the result better be a healthier, more sustainable community; otherwise the “cash infusion” is just postponing demise.<br />
<br />
<b>Maximizing downloads is less important than maximizing engagement</b><br />
While commercial interests and committers’ individual career goals may be enhanced by focusing on achieving the highest possible levels of downloads and industry buzz, this by itself does nothing for the community. The community indirectly benefits as a result of users who come to know about it via the “buzz” and later decide to engage. But if maximizing on sheer numbers of “free beer consumers” in any way reduces user engagement or discourages new volunteers from getting involved, it does harm to the community. A critical juncture is what happens when an OSS project becomes commercially important. Inevitably, the need for “stability” starts popping up in community discussions and someone proposes that the project should move to “review then commit” (nothing gets committed until *after* it has been reviewed by the project committers). Then comes the decision to have a “high bar” for commit. This will “stabilize” the code in the short term, allowing vast hordes of free beer drinkers to download and use it “with confidence” and generate ever more positive industry buzz. But it will kill the community over time. I am not suggesting that this *has* to happen. The Apache httpd and Tomcat projects, and many others, have managed to have it both ways - lots of downloads and “buzz” and healthy communities. But they have had to work at it and stay focused on maintaining an environment where new volunteers are welcomed into the community, processes are transparent, there is genuine openness to new ideas and it is not hard for new contributors to learn about the project and find useful things to work on.<br />
<br />
<b>Problems are worth more than solutions</b><br />
At Apache, we often repeat the mantra, “community before code.” That means that if you ever have to decide between the interests of the community and the most expedient way to ship working software, the community wins. We take time to talk about things and come to consensus and we make technical decisions based on consensus - even if that sometimes takes longer or results in less elegant code committed to the repository. From the standpoint of the community as economic agent, its ability to attract and retain volunteers is paramount, so it makes sense that we *use* the code to feed the community, and not vice-versa. An interesting consequence of this is that problems - whether they be bug reports or ideas for enhancements - are more valuable to the community than “donations” of code. Large “code dumps” of finished code actually have negative value, because they distract the community with generally boring things like licensing, repackaging, and documentation and add an additional support burden. Small contributions of code or ideas that raise interesting problems are sustaining food for the community. Here again, the actual interest of the community as an economic agent do not correspond exactly to the interests of those consuming its “product.” This is not surprising, because the real “customers” of an OSS community are the members of the community itself, who are typically a tiny subset of the user base of the software that is produced. <br />
<br />
<b>Diversity is the risk management strategy</b><br />
Just as corporations have to worry about mismanagement, market forces or other externalities destroying their business models, OSS communities have to worry about internal or external problems forcing them to lose their volunteers. Just as businesses diversify to spread risk, OSS communities “diversify” - but in the case of OSS communities, diversification means cultivating a diverse membership. Having all key contributors work for the same company, for example, represents a material risk (assuming they are all paid to contribute). Or having them all share a narrow view of *the one right way* to do whatever the code does. Diversity in open source communities is a natural hedge against technological obsolescence and collective loss of interest. Software technology is faddish and talented developers are fickle - easily attracted to the next new shiny object. To survive in the market for eyeballs and interest, OSS communities have to be places where new ideas can happen. This means they have to attract people who can have different ideas than what the community already has - i.e., they have to be constantly diversifying.Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com0tag:blogger.com,1999:blog-4528669273992673185.post-6786653515824314612011-10-10T17:59:00.000-07:002011-10-12T15:34:12.654-07:00Rethinking authenticationAuthentication may end up being the <b><i>next big thing</i></b> for mobile devices after the "killer app" that led to us all walking around with them - i.e., voice. The whole concept of "user authentication" is due for an uplift and all of us effectively becoming net <a href="http://en.wikipedia.org/wiki/Point_of_presence">POPs</a> via our mobile devices opens up some interesting possibilities in this area.<br />
<br />
Traditionally, authentication is something that punctuates and intrudes on our experience as we do things that only we should be able to do - e.g. access financial accounts, use credit cards, check in to hotels, flights, exclusive events, etc. A person or automated system gating our access to something has to get to a sufficiently high level of confidence that we are who we say we are in order to let us in. Authentication puts gates in front of us and we present credentials to get the gates to open.<br />
<br />
Having an "always on" POP attached to us allows us to think about the problem differently. Instead of authenticating at experience-interrupting gates, we can think about continuously updating our estimate of the probability that the person attached to a mobile device is the person who <b>should</b> be attached to that device (call this the <i>right binding probability)</i>. As I walk around and do stuff, take calls, get visually identified, etc., my device can provide a stream of "naturally authenticating information" (eventually based on biometrics, but also including behavioral information as well as the outcome of authentication / identification events). When I want to do something that only I can do, my authentication state can be pushed in front of me, opening gates and eventually even eliminating most of them altogether in favor of challenges based on thresholds of the right binding probability. <br />
<br />
There are obviously privacy considerations to think about here and at the end of the day, it will come down to how much "observation" we are going to allow in order to make authentication more convenient for us and our identities more secure. Just allowing the phone to identify us via voiceprint and to report this event to an authentication service that we opt in to could provide a convenient second factor for financial transactions - again, without interrupting experience.<br />
<br />
Updating right binding probabilities based on authenticating events presents an interesting mathematical modelling problem. Each event should have an immediate impact, but its effect should decay over time. A relatively strong event like voiceprint identification should create a significant bump and a weaker event like crossing a geo fence into a common haunt at a regular time should contribute less. But how, if at all, should the second event affect the decay of the first event's effect? It seems we need to keep a rolling window of recent events, including their times and an updating algorithm that looks at both existence of event types over backward-looking time intervals as well as sequencing.Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com0tag:blogger.com,1999:blog-4528669273992673185.post-6193918408483884922011-09-27T21:09:00.000-07:002011-09-27T21:14:21.077-07:00Client Oriented ArchitectureFor no particular reason, I have been thinking a fair amount recently about the <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.20.1495&rep=rep1&type=pdf">CAP theorem</a> and how the basic problem that it presents is worked around in various ways by contemporary and even ancient systems.<br />
<br />
I remember years ago as a freshly minted SOA zealot, I was confused by the pushback that I got from mainframe developers who insisted that client applications needed to have more control over how services were activated and how they worked. I always thought that good, clean service API design and "separation of concerns," along with developer education and evangelism would make this resistance go away. I was wrong.<br />
<br />
I still think the basic idea of SOA (encapsulation and loose coupling) is correct; but once you shatter the illusion of the always-available, always-consistent central data store, you need to let the client do what it needs to do. The whole system has to be a little more "client-oriented."<br />
<br />
The <a href="http://s3.amazonaws.com/AllThingsDistributed/sosp/amazon-dynamo-sosp2007.pdf">Dynamo Paper </a>provides a great example of what I am talking about here. I am not sure it is still an accurate description of how Amazon's applications work; but the practical issues and approaches described in the paper are really instructive. According to the paper, Dynamo is a key-value store designed to deliver very high availability but only "eventual consistency" (i.e., at any given time, there may be multiple, inconsistent versions of an object in circulation and the system provides mechanisms to resolve conflicts over time). For applications that require it, Dynamo lets clients decide how to resolve version conflicts. To do that, services maintain <a href="http://en.wikipedia.org/wiki/Vector_clock">vector clocks </a>of version information and surface what would in a "pure" SOA implementation be "service side" concerns to the client. To add even more horror to SOA purists, the paper also reports that in some cases, applications that have very stringent performance demands can bypass the normal service location and binding infrastructure - again, letting clients make their own decisions. Finally, they even mention the ability of clients to tune the "sloppy quorum" parameters that determine effective durability of writes, availability of reads and incidence of version conflicts.<br />
<br />
Despite the catchy title for this post, I don't mean to suggest that SOA was a bad idea or that we should all go back to point-to-point interfaces and tight coupling everywhere. What I am suggesting is that just having clean service APIs at the semantic, or "model" level and counting on the infrastructure to make all decisions on behalf of the client doesn't cut it in the post-CAP world. Clients need to be allowed to be intelligent and engaged in managing their own QoS. The examples above illustrate some of the ways that can happen. I am sure there are lots of others. An interesting question is how much of this does it make sense to standardize and what ends up as part of service API definitions. Dynamo's context is a concrete example. Looks like it just rides along in the service payloads so is effectively standardized into the infrastructure.<br />
<br />
<br />
<br />Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com0tag:blogger.com,1999:blog-4528669273992673185.post-29746855862967052432011-09-04T19:29:00.000-07:002011-09-04T19:30:38.348-07:00Scaling DevOpsIn a <a href="http://www.infoq.com/presentations/Cooperation-Collaboration-Awareness">great InfoQ talk, </a>John Allspaw (Flickr, Etsy) presents a compelling argument for really aggressively breaking down the barriers between IT development and operations. The talk presents lots of simple examples that anyone who has ever managed development and/or operations can relate to. It also shows great tech leadership attitude, IMO.<br />
<br />
One thing that Allspaw mentions off hand is that the organizational scalability of the model is still being proved out. It is ironic that the secret behind manageable massive scalability in some of the largest web sites may be small team size. When the whole tech team consists of 20 people, it is not hard to get them all in a room and develop and maintain a completely shared vision.<br />
<br />
There are two questions that I have been thinking about related to this. First, how do you scale it organizationally - i.e., how do you make it work in large organizations with multiple applications? Secondly, while I think <a href="http://en.wikipedia.org/wiki/DevOps">the basic principles</a> of DevOps are great, how do you manage the collision with ITIL and other traditional management processes and structures? I find myself more than a little amused by<a href="http://teddziuba.com/2011/03/devops-scam.html"> this post</a> that basically points out the lack of any kind of blueprint or methodology worked out beyond "break down the silos" and do the obvious in terms of config management, continuous integration and "infrastructure as software."<br />
<br />
The scaling problem here is the same problem faced by "new management" principles such as the <a href="http://stevedenning.typepad.com/steve_denning/2011/01/reinventing-management-part-4-coordination-from-bureaucracy-to-dynamic-linking.html">transition from bureaucracy to dynamic linking</a> advocated by Steve Denning. Part of what is needed is the management equivalent of <a href="http://en.wikipedia.org/wiki/Inversion_of_control">inversion of control</a> - some kind of "execution framework" enabling small, collaborating but decentralized teams to achieve their goals without either having a static master plan or detailed knowledge of everything everyone else is doing [1]. The other is a scalable approach to prioritization, planning and performance measurement. Again without a static top-down process, we need a way to develop plans and measures that maintain direct and strong connection between each small team with the end customer and product. <br />
<br />
The second question is even harder. Allspaw acknowledges for example that trying to throw out traditional change management altogether is a bad idea, even with the greatest, most well-coordinated team. For some changes, you need to, as he puts it "get out the stack of ITIL books" and follow a more traditional process. The problem here is both what to aim for and how to engineer the transformation, especially when the point of departure is a large group with traditional processes and detached, bureaucratic management. <br />
<br />
<hr />
[1] For another way to see the analogy here, have a look at Robert C Martin's, <a href="http://www.objectmentor.com/resources/articles/dip.pdf">Dependency Inversion Principle</a>. Look at the small, self-directed teams as the "lower level modules," senior management as "higher level modules" and the "abstractions" as the higher level goals and values of the enterprise.Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com0tag:blogger.com,1999:blog-4528669273992673185.post-38219953901386303982011-08-19T12:37:00.000-07:002011-09-04T19:36:10.129-07:00Engineering emergent behaviorComplex adaptive systems seem to be all the rage now. This month's <a href="http://hbr.org/archive-toc/BR1109">Harvard Business Review</a> has several articles on managing complexity. It has become fashionable to use computational complexity to explain failure or lack of engineering control in diverse areas, ranging from <a href="http://www.cs.princeton.edu/%7Erongge/derivative.pdf">derivatives pricing</a> to <a href="http://www.technologyreview.com/blog/arxiv/27068/">strong AI</a>.<br />
<br />
All of this is fun and interesting, though not exactly new. It was new in the 1950's when <a href="http://en.wikipedia.org/wiki/Edward_Norton_Lorenz">Lorenz</a> demonstrated the impossibility of predicting the weather. Or in the 1970's when the "<a href="http://books.google.com/books?id=0E667XpBq1UC&pg=PA87&lpg=PA87&dq=chaos+cabal+uc+santa+cruz&source=bl&ots=VVbHJg79yg&sig=Rk9OQw6fWwCkQpRNU_LIAV0azFg&hl=en&ei=LpVOToicNu7YiALG8r2NAQ&sa=X&oi=book_result&ct=result&resnum=9&ved=0CFUQ6AEwCA#v=onepage&q&f=false">chaos cabal</a>" began developing the ideas of self-organization and emergent behavior in nonlinear dynamical systems.<br />
<br />
So what does all of this mean to us today? In particular, how exactly are policy-makers and leaders supposed to "embrace complexity?" The HBR articles make some reasonable practical recommendations that I am not going to repeat. What I am interested in thinking about here is the general question of what it means to try to engineer or control emergent behavior and how the systems that we inhabit need to change to support meaningful attempts at this.<br />
<br />
<i><b>Unexpected outcomes are to be expected.</b></i><br />
Emergent behavior is by definition unpredictable. Broad patterns can be anticipated and to some extent engineered; but new and different behaviors need to be understood and exploited, rather than attempting to "minimize variance" or ensure mirco-level predictability. In a recent article on open source product evolution, <a href="http://opensource.com/business/11/8/open-business-practice-effect">Tarus Balog</a> talks about one way to cultivate what might be called <i>emergent value</i> by means of what he calls the "practice effect" - release early, release often and allow the community to work with and extend the product. What this comes down to is presenting the interacting agents that make up a complex adaptive system with opportunities for productive evolution rather than trying to push completely predetermined outcomes through the system. Commercial companies trying to leverage open source are learning the art of how to do this. Open source community leaders are learning lessons that are broadly applicable from this experience as well.<br />
<br />
<b><i>Extreme sensitivity to initial conditions means small changes can create big effects.</i></b><br />
Lorenz famously named this the "<a href="http://en.wikipedia.org/wiki/Butterfly_effect">butterfly effect</a>." This is the essence of why long-term behavior of chaotic dynamical systems is not practically predictable and it is what makes managing complex adaptive systems extremely difficult. The best strategy for dealing with this again comes from open source - move things along in what <a href="http://www.betaversion.org/%7Estefano/papers/ac2006.2.pdf">Stefano Mazzocchi</a> dubbed "small, reversible steps." At the Apache Software Foundation, most projects follow a process that we call "commit, then review." Committers make changes to the source code and then the community reviews the changes. "Patches" (where Apache got its name) get applied visibly in small increments and the community provides feedback. If the feedback is not good, the patches get rolled back. Big bang commits of very large amounts of code or sweeping changes are rare. Proceeding via small, reversible steps and observing effects while reversibility is still possible is a great strategy for dealing with complex adaptive systems when this is practical.<br />
<br />
<b><i>There is no kernel.</i></b><br />
The traditional systems engineering model starts with a "kernel" and builds manageable complexity on top of it. Linux and the TCP/IP protocol stack are great examples. Starting with a stable and solid foundation, we build specialized and advanced capabilities on top. Similarly, governments, policies and organizations can be looked at this way, with a stable, centralized core forming the basis for bureaucratically articulated extension points. Complex adaptive systems have no kernels. Their core dynamics are driven by networks of relationships that evolve dynamically over time. That means that to effectively drive change in these systems, you need to focus not on top-down, centralized programs but decentralized connection-oriented interventions. See <a href="http://ucsd.academia.edu/AlanDaly/Papers/211390/Reform_at_the_Edge_of_Chaos_Connecting_Complexity_Social_Networks_and_Policy_Implementation">this article</a> for an interesting analysis of a practical example in education reform. As a side note, check out<a href="http://www.physorg.com/news/2011-08-internet-architecture-hourglass-future.html"> this fascinating analysis </a>of how TCP/IP has itself evolved (or resisted evolution).<br />
<br />
<b><i>Broad and concurrent search for solutions.</i></b><br />
Engineering well-conditioned, deterministic systems always comes down to posing and solving optimization problems sequentially. Public policy and strategic planning in the command and control model of corporate governance has traditionally mimicked this approach. Form a task force or planning team to analyze a problem. Develop a plan. Mobilize the organization to execute. If things go awry, revisit the plan and try something else. In computer science terms, this could broadly be described as depth-first search. Given the weak and only extremely local engineering control available in complex adaptive systems, this approach tends to fail miserably. What is needed is massively concurrent, breadth-first search. Instead of a centralized, top-down approach to engineering emergent behavior, a loosely federated approach distributed across interaction points gets to better solutions faster. Google works this way both literally as a search technology and by <a href="http://online.wsj.com/public/article/SB114601763677436091-RZdaVtvykRAz4EhCKs0KervA0Eo_20060503.html?mod=blogs">some accounts</a> as an organization as well.Phil Steitzhttp://www.blogger.com/profile/06145686483421887008noreply@blogger.com0