Friday, September 15, 2023

How not to get breached (too badly)

The other day, someone asked me if I had ever experienced a major security breach. They were shocked when I responded that I had not. That is because I have been a CTO for the past 18 years, including stints at some bleeding edge SaaS companies that were constantly under attack. I have managed many security incidents, but none that resulted in a major breach. While of course it is possible and even likely that dumb luck has played a role in my success, I think that the following things have certainly helped. Each of the imperatives below have been important to me. I have never achieved perfection in any of them, but I have never stopped pushing.  The imperatives are written from the standpoint of a CTO; but anyone who cares about security can use them to help protect their company.

1. Know your risks and be honest and transparent about them

You should always have a top n list of key risks that you are worried about, and n should be less than or equal to 10.  You should have a weekly conversation with your Security officer that reviews each risk, mitigation plans, compensating controls and any help that the security team needs managing it.  It is critical that these conversations be open, honest, comprehensive and engaged. If you can't understand some of the technical security content, you need to study it and keep asking questions until you understand it.  You need to build a culture on your team that does not sugar coat or hide risks and your response to learning about them needs to be consistently positive, supportive and action-oriented.  You need to never appear to be annoyed by risks or angry at those who report them.  You also need to review critical risks at least quarterly with your partners on the executive team.  These reviews need to be open, interactive, transparent and engaging.  Your objective is not to show that you have things under control, to justify investment or to cover your ass.  Your objective is to proactively cover their ass.  If you do a good job clearly representing risks, mitigation plans and compensating controls, you will get funding and leadership support as a side effect. 

2. Focus on actual risk

Don't let compliance, vendors, talking heads or your own cool ideas distract you from systematically reducing risk.  If you do a good job identifying and managing risk, you will achieve compliance as a side effect.  Attackers don't care how "buttoned up" your security program looks or how beautifully illustrated your risk registry is.  What matters is where you are weak and what you have done to compensate for your weaknesses.  Note that this is exactly the same thing that auditors care about.  Try to put every possible cycle into things that significantly reduce actual risk.  Compensating controls can be ugly and low tech, but they can save your ass.  I am certain that some of the ugliest and most cumbersome shims that I have put in while working on permanent solutions have saved me and my companies from great harm.  

3. Pull on every thread

When you have evidence of an attack or vulnerability, make sure that you investigate everything fully. Don't just celebrate having thwarted or annoyed the attackers into leaving. You need to have people whose job it is to investigate security incidents and you have to give them the time, tools and access to do their jobs.  Ask probing questions and don't assume that your initial analysis of an anomaly is correct.  I wrote a few years back about fully solving problems.  That same thinking applies here - especially the part about "widening bugs."

4. Make security@ a welcoming and helpful place

Respond kindly and helpfully to all security-related questions or concerns from employees.  Don't put the burden on the reporter to establish that an issue is worth investigating.  You want them to come back, not to feel stupid or ignore things.  You also want them to learn.  The most important single thing that any company can do to reduce security risk is to develop or acquire high-quality security awareness training and make sure that everyone takes it.  If your company produces any kind of software (including for internal use), make sure to also acquire - and require - high-quality secure application development training.  It's very important that all training, consulting, reviews and standards promoted by your security team be of the highest quality and backed by friendly people who understand your business and are willing to patiently answer questions and help people understand security concepts.

5. Stay current

Attack types and vectors change all of the time. Many of these changes have no impact on your environment, but you need to make sure you see and assess impact of new threats as they emerge. Here again, you need to have people whose job includes monitoring security advisories and assessing their impact on you.  You need to make sure that you understand clearly what the advisories and your team are telling you.  That means you need to devote a significant amount of your professional development time to keeping up with the changing threat landscape.  As a CTO, you have a unique vantage point that virtually nobody else in the company has.  It is critical that as new threats emerge, you personally think through their implications across your technology estate.  Just as you and your team need to stay current on the threat landscape, so your systems need to remain current in terms of patch levels.  If you do not have fully automated, short-cycle patch deployment capability, this needs to be put in place.  Even if it is in place, it needs to be actively maintained and continuously exercised and you need to eliminate the things that can't be patched.

6. Do real diligence on suppliers

Some companies have well-established supplier risk management capabilities.  Even in those companies, however, sometimes the level of technical security diligence is not where it needs to be.  You need to step back from the questionnaires and boilerplate RFP responses and think clearly about where the material risks are with suppliers and dive deeply into what controls you and they have in place to mitigate them. Again, your vantage point as CTO is critical here.  You can't count on somebody else's checklist to see the full picture.  You need to direct your best spidey-sense toward where vendors may be weak and adaptively probe to make sure that you have discovered all of the risks.  Be especially careful with low-cost vendors.  Finally, make sure that you regularly review vendors' security posture.  If a vendor is acquired or the product or service you are using is slated for end of life, that should trigger a special review and contingency plan.

7. Use architecture as a weapon

Everyone knows that it's way less effective to try to add security to a naively implemented system.  On the opposite side of this are systems that deliver security "natively," exposing services and features in a way that is hostile to attackers.  Zero trust and end-to-end encryption are examples of this.  They are baked into the architecture and inherently hostile to attackers.  Really effective, dynamic and fast credential management and revocation is another great weapon.  The same service that allows applications to quickly get keys and secure communications can help you quickly respond to an attack or vulnerability.  Constantly push for the architecture win-wins that make your systems both more robust and more secure.

8. Constantly question privilege

Make least privilege an organizing principle.  Start with your own.  Do you really need production access?  You are a fat target.  Don't have anything valuable on your laptop and don't let your account be a valuable attack vector.   The same applies to everyone in your company and every process running in your environment.  The less they can do, the less valuable they are as attack vectors.  Constantly ask why individuals or processes need the access that they have and constantly prune.  Just like patching, if your environment does not have the automation, test environments or other infrastructure required to stop individuals or batch jobs from having to have weakly limited production access, you need to fix that.  If offboarding is not bulletproof when it comes to system access, make it so.  Now.  This sometimes looks daunting or even impossible to fix.  Trust me, it never is.

9. Don't acquire, decrypt, transmit or store data that you don't need

I often make the joke that the most secure and highly available system is the null system - the system that does nothing.  To deliver business value, systems need to access and process data.  But many can fully achieve their objectives with a much lighter touch on data.  Here I am reminded of an army adage about conserving energy quoted by Colin Powell in his awesome leadership book:  "Don’t run if you can walk; don’t stand up if you can sit down; don’t sit down if you can lie down; and don’t stay awake if you can go to sleep."  Here is my version for data processing:  Don't receive if you can do without; don't decrypt if you don't need to see;  don't store if you don't need to persist;  don't transmit if can omit.  The wonderful thing about modern distributed architectures is that they don't force you to create massive centralized honey pots of raw data.  So don't.  

10. Don't assume that any zone, subnet, vm or other subsystem can be hardened

I saved the best one for last.  It never ceases to amaze me how coming on 40 years now after the famous Gage/McNeally pronouncement that "the network is the computer," people keep thinking that they can somehow wall off little islands of safety and security.  You need to disabuse yourself and your team of that archaic fantasy.  You need to constantly assume that the bad guys are inside "your" network and focus on limiting what they can do once inside.  Like multi-factor authentication, strong crypto keys and end-to-end encryption, network controls are good architectural weapons, but they are just one weapon.  Like the others, they make your estate a more hostile place for attackers and slow them down; but they need the others behind them.

Some of these imperatives might seem to limit or constrain what you can do with your products or how fast you can move.  They absolutely don't have to.  For example, number 9 does not say you can't store any data. It just says you should limit what you store.  And guess what, if you do the hard thinking in system design to limit things to what you actually need, you can move faster and do more.  Once you get hard core about 8, you will also pick up speed and value because the only way to do it is to automate.  The win-wins in 7 really are all over the place and once you get that thinking baked in, first in your own mind and then in your team and entire company, you will get benefits way beyond just being a harder target.