Reasons for Not Patching Systems
As I started another new class in security, one of the discussions we had was about patching systems. This is not a new subject and comes up in just about every security class I have ever taught. In fact, I have had this type of discussion at work. Every security standard and common practice states that systems needed to be patched. We only need to do a quick look at recent security breaches to see that many of them occur because systems are out-of-date or they are running obsolete software. For example, the WannaCry ransomware attack was effective when executed on Windows XP systems not newer Windows systems, https://www.csoonline.com/article/3227906/what-is-wannacry-ransomware-how-does-it-infect-and-who-was-responsible.html. Microsoft end of life for Windows XP was in April 2014, https://www.microsoft.com/en-us/microsoft-365/windows/end-of-windows-xp-support and https://www.bytesolutions.com/windows-xp-end-life-eol-retirement-dates/. Yet the WannaCry attack occurred in May 2017, https://usa.kaspersky.com/resource-center/threats/ransomware-wannacry. We have all heard of the Equifax breach in 2017, it too was executed against an unpatched system, https://www.csoonline.com/article/2130877/the-biggest-data-breaches-of-the-21st-century.html. The list can go on and on. In fact, a CSO report showed that 60% of breaches involved unpatched systems, https://www.csoonline.com/article/3153707/top-cybersecurity-facts-figures-and-statistics.html.
Although it may be clear on why to patch and update systems, it can be rather deceptive on why systems are not patched. To make it clear, if a system is breached, it is no one’s fault except the business management. It is the business management that will make the hard decision on how much money can be spent on security and this decision will dictate what the implemented security system will look like. At the heart of this decision is risk. Risk is the major reason security systems exist. Without risk management, there is no reason to have a security system in place. The first time I read Principles of Computer Security by Arthur Conklin and Greg White I ran across a great formula for calculating a value of security on an asset that helped make it clear on how risk can be assessed on a particular asset. Although Conklin and White called it the Operational Model of Computer Security, I have seen many similar versions in other books using other names. This model is defined as:
Protection = Prevention + (Detection + Response)
I have never understood the reason for the parentheses, but the basic equation is clear. This allows a cost of security to be assigned to an asset which now can be turned into a business case. As I always tell my students from this point there are three options a business can select from. First if the protection cost is less than the value of the asset, it almost always makes sense to implement the security item. However, if the protection cost is higher than the value of the asset, then the management can select one of two options. If the asset requires some sort of protection, because of a business reason, then the first option is to let the risk be managed by someone else. Another way to look at this is that the asset will be insured. Insurance is a great way to reduce the security cost. It also allows a business to pay a third-party business to assume some of the risk for the asset. The second option is for the management to accept the risk. In this case the business will just accept a loss if something happens to the asset. It will be management to determine what to do the asset.
With all this background information it is time to look at the reasons that patches are not applied. There are many reasons not to apply patches, but the following ones are the ones I have seen in the places I have worked. To start with there is usually no patch management policy in place. Just about every security standard requires patches to be done, and in general there is no explicit patch management policy required. This leaves open many holes in how patch management can be done.
It is clear that operating system updates are done automatically from the OS vendor. However, there is a risk that an OS update may break a mission critical system. This problem could be addressed very easy with a lab environment to match the production environment. Years ago, this would require extra hardware so many businesses would not spend the money for a lab. The issue becomes deploying an update into production without any testing. This is never a bad idea, until one of these patches takes down a mission critical system. Of course, this loss of a mission critical system causes major push back from management. When a patch fails and a major push back in made from management, the message to the staff is do not patch since we cannot test the patch first. This no patch process now increases the risk of a security breach. Today it seems inexcusable that a production like environment is not created. The need for extra hardware does not exist any more since virtual systems can be used to test. Even if a business does not have a large enough system for this virtual lab, there are many low-cost ways of creating a test environment in the cloud, then dumping the test environment when done testing.
A common reason I have heard for not patching is if the system is not broken, do not fix it. I cannot put into words what I think when I hear this. That being said I also understand where it is coming from. If a patch as failed in the past and management has made a big push back over it, then there is good reason not to risk getting management involved. Remember if an updated fails just about everyone knows about the failure, but if the system works no one cares. A business I once worked for used a remote access server, instead of VPN connections, for all remote access to internal systems. At some point the remote access server was replaced with a new remote access server, but the old one was forgotten. This old remote access server was not updated and one day it was breached. This is another common mistake I have seen in the past because the old server must be left in place during the transition to the new server. At some point the old server is forgotten and after the transaction is complete the old server is not decommissioned or updated. This is why audits need to be done on a regular basis, but that would require not only documentation but time to do which cost money. This is an operational cost that management down plays to save money in the short term, but can lead to larger problems and expense in the future if there is a breach. The error here is that most business see audits only as a check box for compliance and if compliance is not needed it is not worth the cost.
Another reason a system may not be update, along a similar line, is when the system is internally produced. It is not uncommon for a developed system to use third party tools in its development. It is also not uncommon to have constant pressure on moving development forward not going back to update things that are complete and working. The problem is the same for development as in system maintenance; some of these third-party tools need updating. With the pressure to meet demands of new features, updating tools can been seen as unnecessary work because the current version is just fine. However, there will be a time when all systems will reach an end of life, or need a patch, and that includes third party tools. At these times the development process takes a big hit to update any tools. The other issue is that sometimes the update of a tool is a breaking change. A breaking change is one where existing code has to be re-written to fit the new change because the original coding has been redesigned by the vendor. These fixes can take more time since the original intent of the code has to be understood before there can be an understanding of how to allow for the breaking changes to be fixed. I have seen tools out-of-date by as many as four major versions. In these cases, updates are not simple and most of the time will not easily intergrade back into the main source control branch while others are still making enhancements to the main source control branch. This issue introduces many risks to the code base and my experience has shown that more often than not there are few, if any unit tests, to identify side effects to a change and this complicates the update all the more.
I have also seen where developers change code for a third-party tool to allow it to integrate into their system easily. This is a real problem because a breaking change, or any update, can affect the code change the developer made. This is why it is best to find a proper way to use a third-party tool rather than change the code to fix the tool to work the “way we want it to work”. Of course, figuring out how to use a tool properly may slow down development, but it is always better to use a tool as it is designed to be used.
Often it is said that there is a lack of resources to handle patch management. On the surface this may seem real because, in general, IT seems to always be short of resources. If this is really true, the management team needs to find a way to correct this shortage issue; otherwise, it will be only a matter of time before there is a breach. However, I have seen that this lack of resources is created by a poor process. If patches are being handled manually of course it will take more resources. If this is the case then changes to the process should be made before new staff is added. I have seen many cases where there is no automated update process. This is very common in internally developed systems. If an update cannot be made by single click, then more work needs to be done to get a single click deployment. This is at the heart of deployment in DevOps.
I have seen many systems that do an update have to be followed up by someone having to come in afterwards to update configuration files. This is not good for any update process and it leaves that component open to having update issues. In general, an update process should be able to update configuration files as well. The main problem to this process is identifying user changes to configuration verse update changes required. This is a problem that can be solved, but in most cases development timelines are too short to allow for this development. It is common that the update process completed as quick as possible and update problems can be addressed later. This “deal with the issue later” has become a bigger problem with Software-As-A-Service, SaaS, design. SaaS allows a business to control the deployed systems more so many businesses see updates as an internal support issue that can be done later. Again, this attitude goes against DevOps.
In short, commitment to risk management on the highest management levels will inherently lead to a patch management policy that will be properly implemented; which in turn will truly reduce the risk of a security breach.
Other References
https://deltarisk.com/blog/we-dont-need-no-stinking-patches-why-organizations-dont-patch/
https://blog.automox.com/6-reasons-companies-dont-patch
https://www.csoonline.com/article/3025807/why-patching-is-still-a-problem-and-how-to-fix-it.html