ALPS blog

The Nightmares of Patch Management: The Status Quo and Beyond

Original article by Trend Micro

As the value of data increases together with continuous discoveries about what can be done with it, the number of targeted attacks and threats that abuse vulnerabilities in systems has also increased. Vendors and manufacturers have noticed this upshot and have been trying to keep up — to both the delight and chagrin of IT teams.


Figure 1. The number of software vulnerabilities per year assigned CVE numbers

Patching and tracking updates for risk mitigation has become a daunting task for administrators and IT teams, especially since this is only a small part of their everyday tasks in an already thinly distributed team. In ideal scenarios, organizations update their policies every couple of years, alongside the security strategies and objectives that their actionable accomplishments are built on to ensure continuous availability, efficiency, and mobility. It’s also important to note that while small and medium businesses (SMBs) have been observed as catching up — in some ways reportedly performing better at patching and managing their systems on a regular basis than their bigger supply chain counterparts — overall, more can be done to implement and update a patch management policy in light of technological and cybersecurity developments.

A review of the patch management of the past years is provided in the original article.

In general, patching for endpoints have become simpler, automated, and more streamlined compared to past methods. Patching on the server side, however, tells a different story of challenges altogether. The reluctance to immediately implement the necessary measures for servers is mainly due to downtime costs: Applying the patches create a likelihood that deploying these patches immediately breaks the continuity of existing functionalities. Among a number of concerns, these business discontinuity issues come in the form of patches themselves having bugs, introducing new vulnerabilities upon deployment, and failing to address the disclosed gap. Additionally, vendors include “extras” in the patches that might turn out to be potentially unsafe, and might require the necessary reboot of systems. In particular, for industrial control systems (ICSs), reboots for patching are considered problematic due to how they affect public integrated systems.

The cloud, not to mention its respective advantages, has noticeably grown in popularity. Patching has also been less of an issue as cloud service providers (CSPs) have made this service convenient for their subscribers. On the flipside, this growth has also attracted cybercriminals who are interested in the cloud’s illicit potential. In moving their on-premises hosted applications and servers to cloud containers and platforms, there is a possibility that organizations might neglect the necessary security measures when dealing with the differences in structure. Cybercriminals have caught on and found the most profitable ways to catch enterprises off-guard, leaving more gaps open with misconfigurations and container flaws.

Challenges and best practices

Consumer awareness of the need to patch and update operating systems heavily contributes to companies’ growing consciousness with regard to improving their cybersecurity postures despite difficulties in implementing them. The market’s demand for better online security, data privacy, and storage has signaled improvements for automated patch deployment. As a result, this demand prompts manufacturers and vendors to keep up. Observers note that there is a growing industry that aims to help companies and users to deploy patches automatically.

Among the challenges that are often mentioned, the following are common to implementing patch management procedures on a regular basis:

  • Resources. At the most basic level, not all vendors deploy fixes on the same day as Patch Tuesday. As a result, system administrators can’t keep track of all the different release cycles needed for all the systems that are likely connected and used.

In addition, companies have long sounded the alarm on the growing gap in skilled cybersecurity personnel for companies — this problem persists until today and presents a challenge to keeping an adequate cybersecurity posture. The surge in production-related technologies and systems also potentially muddles the tracking necessary for management. While companies’ IT teams and administrators know patching is a fundamental component for handling vulnerabilities, there is also a gap in confidence and training for the identification, analysis, and implementation for vulnerability management.

Best practices. There is yet to be an established standard for informing industry practitioners and IT teams of all release cycle intervals— that is, if they can deploy fixes on a regular basis at all. While not all vendors can automate their releases, information is often available on their respective news or blog sites. Occasionally, vendors are also capable of making email lists, updates, and RSS feeds that can be integrated into specific mailboxes.

Furthermore, rather than seeing patch and vulnerability management as a traditional IT and security operations task, these should instead be considered part of a wider security strategy, especially for DevOps and operations security. An organized inventory is one of the primary procedures for asset management, and keeping track of all business assets that are involved in operations is the only way that these can be monitored for accurate analysis. Traffic and behavior patterns, operating system versions, network addresses, credentials, and compliance checks are just some of the few components that can contribute to accurate monitoring, data analytics, and anomaly detections. Keeping in mind that most patches are made available within hours of public disclosure, immediate implementation via automation can narrow down the risk window significantly, either with virtual patches or permanent updates.

  • False positives and misdirected priorities due to fragmented and siloed solutions. A solution that is focused on one environment (for example, email detection) might falsely flag a vulnerability that is covered by a different solution (such as a cloud server solution), or vice versa. In this simplified example, the disconnected solutions might lead analysts and security teams to chasing false positives or attending to low-risk vulnerabilities while leaving critical gaps open. This can also lead to unidentified vulnerability backlogs that can remain unpatched, leaving systems at a higher risk to exploits and intrusion.

Best practices. Seeing the solutions and issues from a centralized perspective allows for better focus and insight — not just during an incident but also when hunting for the potential threat openings. Utilizing technologies such as security information and event management (SIEM), artificial intelligence (AI), data analytics, and threat intelligence, among others, allows security operation centers (SOC) and IT teams correlated detections across different environments for a bigger context in investigations and responses. This visibility allows for centralized, proactive, and defragmented monitoring, tracking, analysis, identification, and detection of vulnerable areas, thus reducing false positives and accelerating integrated responses.

  • New and forgotten openings due to the foray into cloud solutions. In the rush to offer new services and products, companies have increased customer interaction with applications. However, as they improve efficiency by moving to the cloud, companies lose track of details that address their security postures. When migrating to the cloud, it is crucial for companies to ask some integral questions; for example, are the apps designed securely? Are the APIs safe? Were the apps fixed for flaws? Were the containers properly customized and configured? Are authentication and authorization tokens correctly implemented?

Best practices. Whether in the cloud or on-premises, testing applications and cloud containers with the same rigor and attention remains paramount. It’s important to have a checklist of the governance and structure policies of both containers and apps during their foundational builds to ensure subsequent security and functional efficiency of developments. Developers must ensure that base images are vulnerability-free and scanned in their entirety before deploying into production. They must also establish that base images are clear allowed to build upon, making sure that the packages, binaries, and libraries are secure from the onset.

Companies must also include due diligence on their respective known, high-risk, and common vulnerabilities to ensure that the necessary updates and solutions are responded to. For any analysis prior to subscription, companies should look into their historical data on high-risk windows between disclosures and deployments, change management, deployed patch applicability, and issue and risk management. It is equally important to conduct regular scans from containers to keys, compliance to malware variants, and third-party software to teams. The differences in the upkeep and audit of the infrastructures might be unnerving, but the foundational vulnerability checks and scans can reduce risks in the long run.

  • Remote work and gaps in vulnerability management. While remote work arrangements are not new, enterprises have been challenged to implement at scale for all workers and their devices as a result of the abrupt changes brought on by the pandemic. While it is possible that administrators who considered implementing patches experience difficulties with everything on-site, these difficulties are heightened when both the users and the systems that need patching are geographically dispersed. Some systems require the administrators to physically interact with the machines that need to be updated, while an uncertain timeline and implementation puts more organizations at higher risk. Especially for legacy ICSs, deploying fixes remotely and blindly is almost unimaginable due to the delicate balance and integrations heavily dependent on the feasibility of patching.

Best practices. Not all vulnerabilities are created equal. Out of the total number of security openings disclosed, only a few openings are exploited at large and even less are used for intrusion, as shown in Figure 1. While these numbers are not to be taken lightly, enterprise administrators and users can benefit from the prioritization and identification of the vulnerabilities that need to be patched immediately. Some of the necessary measures include having good vulnerability information sources, system and software inventories, vulnerability assessments, scanning and verification tools, patch deployment measures (for example, agent-based vs. agentless deployments), and continuous improvement plans for risk mitigation.

Although these numbers are not to be taken lightly, company administrators and users can benefit from prioritising and identifying vulnerabilities that need to be patched immediately. Necessary measures include good vulnerability information sources, system and software inventories, vulnerability assessments, scanning and verification tools, patch deployment measures (e.g. agent-based vs. agentless deployment), and continuous risk mitigation improvement plans.

Where patch deployment requires physical on-site interaction, administrators need decision support on which gaps to prioritise from a variety of disclosures. This is supported by the ability to identify gaps found in specific environments and focus on fixing them – for example with virtual patches. For ICSs in particular, this prioritisation becomes essential when system administrators and IT teams weigh up whether to patch systems or follow certain procedures and models.



Featured News