ALPS blog

by David Fiser

The now-infamous Orion Solar Winds case caused a lot of noise in both the cybersecurity community and the public sphere, while attention on supply chain attacks all over the world has also increased. The attacks themselves are not new; in fact, these types of threats have been prevalent for a while. As all attacks do, they naturally evolve into more advanced forms. In the following article, we discuss previous examples of such attacks and dive into possible next scenarios.

Where are the Weak Links?

First, we establish a concrete definition for a supply chain. The generic definition is “a process of getting a product to the customer.” As we are in the computer world, the product here is developed software. Thus, we first focus on the software development process itself.

Software developers, no matter what methodology they use, should split complex projects into simpler tasks. This is so that tasks can be defined and assigned to specific developers inside the favored issue-tracking software. Then the developer will write a piece of code that they will test and push the changes into source code management (SCM) software.

Figure.1

Figure 1. Outline of development process
The code will be processed by a continuous integration and continuous delivery (CI/CD) pipeline that will execute specific tasks in order to perform builds, tests, or the final deployment of the project choice.

Every step inside our defined supply chain process has its own security risks and impacts. For example, some security breaches might start with malicious users of issue-tracking software. This user could be an angry insider taking advantage of secure information; they could also be someone who uses vulnerability exploitation. These scenarios can happen, but the impact will most likely be low as there are many other steps within the chain. The other parts of the supply chain will most likely be subject to malicious tasks. Therefore, we won’t spent too much time in this scenario, with one exception — authentication.

Since the strength of the whole supply chain is defined by its weakest part, ignoring security best practices (such as password reuse or missing proper authentication mechanisms) means the malicious actor can get access to systems with greater impact upon exposure.

Developers and SCM

Another integral part of the chain is the developer, a person who is needed for writing code. It is important to consider the human side of things as a weakness and possible security risk. We are not proposing to replace the developer with machines, but rather to educate them on the risks that they can face.

Recently, we have seen threat actors targeting vulnerability researchers. From a technical perspective, the interesting fact is that this threat was hidden inside a Visual Studio project file — specifically inside the prebuild event — and spawned a PowerShell-driven payload. There are two notable connection takeaways from that scenario. The first is the level of trust between “peers” and the second (which is most applicable to developers) is bigger source-code adoption and third-party project integrations. One should take care with regard to what source code and which projects they integrate. This requires a need for verification for source of the code, software, plug-ins, or containers used. We previously tackled the problems in one of our previous articles about online coding platforms.

The developer also has access to the next part of the chain, the SCM. This involves credentials, and because the process of repeatedly entering credentials with single access is frustrating, they are stored. Together with credential storage, there is a risk of credential leaks, and the impact is greater when they are stored in unencrypted form.

We can see a common pattern in the DevOps community: Developers will store credentials in unencrypted form (for example, text-based configuration files or environmental variables), and this includes service tokens, usernames, and email addresses.

Figure.2

Figure 2. An example of storing credentials and tokens in plaintext inside “secure” environments
Although it seems that these security issues are excessive, the cliché that the chain is only as secure as its weakest part holds true. Thus, precautions that lower risk (for example, using password vaults), should be worth consideration at least.

Figure.3

Figure 3. An example of usage hardcoded plaintext credentials inside source code
In essence, developers should not push hardcoded credentials to the SCM; for example, when using infrastructure as code (IaC), they should think about credential storage in general.

CI/CD Pipeline

The next part of the chain is the CI/CD pipeline. There is strong competition among software vendors; familiar names include Jenkins, GitLab, TeamCity, Azure DevOps, and others.

The pipeline inside the CI/CD software can divided into the following:

Figure.4

Figure 4. CI/CD pipeline
System Access

From a security perspective, access to such systems has to be limited and not accessible to everyone. It should be hidden in the corporate network, accessible only for users with specified roles and thus, special access rights. Even simple exposure to the internet is a security risk as a vulnerability might exist and be actively exploited by threat actors, as we have already seen before.

System Configuration

Another possible source of security risk and related issues is misconfiguration. Certain configurations make sense for specific environments. For example, an enterprise requires multiple users to have access to the system, but everyone has different roles. In this situation, role-based security should be configured. However, there are traps involved in these configurations, as we previously described in the case of Jenkins. With the rise of software complexity and configuration options, testing for common software misconfiguration issues is highly encouraged.

Obtaining Source Code

We can distinguish two patterns here:

SCM is integrated into CI/CD, as in the case of the Azure DevOps server
The SCM is running on different environments; thus, access tokens or credentials have to be stored inside the system

The way software stores the source code credentials is crucial as their exposure can lead to malicious source code modifications. However, it is notable to mention that we have seen insecure practices used inside CI/CD software and discovered unencrypted credential storage vulnerabilities on 3rd party extensions (as in the case of Jenkins plug-ins).

Build Environment Configuration

The security aspects here are of course related to the passed credentials. There is a risk of passing unencrypted credentials and their storage in the forms of build artefacts. Essentially, some configuration prerequisites contain sensitive information that will generally be left behind after a build.

Build

This is the step of compiling actual source code into target binary form. We have seen attacks on this phase in the Solar Winds case as well as the CCleaner case. There are logical reasons that an attacker would attack during this phase. First, since it is one of the last steps in the supply chain, this means that there will not be much verification afterward. Secondly, the produced executable binaries are typically digitally signed these days. This means that any code modification that will happen after code signing will lack a valid digital signature. The hash of the binary would not match, and because it is encrypted by a private key, it would not be possible to generate a valid hash without its knowledge. This is why the build process is a suitable target for supply chain attacks. However, not all software are shipped with a digital signature.

When we mention digital signing, we have to emphasize the need to keep the private key of the certificate secret. Exposure of this private key would mean that attackers can digitally sign whatever software they want with that certificate, and thus, it must be revoked.

Deployment or Product Delivery

The last part of the chain that we can also include in CI/CD is deployment. First of all, the deployment process can be specified inside the CI/CD software and triggered upon a successful build or test. Thus, architecture information might be present here together with the credentials and other sensitive information, emphasizing the need for their proper management.

The easiest way for an attacker is to replace the whole software product. Malicious code injection is also used and is suitable for certain applications (not digitally signed). As an example, we can see code injection attacks on WordPress-powered websites, where malicious code snippets are inserted as a result of vulnerability or security weakness exploitation.

An executable binary code injection is also possible, but it is more dependent on the compiled binary and an attacker’s skills; also if the binary is digitally signed, the attack is easily discoverable, and not efficient. A simple binary replacement is more convenient for the attackers. However, hacking into a delivery server (servers that allow users to download actual software products) where the binary is stored is not easy task and would require significant resources. Instead, tactics like phishing, scam campaigns, or DNS spoofing attacks have been launched alongside the misuse of known software brands (even security vendors) to deliver malicious or grayware payloads.

Conclusion

This blog post simply outlines the basic issues concerning supply chain security. The problems of these types of attacks are too complex to cover in one post. This is a particularly pressing issue because enterprises are expected to run majority of their workload on the cloud by the end of 2021, as we discussed in our yearly predictions. This means that, in response to the health crisis, many organizations are haphazardly adopting new software without fully understanding the full scope of the security concerns.

To learn more about documented supply chain attacks, visit this GitHub page.