DevOps accelerates everything from code to production. Manual approvals for releases and deployments quickly become a thing of the past. DevOps teams need to embrace security and compliance in their day-to-day operations. This is especially true in a heavily regulated environment. Big risks are not accepted, but DevOps teams in these environments are forced to be super-efficient. It forces us to think about finding the balance between speed and control. A common approach to handle topics like this is the four-eyes principle. In this article, we’ll cover what it is and zoom in into some practical examples on how to practice it in a DevOps-driven environment.
What is it?
Before directly jumping into the details, let’s first demystify something. The four-eyes principle already exists for some time. It’s not a new hype that was born in the DevOps era. Instead, it already existed long before personal computers were common.
Definition
Simply speaking, according to the website of openriskmanual, the definition is:
The Four Eyes Principle (also Two-person rule) is a widely used Internal Control mechanism that requires that any activity by an individual within the organization that involves Material Risk profile must be controlled (reviewed, double checked) by a second individual that is independent and competent.
Given the definition let’s highlight the words in bold:
- Internal Control mechanism: this means it’s a way to control things internally rather than externally. Internally from the perspective of a DevOps team makes sense. Fewer external dependencies speed things up. Great.
- Double checked: connected to a certain risk, the activity that follows the original one needs to be checked by another person other than the individual who triggers the action
- Independent and competent: it means that the person who approves or declines the action should not be biased. He/she needs to fully understand the situation about what is going on and make an unbiased decision to approve or not.
It’s important to clarify this using some practical examples:
Examples from the physical world
Some examples from the physical world are:
- When flying an airplane, safety procedures require two pilots to be present in the cockpit all the time. If one makes a mistake (or turns ill), the other one can and should take over.
- Some materials like nuclear waste require two persons to handle. Think of signing it off before it’s been put to transport.
- In banks, it requires two persons to open a safe which contains a lot of cash or other valuable items like jewelry.
- Double check a certain financial product like a mortgage for a so called “high risk” customer.
In the DevOps world, there is no need to be in the same area. All are virtual, yet the principles remain.
Why do you need it?
The four-eyes principle helps to avoid risks that are too high to go over too easily. From a DevOps perspective, there is not much risk of deploying an application to a developing environment. There are no customers which actually use this environment, only other developers. Thus the risk relatively low. When deploying to a production environment, the risk is high. If a deployment fails and customers do face a broken application they run away. Then the risk becomes much higher.
Practical examples
These examples from the DevOps world help to keep the risks low and speed up at the same time:
- Apply the four-eyes principle when there is a commit of > 100 lines (just an example). The more lines are changed, the bigger the risk might be since a lot has changed. One catch here, even a commit of just one line can have a dramatic impact if it’s incorrect. Consider a calculation that should return 100 instead of 1000 this also has a big impact on the outcome.
- New files added: if a lot (say 5+) files are added to the code repository, it’s useful that someone else takes a look and approves or rejects this. Perhaps the first person accidentally committed local files that only work on his machine (f.e. config files which contain sensitive data), it’s good to reject this commit.
- Suppose a developer changes the code to fix a critical vulnerability Sometimes you want a second person to check the solution to make sure there are no new bugs. This helps the first person to avoid wasting precious time to fix a potential new bug.
Who sets the rules
It’s interesting to decide who sets the rules for these kinds of examples. A simple answer might be: the DevOps teams since they build and operate the CI/CD pipelines and the code repositories.
However, keep in mind that every team might choose a different set of rules and no one else but the team knows those rules. Who is then end responsible for it? Remember, the business representative accepts or rejects risks in terms of security vulnerabilities. If the business representative is part of the team, the rules are still pretty much invisible to the risk (CISO) department.
So, as most issues that deal with risks, it’s good to have the CISO department work together with other (Cloud and security) departments and with the DevOps teams to enforce the rules for these kinds of issues. This way, teams are end responsible, but the rules are governed and also covered from an auditing perspective.
Business rationale
As seen in the previous examples, there are a lot of (daily) interactions. In a DevOps world, there is no time to constantly communicate such decisions to the representative stakeholders.
For example, it’s too slow to let a security officer manually check things and approve or reject. He/she needs to be in the team and understand the context of the application to judge a change which requires finding the balance between a security risk and the potential business benefits.
So there is a business rationale to implement the four-eyes principle.
CI/CD pipelines
In the past, release managers manually decided a certain release was “good enough” to be pushed out to production. Now the release manager’s role is over of very much limited. CI/CD pipeline play a vital role in the four-eyes principle.
The pipeline should have quality gateways that determine if a release is “good enough” to proceed to the next stage. Quality gates include security checks and various tests (integration and performance tests to name a few). Every stage has a more restrictive quality gate to ensure the quality of the release is sufficient. A release manager or other team member can decide to push on when all requirements are met.
Let’s make things practical now to implement this principle into your pipelines.
Jenkins
Classic Jenkins (Not Jenkins X) requires the pipeline input step plugin to enable an interactive input stage in your pipeline. This interactive input step stops your pipeline and requires a person to continue or abort the current pipeline.
The dialog box looks like this:
Ready to go? Proceed or Abort
It’s a pretty basic implementation of the four-eyes principle since there is no hard requirement that “another person” than the person who triggered the pipeline actually decides what to do.
From a code perspective things look like this:
lock('helm-chart') {
input 'Deploy this version?'
stage 'deploy'
node('k8s') {
sh 'helm upgrade --install sample-app ./sample-app'
}
}The following pipeline snippet locks the resources of the current application. Since the input step requires human intervention (timing is unpredictable) it makes sure there won’t be a conflict between multiple pipelines running in parallel. The pipeline asks the user to answer “yes” or “no” to deploy a certain version of the application. If yes, the application is deployed using a Helm chart. If no, the pipeline stops here. Besides this, it is useful to add a timeout to this section so the pipeline stops and is cleaned up when there is no timely response. To read more about this, please check out this blog.
AWS CodePipeline
AWS brings this feature to enable a “manual approval action” to your pipeline. You need to add a Manual_Approval stage to your pipeline script. A snippet looks like this:
{
"name": "manualApprovalStage", "actions": [ { "name": "manualApprovalAction", "actionTypeId": { "category": "Approval", "owner": "AWS", "version": "1", "provider": "Manual" }, "inputArtifacts": [], "outputArtifacts": [], "configuration": { "NotificationArn": "arn:aws:sns:<hidden>:ManualApprovalTopicThe text in bold is most important, since it provides the essentials of the approval stage. Optionally, you can let AWS CodePipeline send a notification to Amazon SNS (Simple Notification Service) when an approval stage is ready to be reviewed. Fill in the elements under the “configuration” element.
Be sure to check out the pipeline reference structure for all options which are available.
Azure DevOps pipelines
Compared to AWS or Jenkins, Azure presents the most advanced ways to control your CI/CD pipelines. Azure uses the concept of Approvals and Gates. You can include pre-deployment and post-deployment conditions in every step of the pipeline which enables a second person to approve or reject a change. This also includes prompts so that users can handle a manual task before the pipeline continues.
Consider the following example: a user (other than the user who started the pipeline) should manually approve or reject the deployment of an application. Therefore you need to edit a release pipeline and switch the “pre-deployment approval on”. In this step, you need to select the users which are allowed to approve or reject the next step of the pipeline.
Other use cases are:
- Let users manually sign out of an application before the deployment continues
- Check for open incidents (in Azure boards or Service Now) before the deployment continues
- Let the pipeline user manually enter a specific parameter (e.g. a number to specify how many containers should spin up to run in parallel)
Since there are so many options available and also integrations with other tools, check out the “control deployments, approvals & gates” webpage.
Conclusion
In this article I highlighted the history and main concepts of the four-eyes principle. I also emphasized why you need to implement it. In a DevOps world, it’s obvious you need an automated implementation. All three major cloud providers (AWS, Azure) as well as popular CI/CD platforms like Jenkins provide ways to help you automate this step. I hope this article inspired you to start experimenting with it.
