Edgewise is now part of the Zscaler family. Learn More

DevSecOps: Finding The Rainbow Unicorn

Call to action for security pros — how to stop being a roadblock and become a hero

Today’s typical organization uses over 1,100 cloud applications or services, many of which are developed and deployed internally. All of which represent increased surface area and risk. We live in an app-driven world, and while apps facilitate business productivity, without input from the security team they can introduce unintended and potentially dangerous vulnerabilities.

DevOps has accelerated rapid software/application development and the establishment of the continuous integration, continuous delivery pipeline (CI/CD). Organizations have come to rely on this as a fact of doing business, keeping customers happy, and driving up revenues. Since the DevOps movement first started, security teams have struggled to insert themselves into this faster-paced process. While security should be baked into the development/deployment process (we’ve all seen the wreckage when it isn’t), it’s often not...because security is viewed as an impediment to velocity, slowing down release cycles.

As a way to force the conversation and ultimately the process, the security industry invented “DevSecOps,” injecting both security principles and security individuals into the teams responsible for building and deploying business services. If DevOps is an acceleration ramp for the business, the “Sec” in DevSecOps is intended to be the guardrails, helping companies minimize risk while maintaining speed. However, the reality today is, for most companies, DevSecOps is barely more than lip service or something tacked onto the end of the release cycle as a periodic scan or audit—the antithesis of the continuous integration philosophy behind DevOps. This is due in part to the conflicting mindsets of DevOps (developers and network engineers alike) and security analysts. Security teams need to adjust perspective and move from a traditional “advise and consent” role to an active “enable and refine” role; the principles that have made DevOps so successful can be easily applied to DevSecOps.

Embed security experts within the team

The success of DevOps lies in the fact that while all team members actively participate in all aspects of the release cycle—with developers testing their code and building deployment plans, and operators understanding the features being deployed—there are still subject matter experts (SMEs) within the team whose primary role is their area of expertise. Similarly, for DevSecOps to work, security engineers must be part of the team, not just offer guidance but actively contribute to the release and participate in ongoing discussions and decisions.

To help change the perception of security as a roadblock, the security SME needs to do more than advise and pump brakes; they need to enable and empower fellow team members. Most problems can be solved by simply having the right people in the room at the same time.

For example, a common occurrence in DevOps is the adoption or usage of some newfangled technology or open source third-party library. These are the types of things that (rightly) make security professionals cringe because they can introduce unknowns. By educating team members on risks, then taking it one step further and running vulnerability scans and static/dynamic code analysis, the security SME can support the process instead of just raising red flags. Ideally, it means a change from acting as a security analyst, someone who typically breaks things and identifies vulnerabilities, to becoming a security engineer, someone who actively builds more secure systems, educates colleagues, and operates as part of the team while maintaining a high level of dedicated security expertise.

From continuous testing to continuous security

The continuous testing and continuous integration inherent in DevOps allows development and operations teams to affect small changes and discover issues without breaking workflows. This is achieved through automation—automated builds, tests, and deployments.

Similarly, security within DevSecOps needs to be continuous and automated. This requires the adoption of new security tools that facilitate the process. There are many security technologies available to help enable velocity, including automated static and dynamic application security testing (SAST/DAST), continuous vulnerability scanning, third-party and open source security ratings, and more. Just like with continuous integration, everything doesn’t have to be tackled at once. Make incremental gains and build upon successes, and be sure to pick technologies that can be easily integrated with the existing environment. The embedded security engineer should drive implementation and work with the entire team to understand the results so that the role isn’t reduced to security imploring DevOps to adopt unfamiliar technology.

The continuous nature of DevOps means setting a standard for continuous security monitoring (if the team is to become truly DevSecOps). In a continuous deployment model, applications receive regular updates, plus they may also undergo multiple configuration changes or changes to third-party dependent services, especially in the cloud where the web of dependencies can be unsettlingly expansive. Therefore, traditional or manual point-in-time security assessments are insufficient. Just as alerting systems to monitor uptime and response time for most cloud applications exist, systems should be in place to monitor for newly discovered vulnerabilities or suspicious network activity. These are business enabling—not inhibiting—systems; the goal is not to eliminate security risk completely (though that would be nice), it is to ensure risk is assessed (minimally) at the same pace as business improvements.

Register for our free webinar, "Putting the Monster in a Box: Zero Trust Microsegmentation for Victory," with guest speaker, Forrester Principal Analyst, Dr. Chase Cunningham.

Security that is portable and consistent

Modern development environments use cloud services and containers because they facilitate portability. When an application or workload is developed, its container is configured so that it can be deployed identically (or near identically) across the pipeline: from development, to testing, to staging, to production. Ideally, the work put in place to ensure applications have the proper memory, processor allocation, and other environmental factors needs to happen only once. This simplifies the work of the release engineer and operations.

Unfortunately, security has been a laggard in this arena, due to the incompatibility of security tools with modern infrastructure and the fact that security is often overlaid at the tail of the process. For example, security settings applied to a staging environment often have to be completely rebuilt from scratch on the production environment. This means full testing cannot be done until later stages, and it can also introduce operational problems at each step (e.g., if access rights are not configured properly). As a consequence, cloud applications and environments are often provisioned with terribly over-permissive settings to avoid operational problems. Network and communications controls are the worst offenders, since addresses, subnets, and physical topology are the most volatile aspects of the applications as they progress through the pipeline.

By incorporating modern security tools and practices early and often, DevSecOps can build security policies that travel with workloads and containers and persist regardless of where they are deployed. The fundamental principle of least privilege is crucial to securing all applications, and DevSecOps means applying this principle in the same portable manner as the operational configuration settings.

Flail fast, fail fast

This is a tough area for most security professionals. Security understandably requires a certain level of cautiousness that DevOps teams generally don’t have. This is not to say DevOps is reckless, just that’s its tolerance for incremental change and even failure is higher because steps can usually be rolled back. For security to become a first-class citizen within DevSecOps, it must adapt to this reality and find a balance that does not cast security as an impediment.

For example, when rolling out security policies, instead of introducing every rule at once and causing significant friction and full regression, consider rolling them out incrementally. When implementing a new application scanning tool, consider scanning just one component and refining that process until it is fully reliable before expanding to other components. It’s actually not hard to take most security initiatives and improvements and break them into smaller, less-intrusive tasks, but it does require a shift in security’s mindset. Of course, automation and tooling are key to being successful, but the first shift is that of culture and collaboration.

As the culture shifts, and security—both in people and in process—will become more ingrained into DevOps and the promise of DevSecOps will no longer be lip service. If DevOps is the culmination of the road builders with the car builders, DevSecOps is the addition of the guardrail builders to the team.


Stay on the cutting edge. Subscribe to our blog.

Harry Sverdlove, Founder and CTO

Written by Harry Sverdlove, Founder and CTO

Harry Sverdlove, Edgewise’s Chief Technology Officer, was previously CTO of Carbon Black, where he was the key driving force behind their industry-leading endpoint security platform. Earlier in his career, Harry was principal research scientist for McAfee, Inc., where he supervised the architecture of crawlers, spam detectors and link analyzers. Prior to that, Harry was director of engineering at Compuware Corporation (formerly NuMega), and principal architect for Rational Software, where he designed the core automation engine for Rational Robot.