With emerging ideas, innovation, and talents, the lines
between DevOps, release engineering, and even security are rapidly blurring. I
invite you to sit down for a moment with Principle Consultant, J. Paul Reed,
and listen to his take on what the intersection between these once
individualized fields entails, and may even foreshadow.
morning, Paul. There’s a lot those pursuing DevOps can learn from Release
Engineering practices. I know you’ve got a lot of experience to share, so let’s
J. Paul Reed: Good morning, it's
good to be here. My background is release engineering, although these days I am
actually called a DevOps consultant. I have about 15 years experience doing
that. That's what my
presentation is about: sort of the intersection between DevOps, Rugged
DevOps, and release engineering and wanting to explore that with the security
and Rugged DevOps communities.
Derek: In your
presentation, you touched on the culture between security and DevOps and also
release engineering -- that a number of organizations have challenges with and
that's the Culture of No. There's a lot of, "Hey, we want to move faster
at higher velocity. We have new requirements that we're trying to push out to
market, and we have these new practices that we're moving forward with. Can
security come and play with the DevOps team?"
J. Paul: I
actually put up a tweet that a lot of people liked on one of my
slides: "If your answer to every question is ‘no,’ do not be surprised
when people start pouring effort into ways to not even ask." This idea
that if your answer to everything is “no,” then that is seen as a bug or a
blockage like on the Internet, and organizations will just route around it. I
think security found that out in a very visceral, hard way. In release
engineering, it’s the same thing.
“If your answer to every question is ‘no,’ do not be
surprised when people start pouring effort into ways to not even ask.”
One of the reasons that Git became so popular
is because developers didn't have to ask if there were permission to create
branches. They created an entire infrastructure and ecosystem around not having
to ask. I think that's one of the risks we run, and it's one of the
One of the interesting things we’re finding with DevOps ...
because that idea of getting new traction and people do want to move faster …
is we can frame the work that we do in the context of that pipeline. By
identifying and optimizing some of the business value that is part of that
pipeline, businesses are receptive. Developers are receptive. Different parts
of the business are receptive in ways I've almost never seen in my career, and
it's great to be a part of that. From a Rugged DevOps or security perspective,
I think if we could move that work into the pipeline, not only do we make it
visible in terms of the costs and trade-offs, but then also we could possibly
do more. It's part of that whole. There are lots of presentations talking about
this idea of shift left … that you can shift that work from your perspective
further up into the stream so that you can address it earlier and actually have
a chance at fixing the problem.
In talking with Josh
Corman and a lot of the Rugged DevOps people, they always talk about how at
the end of that process, they would rubber stamp: “Yes, this is secure.” Because
even if wasn't really secure and it was bad, what were you going to do? As a
release engineer, that resonated with me because we felt like that all the
time. We were kind of doing a bunch of work at the end, and there was no time
to do it right. So a lot of times, it was skimped on.
Derek: When you
think about the way traditional security works, how early can we think about
Rugged DevOps shifting left?
J. Paul: Yeah, I
don't think it's so much about getting everything right at the beginning, per se.
I think that the question is how far forward can we shift into that process. I
think if you can shift that all the way to the beginning, that is possible. The
beginning is where you define your pipeline.
A lot of people define that pipeline as commits, that is
developers writing code. Some people will define it actually at the product
management stage, so even earlier than that. Or that kind of agile story phase,
I think you could certainly integrate it there. This is sort of what I was
exploring in my presentation. I open with the slide on what is the intersection
of release engineering and Rugged DevOps, and I say I don't actually know. It's
a very emergent field.
“There's no shortcuts to production...They put the
financial resources and the engineering resources into building the pipeline
that moves code quickly through it.”
I spend the next few slides talking about sort of the
crossover in making that bar. There are a lot of similarities there. I think
when you're talking about pushing that stuff forward, it’s about the more tools
that you can make part of that pipeline, like release engineering tools. So for
us, that might be something like: How do we track what developers create as
dependencies in the work that they're doing? So how do we make that a little bit
easier in time for them to say, "Yeah, I'm using this version of that, and
it's integrated here from a release engineering perspective." Then from
the security perspective, you can take that information and use it to do
different types of security testing or penetration testing. If you can move
that earlier in the process, that's what it will do. Then how early you do that
really is a function of how good you get at this sort of thing.
I don't think we've seen this with security entirely yet.
We're still recognizing the value with release engineering and companies are
hitting it out of the park. They just put everything in there and continue
delivering pipeline. There's no shortcuts to production. There's no back door
to get stuff deployed. They put the financial resources and the engineering
resources into building the pipeline that moves code quickly through it. Then
once you do that, you can augment that pipeline with more and more features, if
you will. One of those might be moving security way forward in that process.
Derek: Are there
old ways to do things that just won't work in the new universe and you have to
adopt new tools or practices?
J. Paul: I do
hear a lot of, "Well, we can't do X because of Y" -- “Y” being one of
those cut old ways that you're talking about. One of the things we continually
see at conferences is idea of the answer being, “We can't do X because of the
old way.” In fact, in security, you see this all the time: "I can't do X
because of audit compliant stuff." But case study after case study says:
If you're willing to rethink the framing on the way you do audit compliance and
work with your auditor -- if you're willing to look at the problem slightly
differently -- then you can achieve those results. Because we have all this
proof, when people say, "Oh, we can't do X because of the old way,"
my question is, “Are we thinking of the problem in an old frame or in a more
traditional framing that is not sewn enough?”
Now that's not to imply the concerns people bring up are invalid.
That's the initial question that you had, which was about people. If they have
a lot of knowledge, they might be worried, "Well, I can't automate things
this way as well as I can test them." I talked in my presentation about
how release engineering is undergoing a fundamental shift. I'm very upfront
about the fact that if you are a release engineer and you are not building
continuous delivery pipeline and involved in the support and service of that
continuous delivery pipeline, your job is probably not going to be there in
five to 10 years. That's just the way the world works. A lot of people think,
"Oh, okay, that's unfortunate or whatever."
“If you are a release engineer and you are not
building continuous delivery pipeline and involved in the support and service
of that continuous pipeline, your job is probably not going to be there in five
to ten years”
I'll give you a QA example that I thought was really
Organizations spend a bunch of time automating a test and
the initial response is, "Well, if you automate all of those tests, what
are the QA engineers going to do?" It turns out that because QA engineers
are so good at looking at a product and coming up with the requirements, they
need a lot of that totally valuable knowledge forwarded into the value stream.
They are having those QA engineers doing requirements analysis and working with
product management to firm up the actual requirements that go into the continuous
delivery pipeline. What was fascinating about it was that it's not that the
organization was, "We are going to automate you out of a job and then
we're going to fire you, so go automate yourself into a script." People
are like, "I'm a person, not a machine." You have that whole
conversation, and they end up doing more interesting work.
They put them working on that continuous delivery pipeline
in the requirements analysis. It's totally different than what you might
expect. It's going to be the same with security and release engineering. For
security especially, we're going to see a lot of that work go. There's a set of
compliance work you can do in an automated fashion. Once that is automated, I
see a lot of discussion about red team, blue team … kind of wargaming type of
thing. And it frees up time to do that and to work as a team in that way.
Because you can't automate all those things, or at least today you can't. I
think everybody in the security space would agree that it's more interesting
work than running around, if you’ve got a huge project, with a black binder
with a bunch of rules.
Derek: One of the
concepts that really resonated with you was the software supply chain. How does
that concept fit with doing release engineering right and doing Rugged DevOps
right or incorporating security into DevOps?
J. Paul: Yeah,
the supply chain idea is something that was fascinating the first time I heard
it. In fact, it's one of the things that Josh and I spent a bunch of time
talking about it when we first met. I think it's a great way to frame a
problem. I'm sad that I didn't think of it, actually, and the reason is because
release engineers think about that all the time. We've thought that was our
role for 20 to 30 years, for as long as release engineering has been around.
It’s this idea of knowing what the dependencies are, dependency management
tracking and trying to make sure that you don't pull in bad dependencies --
whether they are tainted because of the license or containing malicious
software. This problem has only gotten worse with open source software, and
that's also something that from a supply chain perspective we talk a lot about.
“I told this story about an engineer who was missing a
DLL from the build. They just Googled for the DDL and downloaded it, and threw
it on all the build machines. That was pretty scary.”
That was one of the things that I wouldn't think keeps
release engineers up at night as much as it keeps security engineers up at
night. Where is our software coming from, and what issues may it have in it?
That's not something traditionally developers, for whatever reason, seem to
think about and that's not to denigrate them. A lot of times they're under
deadlines, like we are. They go to the Internet. They grab whatever version of
the library. In fact, the one I usually see is the upgraded version because
there's some API that they need or something like that. There's a concern
there, when you think about it, of where that's coming from. I told this story
about an engineer who was a missing DLL from the build. They just Googled for
the DLL and downloaded it, and threw it on all the build machines. That was pretty
One of the slides in the presentation I think is really
critical is: “If you have one vulnerable library in your product, that is a
security problem. If you've got multiple versions of the same library and
multiple versions of those are vulnerable, that's a release engineering
problem.” That's one of the best ways upfront that release engineers can
contribute to Rugged DevOps and contribute to the security space in terms of
helping to detangle that problem. More interestingly, once you've detangled
that problem, you have to figure out how to make it so that that just doesn't
turn into spaghetti again.
I've detangled that problem multiple times usually, by the
way, not so much in a security context but in a licensing context.
“If you have one vulnerable library in your product,
that is a security problem. If you've got multiple versions of the same library
and multiple versions of those are vulnerable, that's a release engineering
The way you do that, again, is shifting left. Moving that
forward where you have a way that as developers put libraries into the product,
new code that isn't written by them because there's a dependency there that was
well documented. You can do that audit in kind of a continuous fashion so that
maybe an artifact that you build is a list of library conversion. Then from an
automated security testing perspective, we can compare that against a list of
CD use or known issues.
Derek: I did a
lot of research at Sonatype on the software supply chain and one stat boggles
my mind. Out of the top 100 components companies were downloading, they
downloaded an average of 27 versions of each of those components in a single
year. When you think about the complexity and the technical debt, and if
there's security debt in that at all … you only need 100 parts and yet you're
using 2,700 parts. Why would you ever want to do that?
J. Paul: One thing
I'll point out is that I think the industry's moving in some sense in the wrong
direction. What mean by that is you've got your Java you've built in this in to
make it really, really, really easy. From the command line, you just pick up
libraries from the Internet. Who knows where they came from. Node makes this trivial. In fact, Node was
built around npm, the package manager. All of that is online. In fact, it's
even worse. One of the things I get called in to help with a lot these days is
... and I kind of giggle at this, just because of the dichotomy … people were
so interested in Git for so long because it was like offline Git, offline
commit. It's great, right? You can build offline, and people always use the example
of when I'm commuting home on the train, I can commit blah, blah, blah, and
that was the big reason for doing it.
Now we've moved with Node and some of the tooling around
Java so that our software builds literally require us to talk to the Internet
to download packages. There's this big push for offline operations. But it's
fine that no download needs 68 billion versions of libraries, and everything is
"self-contained." But if you're going to look at a Node package, it's
got versions of those things stocked in there. That's a feature, not a bug.
Right? In certain platforms ... you see this with RubyGems, when the Ruby Gems
site went down, nobody could deploy their web applications. That's a
fundamentally broken engineering design in my opinion. Not that it's easy for
developers to get that. But that our build processes, our deployment processes,
rely on those things. And they rely on us as developers to say, "I want
version 1.2.4 of that library, and that 1.2.4. is the same version that you use."
I posted a slide about versioning -- and that's a very
release engineering problem. As an example, Open
SSL made a mistake in their versioning and instead of bumping the version
like they should have, they repackaged binary. I suspect the reason that they
did that is because they published all the CVEs with that version number and
everybody is like a hawk watching Open SSL. So they couldn't bump the version
number easily. Open SSL can't be flexible in their release engineering anymore
because they've been so traditionally horrible at it. Right? We've made it
really easy to stuff all of those components into our products, but we really
don't know what we're stuffing in there.
If you look at it, we end up worrying about a lot of the
same things. I think a lot of the nuts to crack, if you will, in the Rugged
DevOps community are maybe 50 to 80% release engineering problems.
Strengthening that extra feature of security in there, to make that part of it,
especially with the supply chain, will work really well.
“A lot of the nuts to crack, if you will, in the
Rugged DevOps community are maybe 50 to 80% release engineering problems.
Strengthening that extra feature of security in there, to make that part of it,
especially with the supply chain, will work really well.”
Derek: J. Paul
Reed, thank you very much. It was a pleasure talking to you, I really enjoyed
the conversation. We'll look forward to seeing you again soon.
J. Paul: Awesome.
If you loved this interview and are looking for more great
stuff on Rugged DevOps, I invite you to download this awesome research paper
from Amy DeMartine at Forrester, “The Seven Habits of
As Amy notes, “DevOps practices can only increase speed and
quality up to a point without security and risk (S&R) pros’ expertise. Old
application security practices hinder speedy releases, and security
vulnerabilities represent defects that can leave a company open to cyberattacks.
But DevOps practitioners can leap forward with both increased speed and quality
by including S&R pros in DevOps feedback loops and including security
practices in the automated life cycle. These new practices are called Rugged