DTCC Connection met with Marty Mroz, DTCC Managing Director of Enterprise DevSecOps, and Marc Masri, DTCC Executive Director of Enterprise DevSecOps, to discuss how the firm’s IT department stayed on top of the Log4j vulnerability in December, protecting clients and the financial services industry from the biggest threat facing Java applications in history.
Related: Advancing Cybersecurity. Together.
DC: What is “Log4j” and how did its vulnerability pose a threat to DTCC, in particular?
Mroz: Log4j is what many consider to be the most widely accessed core library in Java applications and is used to log error messages. Recently a severe vulnerability was discovered that allowed cyber attackers to break into systems, steal data, and execute other nefarious activities. This zero-day exploit – meaning it requires immediate mitigation – impacts just about every Java application, including the hundreds we operate at DTCC. When we became aware of the threat, our technology and cybersecurity teams came together quickly to mitigate this risk.
“In times of crisis, when systemically important utilities like DTCC are under pressure to guarantee that our financial system remains safe and stable, it’s the automated, efficient, and repeatable processes provided by DTCC’s Information Technology capabilities that blend Development, Security, and Operations that make a major difference.” – Anchal Gupta, DTCC Managing Director of Enterprise Product and Platform Engineering
DC: What steps did DTCC need to take to respond to this threat?
Masri: At DTCC we are acutely aware of evolving cybersecurity risks and it’s our top priority to stay ahead of emerging threats as we stand at the forefront of safeguarding the global financial markets. I am pleased to say that while the Log4j vulnerability is a significant threat, our tried-and-true processes, and technical capabilities that we leverage as part of our normal course of business positioned us for a swift and effective response. Our stringent perimeter defenses provided protection to DTCC as the incident evolved early on. Beyond that, we relied on the hard work and forethought of our technical and risk teams who created the capabilities and tools in our DevSecOps Delivery Pipeline – that we use every day – which enables development squads to react quickly by remediating the risk of impact to our applications from affecting DTCC or our clients.
Related: "What is DevSecOps?" Video
DC: What is the DevSecOps Delivery Pipeline and how did it position DTCC to neutralize this threat and protect its critical systems?
Masri: At DTCC, the DevSecOps Delivery Pipeline (DDP) is a set of automated tools and capabilities that support the end-to-end delivery of secure, tested, resilient code and infrastructure, supporting the critical DTCC products and services that power the financial services industry.
The pipeline is used by our developers and engineers every day as our standard suite of tools and automated capabilities enabling us to build, compile, scan, and deploy secure and resilient code as part of our software delivery lifecycle. We were able to scale up these processes to rapidly release code that protected our applications from Log4j, without having to spend time on interim fixes that would have delayed us from fully rectifying this issue in a timely manner. We quickly dispatched our developers to identify the Java code that needed to be changed. Then we made the necessary changes, pushed them through our pipeline, and within hours we had fully compiled, and tested code ready to deploy to production.
Since we already had built capabilities, such as Free and Open-Source Software (FOSS) scanning, into our pipeline, we were never at the point of compromising our risk posture. Our team had total transparency into what was deployed at all times and it’s been a gamechanger for our developers to manage multiple branches of code while navigating high risk situations like this. Having these capabilities and DevSecOps practices in our muscle memory positioned DTCC to react immediately to the threat and then get back to regular project work. Patching so many applications in such a short timeframe would by other means be a project management nightmare requiring manual top-down coordination; instead, everyday heroes rose to the occasion using everyday processes and tools.
DC: How does DTCC’s commitment to enhancing its DevSecOps capabilities protect clients and the industry?
Mroz: I cannot overstate how much of a threat the Log4j vulnerability is to Java applications. It’s one of the biggest security issues that has existed to date, and if exploited, the potential risk to our business would have been severe. Even during peak moments during this remediation, our capabilities performed flawlessly at a much higher scale than typical. For example, on any given day we typically process up to 1,000 software packaging jobs on average. With our Log4j response, we doubled that throughput. Beyond that, we processed nearly 250 emergency changes through the DevSecOps pipeline in December 2021 alone, when we have averaged just two emergency changes in the same month over the past two years.
In December 2021, DevSecOps scaled up to nearly 250 emergency releases in response to the log4j vulnerability. See how this compares to activity from December 2019 and 2020.
Our ability to manage this risk without needing to deploy a set of interim fixes that would cause significant rework is a testament to the vision of our technical teams, including firewall engineers and cybersecurity risk professionals, who ensure DTCC remains well-positioned to act quickly. We have a set of multi-layered defenses that allowed us to respond without compromising the services we provide to our industry clients and partners. It’s reflective of the strong collaboration between our IT and technology risk functions and our collective risk management mindset that enables DTCC to stay ahead of ever-evolving cyber risks.