Scaling Appsec at Netflix
The Application Security team at Netflix is responsible for securing the applications that help run the Netflix business and the streaming product. Our customers are primarily engineering teams that produce software deployed within our cloud infrastructure. In addition to this, we run the Netflix bug bounty program and provide product security incident response capabilities.
The Netflix cultural values of ‘Context not Control’ and ‘Freedom and Responsibility’ highly influence how we do Security at Netflix. Our goal is to enable Netflix engineering teams to build secure software while providing them the appropriate security context to make decisions. You can read more about the impact of Netflix culture on our security approach here.
One of Netflix’s current technology challenges lies in the design and creation of an app ecosystem that enables the Netflix Studio to scale as we create original programming around the world. As our engineering customer base grows, the value of automation to scale ourselves is increasing.
Our team’s work falls into three categories:
- Operational Appsec capabilities — This includes traditional Appsec activities like bug bounty triage, pentesting, threat modeling, vulnerability management, and product security incident response.
- Security Partnerships — Security Partnerships are aimed at driving holistic security improvements to drive down risk.
- Appsec Automation — Appsec Automation aims to build a comprehensive app inventory and enable self-service security guidance.
Operational Appsec capabilities are valuable but highly interrupt-driven. This doesn’t align well with focused engineering work needed to scale our services. Over the past few quarters, we reorganized our team in two squads based on focus areas to manage our work better. The team is now organized into Partnership and Automation squads. The Operational Appsec work is shared by both squads as a weekly rotation for team on-call. This provides the team the ability to do focused work while sharing the interrupt load.
The goal of the Appsec automation squad is to provide consistent, actionable, self-service security guidance to developers. We aim to have a single view for developers for all actions needed to keep their applications healthy from a security standpoint. Netflix engineering invests in the concept of an Infrastructure and Security Paved Road. This provides well-integrated, secure by default central platforms to engineers at Netflix so they can focus on delivering their core business value. A great example of our infrastructure paved road is Spinnaker, our Continuous Delivery Platform. Spinnaker enables developers to release software with a high velocity without having to directly manage their cloud resources. Similarly, there are security paved road solutions for authentication, authorization, secret storage, and TLS certs. We believe that driving adoption of these security controls reduces more application risk in our ecosystem than vulnerability remediation does.
In the past, we have primarily invested in automation for vulnerability identification (static code scanning, dynamic testing, grep for anti-patterns, etc) in line with common “DevSecOps” approaches. Lately, we have shifted focus towards driving adoption of the Security Paved Road practices across our application inventory. We believe that growing our automation capabilities in measuring security controls adoption will complement our vulnerability identification services for risk reduction. We continue to invest in higher signal vulnerability identification work for remediating vulnerable third party software. Our self-service guidance views surface open vulnerabilities as well as security paved road controls that need to be implemented based on app risk. We consider various factors in determining app risk, e.g.: exposure to the internet, business criticality, types of users, types of data it handles etc.
While our goal is to serve the majority of our customers through our self-service guidance, there will still be certain parts of the ecosystem that need white glove security engagements. The partnership squad partners closely with engineering and product teams that pose high risk (e.g. Payments Engineering) or have the potential for high leverage security work (e.g. driving secure by default for infrastructure paved road). The goal is to identify security risk areas and focus on larger strategic initiatives to drive down risk, as opposed to remediating one-off vulnerabilities. Check out this talk for more details on how our partnership work is executed.
The per-app security assessments approach does not scale in our ecosystem anymore. We are investing in secure by default frameworks and actionable self-service to make security more usable and transparent to developers. In the long term, we want more of the ecosystem to adopt secure by default paved road solutions to reduce security risk. This would allow us to focus partnerships further on harder, high leverage security problems that don’t lend themselves well to automation.
Our goal with this strategy is to step back from our operational responsibilities and focus our efforts on high leverage activities both in our automation and partnership charters. We will continue to gauge the success of this approach with signals like the adoption of security paved road practices and risk reduction from paved road adoption. We are still early on this journey and would love to hear any feedback from our peers in the industry. If this approach to scaling Appsec is exciting to you, check out open roles on our team.
