Data movement for Google services at Netflix

  • Application owners and data scientists were managing security credentials and reusing code to authenticate with Google. They would otherwise have to set up separate GCP cloud projects to authenticate their projects. This made the bar higher for transferring data between Google services and our data warehouse.
  • There was also a huge observability gap about what data was moving from the Data warehouse to Google and vice versa.

Google suite client architecture

Fig 1: Google suite architecture with a Python client accessing Drive APIs and Java accessing Sheets.

Google proxy

  1. The user shares the concerned document with the email address associated with a service account.
  2. The application running inside the Netflix ecosystem has credentials managed by an internal tool called Metatron. This tool also manages credentials on the proxy.
  3. When the client makes a request to modify (or read) a resource on Google the proxy receives the request, verifies the credentials and makes the request to Google on behalf of the client. Results of the request are then returned back to the client.

Client

Fig 2: Google client SDK
Fig 3: Proxy service

Data movement job

Lineage

Fig 4: Lineage logging
  • Source (could be the source table, Google Sheet etc)
  • Destination (destination table, Google Sheet etc.)
  • User / Application information (captured from the Metatron authentication context)

Production

Stay Tuned

Acknowledgements

--

--

--

Learn more about how Netflix designs, builds, and operates our systems and engineering organizations

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Ruby on Rails: HTTP, MVC and Routes

The Automation of Visual Design and How Designers Benefit from Coding

How to perform Joins in PySpark ?

The Mechanics of Kubernetes

EC2 From Zero to Hero

Glossary ● Cloud Native Applications

Occlusion is important in AR

AWS I3 instances are x2.6 faster and 80% cheaper with Redis Flash

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Netflix Technology Blog

Netflix Technology Blog

Learn more about how Netflix designs, builds, and operates our systems and engineering organizations

More from Medium

Selected Readings on Digital Self-Determination for Migrants

Indexing strategies in databases

Dynamics 365/Model-Driven Power Apps: Solving the AADSTS50011 Error

Surface Classification in Automotive Industry