Optimizing Large-Scale Deployments at LinkedIn with Rahul Gade

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Inhalt bereitgestellt von The Data Flowcast. Alle Podcast-Inhalte, einschließlich Episoden, Grafiken und Podcast-Beschreibungen, werden direkt von The Data Flowcast oder seinem Podcast-Plattformpartner hochgeladen und bereitgestellt. Wenn Sie glauben, dass jemand Ihr urheberrechtlich geschütztes Werk ohne Ihre Erlaubnis nutzt, können Sie dem hier beschriebenen Verfahren folgen https://de.player.fm/legal.

1y ago 27:47

MP3•Episode-Home

Scaling deployments for a billion users demands innovation, precision and resilience. In this episode, we dive into how LinkedIn optimizes its continuous deployment process using Apache Airflow. Rahul Gade, Staff Software Engineer at LinkedIn, shares his insights on building scalable systems and democratizing deployments for over 10,000 engineers.

Rahul discusses the challenges of managing large-scale deployments across 6,000 services and how his team leverages Airflow to enhance efficiency, reliability and user accessibility.

Key Takeaways:

(01:36) LinkedIn minimizes human involvement in production to reduce errors.

(02:00) Airflow powers LinkedIn’s Continuous Deployment platform.

(05:43) Continuous deployment adoption grew from 8% to a targeted 80%.

(11:25) Kubernetes ensures scalability and flexibility for deployments.

(12:04) A custom UI offers real-time deployment transparency.

(16:23) No-code YAML workflows simplify deployment tasks.

(17:18) Canaries and metrics ensure safe deployments across fabrics.

(20:45) A gateway service ensures redundancy across Airflow clusters.

(24:22) Abstractions let engineers focus on development, not logistics.

(25:20) Multi-language support in Airflow 3.0 simplifies adoption.

Resources Mentioned:

Rahul Gade -

https://www.linkedin.com/in/rahul-gade-68666818/

LinkedIn -

https://www.linkedin.com/company/linkedin/

Apache Airflow -

https://airflow.apache.org/

Kubernetes -

https://kubernetes.io/

Open Policy Agent (OPA) -

https://www.openpolicyagent.org/

Backstage -

https://backstage.io/

Apache Airflow Survey -

https://astronomer.typeform.com/airflowsurvey24

Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

82 Episoden

#IT Management #Tech #Industries #Airflow