The Apache Software Foundation Announces Apache®

New Post Public Reply Private Reply Replies (0) Message Board
News Desk 2022
21
The Apache Software Foundation Announces Apache® Airflow™ as a Top-Level Project

Wakefield, MA, Jan. 08, 2019 (GLOBE NEWSWIRE) -- The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today Apache® Airflow™ as a Top-Level Project (TLP).

Apache Airflow is a flexible, scalable workflow automation and scheduling system for authoring and managing Big Data processing pipelines of hundreds of petabytes. Graduation from the Apache Incubator as a Top-Level Project signifies that the Apache Airflow community and products have been well-governed under the ASF's meritocratic process and principles.

"Since its inception, Apache Airflow has quickly become the de-facto standard for workflow orchestration," said Bolke de Bruin, Vice President of Apache Airflow. "Airflow has gained adoption among developers and data scientists alike thanks to its focus on configuration-as-code. That has gained us a community during incubation at the ASF that not only uses Apache Airflow but also contributes back. This reflects Airflow’s ease of use, scalability, and power of our diverse community; that it is embraced by enterprises and start-ups alike, allows us to now graduate to a Top-Level Project."

Apache Airflow is used to easily orchestrate complex computational workflows. Through smart scheduling, database and dependency management, error handling and logging, Airflow automates resource management, from single servers to large-scale clusters. Written in Python, the project is highly extensible and able to run tasks written in other languages, allowing integration with commonly used architectures and projects such as AWS S3, Docker, Apache Hadoop HDFS, Apache Hive, Kubernetes, MySQL, Postgres, Apache Zeppelin, and more. Airflow originated at Airbnb in 2014 and was submitted to the Apache Incubator March 2016.

Apache Airflow is in use at more than 200 organizations, including Adobe, Airbnb, Astronomer, Etsy, Google, ING, Lyft, NYC City Planning, Paypal, Polidea, Qubole, Quizlet, Reddit, Reply, Solita, Square, Twitter, and United Airlines, among others. A list of known users can be found at https://github.com/apache/incubator-airflow#w...he-airflow

"Adobe Experience Platform is built on cloud infrastructure leveraging open source technologies such as Apache Spark, Kafka, Hadoop, Storm, and more," said Hitesh Shah, Principal Architect of Adobe Experience Platform. "Apache Airflow is a great new addition to the ecosystem of orchestration engines for Big Data processing pipelines. We have been leveraging Airflow for various use cases in Adobe Experience Cloud and will soon be looking to share the results of our experiments of running Airflow on Kubernetes."

"Our clients just love Apache Airflow. Airflow has been a part of all our Data pipelines created in past 2 years acting as the ring-master and taming our Machine Learning and ETL Pipelines," said Kaxil Naik, Data Engineer at Data Reply. "It has helped us create a Single View for our client's entire data ecosystem. Airflow's Data-aware scheduling and error-handling helped automate entire report generation process reliably without any human-intervention. It easily integrates with Google Cloud (and other major cloud providers) as well and allows non-technical personnel to use it without a steep learning curve because of Airflow’s configuration-as-a-code paradigm."

"With over 250 PB of data under management, PayPal relies on workflow schedulers such as Apache Airflow to manage its data movement needs reliably," said Sid Anand, Chief Data Engineer at PayPal. "Additionally, Airflow is used for a range of system orchestration needs across many of our distributed systems: needs include self-healing, autoscaling, and reliable [re-]provisioning."

"Since our offering of Apache Airflow as a service in Sept 2016, a lot of big and small enterprises have successfully shifted all of their workflow needs to Airflow," said Sumit Maheshwari, Engineering Manager at Qubole. "At Qubole, not only are we a provider, but also a big consumer of Airflow as well. For example, our whole Insight and Recommendations platform is built around Airflow only, where we process billions of events every month from hundreds of enterprises and generate insights for them on big data solutions like Apache Hadoop, Apache Spark, and Presto. We are very impressed by the simplicity of Airflow and ease at which it can be integrated with other solutions like clouds, monitoring systems or various data sources."

"At ING, we use Apache Airflow to orchestrate our core processes, transforming billions of records from across the globe each day," said Rob Keevil, Data Analytics Platform Lead at ING WB Advanced Analytics. "Its feature set, Open Source heritage and extensibility make it well suited to coordinate the wide variety of batch processes we operate, including ETL workflows, model training, integration scripting, data integrity testing, and alerting. We have played an active role in Airflow development from the onset, having submitted hundreds of pull requests to ensure that the community benefits from the Airflow improvements created at ING. We are delighted to see Airflow graduate from the Apache Incubator, and look forward to see where this exciting project will be taken in future!"

"We saw immediately the value of Apache Airflow as an orchestrator when we started contributing and using it," said Jarek Potiuk, Principal Software Engineer at Polidea. "Being able to develop and maintain the whole workflow by engineers is usually a challenge when you have a huge configuration to maintain. Airflow allows your DevOps to have a lot of fun and still use the standard coding tools to evolve your infrastructure. This is 'infrastructure as a code' at its best."

"Workflow orchestration is essential to the (big) data era that we live in," added de Bruin. "The field is evolving quite fast and the new data thinking is just starting to make an impact. Apache Airflow is a child of the data era and therefore very well positioned, and is also young so a lot of development can still happen. Airflow can use bright minds from scientific computing, enterprises, and start-ups to further improve it. Join the community, it is easy to hop on!"

Availability and Oversight Apache Airflow software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For downloads, documentation, and ways to become involved with Apache Airflow, visit http://airflow.apache.org/ and https://twitter.com/ApacheAirflow

About The Apache Software Foundation (ASF) Established in 1999, the all-volunteer Foundation oversees more than 350 leading Open Source projects, including Apache HTTP Server --the world's most popular Web server software. Through the ASF's meritocratic process known as "The Apache Way," more than 730 individual Members and 7,000 Committers across six continents successfully collaborate to develop freely available enterprise-grade software, benefiting millions of users worldwide: thousands of software solutions are distributed under the Apache License; and the community actively participates in ASF mailing lists, mentoring initiatives, and ApacheCon, the Foundation's official user conference, trainings, and expo. The ASF is a US 501(c)(3) charitable organization, funded by individual donations and corporate sponsors including Aetna, Alibaba Cloud Computing, Anonymous, ARM, Baidu, Bloomberg, Budget Direct, Capital One, Cerner, Cloudera, Comcast, Facebook, Google, Handshake, Hortonworks, Huawei, IBM, Indeed, Inspur, LeaseWeb, Microsoft, Oath, ODPi, Pineapple Fund, Pivotal, Private Internet Access, Red Hat, Target, Tencent, and Union Investment. For more information, visit http://apache.org/ and https://twitter.com/TheASF

© The Apache Software Foundation. "Apache", "Airflow", "Apache Airflow", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.

# # #

Sally Khudairi Vice President The Apache Software Foundation press@apache.org +1 617 921 8656

Scroll down for more posts ▼

Top 10 Most Recent News Articles

FAMID's Six-Month IVDR Win Opens Europe's Market Door

Updated Category News Views 25

A six-month regulatory sprint is the sort of result that makes a crowded diagnostics field pause and squint. DNV has issued an IVDR certificate to FAMID Biomedical Technology for its E7 Oncoprotein Rapid Test Kit, a Class C in vitro diagnostic intended to aid diagnosis of cervical epithelial cell lesions through detection of HPV E7 oncoprotein in cervical cell samples....

Continue Reading
A Parish Memoir That Sends Its Proceeds Back Home

Updated Category News Views 142

Some books arrive with a thunderclap; this one shows up carrying a folding chair, a prayer book, and a memory that refuses to leave. You Give Them Something to Eat, the new memoir from Fr. Michael J Rieder, is built from more than a hundred short chapters rather than one grand, polished life story. That choice fits the material. Parish life is rarely a clean parade of...

Continue Reading
Litmus Gets $475 Million as Music Rights Mature Further

Updated Category News Views 115

Music royalties just got another serious pile of institutional money, and Litmus is holding the shovel. The company announced roughly $475 million of new capital commitments from Carlyle and investment funds managed by KKR, a notable vote of confidence in a platform that did not even exist before 2022. This is not some celebrity vanity vehicle buying a famous chorus and...

Continue Reading
Vusion Lands a 2,000-Store dm EdgeSense Rollout in Germany

Updated Category News Views 150

A drugstore shelf rarely gets anyone's pulse racing, but 2,000 of them wired into a live digital system is a serious retail deployment. Vusion has expanded its partnership with dm-drogerie markt to roll out its newest EdgeSense platform across about 2,000 German stores over the next three years. This is not a glossy one-store pilot with a ribbon-cutting and a few tablets...

Continue Reading
MIND Takes Its Copilot Security Bet Across Microsoft 365

Updated Category News Views 81

Copilot’s promise gets a lot less charming the minute confidential files start wandering around unchecked. That is the knot MIND is trying to untie with its expanded, generally available integrations across Microsoft Copilot, Microsoft 365, Purview, Teams, SharePoint, OneDrive, Edge and Exchange. The Seattle data-loss-prevention company announced the collaboration on...

Continue Reading
Vertex Extends Genomics Pact With $220M in Potential

Updated Category News Views 177

Eight years is long enough to separate a useful science partner from a glossy slide deck, and Vertex’s decision to add four more years with Genomics deserves attention. This is not a splashy acquisition, nor is it a clean revenue number investors can plug into next quarter’s model. It is a deeper commitment to the slow, expensive, wonderfully unforgiving business of...

Continue Reading
Perform.AI Tests Whether Stores Can Sell to Agents

Updated Category News Views 98

Shopping bots are about to become one more fussy customer standing in the aisle, except this one reads machine-facing files, rejects broken checkout steps, and does not care how pretty the homepage looks. Perform.AI has launched a free UCP and ACP Readiness Checker aimed at online retailers that need to know whether AI agents can find products, purchase them, and follow...

Continue Reading
Nightguards and TMJ Appliances Require Different Calls

Updated Category News Views 115

Two clear plastic devices can invite a costly wrong turn. Dr. Terrance O’Keefe’s explanation, published Nov. 18, 2026, draws a line that patients too often blur: a nightguard protects teeth from grinding, while a TMJ appliance is meant to address joint and muscle strain tied to temporomandibular disorders. They may look alike sitting in a case. Their jobs are not...

Continue Reading
Fidelis Puts $140,000 Behind Statewide Postpartum Care

Updated Category News Views 83

A missed postpartum appointment can look like a small administrative failure until it is not. Fidelis Care’s $140,000 round of Here for Your Health maternal health grants is aimed squarely at that unglamorous but consequential gap: keeping new mothers connected to care after delivery, particularly in underserved New York communities. Small Dollars, Very Specific...

Continue Reading
Specialty Chemicals Need Demand Before Capacity Bets

Updated Category News Views 96

Empty reactor space does not get filled because everybody suddenly loves reshoring. SOCMA's 2027 Contract Manufacturing Outlook makes that stubborn fact hard to dodge: nearly 70% of surveyed respondents expect North American manufacturing activity to rise over the next three years, yet 79.5% say stronger, more predictable customer demand is critical before future...

Continue Reading

Top 5 Most Recently Viewed Articles

New Terminal One Partners with Plaza Premium to Elevate Travel

Updated Category News Views 234

New Terminal One & Plaza Premium Group Collaboration The New Terminal One at JFK International Airport has joined forces with Plaza Premium Group (PPG), a leading name in airport hospitality, to create two state-of-the-art lounges. Together, they aim to transform the travel experience at this vital international hub, covering close to 14,000 square feet and promising...

Continue Reading
Market Update: S&P 500 Fluctuates After Holiday Trading

Updated Category News Views 337

S&P 500 Experiences Mixed Results in Light Trading The recent trading session saw the S&P 500 close on a flat note after a day of ups and downs, all in the backdrop of lower-than-usual trading volumes, a trend typical of the holiday breakout. As of the market close, the S&P 500 had dipped by 0.1%, while the NASDAQ Composite mirrored this decrease. In contrast, the Dow...

Continue Reading
Brand Engagement Network Expands Horizons with Cataneo Acquisition

Updated Category News Views 104

Brand Engagement Network Expands Strategic Capabilities Jackson, Wyo. – brand Engagement Network, Inc. (NASDAQ: BNAI) has recently made headlines by agreeing to acquire a German media technology powerhouse, Cataneo Gmbh. This acquisition not only highlights BEN's commitment to growth but also demonstrates its ambition to transform how brands use AI to engage with their...

Continue Reading
Metagenomi Showcases Advanced Adenine Base Editors for Gene Therapy

Updated Category News Views 195

Introducing Metagenomi's Innovative Adenine Base Editors Metagenomi has unveiled its cutting-edge Adenine Base Editors (ABEs) at a recent event, highlighting their impressive capabilities in genome editing. These novel tools achieved over 95% targetability in gene modification, demonstrating exceptional performance and efficiency. Key Highlights from the Presentation...

Continue Reading
Streamlining IT Procurement: SysAid Launches on AWS Marketplace

Updated Category News Views 49

SysAid Now Accessible via AWS Marketplace SysAid, a leading Enterprise and IT Service Management solution provider, has made an exciting announcement regarding its availability on AWS Marketplace. This curated digital catalog is designed to simplify the process of discovering, procuring, and deploying software solutions. With AWS Marketplace having over 290,000 active...

Continue Reading