Community Effort Driving Standardization of Apache Spark

New Post Public Reply Private Reply Replies (0) Message Board
News Desk 2018
Community Effort Driving Standardization of Apache Spark Through Expanded Role in Hadoop Projects

PALO ALTO, CA--(Marketwired - Jul 1, 2014) - Open source contributors, Cloudera, Databricks, IBM, Intel, and MapR announced today that they are joining efforts to broaden support for Apache Spark (Spark), while simultaneously standardizing it as the framework of choice by bringing popular tools from the MapReduce world to this new engine.

Spark has quickly become a standard in many Hadoop distributions, with rapid customer adoption and use in a variety of use cases, ranging from machine learning to stream processing workloads. To further support this growth, these five vendors have come together to collectively broaden the range of tools and technologies in the Hadoop ecosystem that leverage Spark as an underlying processing engine.

Today, besides being used independently as a programming language, Spark is used as the basis for several projects including:

1. Spark Streaming for continuous data processing 2. MLLib for a machine learning toolkit 3. GraphX for graph analytics capabilities

In recent months, other projects have also added support for Spark, as evinced by recent efforts to port Crunch, Mahout, and Concurrent's Cascading framework to Spark.

This collaborative new effort expands upon the Spark momentum to include several key Hadoop projects -- starting with the Apache Hive SQL engine (Hive). Using Spark as the underlying execution engine, this effort will improve the performance of batch SQL jobs in Hive, while seamlessly maintaining compatibility with the core Hive code base.

Simultaneously, the group is investigating ways to adapt Apache Pig to leverage Spark, as well as other popular tools, such as Sqoop and Search. By making Spark the execution layer of choice, this group is driving consolidation and standardization around Spark as the evolution of MapReduce for modern hardware.

This effort highlights the power of open source communities, with marketplace competitors coming together to help shape a common execution layer, thus creating a community standard. End users benefit by having a widely supported execution layer, preventing lock-in, while continuing to use their tools of choice. Further, the simplicity of having to manage and learn a single engine reduces operational costs.

Spark is an open source data analytics framework originally developed in the AMPLab at UC Berkeley. Quickly embraced for its inherent advantages, such as improved data processing and in-memory capabilities on Hadoop, Spark offers application performance gains - up to 100 times faster than Hadoop MapReduce for certain applications. Spark has attracted the attention of the open source community and vendors alike.

Hive is a data warehouse infrastructure initially developed by Facebook Inc. and built on top of Hadoop. Hive was created to query and manage large datasets stored across a cluster of servers. Hive continues to remain a popular choice for SQL batch processing and it offers many advantages to customers. There is an active community including enterprise vendors Cloudera, IBM, Intel and MapR, committed to furthering Hive based on cutting edge industry standards.

Supporting Quotes

"The ecosystem of software related to Apache Hadoop is constantly evolving and expanding to include new ways to process Big Data. The rapid changes and additions signify a need among our customers for standardization on platform functionality to simplify the usage of Hadoop for business applications," said Doug Cutting, co-creator of Hadoop and chief architect for Cloudera. "By creating an industry standard with Apache Spark, we help our customers realize a faster time to value from their deployments."

"Since Spark was open sourced it has generated rapid interest -- with over 200 contributors from 50+ organizations collaborating around the project; establishing itself as the most active project in the Hadoop ecosystem," said Arsalan Tavakoli-Shiraji, Business Development Lead at Databricks. "With such a groundswell of support and clear benefits for realizing sophisticated analytics, we believe Spark is the future of data processing on Hadoop."

"Big Data is an inflection point for next generation businesses. IBM is committed to Spark for data processing in Hadoop and to open source technologies across the Hadoop ecosystem. Big data is providing innumerable benefits to our customers as we focus on making Hadoop withstand the demands of the enterprise adding administrative, discovery, development, provisioning, security and governance, along with best-in-class analytical capabilities," said Anjul Bhambhri, Vice President, Big Data.

"We are committed to accelerating the evolution of Hadoop through our strategic alliance with Cloudera and Intel contributions to key open source projects in the ecosystem," said Vin Sharma, director, Strategy & Business Development for Intel's Big Data Solutions. "Spark is delivering one of the most exciting innovations in data processing into the Hadoop platform. By working with the developer communities to establish Spark as the back-end for Hive, Pig, and other Hadoop projects, Intel aims to bring the latest advances in memory and processor technologies into widespread use by enterprises grappling with hyperscale analytics."

"We believe Spark on Hadoop is a game changer for any business and moves Hadoop closer to supporting real-time, operational applications." said M. C. Srivas, CTO and Co-founder of MapR Technologies. "MapR is dedicated to collaborating, contributing to and supporting these open source projects and customers will benefit immensely from the Hive and Spark communities coming together to significantly enhance Hadoop and move it forward."

Cloudera, Databricks, IBM, Intel, and MapR agree that by bringing Spark more widely to Hadoop communities, the outcome will be a rich and unified ecosystem that will deliver the next level of performance in Hadoop deployments. A proposal for Apache Hive on Apache Spark has been submitted and companies anticipate work immediately upon its approval. Hive on Spark will work within the context of the existing Hive community and establish Spark as the back end standard to improve Hive performance.

About Cloudera Cloudera is revolutionizing enterprise data management by offering the first unified Platform for Big Data, an enterprise data hub built on Apache Hadoop™. Cloudera offers enterprises one place to store, process and analyze all their data, empowering them to extend the value of existing investments while enabling fundamental new ways to derive value from their data. Only Cloudera offers everything needed on a journey to an enterprise data hub, including software for business critical data challenges such as storage, access, management, analysis, security and search. As the leading educator of Hadoop professionals, Cloudera has trained over 22,000 individuals worldwide. Over 1,000 partners and a seasoned professional services team help deliver greater time to value. Finally, only Cloudera provides proactive and predictive support to run an enterprise data hub with confidence. Leading organizations in every industry plus top public sector organizations globally run Cloudera in production. www.cloudera.com

Connect with Cloudera Read our blogs: http://www.cloudera.com/blog/ and http://vision.cloudera.com/ Follow us on Twitter: http://twitter.com/cloudera Visit us on Facebook: http://www.facebook.com/cloudera

Cloudera, Cloudera's Platform for Big Data, Cloudera Enterprise Data Hub Edition, Cloudera Enterprise Flex Edition, Cloudera Enterprise Basic Edition and CDH are trademarks or registered trademarks of Cloudera Inc. in the United States, and in jurisdictions throughout the world. All other company and product names may be trade names or trademarks of their respective owners.

About Databricks Databricks ( databricks.com ) was founded by the creators of Apache Spark, and are using cutting-edge technology based on years of research to build next-generation software for analyzing and extracting value from Big Data. They believe Big Data is a tremendous opportunity that is still largely untapped, and are working to revolutionize what enterprises can do with it. They are venture-backed by Andreessen Horowitz.

About IBM Big Data and Analytics IBM has established the world's deepest portfolio of Big Data and Analytics technology that spans research and development, solutions and software. IBM has invested $24 billion to build its capabilities in Big Data and Analytics through R&D and more than 30 acquisitions. Today, more than 15,000 analytics consultants, 6,000 industry solution business partners, and 400 IBM mathematicians are helping clients use big data to transform their organizations. For more information about IBM Big Data and Analytics, please visit http://www.ibm.com/big-data/us/en/big-data-and-analytics/ and http://www.ibmbigdatahub.com/

About MapR Technologies MapR delivers on the promise of Hadoop with a proven, enterprise-grade platform that supports a broad set of mission-critical and real-time production uses. MapR brings unprecedented dependability, ease-of-use and world-record speed to Hadoop, NoSQL, database and streaming applications in one unified distribution for Hadoop. MapR is used by more than 500 customers across financial services, retail, media, healthcare, manufacturing, telecommunications and government organizations as well as by leading Fortune 100 and Web 2.0 companies. Amazon, Cisco, Google and HP are part of the broad MapR partner ecosystem. Investors include Lightspeed Venture Partners, Mayfield Fund, NEA, and Redpoint Ventures. MapR is based in San Jose, CA. Connect with MapR on Facebook , LinkedIn , and Twitter .

Press Contacts Deborah Wiltshire Cloudera +1 (650) 644-3900 ext. 5907 press@cloudera.com Alex Koritz Databricks 801-461-9795 alex@methodcommunications.com Faye Abloeser Communications Manager IBM Software abloesser@us.ibm.com 908-770-0762 Beth Winkowski MapR Technologies, Inc. (978) 649-7189 bwinkowski@maprtech.com

Scroll down for more posts ▼

Top 10 Most Recent News Articles

Specialized Metal Makers Become the New M&A Prize Again

Updated Category News Views 44

A forgotten heat-treatment line can suddenly be worth more than a shiny new mill when buyers realize nobody can duplicate its certifications, workforce, and customer approvals overnight. That is the basic truth sitting underneath BGL's Nov. 19, 2026 assessment of the U.S. metals market. The easy headline is consolidation. The useful takeaway is much narrower: specialized...

Continue Reading
Bella Rio's Proposed Preferred B Share Giveaway Plan

Updated Category News Views 46

A founder handing over two million of his own common shares is the sort of detail that makes investors stop stirring their coffee. Bella Rio Consulting, the company behind ROCK'N Vodka and Bella Rio Bourbon, says co-founder and Vice President Andy Roiniotis proposes to contribute 2,000,000 personally held Common shares for reclassification into Preferred Series B shares....

Continue Reading
COFE+ Barista Test Puts Robotic Coffee on the Clock

Updated Category News Views 40

A swan latte taking 18 seconds less is not nothing, even if it is hardly a verdict on the future of coffee. COFE+, the seventh-generation robotic barista from Shanghai Hi-Dolphin Robot Technology, beat an award-winning human barista across five drinks at a live November 14 competition in Yiwu. That is a clean, visible demonstration of what machines are increasingly built...

Continue Reading
NEWWAVE Turns 15 and Chases a Bigger Tech Partner Role

Updated Category News Views 97

Fifteen years is about when an outsourced engineering shop has to prove it can be more than extra hands on a keyboard. NEWWAVE’s new identity and “Trusted Technology Partner” positioning are an attempt to make that jump: from carrying out software work to helping clients decide what work ought to be done, then sticking around after launch. It is a sensible ambition,...

Continue Reading
UnitedHealth’s CMS Audit Shock Puts Disclosures Under Lens

Updated Category News Views 123

Medicare Advantage's audit bill just landed with a nasty thud. UnitedHealth Group shares fell more than 10% on November 18 after the Centers for Medicare & Medicaid Services proposed new anti-fraud, audit, and repayment rules for the program. For holders of NYSE:UNH, this was not some sleepy policy memo buried in a federal filing cabinet. It was a hard reminder that...

Continue Reading
CATL Builds Türkiye Battery Service Muscle in Istanbul

Updated Category News Views 82

Battery warranties look mighty impressive until a pack needs work on a Tuesday morning and nobody nearby can touch it. CATL's new NING SERVICE Experience Center in Istanbul goes straight at that unglamorous problem. The facility, announced November 19, 2026, is described as Türkiye's first dedicated battery depot repair center and an integrated sales-and-service hub. It...

Continue Reading
AltaPointe PATH Outreach Puts Housing Waits in Focus

Updated Category News Views 128

Two years is a brutal length of time to ask somebody without stable housing to wait for an income-based apartment. That number hangs over AltaPointe Health's PATH program in Mobile, Alabama, and it ought to. The organization helped 16 people secure permanent housing in the past fiscal year, a figure that can look small if somebody is just skimming a report. It looks very...

Continue Reading
CCTV-4 Revisits China's Enduring Academy Tradition

Updated Category News Views 57

Stone courtyards, old books and hard questions are getting prime time on CCTV-4, which is a more interesting programming bet than it may first appear. Enduring Chinese Academies, a ten-episode cultural exploration series from China Media Group's Global Chinese Programming Center, is built around a simple theme: “One academy, one spirit.” It begins airing November 22...

Continue Reading
We Are All Human Gala Puts Latino Leadership Center Stage

Updated Category News Views 31

Recognition can sound like soft stuff until a room full of decision-makers decides whose work deserves the microphone. That was the real point of We Are All Human’s sixth annual Gala at Cipriani Wall Street in New York on Nov. 20, 2026: put Latino leadership, cultural influence and economic contribution in plain view rather than treating them as a decorative footnote....

Continue Reading
Colossal's Halloween Campaign Wins Top Communicator Honor

Updated Category News Views 76

A horror-themed fan competition putting money toward hospitalized children is not the usual awards-season material, which is precisely why Colossal's latest recognition lands with some weight. The Phoenix-based professional fundraiser received an Award of Excellence, the top honor at the 32nd annual Communicator Awards, for its Face of Halloween campaign in the Innovation...

Continue Reading

Top 5 Most Recently Viewed Articles

Oncotelic Therapeutics 2024 Financial Year Report Summary

Updated Category News Views 219

Oncotelic Therapeutics: A Year of Growth and Development Oncotelic Therapeutics, Inc. (OTCQB:OTLC), a pioneer in innovative treatments for rare diseases, is celebrating a successful fiscal year 2024. The company, known for its work in serious conditions such as pancreatic cancer and pediatric tumors, has reported significant advancements in its clinical and operational...

Continue Reading
OPC Energy Financial Results Coming: What Should You Know?

Updated Category News Views 262

Eyes on OPC Energy's Upcoming Financial Reveal Don’t sleep on March 12, 2026, folks. That’s the day OPC Energy Ltd. (TASE: OPCE) is ready to spill the beans on their fourth quarter and full year financial results for 2025. This isn't just another run-of-the-mill earnings call; it’s a pivotal moment that could set the stage for the company’s future in the energy...

Continue Reading
Zettabyte Appoints Sam Lawn as CFO to Advance AI Solutions

Updated Category News Views 219

Sam Lawn Takes Charge as Global Chief Financial Officer at Zettabyte PALO ALTO and TAIPEI — Zettabyte, an innovative leader in AI solutions with a focus on GPU infrastructure, has officially appointed Sam Lawn as the Global Chief Financial Officer, effective immediately. With this strategic move, Zettabyte aims to bolster its financial leadership and operational...

Continue Reading
AI Workforce Readiness Gap Splits C-Suite, Says Survey

Updated Category News Views 27

CHROs Sound Alarm on AI Readiness Here's the skinny: There's a growing split in the boardroom about rushing headlong into AI. While loads of execs are chomping at the bit—believing AI's about to boost everything from bottom lines to top lines like nobody's business—CHROs aren't so sold that their teams can handle the fast-paced AI implementation. They're seeing the...

Continue Reading
Tejon Ranch Co. Delivers Positive Results in Q3 2025

Updated Category News Views 183

Tejon Ranch Co.: A Review of Q3 2025 Performance Tejon Ranch Co., a prominent diversified real estate development and agribusiness company, has recently reported its financial performance for the third quarter of 2025. The company showcased a notable turnaround from the previous year, presenting a story of resilience and strategic management. Financial Highlights of the...

Continue Reading