June 25th, 2014

H2O – The Killer-App on Spark

RSS icon RSS Category: Uncategorized
Spark H2o

Spark H2o
Summary:
In-memory big data has come of age. Spark platform with it’s elegant API and architecture has captured developer’s hearts. Machine learning as an API for big data is just as real. R and predictive analytics on Big Data has become the center of the space. H2O has established a leadership in scalable ML having focused over the past two years. Spark captured developer’s hearts and minds of developers at the same time.
Sparkling Water brings together best of the both worlds!

H2o_killer_app_spark
Backdrop: Over the past few years, we watched Matei & Ion build a thriving open-source movement and a great development platform for in-memory big data, Spark. At the same time, H2O built a great open source product with a growing customer base focused on scalable machine learning and interactive data science. These past couple of months Spark and H2O teams started brainstorming to bring the best of H2O’s Machine Learning and Spark’s platform. The result is Sparkling Water which brings to Spark the power of of H2O’s fast big data Machine Learning.
Sparkling Water
Users can in a single invocation and process, get the best of Spark – It’s elegant APIs, RDD, simple context, multi-tenancy and H2O’s speed, columnar-compression, in-memory scale and fully-featured Machine Learning and Deep-Learning algorithms.
Easy single-process integration for end-users, reading and writing from Tachyon and RDD is a first step and now available. Data gets parsed and exchanged between Spark and H2O via Tachyon. And a single SparkDriver can setup context and run SQL and ML from same process.
H2o_spark_tachyon
On the longer-term roadmap is H2ORDD which brings the the speed, compression and production-ready in-memory engineering to Spark’s core.
H2o RDD
This allows seamless use of H2O’s Deep Learning and Advanced Algorithms to Spark’s user community.H2O as the killer machine learning application for the Spark Platform will further empower application developers on Spark.
MLLib and H2O: MLlib is a library of efficient implementations of popular algorithms directly built using Spark. Our overarching goal is to see Spark succeed and so we believe that customers should have the choice to select the best tool for meeting their needs in the context of Spark. That’s why we think it is fantastic that Mahout will be porting their algorithms to Spark, and why we’re thrilled 0xData is bringing all the capabilities of H2O to Spark. Overtime, H2O’s ML algorithms and library of legos will accelerate efforts that are started in the community.
We think it is great that we’re moving towards a tighter integration where H2O can be used naturally with the rest of Spark’s capabilities.
What’s next? Sparkling Water code is here:
https://github.com/0xdata/h2o-sparkling
Steps to get it installed and use Tachyon for interoperability are described Installation and Test
Demo Code

object AirlinesDemo extends Demo {
  override def run(conf: DemoConf): Unit = {
    // Prepare data
    // Dataset
    val dataset   = “data/allyears2k_headers.csv”
    // Row parser
    val rowParser = AirlinesParser
    // Table name for SQL
    val tableName = “airlines_table”
    // Select all flights with destination == SFO
    val query = “””SELECT * FROM airlines_table WHERE dest=”SFO” “””
    // Connect to shark cluster and make a query over prostate, transfer data into H2O
    val frame:Frame = executeSpark<a href="dataset, rowParser, conf.extractor, tableName, query, local=conf.local">Airlines</a>
    Log.info(“Extracted frame from Spark: “)
    Log.info(if (frame!=null) frame.toString + “\nRows: “ + frame.numRows() else “<nothing>“)</nothing>
    // Now make a blocking call of GBM directly via Java API
    val model = gbm(frame, frame.vec(“isDepDelayed”), 100, true)
    Log.info(“Model built!”)
  }
  override def name: String = “airlines”
}

Leave a Reply

Fallback Featured Image
Machine Learning in the Age of AutoML

“Just as electricity transformed almost everything 100 years ago, today I actually have a hard

May 15, 2020 - by Parul Pandey
Are All Your AI and ML Models Wrong?

We are living in unprecedented times. Our society and economy are experiencing shocks beyond anything

May 5, 2020 - by James Orton
Lessons of COVID-19 and Moving Forward: Key Takeaways

This week, we hosted our second virtual panel focused on how AI can empower healthcare

May 1, 2020 - by Ingrid Burton
Running H2O cluster on a Kubernetes cluster

H2O is an open-source, in-memory platform for distributed, scalable machine learning. A perfect match for

April 14, 2020 - by Pavel Pscheidl
H2O Release 3.30 (Zahradnik)

There’s a new major release of H2O, and it’s packed with new features and fixes! Among

April 7, 2020 - by Michal Kurka
Brief Perspective on Key Terms and Ideas in Responsible AI

INTRODUCTION As fields like explainable AI and ethical AI have continued to develop in academia and

April 2, 2020 - by Benjamin Cox

Join the AI Revolution

Subscribe, read the documentation, download or contact us.

Subscribe to the Newsletter

Start Your 21-Day Free Trial Today

Get It Now
Desktop img