March 3rd, 2015

Sparkling Water Certified by Cloudera

RSS icon RSS Category: Uncategorized
cloudera_connect

Last month before the H2O.ai team publicly announced Sparkling Water at Strata San Jose we made sure that the product was backed and certified by some major partners. This includes approval from databricks itself as well as Cloudera.
cloudera_connect_combined_logo

Integration Testing for Cloudera

For Cloudera, testing was mainly geared toward deployment and sustainablility of a Sparkling Water cluster. Some key points that came up during the certification process include:

  • Compability. Because Spark is bundled with Cloudera with each release and the user can choose to install either Standalone Spark or a Spark on Yarn through the Cloudera manager, the test must use the Spark installation that comes bundled with Cloudera. Sparkling Water was also tested on Standalone Spark as well as one launched with Yarn:

    # SPARK_HOME set to Cloudera bundled Spark
    export SPARK_HOME="/opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/spark"
    export MASTER="spark://mr-0xd1-precise1.0xdata.loc:7077"
    cd $SPARK_INSTALLATION
    # Run test script with standalone Sparkling Water
    bin/sparkling-shell --num-executors 8 --executor-memory 5g -i cloudera_cert.scala
    # Run test script with Sparkling Water launched on YARN
    bin/sparkling-shell --num-executors 8 --executor-memory 5g --master yarn -i cloudera_cert.scala

    The cloudera_cert.scala script included a test of import and parse of data, moving data between Spark and H2O, using Spark’s SQLContext and H2O’s Algorithms.
  • Security. Sparkling water had to be able to launch and run on a secure cluster. An authentication layer was added by utilizing Kerberos, an authenication server, on the Hadoop servers, and tests that ran on the unsecured servers had to pass on Kerberized servers. To run Sparkling Shell on Kerberos, the user will first initialize the kerberos ticket and then run the scala script with Sparkling Shell:

    kinit 0xdata@CLOUDERA-CERT
    cd $SPARK_INSTALLATION
    bin/sparkling-shell --num-executors 8 --executor-memory 5g -i cloudera_cert.scala
  • Missing Class Paths. Sparkling water had to be able to run with CDH5.3, and to accomplish this the cluster had to be upgraded from the then-current version of Cloudera from 5.3.0-1.cdh5.3.0.p0.30 to 5.3.1-1.cdh5.3.1.p0.5. Though it was only a minor release update this fixed any class path issues we found in the older version for joda jar files:

    java.lang.NoSuchMethodError: org.joda.time.DateTime.now(Lorg/joda/time/DateTimeZone;)Lorg/joda/time/DateTime;
  • Port Collisions. At the moment if there is a port collision when H2O is launched from a Spark launched on Yarn, executors that can’t find an available port will kill itself and try to restart. H2O however cannot accept executor deaths and so when an executor dies so does the H2O cluster. One way to avoid a port collision for now is to specify the base port you want to launch H2O at by passing the configuration spark.ext.h2o.port.base during the launch process:

    bin/sparkling-shell --num-executors 8 --executor-memory 5g --conf spark.ext.h2o.port.base=63331

Getting Started

To get started yourself, download Sparkling Water and get started with our Github Examples.

Leave a Reply

What are we buying today?

Note: this is a guest blog post by Shrinidhi Narasimhan. It’s 2021 and recommendation engines are

July 5, 2021 - by Rohan Rao
The Emergence of Automated Machine Learning in Industry

This post was originally published by K-Tech, Centre of Excellence for Data Science and AI,

June 30, 2021 - by Parul Pandey
What does it take to win a Kaggle competition? Let’s hear it from the winner himself.

In this series of interviews, I present the stories of established Data Scientists and Kaggle

June 14, 2021 - by Parul Pandey
Snowflake on H2O.ai
H2O Integrates with Snowflake Snowpark/Java UDFs: How to better leverage the Snowflake Data Marketplace and deploy In-Database

One of the goals of machine learning is to find unknown predictive features, even hidden

June 9, 2021 - by Eric Gudgion
Getting the best out of H2O.ai’s academic program

“H2O.ai provides impressively scalable implementations of many of the important machine learning tools in a

May 19, 2021 - by Ana Visneski and Jo-Fai Chow
Regístrese para su prueba gratuita y podrá explorar H2O AI Hybrid Cloud

Recientemente, lanzamos nuestra prueba gratuita de 14 días de H2O AI Hybrid Cloud, lo que

May 17, 2021 - by Ana Visneski and Jo-Fai Chow

Start your 14-day free trial today