H2O AI and Cloudera
H2O and Cloudera bring machine learning to data managed on Cloudera. Data scientists train models on that data with open-source H2O-3, H2O Sparkling Water on Apache Spark, and H2O Driverless AI, our AutoML, which Cloudera Machine Learning is certified to launch. H2O MOJO scoring artifacts are certified to run in Cloudera DataFlow, at the edge with Apache NiFi or in Apache Flink streams.
Partnership overview
The challenge of machine learning at scale
Organizations have invested in data management, stream processing and AI platforms to create value from their unique data and domain-specific insight. Training machine learning models on big data is complex, compute-intensive and hard to optimize. Deploying those models into a stream processing system for low-latency inference takes significant model and machine learning engineering.
Data wrangling, model training and record scoring often happen in legacy systems that do not scale. Data engineers end up using multiple tools and writing brittle integration code, and data scientists downsample their training data sets to cope with big data.
Integration overview
Data scientists use the familiar R, Java and Python interfaces to train machine learning models on big data in Hadoop HDFS or S3, with open-source H2O-3 on Cloudera CDP, CDH or HDP. This simplifies data science at scale. They can also automate machine learning with H2O Driverless AI, our AutoML, on data managed by Cloudera.
Data engineers prepare and wrangle data for machine learning at scale on Cloudera with Apache Hive, Apache Impala, Apache Spark, Apache NiFi and Apache Flink. Stream processing developers deploy low-latency inference pipelines (H2O-3 MOJO or Driverless AI MOJO) into data flows (Apache NiFi) or streaming flows (Apache Flink).
Scaling machine learning
H2O-3 is an open-source, distributed, in-memory machine learning platform with linear scalability. It runs from R, Python or Java and scales from a single machine to Hadoop or Kubernetes clusters. Its AutoML runs through the algorithms and their hyperparameters automatically and produces a leaderboard of the best models.
H2O Sparkling Water combines the fast, scalable machine learning algorithms of H2O-3 with Apache Spark. Users prepare data in Spark, feed the results into H2O-3 to build a model and make predictions, then use the results again in Spark.
H2O-3, Sparkling Water and Enterprise Steam are certified on Cloudera CDP, CDH and HDP.
Low-latency inference for IoT and streaming
Cloudera DataFlow (CDF) is a scalable, real-time streaming data platform that ingests, curates and analyzes data for insight and immediate action. CDF provides:
- Edge and flow management (Apache NiFi, MiNiFi, Edge Flow Manager)
- Streams messaging (Apache Kafka)
- Stream processing and analytics (Apache Flink)
Real-time predictive analytics and intelligent IoT use cases, such as predictive maintenance, asset tracking, patient monitoring, utility monitoring and smart cities, need machine learning models deployed to the edge or into streaming engines with low-latency inference.
H2O provides an embeddable, low-latency deployment artifact: the MOJO. Both the H2O-3 MOJO and the Driverless AI MOJO are certified to deploy to CDF at the edge (Apache NiFi) or to power analytics in Apache Flink.
Enterprise AutoML
Driverless AI helps data scientists work on projects faster, using automation to finish key machine learning tasks in minutes or hours, not months.
It delivers automatic feature engineering, model validation, model tuning, model selection and deployment, machine learning interpretability, bring your own recipe, time series, and automatic pipeline generation for model scoring. Driverless AI is an extensible, customizable data science platform that addresses a variety of use cases, for organizations in every industry.
Cloudera Machine Learning (CML) is certified to provision and launch Driverless AI and to import data from Cloudera CDP, CDH and HDP.