March 24th, 2014

Data Munging in H2O+R

Category: Uncategorized
Fallback Featured Image

Over the weekend we fielded a question from one of our users about the basics of data munging in H2O through R – and it was a good question, so I wanted to share the response with a wider audience – namely you guys.

There are a few quick things about data munging in  H2O+R:
– It often looks and feels like you are manipulating data in R; we designed it to work that way.  However, for all of the trappings of old R, once you've passed data to H2O, all of your data munging is taking place on the H2O cluster, and the information you see is being passed to R through JSON.
– This means that you aren't limited by the ceiling on R's ability to handle data, you are now limited by the total amount of memory you initially allocated to your cluster.
– It also means that some commands should be undertaken with care. For example, it's now possible to manipulate datasets with thousands of factor levels. Asking H2O+R to return a table displaying information from high cardinality factors can produce results of overwhelming volume.
-If it's necessary to pass data back and forth from H2O to R (i.e., if you want to manipulate data in the R environment, and not on the H2O server) the R calls str(), as.data.frame() and as.h2o() are your new best friends*.
In this context as.data.frame(h2o data set) turns data into a data frame in the R environment.  Use it wisely – it's entirely possible to ask H2O to pass R many millions of observations and quickly exceed R's capacities – turning your R session into a paperweight. I highly recommend that if you must take data from H2O into R, that you take only the data you absolutely need- pass in a single column or two, but avoid moving the whole large data set if you can.
On the other hand, as.h2o(R data set) can take data from R and pass it to the server. Here the limitation is the amount of memory allocated to H2O, and frankly, R will top out long before H2O does – so if you can work with data in R, you can surely pass it to your cluster.
Double checking that your data were communicated correctly uses str(data set). It's a good double check when you've passed information back and forth to ensure that factors are still being treated as factors, that NA's have been treated appropriately, and all of the other little details that are important moving forward with analysis.
*There are examples in the attached R script, as well as the data set for those examples.  It's easy enough to cut and paste or command+enter through the R file, but you will need to 1. Start your own instance of h2o (or at least have one available) before getting to work and 2.Specify your own file path in line 7 – the path that is included is a place holder, and it will only match your path by accident.
Get the Data-     https://raw.githubusercontent.com/0xdata/h2o/master/smalldata/cebexpanded.csv
Get the R script –  https://github.com/0xdata/h2o/blob/master/R/examples/QuickExampleMar24.R

Leave a Reply

H2O.ai Automatic Machine Learning on Red Hat OpenShift Container Platform Delivers Data Science Ease and Flexibility at Scale

Last week at Red Hat Summit in Boston, Sri Ambati, CEO and Founder, demonstrated how

May 14, 2019 - by Vinod Iyengar
6 Tips to Having it All

I posted this blog on Medium two years ago, thought I'd share a slight rework

May 12, 2019 - by Ingrid Burton
AI/ML Projects — Don’t get stymied in the last mile

Data Scientists build AI/ML models from data, and then deploy it to production – in

May 3, 2019 - by Karthik Guruswamy
Hortifrut uses AI to Determine the Freshness of Blueberries

Who doesn’t love sweet, delicious blueberries? Providing a steady supply of beautiful, tasty berries to the

May 2, 2019 - by Ingrid Burton
Fallback Featured Image
Can Your Machine Learning Model Be Hacked?!

I recently published a longer piece on security vulnerabilities and potential defenses for machine learning

May 2, 2019 - by Patrick Hall
Fallback Featured Image
H2O Driverless AI Updates

We are excited to announce the new release of H2O Driverless AI with lots of improved

April 25, 2019 - by Venkatesh Yadav, VP Customer Success

Join the AI Revolution

Subscribe, read the documentation, download or contact us.

Subscribe to the Newsletter

Start Your 21-Day Free Trial Today

Get It Now
Desktop img