July 9th, 2013

Building A TB-Scale Math Platform @ Uberconf 2013, Denver

RSS icon RSS Category: Uncategorized
Fallback Featured Image


Building A TB-Scale Math Platform
Datasets have gotten to PB-scale, but the modeling you can do has been limited to a single-node (e.g. R, SAS) or stuck inside the database or takes hours on Hadoop-like technologies. We have built a simple clustering package, and are using it to do distributed analytics on the sum of all ram in a cluster.

This talk focuses on how the clustering technology, plus a Java-based vector math API, is being used to build full algorithms like GLM/GLMNET, Random Forest and K-means. These algorithms are complex multi-pass programs and traditional distributed programming models expose the distributed boundaries making the algorithms hard to reason about. We have a basic JDK for doing at-scale math, we can run most Plain Olde Java in (distributed) inner loops, communicate via a K/V store with exact Java Memory Model consistency (not lazy consistency). Adding more cpus makes these algorithms run faster, and adding more ram allows larger datasets. We are bringing back Moore’s Law!
Cliff will be presenting.

Leave a Reply

What Are Feature Stores and Why Are They Important?

Machine learning (ML) models are only as good as the data fed into them. In

January 18, 2022 - by Adam Murphy
A Beginner’s View of H2O MLOps

Note: this is a community blog post by Shamil Dilshan Prematunga. It was first published

January 15, 2022 - by Jo-Fai Chow
Shapley Values – A Gentle Introduction

If you can't explain it to a six-year-old, you don't understand it yourself. - Albert

January 11, 2022 - by Adam Murphy
The Bond Market & AI: How MarketAxess Brings it All Together

The vast majority of the equities market trades electronically while the bond market is still

January 11, 2022 - by Ian Gomez
H2O Release 3.36 (Zorn)

There’s a new major release of H2O, and it’s packed with new features and fixes! Among

January 7, 2022 - by Michal Kurka
1st Place Winner’s Blog – Kaggle 2021 Data Science and Machine Learning Survey

Kaggle, the largest global community of data scientists, conducted the 5th annual industry-wide survey that

January 4, 2022 - by Shivam Bansal and KunHao Yeh

Start your 14-day free trial today