- BERT (Bidirectional Encoder Representations from Transformers) is a natural language processing model Google introduced in 2018.
- It reads the words on both sides of a word at once, so the same word gets a different meaning in different contexts.
- BERT turns text into numbers that machine learning models can use alongside other data.
- Common uses include sentiment analysis, chatbot question answering, summarization, and search.
- BERT is a transformer encoder, pre-trained on plain text without labels.
What is BERT?
BERT, short for Bidirectional Encoder Representations from Transformers, is a machine learning (ML) framework for natural language processing. In 2018, Google developed this algorithm to improve contextual understanding of unlabeled text across a broad range of tasks by learning to predict text that might come before and after (bi-directional) other text.
Examples of BERT
BERT is used for a wide variety of language tasks. Below are examples of what the framework can help you do:
- Determine if a movie’s reviews are positive or negative
- Help chatbots answer questions
- Help predicts text when writing an email
- Can quickly summarize long legal contracts
- Differentiate words that have multiple meanings based on the surrounding text
Why is BERT important?
BERT converts words into numbers. This process is important because machine learning models use numbers, not words, as inputs. This allows you to train machine learning models on your textual data. That is, BERT models are used to transform your text data to then be used with other types of data for making predictions in a ML model.
Bert vs Other Technologies & Methodologies
BERT vs GPT
Along with GPT (Generative Pre-trained Transformer), BERT receives credit as one of the earliest pre-trained algorithms to perform Natural Language Processing (NLP) tasks.
BERT vs transformer
BERT uses an encoder that is very similar to the original encoder of the transformer, this means we can say that BERT is a transformer-based model.
BERT vs word2vec
Consider the two examples sentences
- “We went to the river bank.”
- “I need to go to the bank to make a deposit.”
Word2Vec generates the same single vector for the word bank for both of the sentences. BERT will generate two different vectors for the word bank being used in two different contexts. One vector will be similar to words like money, cash, etc. The other vector would be similar to vectors like beach and coast.
BERT vs RoBERTa
Compared to RoBERTa (Robustly Optimized BERT Pretraining Approach), which was introduced and published after BERT, BERT is a significantly undertrained model and could be improved. RoBERTa uses a dynamic masking pattern instead of a static masking pattern. RoBERTa also replaces next sentence prediction objective with full sentences without NSP.
BERT in H2O AI
H2O Driverless AI uses pre-trained BERT models to turn text into features for its models, with GPU acceleration. See the Driverless AI transformations docs.