Artificial intelligence is no longer just for science fiction movies. Thanks to powerful machine learning technologies, you can use AI to add value to your life in a really impactful way. BERT, or artificial intelligence that learns from you and your friends, is one of these ways AI brings value to our lives.
What is BERT?
BERT name stands for Bidirectional Encoder Representation from Transformers(BERT). Google created BERT, a transformer-based machine learning approach for pre-training natural language processing (NLP). Transformer is an attention mechanism used by BERT that recognizes contextual relationships between words (or subwords) in a text.
Transformer comes with two independent processes in its basic configuration: an encoder that reads the text input and a decoder that generates a task prediction. Only the Transformer’s encoder mechanism is required since BERT’s aim is to provide a language model.
BERT is a new approach to learning language representations that provide cutting-edge results across a wide range of Natural Language Processing(NLP) activities.
How does BERT work?
As the BERT’s meaning suggests, it works with pre-training and fine-tuning as the two processes in the BERT architecture. The model is pre-trained using unlabeled data across various pre-training tasks.
The BERT model is initialized with its pre-trained parameters for fine-tuning, and all of the parameters are adjusted using labeled data from the subsequent jobs. Even though they are all initialized with the same pre-trained parameters, the downstream tasks each have their own fine-tuned models.
The pre-training stage uses two strategies:
- Masked Language Modelling(MLM)
- Next Sentence Prediction(NSP)
It is simple to fine-tune BERT. The Transformer’s self-attention mechanism enables BERT to mimic a variety of downstream tasks, by changing out the relevant inputs and outputs.
A typical practice for applications using text pairings is to individually encode the text pairs before using bidirectional cross attention. Instead, BERT combines these two steps via the self-attention mechanism since self-attention effectively integrates bidirectional cross attention between two phrases when encoding a concatenated text pair.
How to use BERT?
BERT models can be fine-tuned to perform several tasks like
Sentiment Analysis
- Question-Answering
- Text Generation
- Named Entity Recognition
- Text Prediction
- Text Summarization
Hugging Face offers a transformers library that gives us access to several open-source BERT models. The ‘pipeline’ class of this library allows us to use several BERT models for various tasks. We will see a few examples of how BERT models perform different tasks.
Firstly, we install the transformers library as shown below
| pip install transformers |
Now, we will see how to perform sentiment analysis using this transformers library by loading a BERT model fine-tuned for the sentiment analysis task.
| from transformers import pipeline Classifier = pipeline(‘sentiment-analysis’, model = ‘distilbert-base-uncased-finetuned-sst-2-english’) result = Classifier(‘I like you’) print(result) |
In the above code, we imported the pipeline class from transformers and created an instance of ‘sentiment-analysis’ pipeline with a custom BERT model. In the next step, we passed a sentence to this pipeline instance for classification and printed the result in the next line. The output of the above code is as follows
| [{‘label’: ‘POSITIVE’, ‘score’: 0.9998695850372314}] |
We can see that the label is ‘POSITIVE’ which means the sentence we passed is a positive sentence and the confidence score is 0.99. This means that there is a 99% chance that the given sentence is positive. Similarly, we can perform other tasks.
Conclusion
In conclusion, the BERT language model, which is extremely sophisticated and complex, aids in automating language comprehension. By using Transformers architecture and training on enormous amounts of data, it can achieve state-of-the-art performance and revolutionize the NLP industry.
The future of unfinished NLP milestones appears promising thanks to open-source BERT libraries and the amazing AI community’s efforts to keep enhancing and sharing new BERT models.
Leave a Reply