LaMDA, in particular, was built for dialogue (it’s in the name after all—Language Model for Dialogue Applications), which is why it was so good at responding in conversation.
Computers face the challenge of how to deal with the complex, unstructured mass of data that is human language. Computers don’t communicate the way we do, but with the latest developments in deep machine learning technology they are becoming more and more able to approximate it.
Scientists do the work of training an NLP engine to be able to “understand” human speech through a step-by-step process:
Tokenization. This refers to the process of reducing a bolivia mobile database text into smaller parts called “tokens”. Tokens are often single words, though they can be larger or smaller units as well. These are the building blocks for the way computers will process natural language.
Stemming/lemmatization. Raw tokens often include words in their complex forms, such as plural, participle, or past tense. The next task is to reduce these words into tokens in their root form.
Part-of-speech tagging. After the creation and editing of tokens, scientists need to label them according to the type of word they are, whether noun, verb, or adjective, and so on.
Stop word removal. This refers to the removal of words that don’t contribute significantly to the meaning of a group of text, such as “a” or “the”.
Scientists do this for large amounts of training data called corpora, which helps the machine become better at processing the text.
Based on how the machine is trained, NLP systems can be used for a wide variety of purposes, which we’ll look into later.