Natural language processing (NLP) is, at its core, getting computers to communicate with humans in a natural language like English or Russian or Khmer. Because computers are machines, this process means encoding the natural language into a form that computers understand and can process, and then reversing that process to communicate in a natural language for the human.
In dialogue systems like Alexa or Siri, the process of converting natural language into a logical form the computer can work with is called Natural Language Understanding (NLU). Then the dialogue system takes this logical form and performs some task in the Dialogue Manager, such as change the topic of the conversation or log away some information in the database. The conversion back to human-speak is called Natural Language Generation (NLG).
NLP is still an evolving field, and there are many varied approaches to NLP solutions. Approaches aside, the fundamentals of NLP involve reducing natural language into smaller pieces which can be better analyzed and understood. A written document can be broken into sentences, which are broken into phrases, then words, then morphemes, then characters. We can call these natural language units. Each unit provides important information about the meaning of the text.
By looking at a lot of text, a computer using machine learning algorithms can find statistically probable patterns in these language units and leverage this information to make decisions about the text. For example, a simple language model trained to look at sequences of words would know that sequences like “I like chocolate” are more probable than sequences like “I paint chocolate” and even more probable than “I chocolate paint.” This can help the system disambiguate words with the same orthographic representation. For example, if the system processes the sentence “She beat the record” it can detect that beat is the verb meaning “to defeat someone or something” and not the noun used in music or the adjective meaning tired because of the likelihood of a verb appearing in that position. Without a language model, beat and other words with the same spelling would be ambiguous.
In addition to learning patterns from data, computers can be told things about language and meaning overtly. For example, we can write rules about language, such as “a noun phrase followed by a verb phrase can be a complete sentence” and “adjectives modify the nouns that follow them.” These sorts of rules, often called grammars, can help the NLP system deconstruct or construct natural language out of the smaller language units.
Another example is saving information about language and meaning in an ontology. An ontology is a graphical database of meaning and relatedness. Certain concepts are related to others in different ways–for example, a dog isA mammal. We now know that dogs and animals are related with hypernymic relation isA and we can infer that dogs have characteristics that all mammals have, such as having hair. We could have other relations as well, such as partOf or located: carburator partOf car and Taj Mahal located Agra. If we want to know all parts of a car, we can just consult the ontology. With an ontology or other knowledge resource, the NLP system can leverage the real-world context behind a word or phrase, much in the way a human would.
AllyO leverages NLP to better understand and answer applicants in a chatbot setting. AllyO has a dialogue system that interprets the intent of the applicants’ messages, extracts any relevant information the applicant provides and responds appropriately to the applicants’ messages. The goal of this dialogue system can vary somewhat, but usually involves screening applicants and setting up interviews. To accomplish these goals, AllyO leverages a hybrid machine learning and knowledge representation approach.
First, AllyO’s dialogue system breaks the user’s message into language units–sentences, words, morphemes, and so on. It analyzes these and provides further information about the sorts of language units the message contains–What is the part of speech? What is the shape of the word? What do we know about the meaning of this word or phrase, based on where it appears in our ontology? The system corrects any misspellings in words it does not recognize and then moves along to argument extraction and slot filling.
AllyO’s NLU extracts any important entities in the message, such as contact information or locations. These can later be stored as data provided by the applicant and shared with the hiring manager. These are also used to inform the actions that the dialogue manager takes in response to the applicant’s message.
A machine learning classifier analyzes the message and its language units, extracted information, and more of the message’s features to determine the applicant’s intent. Intent can be thought of as the goal of the applicant’s message–Was their goal to share a phone number? Was it to learn information about the health benefits at this company? This intent, along with the extracted information, is synthesized into a semantic representation. Now we have effectively used NLU to convert the natural language message into a formal representation the computer can understand.
This semantic representation is sent to the dialogue manager (DM), which takes all of the information provided and makes a decision about what to do next. The DM uses a variety of internal rules to determine what to do with the information provided and how to best respond to the applicant–Should we ask the applicant for additional information to narrow down the job search? Can we answer the applicant’s question about health benefits? Does the applicant meet the requirements for the job they are interested in?
Once the DM has made a decision, the natural language generation (NLG) unit generates an appropriate response to the user. This NLG is customer-specific so that the customer can control how the bot responds to its applicants. Then, the applicant responds and the process starts all over again!
How can you assess the NLP skills of a dialogue system that claims to be AI? Here’s a little cheat sheet of do’s and don’t’s to get you started
See how AllyO uses NLP in our chatbots, schedule a demo here
Stay up-to-date with the latest insights and trends from AI recruiting brought to you by AllyO Blog!