Insights Into Natural Language Processing for Recruitment

What is NLP?

Natural language processing (NLP) is, at its core, getting computers to communicate with humans in a natural language like English or Russian or Khmer.  Because computers are machines, this process means encoding the natural language into a form that computers understand and can process, and then reversing that process to communicate in a natural language for the human.  

In dialogue systems like Alexa or Siri, the process of converting natural language into a logical form the computer can work with is called Natural Language Understanding (NLU).  Then the dialogue system takes this logical form and performs some task in the Dialogue Manager, such as change the topic of the conversation or log away some information in the database.  The conversion back to human-speak is called Natural Language Generation (NLG).  

NLP is still an evolving field, and there are many varied approaches to NLP solutions.  Approaches aside, the fundamentals of NLP involve reducing natural language into smaller pieces which can be better analyzed and understood.  A written document can be broken into sentences, which are broken into phrases, then words, then morphemes, then characters. We can call these natural language units.  Each unit provides important information about the meaning of the text.

By looking at a lot of text, a computer using machine learning algorithms can find statistically probable patterns in these language units and leverage this information to make decisions about the text.  For example, a simple language model trained to look at sequences of words would know that sequences like “I like chocolate” are more probable than sequences like “I paint chocolate” and even more probable than “I chocolate paint.”  This can help the system disambiguate words with the same orthographic representation. For example, if the system processes the sentence “She beat the record” it can detect that beat is the verb meaning “to defeat someone or something” and not the noun used in music or the adjective meaning tired because of the likelihood of a verb appearing in that position.  Without a language model, beat and other words with the same spelling would be ambiguous.

In addition to learning patterns from data, computers can be told things about language and meaning overtly.  For example, we can write rules about language, such as “a noun phrase followed by a verb phrase can be a complete sentence” and “adjectives modify the nouns that follow them.”  These sorts of rules, often called grammars, can help the NLP system deconstruct or construct natural language out of the smaller language units.  

Another example is saving information about language and meaning in an ontology.  An ontology is a graphical database of meaning and relatedness. Certain concepts are related to others in different ways–for example, a dog isA mammal.  We now know that dogs and animals are related with hypernymic relation isA and we can infer that dogs have characteristics that all mammals have, such as having hair.  We could have other relations as well, such as partOf or located:  carburator partOf car and Taj Mahal located Agra.  If we want to know all parts of a car, we can just consult the ontology.  With an ontology or other knowledge resource, the NLP system can leverage the real-world context behind a word or phrase, much in the way a human would.

 

How NLP for AllyO’s Recruitment Chatbot Strengthens Candidate Experience

AllyO leverages NLP to better understand and answer applicants in a chatbot setting.  AllyO has a dialogue system that interprets the intent of the applicants’ messages, extracts any relevant information the applicant provides and responds appropriately to the applicants’ messages.  The goal of this dialogue system can vary somewhat, but usually involves screening applicants and setting up interviews. To accomplish these goals, AllyO leverages a hybrid machine learning and knowledge representation approach.  

Allyo bot

First, AllyO’s dialogue system breaks the user’s message into language units–sentences, words, morphemes, and so on.  It analyzes these and provides further information about the sorts of language units the message contains–What is the part of speech?  What is the shape of the word? What do we know about the meaning of this word or phrase, based on where it appears in our ontology? The system corrects any misspellings in words it does not recognize and then moves along to argument extraction and slot filling.

AllyO’s NLU extracts any important entities in the message, such as contact information or locations.  These can later be stored as data provided by the applicant and shared with the hiring manager. These are also used to inform the actions that the dialogue manager takes in response to the applicant’s message.

code

A machine learning classifier analyzes the message and its language units, extracted information, and more of the message’s features to determine the applicant’s intent.  Intent can be thought of as the goal of the applicant’s message–Was their goal to share a phone number?  Was it to learn information about the health benefits at this company? This intent, along with the extracted information, is synthesized into a semantic representation.  Now we have effectively used NLU to convert the natural language message into a formal representation the computer can understand.

code

This semantic representation is sent to the dialogue manager (DM), which takes all of the information provided and makes a decision about what to do next.  The DM uses a variety of internal rules to determine what to do with the information provided and how to best respond to the applicant–Should we ask the applicant for additional information to narrow down the job search?  Can we answer the applicant’s question about health benefits? Does the applicant meet the requirements for the job they are interested in?

Once the DM has made a decision, the natural language generation (NLG) unit generates an appropriate response to the user.  This NLG is customer-specific so that the customer can control how the bot responds to its applicants. Then, the applicant responds and the process starts all over again!  

How to test the NLP of a system

How can you assess the NLP skills of a dialogue system that claims to be AI?  Here’s a little cheat sheet of do’s and don’t’s to get you started

What not to do:

  • Ask it questions out of its domain
    Cooperate with the system–there’s no need for a customer service chatbot to tell you about the weather for the day or the value of the company’s stock.  Stay in its lane.
  • Try to trick it
    Again, the people who will be using the dialogue system will be trying to cooperate with it–that’s the beauty of task-oriented systems like AllyO: both the applicant and AllyO want to find a job that’s a good fit.  Don’t say things in an especially confusing way just to trick it. No real user is going to do that, so it isn’t a good indicator of the value of the product.

What to do:

  • Talk to it like you would to a normal person
    If the system asks you a question, don’t lob it an easy answer.  For example, if the system asks you to choose from a list, don’t necessarily answer with “#1.”  Instead, try things like “the first one” or “the manager position.”
  • Check to see if it understands you the same way if you say the same thing with different wording
    Just try asking or answering the same question with different phrasing.  For example, tell the system “my background is in foodservice” and then restart and try again with “I have worked several years with foodservice.” Try using synonyms like “work from home” and “remote work.”  
  • Check to see if it can handle telling it multiple things at the same time
    For example, you can chain information like “My email address is johnsmith@allyo.com and my phone number is 1-800-123-4567” and see whether the system understands both elements.  
  • Check to see if it can be flexible
    For example, see if you can answer questions out of order, change the topic, or request to change an answer you provided before.  
  • See if it remembers what you’re talking about by using pronouns
    “I worked for Amazon for three years” allow the system to respond “Actually, I think I worked for them 4 years.”  See if it understands that “them” is “Amazon.”

See how AllyO uses NLP in our chatbots, schedule a demo here


Kathleen Preddy

Kathleen is a natural language processing (NLP) engineer with experience in dialogue systems and semantic search. As a NLP Applied Scientist at AllyO, Kathleen designs and implements the natural language understanding (NLU) component of the dialogue system with the goal to better understand applicants. She holds her M.S. in Computational Linguistics with a specialization in semantic parsing from the University of Washington. Her research interests also include information extraction, semantic representations, multilingual NLP, ontologies, and question answering. Kathleen is a member of the Association for Computational Linguistics.

SUBSCRIBE TO ALLYO BLOG

Stay up-to-date with the latest insights and trends from AI recruiting brought to you by AllyO Blog!


You might also like

Filter

Schedule Demo Widget