A Complete Guide to Natural Language Processing (NLP)

What is Natural Language Processing (NLP)?

“Ugh, why doesn’t this thing understand me?” moan chatbot users everywhere. When that happens, it's either because the bot's Natural Language Processing (NLP) isn't up to snuff or is non-existent.

Other times, it's super cool to chat with a bot. It understands exactly what you intend to say, no matter how you word it. It remembers context from what you've previously asked, allowing the conversation to flow naturally. It determines the right response based on what it has learned. And it delivers that response in a human-like format.

NLP is what makes that possible. It allows bots to communicate with end users in a way that mirrors human conversation.

NLP is an artificial intelligence (AI) technology that uses algorithms to ingest what humans are saying, understand what they're trying to communicate, figure out what to do with that input, and generate appropriate responses. It's how chatbots chat and handle end-to-end interactions in the customer's language.

Arguably, the most important part of NLP technology is in the understanding. Natural Language Understanding (NLU) allows a bot to understand the meaning, or "intent" often hidden in human language. Without this piece of the NLP system, AI wouldn't be able to do its thing: communicate with users.

Understanding intent is different than recognizing what is being said. For instance, language models that parse text and speech recognition technology, used by voice-activated chatbots like Siri and Alexa, aim to understand the "what" of a user's query. NLU goes a step further by identifying the meaning of those words and understanding what it is that the user is trying to achieve.

How Does Natural Language Processing Work?

First, let's get clear about one thing. NLP isn't a search engine. A search engine looks for keywords in your text and spits out as many results as it can, leaving you to sift through the results to find what you're looking for. (Think Google search.) NLP technology works differently.

When someone uses a chatbot or dynamic search bar powered by NLP, they don't have to rely on exact keyword matches. Users can speak or type in full, even complex, sentences. NLP breaks down the sentence structure, uses knowledge of idioms, broken grammar, and idiosyncrasies, and recognizes patterns to parse out what the user is trying to say, even if 10 users say it 10 different ways. In doing so, it finds the intent of the user's input so that it can deliver an appropriate response.

The most successful chatbots are designed to do a handful of things, not all the things. For example, a bank bot may be programmed to answer frequently asked questions, help customers choose the right account type, and execute simple transactions like fund transfers. But this same chatbot will not be able to tell customers who won the game last night or book a flight to Hawaii. It won't have the NLP knowledge to understand or speak about anything outside of the universe it's been programmed to know.

This is important because when the chatbot is breaking down human language to find the user's intent, it's using the context of its known universe. The chatbot knows a number of intents that have been classified based on what the chatbot is programmed to do. When it recognizes one of those intents, it can pull the right response from the right resources to serve to the end user.

Key Components of NLP

Let's break the process down into the AI elements. First the machine has to understand the intent (natural language understanding – NLU), then it has to figure out what the answer is (machine learning – ML), and finally it has to spit out (or generate) that answer in human language rather than in machine language (natural language generation – NLG).

 

1. Understanding the intent: Natural Language Understanding (NLU)

Understanding what the customer is trying to do (their intent), requires breaking down sentences into their component parts. Remember diagramming sentences in grammar class? That's NLU's specialty: identifying parts of speech, otherwise known as part-of-speech tagging. There's a whole lot of technology that goes into making understanding human language seem effortless.

• Topic analysis:
  Discovering the meaning or "intent" of what the user has said. This includes:
• Contextual extraction:
  Understanding the current context.
• Syntactic analysis:
  Analyzing the syntax, or structure of the sentence, and the roles of the words used.
• Entity extraction:
  Finding entities like a person, place, organization, or event, and determining how important those entities are.
• Semantic analysis:
  Concluding the meaning of words based on context, which is especially important when one word has multiple meanings.
• Sentiment analysis:
  Identifying how the user feels (mood, emotion, opinions).

2. Determining the answer: Machine Learning

Now that the machine knows the intent, how does it categorize that intent in order to create an appropriate response? The machine turns the human language into binary machine code so it can find the answer using its algorithms, such as neural networks that recognize and classify patterns, similar to how the human brain works. In other words, this is how AI converts text into structured data that the machine understands.

• Content categorization:
  Grouping intents into categories to be used by the machine.
• Machine translation:
  Converting human text or speech into machine language.
• Machine language:
  Binary code that the machine can act on.

3. Generating the response: Natural Language Generation (NLG)

Then it has to translate the answer from machine code back into human language using meaningful words, phrases, and sentences the user will understand. When NLG is dialed all the way up, this is where chatbots are able to not just offer a response in a human-like way, but write their own answers based on its own autonomous deep learning. This can be a little scary for brands (just ask Microsoft who deployed and killed its Twitter chatbot, Tay, within 24 hours after it started spitting out offensive content). Many brands want to have access to the underlying technology to control what language is generated.

• Sentence generation:
  Creating sentences from machine code.
• Document creation:
  Structuring sentences into a compelling narrative.
• Document summarization:
  Generating synopses of large bodies of text.

Natural Language Processing in Action

The earliest phase of NLP in the 1950s was focused on machine translation, in which computers used paper punch cards to translate Russian to English. Now machine translation is a routine offering and natural language processing techniques have flourished.

Forward-leaning brands now use NLP in AI chatbots to create great customer experiences in conversational interactions, personalize responses to the customer based on what they know about them, provide answers, and facilitate real transactions. The result is tailored engagement outcomes that create remarkable experiences for the end users.

 

Natural Language Processing Examples:

Providing Customer Service and Support
AI chatbots with NLP are helping customers locate the nearest bank branch, make decisions on what products to order, find warranty information, and a lot more, acting as a stand-in for front-line customer service.

Allowing Customers to Execute Transactions
AI chatbots with NLP can be used to help customers complete transactions. In a conversation with a chatbot, customers can book a movie ticket, order lunch, make a hair appointment, change a reservation, and more.

Analyzing Sentiment & Feedback
Organizations can use NLP to monitor trending issues on social media to see how their brand is doing, understand voter sentiment during a political campaign, analyze the feedback of customers who reach out to call centers, and more.

Analyzing Free Text
There's a lot of valuable information in free text like lengthy customer reviews of products, patient medical records, and online articles and blog posts. NLP can be used to mine masses of free text to find relevant data that organizations can use to transform their products, services, or customer experience.

Making Enterprise Knowledge More Accessible
NLP allows users to search through FAQs or other data sets as if they were talking to another person. It may be in a chatbot conversation or enterprise search like a dynamic search bar. Since NLP doesn't rely on matching keywords to find answers, users can more easily find what they're looking for.

Generating Automatic Transcripts
Human language is not precise. Since NLP analyzes all aspects of human language, it can be used to create more accurate speech-to-text transcripts. Important meetings and customer conversations can be automatically transcribed for future reference or data mining.

Subscribe via iTunes, Spotify and more.

Why is Natural Language Processing Important?

In today's competitive CX landscape, brands must continue to improve and evolve the customer experience, or else risk being left behind by competitors. As NLP continues to advance, brands are able to create more desirable outcomes on AI-assisted channels.

Computer science fields of study continue to push NLP forward, like computational linguistics. This includes creating computational models of various types of linguistic phenomena which can then be applied to online conversations to filter out unwanted language or to online search for retrieving documents and websites that best meet the combination of search terms.

With businesses and customers embracing cloud computing more than ever, NLP helps make automated services easier, more accurate, and more efficient. For customers, self-serving is more streamlined when they can use their own words to have conversations with chatbots or to search through FAQs. Their questions are answered immediately and efficiently. For companies, this results in decreased call center volume, allowing them to be more efficient and freeing their agents to focus on higher value issues that require a human touch.

On that note, NLP also plays an important role on the agent's side. Virtual assistants using NLP follow along with the conversation between agents and customers to discern the intent and offer possible responses. This helps the agent personalize the experience and work more efficiently to find resolutions.

NLP unlocks the potential of data analysis, previously hidden in troves of text, as language models continue to evolve. Larger sets of data like customer reviews can now be analyzed with machine learning algorithms. It finds the underlying structure in the data to help businesses gain aggregated insights and improve the customer experience.

Last, but certainly not least – for those of us still waiting on the future The Jetsons promised in the 1960s – natural language processing gets us that much closer to an automated and easier lifestyle. Don't we all need a household robot like Rosie? She'll be part of the family with NLP capabilities.