Overview of natural language processing

Overview and definition of NLP

Natural language processing (NLP) is a fascinating branch of AI that allows people and machines to communicate with one another in everyday language. The way you give voice commands to Siri or get translations on Google are both examples of NLP in action, because both show software understanding and responding to human language.

How does NLP relate to AI?

Most people aren’t programmers or software power-users, which is one of the reasons natural language processing is so useful. Software can have a steep learning curve, and NLP allows you to use even complex software without becoming an expert. In fact, NLP allows you to simply describe what you want to an AI in the same way you might explain it to a friend.

The term AI covers a wide range of technologies, but the NLP branch focuses on the challenges of human language. To be truly useful, NLP solutions have to do more than just catch the words you say. NLP systems have to understand the context and intent behind your words. To make that possible, NLP developers use other AI technologies like machine learning and deep learning.

Why is natural language processing important?

Natural language processing allows you to use technology that might otherwise be difficult to use. It also allows computers to understand text and speech in a way that they couldn’t before NLP. Here are some of the benefits of NLP:

NLP helps people be more productive

At work, natural language processing can increase your efficiency by allowing you to use automation for repetitive or time-consuming tasks. For example, a customer service department might use NLP-powered chatbots to handle routine customer inquiries. Or, an accounting department might use NLP-enabled systems to pull key information from invoices and receipts and use it to populate a database or a spreadsheet.

As additional benefits, automating data entry and processing reduces the chances of human error and accelerates workflows. When a system can understand human language well enough to take over basic tasks, it increases productivity by allowing users to focus on higher value tasks.

NLP helps improve customer experiences

When you try to reach a business by phone but can’t get past a confusing phone tree, you’re experiencing a poorly trained interactive voice response system. But a well-trained NLP chatbot can give customers a more intuitive experience. An e-commerce site with an NLP-enabled system, for example, can analyze what products a customer has been browsing on the site. By understanding which products are of interest to a customer, the system can then suggest products that the customer is likely to want.

NLP-powered customer service chatbots improve experiences by answering questions or solving problems quickly. These chatbots can be designed to have a complete history of a customer’s interactions and recognize problems the customer has had in the past.

NLP creates new insights

If you have ever sent an email to a business to ask a question, offer advice or register a complaint, it can seem like no one even reads it. And that might be the case, because few businesses have the time to read every piece of customer feedback they get. But NLP-enabled AI does have the time. It can sift through enormous datasets like customer conversations on web sites. It can then give businesses a reliable summary of those discussions so that the business can correct the issue.

How does natural language processing work?

Here’s how NLP makes human and machine conversations possible. The following steps apply to language in the form of text. NLP for spoken language is a little different but follows the same general principles.

Let’s use a sample sentence to show how the process works:

“I like demonstrating how natural language processing works.”

The algorithm begins with text preprocessing.

Text preprocessing

Text preprocessing refers to simplifying the text people create to make it easier for NLP algorithms to process human language.

• Tokenization

Tokenization is the process of breaking down the words and punctuation in a sentence into tokens. Tokenization is important because it’s more efficient for NLP algorithms to process tokens than text when performing tasks like indexing and search. The sample sentence has eight words in it: “I like demonstrating how natural language processing works.” It also has a period, so you get eight tokens by counting the words and one by counting the period, for a total of nine tokens.

• Lowercasing

Lowercasing is the process of converting all tokens into lowercase tokens to make the dataset simpler. In the sample sentence, one of the tokens was “I.” To avoid ambiguity and increase efficiency, the lowercasing step converts those uppercase tokens into “i”. The rules for lowercasing get more complicated in other parts of the process.

• Removing stop words

Another way of simplifying text for natural language processing is to remove words that don’t have significant meaning, which are called stop words. In the sample sentence, the words “i” and “how” would typically be designated stop words. When an algorithm removes them, you have seven tokens left, which are “like”, “demonstrating”, “natural”, “language”, “processing”, “works”, and “.”.

• Stemming and lemmatization

Even with the seven tokens remaining there’s room for more simplification. One method, stemming, is cutting a word down to its base or root form. The token “demonstrating” is a word that builds on the stem “demonstr” just as “natural” builds on “natur,” so the model replaces the original tokens with the stems “demonstr” and “natur”.

A word can have different meanings in different contexts, and lemmatization is the process of determining the right meaning in a particular context. The sample sentence includes “like”, which can mean “enjoy” or “similar to”. In this case, lemmatization would result in assigning the context where “like” means “enjoy”.

Text representation

The next step is text representation, which is converting words into a numerical format that a machine can process. In addition to numbers, text can be converted into vectors or embeddings, which are more complex formats that provide information like context.

• Bag of words (BoW)

Computers are great at counting, and the BoW measurement counts how many times a word appears in a document. If the words team, game, and score appear frequently in a document, for example, the context is more likely to be sports. The sample sentence only has one instance of each word. The BoW representation would show that each word only occurs once like this:

{“i”: 1, “like”: 1, “demonstrating”: 1, “how”: 1, “natural”: 1, “language”: 1, “processing”: 1, “works”: 1}

• TF-IDF (term frequency-inverse document frequency)

TF-IDF uses a formula based on how often a word appears in an overall dataset made up of many documents to determine a word’s importance. The more often a word shows up, the lower its TF-IDF weight, and the less important it is in a single document. Words like “the” and “a” appear quite often, and so are less important. You would represent the weights of the sample sentence in a form something like this, showing that common words have lower weights and uncommon words have higher weights:

{“i”: 0.1, “like”: 0.1, “demonstrating”: 0.3, “how”: 0.1, “natural”: 0.2, “language”: 0.2, “processing”: 0.5, “works”: 0.1}

Text analysis

Text analysis is the point at which an NLP algorithm extracts meaning from text. It’s how an algorithm can produce appropriate responses to the user's query.

• Named entity recognition (NER)

Having done some quantitative analysis, NLP algorithms next look for words that are recognizable as naming something. The word apple refers to a type of fruit, but the word Apple refers to a specific company, and an NLP algorithm needs a way of recognizing the difference. The sample sentence, “I like demonstrating how natural language processing works,” includes the words “natural language processing”, which people might recognize as naming an AI technology. This is represented as follows:

NER Output: [(“natural language processing”, “Technology”)]

• Sentiment analysis

Some datasets, like a catalog of lawnmower parts, may not have a significant emotional tone. But a movie review might have a strong emotional tone. If a dataset does have a tone, sentiment analysis is the step in natural language processing that captures it. For the sample sentence, the sentiment analysis might look like this:

Sentiment Output: Positive

Syntax parsing

Every sentence has a grammatical structure. Syntax parsing is the process of analyzing that structure to find nouns, verbs, subjects, etc. This is important because different languages use different syntax, so syntax parsing is vital to machine translation. For the sample sentence, syntax parsing might generate this result:

Syntax Tree: (ROOT (S (NP (PRP I)) (VP (VBP like) (S (VP (VBG demonstrating) (SBAR (WHADVP (WRB how)) (S (NP (NNP Natural) (NNP Language) (NNP Processing)) (VP (VBZ works)))))))))

Depending on the specific algorithm, there will typically be additional steps. The result is a conversation where the machine seems to understand your words and intention and responds to you in natural language.

What are examples of natural language processing tasks?

Controlling a drone with voice commands

Even the most powerful technology has limited value if you don’t know how to use it. Natural language processing makes technology more accessible. It reduces the need for specialized technical knowledge to get the benefits of advanced software or hardware. Users can interact with NLP-enabled systems through natural conversation instead of relying on complex commands, coding, or physical controls.

For example, a remote-control application for small drones can allow you to simply tell a drone to do something like flip in the air, without the need to learn the intricate controls you’d otherwise need to use. The ability to use simple voice commands allows more people to use technology.

Gaining better insights for brand management

People express their thoughts and preferences every day, and businesses have access to much of that data. Businesses already use data like product sales to understand some aspects of customer behavior. With natural language processing solutions, computers can turn social media conversations and online customer reviews into actionable information as well.

The insights from NLP are a different kind of information than traditional sales analytics. Business use sales figures for operational insights like forecasting or resource management, but NLP-based analysis can be more powerful in brand management or enhancing customer experiences.

Preventing information overload

If you have an email address that you use for your job, information overload can seem inevitable. The average employee receives more than 120 emails a day, so it’s not surprising that about 60% of employees simply ignore internal company emails. But if your email application has NLP capabilities, it can help you avoid information overload. It can filter, categorize, and prioritize emails so that the most important ones get the attention they need.

Similarly, NLP systems built into collaboration software can transcribe and summarize meetings. They can even recognize and capture the key points made during a meeting and report on assigned action items. This kind of NLP-enabled automation helps individuals save time and increases overall organizational efficiency.

Natural language processing use cases

NLP is transforming industries. NLP applications improve employee efficiency, make customer experiences better, and enable strategic decision-making, spanning a wide range of sectors. Below are some notable use cases for NLP across different industries.

Natural language processing approaches

Most approaches to natural language processing fall into one of two broad categories. They either take a rules-based approach or a machine learning–based approach.

Rules-based NLP

This approach attempts to identify a set of linguistic rules that a computer can follow to reliably understand and generate human language. As a result, it’s heavily oriented toward linguistic aspects of computational linguistics. Rules-based approaches can be highly effective if you’re using datasets where the language is both controlled and predictable, like legal documents or technical manuals.

Machine learning–based NLP

A machine learning–based approach to NLP uses a variety of statistical methods and algorithmic approaches. Rather than creating the rules in advance, the aim is to allow a computer to learn how to communicate based on a massive dataset. The idea is that once a computer has processed enough examples of human language, the computer will identify the patterns that make for good human language. If you have large enough datasets, machine learning–based approaches to NLP can be quite flexible and remarkably effective.

A brief history of natural language processing

Natural language processing as a branch of AI began to develop in the 1940s. In the 1980s and 1990s, computing solutions became more powerful and machine learning began to mature. More recently, the rise of deep learning, neural networks, and various forms of generative AI has completely transformed natural language processing.

Events in NLP evolution

• 1940s | First NLP developments, including the Weaver Memorandum on machine translation, appear
• 1950s | “Universal grammar” introduced; Turing test proposed; deep learning emerges
• 1960s | Chatbots such as ELIZA emerge and linguistic theories underpinning NLP evolve
• 1970s | Case grammar theory and semantic networks emerge; rule-based systems mature and chatbots proliferate
• 1980s | Early machine learning algorithms develop; speech recognition technologies evolve; computational linguistics expands and RNN theory emerges
• 1990s | Statistical models and LSTMs emerge and machine translation gains traction
• 2000s | Machine learning evolves rapidly and neural language models develop; vast amounts of spoken and textual data become accessible
• 2010s | A significant increase in available data and compute enables deep learning to evolve rapidly
• 2020s | Large language models, pre-trained language models, and transformers gain prominence

Technology overview of natural language processing

Natural language processing is an umbrella term that covers a range of technologies and techniques that make it possible for machines to understand and to produce human language. Every technology that enables NLP falls under one of those two capabilities.

NLP sub-categories

Natural language processing covers a range of technologies and techniques. But the primary purpose of NLP is to make it possible for machines to understand and produce human language. These two capabilities are the main components of natural language processing.

• Natural language understanding (NLU): Natural language understanding focuses on comprehending and interpreting human language. To do that, NLU systems must be able to parse syntax, analyze semantics, and understand the way context affects meaning in human language. This might take forms such as understanding spoken questions or performing tasks based on spoken directions.
• Natural language generation (NLG): Natural language generation focuses on producing human-like text or speech. To do that, NLG systems must be able to turn unstructured data into natural-sounding language. This might mean summarizing information or even carrying on conversations.

The role of machine learning in NLP

What could a computer do if it could teach itself new skills? That’s what machine learning is. Machine learning is when computers learn how to do tasks on their own without any specific instructions.

For NLP, machine learning takes the form of creating models that enable both natural language understanding and natural language generation. It uses techniques including supervised learning, which refers to training models on data that has labels, and unsupervised learning, which is training models on data that doesn’t have labels.

The role of deep learning in NLP

Deep learning is a specific form of machine learning. It uses neural networks that have multiple layers, which is why “deep” is in the name. The “learning” refers to the use of algorithms that identify and then model complex patterns in datasets. Deep learning is important in NLP because it has made NLP much better at certain tasks. These include translating between languages, analyzing the sentiment in a dataset, and generating text.

How neural networks enhance NLP

Neural networks are based on the idea of using the human brain as a model for how to process data. Neural networks enable NLP systems to be highly accurate in both understanding and generating human language. Neural networks can have various architectures and are key to enabling applications like a virtual assistant, chatbot, or automated text analysis.

Computational linguistics and NLP

Computational linguistics is the field of study that combines computer science and linguistics to focus on natural language processing. It creates a theoretical foundation for enabling computers to understand human language.

• Syntax

Studies the structure of sentences and the rules that make sentences grammatical or ungrammatical.

English syntax: “The cat sits on the mat.”

Incorrect English syntax: “Cat the on mat sits.”

• Semantics

Studies meaning in language, including how words and phrases represent objects, actions and ideas.

Sentence: “The cat is on the mat.”

Semantic interpretation: The meaning is that there’s a cat physically located on top of a mat.

• Pragmatics

Studies the way context influences the interpretation of language.

Sentence: “Can you pass the salt?”

Pragmatic interpretation: Although it’s a question about ability, the context shows that you should understand it is a polite request for someone to pass the salt.

Computational linguistics is important because it connects the dots between linguistic theory and real-world applications of NLP.

Read more