Ready for another shot at a deep dive into the world of NLP?

So far, we have explored the fundamentals of Natural Language Processing — understanding and production at the same time, noise removal using tokenization, stop word removal, and stemming, and shapes that text can take using methods like TF-IDF, dependency parsing, and even word clouds – Zipf’s Law: A funny but interesting rule, which shows how often words are repeated in language. We also looked at what is commonly called Bag-of-Words, n-grams, One-hot encoding, TF-IDF, and more succinctly-collected byte-pair encodings into the minds of words.

Coming up, we will be understanding the convergence and divergence of textual similarity estimation among machine systems; classic similarity measures include Levenshtein Distance, Jaccard Index, Smith-Waterman Algorithm, and Cosine Similarity; more recently introduced are Soft Cosine Measure and Word Mover’s Distance (WMD). The basis of the Dice Coefficient, Hamming Distance, Euclidean Distance, and Manhattan Distance (L1 norm) will also be discussed.

There follows the area of Part-of-Speech (POS) Tagging, which is among the earliest and most significant tasks in NLP. We shall discuss standard techniques like Hidden Markov Models (HMMs) and the Viterbi Algorithm, and enter into more advanced techniques like Conditional Random Fields (CRFs) and Maximum Entropy Markov Models (MEMMs). The exploration will consider how modern Transformer-based POS taggers, such as BERT and rule-based POS tagging methods, herald a new generation of linguistic analysis.

Let’s keep it simple, realistic, and insightful, and discover the layers of language at a depth previously unknown.

Imagine that you are teaching a child to read. They first learn to identify letters, followed by words, sentences, and later on, even simple jokes. This is how NLP works: it analyzes language and breaks it into five concise steps of activities, each step standing on its own. Now, let’s move on to understanding each step better.

Phases of NLP

Lexical Analysis: Breaking It Down (The Word Detective)

NLP portions the text and breaks it off into meaningless pieces, like taking the Lego kit apart to see each brick. Lexical Analysis parses the text into a sequence of tokens such as meaningful elements, words, or symbols. 

• Tokenization: Juggling sentences into words/symbols.
  • “Don’t panic!” → [“Don’t”, “panic”, “!”]
  • It has special considerations: “U.S.A.” (it isn’t three tokens, it is merely one) or “🤣” counts as a “laughing emoji.”
• Normalization: Making the words consistent.
  • “Running” becomes “run” (stemming).
  • This “cat” and “Cat” would also read the same word (lowercasing)

Syntactic Analysis: Understanding the Structure of a Sentence(The Grammar Police)

Right now, NLP is figuring out relations between words the way we would do it in English, diagramming sentences. This refers to a sentence’s grammatical relationships as clarified through the analysis of the given structure. 

• Part-of-Speech (POS) Tagging: Labels the words-nouns, verbs, etc.
  “She [PRON] eats [VERB] sushi [NOUN].”
• Parsing: Maps relationships between words.
  “The cat chased the dog” →  The Cat (subject) → chased (action) → The Dog (object)
• Grammar Checks: It catches things like “He go to school” (should be “goes”).

Semantic Analysis: What Does This Say? (The Meaning Decoder )

NLP now moves beyond grammar and into extracting meaning. Semantic Analysis is the process of extracting meaning from a text is through the interpretation of words, phrases, and complete sentences in context. 

• Word Sense Disambiguation
  “Bank” → bank (financial institution) or bank (alongside the river)? (Depending on the context!)
• Entity Recognition
  “Apple released a new iPhone.” → “Apple” means the company, not the fruit.
• Literal vs. Figurative Meaning
  “Break a leg!” → Good luck-not a medical warning.

Discourse Analysis: Linking Cohesion in Meaning (Reading Between Sentences)

NLP studies the relationship between sentences to maintain meaning across coherence. In a nutshell, the process of learning how sentences fit together to comprise coherent meaning across a text is Discourse Analysis. 

• Pronoun Resolution
  “Anna said she’s late.” → “she” = Anna
• Cause and Effect:
  “It rained. The picnic was canceled.” → Rain → cancellation.
• Topic Flow:
  Detects instances when a conversation flips from “travel plans” to “best luggage brands.”

5. Pragmatic Analysis: Meaning Beyond Speaking (The Mind Reader)

NLP assesses the unspoken portion of communications. The process of inferring intent, implications, and real-world context beyond literal meaning is referred to as Pragmatic Analysis.

• Detecting Tone
  “Great job…” → Honest or sarcastic?
• Indirect Requests
  “It’s cold in here.” → “Please close the window.”
• Cultural Nuances
  “That’s an interesting idea.” → “I hate it.” (in some cultures!)

Real-World Magic: Places Where You Witness These Phases

Search Engines: A query such as “cheap flights to Paris under $500” means that Google processes the words as follows:

Lexical: “Cheap flights” “Paris” “$500” Syntactic: “Flights to Paris,” not “Paris flying.” Semantic: “Under $500” is a price filter. Pragmatic: It’s about booking rather than just interest.

Alexa interprets “Play workout music” and “Play music about working out” differently. When the next occasion arises on which you say something, unfortunately, to Siri, and it misunderstands you, just remember the phrase that failed now!

Similarity Measures

Similarity measures are means or methods of comparing two objects, such as two words, two sentences, or two documents, and checking how similar they are to one another, such as sequences like DNA, to see how different or alike their meanings are. This is analogous to comparing two songs, photographs, or shopping lists to see similarities or differences. They are the mathematical tools behind many technologies: when the search engines bring up documents that are close matches to your query, when Spotify recommends songs, or when the chatbot is posing relevant responses. Each of such measures is implemented differently depending on the kind of data with which it is working, text, numerical values, vectors, or biological sequences, and what kind of similarity is being sought.

Levenshtein distance

Edit distance or Levenshtein distance is the measure of alteration in terms of insertion, deletion, or substitution of every single character in a string to get a second string, intended for measuring distance simply between the two words, thereby being useful in spell-checking, autocorrection, and interpretation of slightly mistyped inputs. To turn back into satisfaction via book, two edits are required: substituting one “o” for an “A” and replacing the second “o” for a “C.” The greater the number of edits, the greater the distance between the two words. It measures the number of single-character changes that would have converted one word into another.

Example: “flaw” → “lawn” taken as 2 edits (f→l, add n).

Jaccard

Jaccard is an index that compares two sets and gives their intersection divided by the union of the two sets. It is very useful in comparing tags, keywords, and even in mutual friends. For example, two friends might like entirely different genres of movies. Jaccard would quantify how similar the two friends’ tastes are, but only based on the genres that both enjoy.

Example: A = {apple, banana}, B = {banana, cherry} → Jaccard = 1 / 3 = 0.33

Smith-Waterman Algorithm

This algorithm is assigned for local sequence alignments; that is, rather than treating the two sequences as a whole, the algorithm tries to find the most similar sub-parts. The method is used extensively in bioinformatics for the matching of small DNA or protein sequence parts. It allows greater flexibility by being able to incorporate extra information or noise at the beginning or the end of a sequence than global methods do.

Cosine Similarity

Cosine similarity uses the angle between two sample vectorized texts to determine whether they should be considered similar. Whether a document has five or fifty pages, if it is spelled in similar fonts, it is similar because the actual content is matched. This type of measurement is more common in the representation of needs to compare documents, inform searches, or drive chatbot responses since it is more tuned to the relation between words than their static count. 

Dice Coefficient

It is similar to Jaccard but with a bit more sensitivity with respect to common elements. It doubles the importance of a shared element between two sets and divides by the total size of both. It has been used, for example, in the case of very short texts or token-based comparisons, like finding out whether or not two search queries reflect the same idea. Dice Coefficient is Similar to Jaccard, but with increased weight for common items.

Hamming distance

Hamming distance is used when comparing two vectors of equal length. It simply counts the number of positions in which the corresponding characters in two vectors are different. This measure is mainly applied in error detection and data transmission (it may even check whether a binary code has been received correctly), and where even a single character misalignment may be more significant. Hamming distance counts mismatches of characters between strings of equal length. It is best suited for comparing binary data and error detection.

Example:  “10101” vs “11100” → mismatches = 3 → Distance = 3

Euclidean Distance

The “as-the-crow-flies” distance, as it is commonly called, is a measure of direct distance in space, be it 2D, 3D, or higher dimensions, between two endpoints. This is used for clustering similar items in machine learning, especially when comparing numerical metrics like price, ratings, coordinates, and so on. Its primary role it is to find the length of a straight line between two points.

Manhattan distance (L1 norm)

Manhattan distance ( or L1 norm) is a metric resembling the Euclidean one, but it computes distance along the coordinate axes like a taxi moves in a grid of city blocks. This Works very well for models in which all features are to be treated equally and their diagonal shortcuts avoided by Euclidean distance. It measures the distance as far as you can go by walking. You could only go up and down, left and right.

Soft Cosine Measure

This soft cosine measure actually considers the semantic similarity between words besides regular cosine similarity. This means that soft cosine will still show similarity even if we compare sentences on the basis of different words with similar meanings (say, “car” vs. “automobile”). This works wonders in those natural language processing tasks that care more about meaning than words.  It is like Cosine Similarity, but of course, it deals with the relationships between words.

Word Mover’s Distance (WMD)

An advanced similarity measure that uses word embeddings (or numerical representations of word meanings) to gauge how much “effort” is involved in transforming one sentence into another word by word. The concern here is not just with which words are used but with how close to each other their meanings are. Thus, if two sentences convey the same meaning, they might be close, although they might seem completely different. This Measures how much effort it takes to match every word in one sentence to every word that is meaningful in another sentence.

Part-of-Speech (POS) Tagging

To understand grammar, you must observe the different parts of the sentence through the action of the doer or the “doing” action, where to place the doer and the action. This is where POS (Part-of-Speech) tagging comes into the picture. In natural language processing, POS tagging is where the words in a sentence get tagged according to their part of speech, namely those words or lexemes that are nouns, verbs, adjectives, et cetera. 

For example, let’s take the following sentence: “The cat sleeps on the mat”. Here is the output from POS Tagging:
The – Determiner
Cat – Noun
Sleeps – Verb
On – Preposition
The – Determiner
Mat – Noun

By tagging each word in this way, machines can better understand grammar, context, and meaning, which makes such programs very useful, especially for chatbots, grammar checkers, and translation applications.

Hidden Markov Model (HMM)

A Hidden Markov Model (HMM) works by attempting to uncover the behind-the-scenes operations given the visible clues. For instance, if there were a person behind a curtain, one would not be able to see what he or she would be doing, but perhaps only hear very quiet sounds like footsteps, the running of water, or typing. Those sounds could help guess if the person was walking, showering, or working on a laptop. So, the actual activity was hidden, while the sounds were observable.

They use some observable data, for instance, words in a sentence or sounds in speech, to draw inferences about something that is hidden, like grammatical roles or spoken words. Applications include speech recognition, handwriting detection, and in natural language processing, part-of-speech tagging.

Viterbi Algorithm

Consider you’re on a road trip with your friends to a destination that has multiple routes, but you wish to reach the destination on time; hence, you prefer the fastest route with minimal congestion. The Viterbi algorithm turns out to be a dexterous navigator that provides the most probable sequence of hidden steps (like road choices), given some messy clues from the real world (like traffic reports). Consider solving a mystery in which you have scattered clues (observed events) and piece them together to come up with the most probable hidden story (like the actual path taken). It is widely used for decoding messages in communication technologies, like in the testing of your mobile phones or even in predicting sequences in DNA.

Conditional Random Fields (CRFs)

Consider a scenario as if you were to read a sentence and label its every word according to the role it plays, marking the name, finding the place, or otherwise, it’s just regular text. But the catch is you will not be labeling individual words, but instead the entire sentence, to understand how the words interrelate before deciding. Instead of labeling each word or element one by one, they look at the whole picture the current word connects the previous and the next word. 

It would work like this: For example, the word “Dr.” would be said next to the word “Smith”; CRFs automatically know that “Dr. Smith” is a person’s name, not just two random words. Thus, CRFs ensure that tags have meaning together and not just alone as well.

Rule-Based POS Taggers

Rule-based POS taggers are intelligent editors in a way: they tag parts of speech (POS) such as nouns, verbs, adjectives, and so on to each word without any automation by following a set of handcrafted grammatical rules. These are written by linguists or language experts- they tell the system how to tag a word based on its spelling, position, and neighboring words. The processing typically involves 2 main modules:

• Lexicon Lookup- where each word is assigned all possible POS tags. 
• Disambiguation Rules- within these rules, context-based rules help choose the most likely tag for the word in such specific sentences.

For example, the present participle is most likely if it ends with “-ing” and is preceded by the verb “is”. Or after “the” knowing some noun, append it with a noun. These rules are thus applied to make guesses about the tag.

Maximum Entropy Markov Models (MEMMs)

You walk through a sentence word by word, then try to tag the words with their respective parts of speech, or tag them for something like identifying names or localities. But it may tag that word not by what that word is or what comes after it, but also by everything around it. This is where Maximum Entropy Markov Models come into play.

By combining the concept of a sequence, such as in a sentence, such models say that a context must have some smart way to make decisions. They use maximum entropy to choose the most uniform and fair solution based on the information available. Therefore, they look at the current word, its previous tag, and some good features like words in the vicinity to make the best guess for the current label of the word.

Summary

To summarize, Natural Language Processing (NLP) is a spectacular exploration traversing the human language journey – analyzing words through lexical analysis, grammar checking in syntactic analysis, meaning attribution in semantic analysis, contextual understanding in discourse analysis, and intent interpretation in pragmatic analysis. Tasks outlined above are aided by similarity measures such as Levenshtein, Jaccard, Cosine, and Word Mover’s Distance, among others, which quantitatively assess the resemblance of texts. In robotic word classification, Part-of-Speech (POS) Tagging features prominently, implemented by Rule-Based Taggers, Hidden Markov Models (HMMs), MEMMs, and CRF, which construct language using numerous rules, patterns, and probabilities. All concepts mentioned above articulate the foundation that enables machines to accurately understand, process, and use human language.

If you want to learn more, stay tuned and discover the fascinating world of NLP, AI, and language learning, little by little! Remember, curiosity is always Numpty Neuron’s best friend. Stay Curious! Stay Numpty!