Part 2: The Journey from Raw Data to Structured Understanding through Data Processing

Welcome back to our NLP blog series! In the first part, we explored the key components of Natural Language Processing, like NLU (Natural Language Understanding) and NLG (Natural Language Generation), and got familiar with foundational elements such as corpora and tokens. Now, in this second part, we’ll dive into the essential steps of data processing. From cleaning and normalizing data to tokenization and beyond, we’ll break down how raw data is transformed and prepared for training a model. Keep reading to discover how we get data ready for the exciting world of machine learning!

In Case You Missed It: Previous Parts of Our NLP Journey

• Part 1: The Building Blocks of NLP – Exploring NLU, NLG, and Core Elements

Data Pre-processing

Data pre-processing includes stages that convert raw text into a format which is suitable for data analysis and model building. Data pre-processing is the crucial step which impacts the model performance of any NLP model. Since raw data often contains irrelevant , noisy, or redundant information, pre-processing is crucial to improving the model performance.

Tokenization

Tokenization splits the text into smaller units, called tokens, become the building blocks of NLP. This tokens can be words, subwords, characters, or sentences, depending on the specific application. Tokenization is a building block in many NLP tasks, such as text analysis, machine translation, sentimental analysis, and more, because it enables machines to understand and process human language in a structured format.

Types of Tokenization

1. Word Tokenization
   • This is the process of splitting the sentences into separate words. For Example,
     • Input: “”
     • output: []
   • Word tokenization is straightforward in the language with the spaces (like in English), but more complex in languages where words are not always separated by spaces like Chinese.
2. Subword Tokenization
   • Subwords tokenization step splits words into more smaller units like syllables, prefixes and suffixes. This is more helpful in handling out-of-vocabulary (OOV) words or rare words.
   • Techniques like Byte Pair Encoding (BPE) and WordPiece are popular subword tokenization methods used in models like GPT, BERT and others.
   • Example (Using WordPiece):
     • Input word: “unhappiness”
     • Output: [“un”, “##happy”, “##ness”]
3. Character Tokenization
   • Tokining word at a character level breaks down words into individual characters.
   • Example:
     • Input word: “numptyneuron”
     • Output: [“n”, “u”, “m”, “p”, “y”, “n”, “e”, “u”, “r”, “o”, “n”]
   • While this can be useful in some cases, it makes sequences more lengthy and less efficient in many NLP tasks.
4. Sentence Tokenization
   • Sentence tokenization or sentence segmentation, involves splitting text into individual sentences. This is often used in text summarization or translation tasks.
   • Example:
     • Input: “Welcome to Numpty Neuron. Let’s be Numpty!”
     • Output: [“Welcome to Numpty Neuron.”, “Let’s be Numpty!”]

Normalization

Think of normalization as giving your text a fresh haircut—it makes everything neat and consistent, ready for analysis. Here’s how we do it:

1. Lower-casing:
   Imagine you’re cleaning up a document where “Apple” and “apple” look different. Lower-casing turns everything into lowercase, so the text becomes uniform. It’s like making sure every word is on the same page.
2. Removing Punctuation:
   Ever notice how commas, periods, and exclamation marks can clutter your sentences? By removing these, we focus purely on the words, making the text cleaner and more straightforward.
3. Removing Special Characters:
   Symbols like @, #, or $ can be distracting. We strip these away to keep our text free of unnecessary clutter.
4. Expanding Contractions:
   Short forms like “can’t” become “cannot,” turning your text into a consistent format. It’s like translating slang into formal language—making everything sound the same.
5. Removing White-space:
   Extra spaces or tabs can make the text look messy. By cleaning these up, we ensure our text is tidy and ready for analysis.
6. Accented Character Normalization:
   Characters with accents, like “é,” can be simplified to “e.” This step helps standardize the text, especially when accents aren’t crucial for understanding.
7. Stemming:
   Stemming is like trimming words down to their basic form. For example, “running” becomes “run.” It’s great for grouping similar words but can be a bit rough around the edges.
   • Porter Stemmer: Cuts off common suffixes but can be a bit aggressive.
   • Lancaster Stemmer: More aggressive, often shortening words even further.
   • Snowball Stemmer: A refined version of the Porter Stemmer, offering a balance between accuracy and simplicity.
8. Lemmatization:
   Lemmatization is a bit like finding the dictionary form of a word. It’s more precise, so “am,” “is,” and “are” all become “be.” It’s like getting to the root meaning of words with a little help from grammar.
   • WordNet Lemmatizer: Uses a large lexical database to find the base form.
   • Spacy Lemmatizer: Part of the Spacy library, using smart models to understand words.
   • Stanford Lemmatizer: Comes from the Stanford NLP toolkit, focusing on accuracy and detail.

Normalization is all about making your text neat and ready for action. Think of it as preparing your data for its big debut—clean, consistent, and ready to impress!

Stopword Removal

Ever wonder why some words seem to pop up everywhere but don’t really add much to the meaning of a sentence? These are called stop words—words like “the,” “is,” and “in” that are so common they can clutter up your text analysis.

Imagine you’re trying to find the key topics in a book, but these frequent, mundane words are hogging all the attention. By removing stop words, you clear out this background noise and let the important words shine.

Example:

• Before: “The Numpty Neuron provides the best insights in AI and psychology for all readers.”
• After stopword removed: “Numpty Neuron provides best insights AI psychology readers.”

See how the focus shifts? Removing stop words helps highlight what’s really important in your text.

Handling Numbers

Ever wondered what to do with those pesky numbers that sometimes clutter up your data? You have two clever options!

1. Removing Numbers: Sometimes, numbers are just noise. By removing them, you clear up space for more meaningful information.
2. Replacing Numbers with Placeholders: Instead of keeping those specific digits, you can swap them out for a simple “NUM” token. This way, you still know numbers were there, but without getting bogged down by the details.

Think of it as tidying up your data—keeping only what you need and making it easier to understand!

Handling Rare or Frequent Words

When dealing with words, balance is key—too few or too many can throw off your analysis.

1. Removing Rare Words: Words that rarely show up may not add much value, so it’s often best to remove them and focus on more common terms.
2. Removing High-Frequency Words: Some words appear so frequently that they lose their meaning in context. By removing them, you make space for the words that truly matter in distinguishing your data.

It’s all about keeping what’s relevant and filtering out the rest!

Handling Emojis/Emoticons

Emojis and emoticons can be fun, but they need a little organization when analyzing text:

1. Removing Emojis/Emoticons: You can simply remove them if they don’t add much to the meaning or analysis.
2. Mapping Emojis to Text: Instead of removing them, you can translate them into words! For example, 😀 can become “happy,” keeping the sentiment while making the text easier to process.

This way, you maintain the emotional context or simplify things for cleaner analysis.

Handling URLs, Email Addresses, and Hashtags

Sometimes, URLs, email addresses, and hashtags don’t serve a purpose in text analysis, so here’s how to manage them:

1. URL and Email Removal: URLs and email addresses are often irrelevant for many NLP tasks, so they can be removed to clean up the data.
2. Hashtag Handling: Hashtags can either be stripped out or used to uncover topics and sentiments by extracting their meaning.

These steps help streamline your text while focusing on more valuable insights

Text Augmentation

To enrich your dataset, you can introduce new variations of the text in creative ways:

• Synonym Replacement: Swap out words with their synonyms to create more diverse versions of the original text while preserving meaning.
• Back-Translation: Translate the text into another language, then back to the original language. This generates fresh variations that can add depth to your dataset.

These techniques help expand your data without changing the core message.

Spelling Correction

Ever catch a typo like “realy” instead of “really”? In data processing, those tiny errors can throw off an entire analysis. That’s where spelling correction comes in. It hunts down mistakes and fixes them, ensuring cleaner, more accurate data. With every correction, machines get a clearer understanding of our language, making your data smarter and more reliable.

Noise Removal

• HTML/XML Tags: We kick out those pesky tags from web-crawled data, leaving only the meaningful text behind. No distractions!
• Non-Alphanumeric Characters: Symbols like @, %, and & can muddle the message. We remove these depending on the context, keeping the focus on what really matters.
• Gibberish Text: Ever seen random characters that make no sense? We detect and eliminate this non-informative text, ensuring only relevant content stays.

By cleaning up this noise, we make sure our data is clear, concise, and ready for analysis!

Text Filtering

• Domain-Specific Vocabulary: Depending on the goal, we either keep or remove specialized terms to make sure the data fits the right context. It’s like filtering out technical jargon when it’s not needed!
• Language Detection: We identify the language of the text and keep only the content that matches the desired language. Perfect for making sure you’re working with the right linguistic data.

With smart text filtering, we fine-tune the data so it aligns perfectly with the task at hand.

Handling Missing Data

• Replacing missing text: When we encounter missing text fields, we have options! We can fill in the gaps with a placeholder to keep things tidy or remove the entries entirely if they don’t add value.

By effectively handling missing data, we ensure our datasets remain robust and ready for analysis!

Text Compression

• Truncating Long Documents: When only part of a document matters, we trim it down to size, focusing on the essential information.
• Summarization: We automatically generate concise summaries of longer texts, capturing the key points while saving time.

Stemming

Stemming is the process of reducing words to their root forms, helping to unify similar terms for better analysis. Here are some popular stemming algorithms:

• Porter Stemmer: One of the most widely used stemming algorithms. It applies a set of rules to remove common suffixes, making it simple and effective for many applications.
• Lovins Stemmer: This algorithm stems words by applying a comprehensive list of rules. It’s known for its broad coverage of terms, though it can be less accurate than others.
• Dawson Stemmer: A lesser-known option that uses specific rules tailored for certain datasets, offering flexibility in stemming choices.
• Krovetz Stemmer: Unlike other stemmers, it focuses on retaining the original meaning of words. It’s particularly useful when context is important.
• Xerox Stemmer: An advanced stemming algorithm that employs complex rules to improve accuracy, making it suitable for technical texts.
• Snowball Stemmer: An extension of the Porter Stemmer, offering improvements and support for multiple languages. It’s great for multilingual applications.
• Hunspell Stemmer: Commonly used in spell checkers, this algorithm supports complex morphology and is excellent for languages with rich inflection.

By using these stemming algorithms, we can enhance our text analysis, making our data cleaner and more coherent!

Summary

In the second part of our NLP blog series, we focused on key data processing steps that prepare raw text for analysis and model training.

We covered tokenization, breaking text into smaller units, and normalization, which tidies up text for consistency. We discussed removing stopwords, numbers, and emojis to highlight important content, as well as techniques like text augmentation and spelling correction to enrich datasets.

We explored noise removal and text filtering for relevance, and emphasized handling missing data for robust datasets. Finally, we looked at text compression techniques and various stemming algorithms to unify similar terms.

In the next part, we’ll dive into text representation, vectorization, word embeddings, Zipf’s Law, and data visualizations. Stay tuned for more exciting insights into NLP! Join us in the next part of our An In-Depth NLP Series journey as we dive deeper into machine learning!