Core Concepts of Text Preprocessing & Tokenization in NLP — Questions & Answers

Test your understanding of essential NLP preprocessing techniques, including Unicode normalization, case-folding, punctuation and whitespace handling, stopword removal, and word-frequency mapping. This quiz is designed to strengthen your knowledge of foundational steps in preparing text data for natural language processing tasks.

This quiz contains 16 questions. Below is a complete reference of all questions, answer choices, and correct answers. You can use this section to review after taking the interactive quiz above.

1. Question 1: Unicode Normalization Purpose

   In text preprocessing for NLP, what is the main goal of Unicode normalization?

   • To add extra symbols to every word
   • To count the number of unique words in a text
   • To convert visually identical characters into a standard form
   • To store data in a compressed format
   Show correct answer

   Correct answer: To convert visually identical characters into a standard form

   Explanation: Unicode normalization ensures that characters with different byte representations but similar appearance, like accented characters, are standardized for consistency. Counting unique words is related to frequency mapping but not normalization. Adding extra symbols or compressing data are unrelated to this specific process. The correct answer focuses on textual standardization, which aids downstream NLP tasks.

2. Question 2: Case-Folding Functionality

   Why do NLP pipelines often use case-folding, such as converting all text to lowercase?

   • To identify the meaning of capitalized words
   • To reduce variability caused by uppercase and lowercase letters
   • To translate text into another language
   • To remove all special characters from text
   Show correct answer

   Correct answer: To reduce variability caused by uppercase and lowercase letters

   Explanation: Case-folding standardizes text, so identical words like 'Dog' and 'dog' are treated the same, simplifying analysis. Recognizing the meaning of capitalized words is not the main aim of this step. Removing special characters is related to punctuation handling, not case-folding. Translation into another language is unrelated to this process.

3. Question 3: Handling Punctuation in Tokenization

   When tokenizing text for NLP, why might punctuation be removed?

   • To increase the number of tokens for better results
   • To ensure every sentence ends with a period
   • To identify different languages within the same text
   • To minimize irrelevant information in word analysis
   Show correct answer

   Correct answer: To minimize irrelevant information in word analysis

   Explanation: Removing punctuation can reduce noise, helping focus on the meaningful content of words for tasks like frequency analysis. Increasing token count is not always beneficial if it adds noise. Ensuring sentences end with periods or identifying languages are not primary reasons for punctuation removal.

4. Question 4: Whitespace Normalization

   What does whitespace normalization achieve in text preprocessing?

   • It removes all words shorter than four letters
   • It merges consecutive spaces and tidies up line breaks
   • It adds random spaces for data augmentation
   • It translates spaces into punctuation marks
   Show correct answer

   Correct answer: It merges consecutive spaces and tidies up line breaks

   Explanation: Whitespace normalization ensures formatting consistency, like replacing multiple spaces with one and removing stray line breaks, which is important for accurate tokenization. Translating spaces into punctuation or adding random spaces are not typical preprocessing steps. Removing short words is a separate possible filtering process, not whitespace normalization.

5. Question 5: Stopword Filtering Motivation

   Why is stopword removal commonly used when preparing text for NLP tasks?

   • To convert numbers into their word forms
   • To remove very common words that contribute little unique meaning
   • To count sentences rather than words
   • To discard all non-English words
   Show correct answer

   Correct answer: To remove very common words that contribute little unique meaning

   Explanation: Stopwords are frequent words like 'the' or 'and' which often add little semantic value, so removing them can highlight more informative terms. Discarding non-English words or converting numbers is outside the scope of stopword filtering. Counting sentences is unrelated to removing stopwords.

6. Question 6: Word-Frequency Map Trade-Offs

   Which is a key trade-off when building word-frequency maps (such as dictionaries) during preprocessing?

   • Translating each word to its synonym
   • Balancing memory usage with lookup speed
   • Increasing the number of stopwords detected
   • Improving punctuation detection accuracy
   Show correct answer

   Correct answer: Balancing memory usage with lookup speed

   Explanation: Storing word counts in frequency maps offers fast access, but uses memory that grows with vocabulary size. Increasing stopword detection or punctuation accuracy relates to other steps. Translating to synonyms is not part of frequency mapping.

7. Question 7: Example of Tokenization

   Given the sentence 'Cats, dogs! Fish?', what is a possible tokenization result after removing punctuation?

   • ['Cats', 'dogs', 'Fish']
   • ['Cats dogs Fish']
   • ['Cats,', 'dogs!', 'Fish?']
   • ['Cats@', 'dogs#', 'Fish&']
   Show correct answer

   Correct answer: ['Cats', 'dogs', 'Fish']

   Explanation: After removing punctuation, each word is separated cleanly, resulting in tokens like 'Cats', 'dogs', and 'Fish'. The second option leaves punctuation attached, which tokenization seeks to avoid. The third contains non-present symbols. The fourth does not split into separate tokens but combines words into one string.

8. Question 8: Normalization and Diacritics

   How does Unicode normalization help with diacritics in words such as 'café'?

   • It converts words into lemmatized forms
   • It reverses the word order in a sentence
   • It ensures words with different accent styles are treated identically
   • It removes all accented letters from the text
   Show correct answer

   Correct answer: It ensures words with different accent styles are treated identically

   Explanation: Unicode normalization ensures that words with the same base letters but different encodings for accents (diacritics) are processed in a standardized way. Removing all accented letters would strip meaning from words. Reversing word order and lemmatization are unrelated processes.

9. Question 9: Case-Folding Side Effect

   What is one possible downside of using case-folding during preprocessing?

   • All special characters are automatically deleted
   • Token boundaries become inconsistent
   • The language of the text is changed
   • Important distinctions such as proper nouns may be lost
   Show correct answer

   Correct answer: Important distinctions such as proper nouns may be lost

   Explanation: Converting all text to lowercase removes cues like capitalization, which could be important for identifying names or sentence starts. It does not delete special characters, which is a different process. Token boundaries are not directly affected. Language does not change due to case-folding.

10. Question 10: Stopword List Customization

    Why might a stopword list need to be customized for specific NLP applications?

    • Stopwords always include technical jargon
    • All stopwords are the same in every language
    • Adding stopwords increases processing speed
    • Certain words may be meaningful in some contexts but not others
    Show correct answer

    Correct answer: Certain words may be meaningful in some contexts but not others

    Explanation: Depending on the domain, some common words could carry specific meaning and should not be removed as stopwords. Stopword lists differ by language and context. Simply adding stopwords doesn't necessarily speed up processing. Stopword lists generally exclude jargon, except in specialized cases.

11. Question 11: Whitespace Impact on Tokenization

    How can inconsistent whitespace in a text file affect tokenization?

    • It converts numbers to words automatically
    • It ensures all tokens are proper nouns
    • It adds new vocabulary to the frequency map
    • It can cause incorrect splitting of words or extra empty tokens
    Show correct answer

    Correct answer: It can cause incorrect splitting of words or extra empty tokens

    Explanation: Irregular spaces or line breaks can split words incorrectly or add empty tokens, lowering the quality of tokenization. Vocabulary isn't increased due to whitespace alone. Numbers to words and token type changes are handled by other processes.

12. Question 12: Dictionary vs. List for Word Frequencies

    Why is a dictionary (hash map) usually preferred over a list for storing word frequencies?

    • Because it orders words alphabetically by default
    • Because it consumes less memory for large vocabularies
    • Because it provides faster lookup and update times
    • Because it automatically removes stopwords
    Show correct answer

    Correct answer: Because it provides faster lookup and update times

    Explanation: Dictionaries (hash maps) give quick access to counts for each word, making frequency updates efficient. They do not order words, remove stopwords, or always use less memory than lists; in fact, they can use more memory but offer much faster operations.

13. Question 13: Basic Stopword Example

    In the sentence 'She and her friend went to the park', which words are most likely to be removed as stopwords?

    • 'and', 'friend', 'her'
    • 'went', 'friend', 'the'
    • 'She', 'friend', 'park'
    • 'and', 'her', 'to', 'the'
    Show correct answer

    Correct answer: 'and', 'her', 'to', 'the'

    Explanation: Common English words like 'and', 'her', 'to', and 'the' are typical stopwords because they don't add much unique meaning. 'She', 'friend', and 'park' are more content-specific and usually kept. 'Went' is a verb, not a stopword.

14. Question 14: Frequency Map Example

    If the word 'apple' appears three times and 'banana' once in a text, what would the frequency map look like?

    • {'apple': 'banana', 3: 1}
    • {'apple': 1, 'banana': 3}
    • {'apple': 3, 'banana': 1}
    • {'apple', 'banana'}
    Show correct answer

    Correct answer: {'apple': 3, 'banana': 1}

    Explanation: In a frequency map, each word is a key and its count is the value, so 'apple' maps to 3 and 'banana' to 1. The other options misrepresent the keys or values, list only words, reverse counts, or mix types.

15. Question 15: Splitting on Whitespace Only

    What is a limitation of simple whitespace-based tokenization?

    • It removes out-of-vocabulary words automatically
    • It increases storage efficiency
    • It does not handle punctuation adjoining words, such as 'hello!'
    • It accurately detects named entities
    Show correct answer

    Correct answer: It does not handle punctuation adjoining words, such as 'hello!'

    Explanation: Whitespace tokenization splits text at spaces and ignores punctuation, so words like 'hello!' will include the exclamation mark, possibly reducing analysis accuracy. It doesn't impact storage efficiency, out-of-vocabulary removal, or entity detection.

16. Question 16: Tokenization Output Type

    After a typical word-level tokenization process, what type of data structure is commonly produced?

    • A set of unique token lengths
    • A single large string containing all tokens
    • A numerical array representing token positions
    • A list of strings, where each string is a token
    Show correct answer

    Correct answer: A list of strings, where each string is a token

    Explanation: Tokenization results in a list of individual word strings, making further text analysis straightforward. It does not create one concatenated string, a numeric position array, or a set of token lengths.