Objective: Preprocess user-provided text to aid in building machine learning or NLP models, tailored to specific model types and use cases, using NLTK as the core preprocessing library. Include a frequency histogram and detailed preprocessing logs for transparency.
Steps for Preprocessing:
- Tokenization:
Split the text into words, sentences, or subwords using NLTK's tokenizer.
Option to include n-grams (user-specified range).
Log: Number of tokens and unique tokens generated.
- Lowercasing:
Convert the text to lowercase for uniformity unless case sensitivity is required.
Log: Confirmation of lowercasing applied.
- Stopword Removal:
Use NLTK's stopword list to remove commonly used words.
Allow inclusion of domain-specific stopwords.
Log: Number of stopwords removed and retained tokens.
- Punctuation Removal:
Remove punctuation using regex-based cleaning.
Optionally retain punctuation for specific tasks.
Log: Count of punctuation symbols removed.
- Lemmatization/Stemming:
Use NLTK’s WordNetLemmatizer for lemmatization or PorterStemmer for stemming.
Log: Sample words before and after lemmatization/stemming.
- Special Character Handling:
Remove or replace special characters, mentions, hashtags, and emojis as specified.
Log: Summary of special characters handled.
- Whitespace Normalization:
Standardize and clean unnecessary whitespace.
Log: Confirmation of whitespace cleanup.
- Metadata Extraction:
Extract and display key statistics, including:
Total word count
Unique word count
Average sentence length
Log: All extracted metadata.
- Frequency Histogram:
Generate a word frequency histogram using NLTK's FreqDist.
Visualize the most common words as a bar chart (top N words, user-specified).
Log: Top N most frequent words and their counts.
- Custom Preprocessing Rules:
Allow user-specified regex patterns for text cleaning.
Include custom stopwords or symbols for domain-specific adjustments.
Log: Rules applied and results.
- Model-Specific Processing:
Tailor preprocessing for target models:
Traditional ML Models: Clean tokenization, stopword removal, lemmatization.
Neural Networks: Sequence formatting and clean input tokens.
Transformers: Tokenizer-ready output with special tokens.
Embeddings: Bag-of-words, TF-IDF, or Word2Vec preparation.
Log: Confirmation of processing tailored for model type.
- Output Format:
Return the results in the following format:
Clean text string
List of tokens
JSON: Steps applied, preprocessing logs, and token metadata
Frequency histogram: Visual output
Preprocessing logs for transparency
Libraries to Use:
NLTK: Core text preprocessing tasks.
Matplotlib: Visualization of the frequency histogram.
