Sentences: 3
Words: 30
First sentence tokens: ['Natural', 'language', 'processing', '(', 'NLP', ')', 'is', 'a', 'fascinating', 'field', '.']
Filtered words: ['Natural', 'language', 'processing', 'NLP', 'fascinating', 'field', 'enables', 'computers', 'understand', 'process', 'human', 'language', 'NLTK', 'provides', 'excellent', 'tools', 'learning', 'NLP', 'concepts']
Original Stemmed Lemmatized
---------------------------------------------
running run run
runs run run
ran ran run
easily easili easily
fairly fairli fairly NLP Packages Overview
This document provides an overview of three powerful Python packages for Natural Language Processing: NLTK, spaCy, and BERTopic.
NLTK (Natural Language Toolkit)
Overview
NLTK is one of the oldest and most comprehensive Python libraries for NLP, originally created for teaching and research.
Key Features:
- Extensive collection of text processing tools
- Access to over 50 corpora and lexical resources (WordNet, TreeBank)
- Text classification, tokenization, stemming, tagging, parsing
- Educational focus with extensive documentation
Best For:
- Learning NLP concepts
- Prototyping and research
- Working with linguistic data structures
- Academic projects and teaching
Limitations:
- Slower than modern alternatives
- Less suited for production environments
- Requires more manual pipeline construction
NLTK Example: Basic Text Processing
NLTK Example: Part-of-Speech Tagging
Part-of-Speech Tags:
The -> DT
quick -> JJ
brown -> NN
fox -> NN
jumps -> VBZ
over -> IN
the -> DT
lazy -> JJ
dog -> NNNLTK Example: Sentiment Analysis
Sentiment Analysis Results:
------------------------------------------------------------
Text: I absolutely love this product! It's amazing!
Negative: 0.000, Neutral: 0.311, Positive: 0.689
Compound Score: 0.871
Text: This is terrible. I hate it.
Negative: 0.694, Neutral: 0.306, Positive: 0.000
Compound Score: -0.778
Text: It's okay, nothing special.
Negative: 0.367, Neutral: 0.325, Positive: 0.309
Compound Score: -0.092
Text: The weather is nice today.
Negative: 0.000, Neutral: 0.588, Positive: 0.412
Compound Score: 0.421
spaCy
Overview
spaCy is a modern, industrial-strength NLP library designed for production use.
Key Features:
- Fast and efficient (Cython-optimized)
- Pre-trained statistical models for multiple languages
- Built-in support for NER, POS tagging, dependency parsing
- Easy integration with deep learning frameworks (PyTorch, TensorFlow)
- Beautiful visualization tools (displaCy)
Best For:
- Production NLP pipelines
- Real-time processing
- Named Entity Recognition
- Document similarity and classification
- Information extraction at scale
Limitations:
- Less flexible than NLTK for research
- Fewer resources for learning basic concepts
- Model-dependent (needs pre-trained models)
spaCy Example: Basic Text Analysis
Tokens and their attributes:
Token Lemma POS Is Stop?
--------------------------------------------------
Apple Apple PROPN False
Inc. Inc. PROPN False
is be AUX True
planning plan VERB False
to to PART True
open open VERB False
a a DET True
new new ADJ False
store store NOUN False
in in ADP True spaCy Example: Named Entity Recognition
Named Entities:
Entity Type Explanation
----------------------------------------------------------------------
Apple Inc. ORG Companies, agencies, institutions, etc.
San Francisco GPE Countries, cities, states
next month DATE Absolute or relative dates or periods
Tim Cook PERSON People, including fictional
Apple Inc.
ORG
is planning to open a new store in
San Francisco
GPE
next month
DATE
.
The CEO, Tim Cook PERSON , announced this during a press conference.
The CEO, Tim Cook PERSON , announced this during a press conference.
spaCy Example: Dependency Parsing
Dependency Parse:
Token Dependency Head Children
--------------------------------------------------
The det fox -
quick amod fox -
brown amod fox -
fox nsubj jumps The, quick, brown
jumps ROOT jumps fox, over
over prep jumps dog
the det dog -
lazy amod dog -
dog pobj over the, lazy spaCy Example: Document Similarity
Document Similarity (using word vectors):
doc1 <-> doc2: 0.839
doc1 <-> doc3: 0.271
doc2 <-> doc3: 0.322
Word Similarity:
king <-> queen: 0.422
king <-> apple: 0.690BERTopic
Overview
BERTopic is a modern topic modeling technique that leverages transformer-based embeddings.
Key Features:
- Uses BERT embeddings for semantic understanding
- Automatically determines optimal number of topics
- UMAP for dimensionality reduction
- HDBSCAN for clustering
- Class-based TF-IDF (c-TF-IDF) for topic representation
- Interactive visualizations
Best For:
- Topic discovery in document collections
- Short text analysis (tweets, reviews, articles)
- Dynamic topic modeling over time
- High-quality, interpretable topics
- Modern alternative to LDA
Limitations:
- Computationally expensive (needs embeddings)
- Requires more memory than classical methods
- Slower than LDA for very large corpora
- GPU recommended for large datasets
BERTopic Example: Basic Topic Modeling
Training BERTopic model...
Discovered 1 topics (excluding outliers)
Outlier documents (topic -1): 0BERTopic Example: Explore Topics
Topic Information:
Topic Count Name
0 0 484 0_the_to_of_and
1 1 16 1_anaheim___
Top Words per Topic:
================================================================================
Topic 0: the, to, of, and, in, is, that, forBERTopic Example: Topic Visualization
#| code-fold: false
# Visualize topics
fig = topic_model.visualize_topics()
fig.show()
# Visualize topic hierarchy
fig_hierarchy = topic_model.visualize_hierarchy(top_n_topics=10)
fig_hierarchy.show()
# Visualize barchart for top topics
fig_barchart = topic_model.visualize_barchart(top_n_topics=5, n_words=10)
fig_barchart.show()BERTopic Example: Find Similar Documents
Topics similar to 'space exploration and satellites':
Topic 0 (similarity: 0.207):
Key words: the, to, of, and, in
Topic 1 (similarity: 0.122):
Key words: anaheim, , , , BERTopic Example: Dynamic Topic Modeling
Topics Over Time:
Topic Words Frequency Timestamp
0 0 the, to, of, and, in 48 2019-12-31 12:01:26.400
1 1 anaheim, , , , 2 2019-12-31 12:01:26.400
2 0 the, of, to, and, in 49 2020-02-19 21:36:00.000
3 1 anaheim, , , , 1 2020-02-19 21:36:00.000
4 0 the, of, and, to, in 49 2020-04-09 19:12:00.000
5 1 anaheim, , , , 1 2020-04-09 19:12:00.000
6 0 the, to, and, of, in 49 2020-05-29 16:48:00.000
7 1 anaheim, , , , 1 2020-05-29 16:48:00.000
8 0 the, to, of, and, that 49 2020-07-18 14:24:00.000
9 1 anaheim, , , , 1 2020-07-18 14:24:00.000
10 0 the, to, and, of, in 49 2020-09-06 12:00:00.000
11 1 anaheim, , , , 1 2020-09-06 12:00:00.000
12 0 the, to, and, of, in 47 2020-10-26 09:36:00.000
13 1 anaheim, , , , 3 2020-10-26 09:36:00.000
14 0 the, to, of, and, in 50 2020-12-15 07:12:00.000Package Comparison
Quick Comparison Table
| Feature | NLTK | spaCy | BERTopic |
|---|---|---|---|
| Primary Use | Education, Research | Production NLP | Topic Modeling |
| Speed | Slow | Fast | Moderate |
| Ease of Use | Moderate | Easy | Easy |
| Pre-trained Models | Limited | Excellent | Uses transformer embeddings |
| Customization | High | Moderate | Moderate |
| Memory Usage | Low | Low-Moderate | High |
| Best For | Learning, Prototyping | NER, Pipelines, Real-time | Topic Discovery |
| Visualization | Limited | Excellent (displaCy) | Excellent (interactive) |
| GPU Support | No | Yes (for training) | Recommended |
| Community | Large, Academic | Large, Industry | Growing |
When to Use Each Package
Use NLTK when:
- Learning NLP concepts
- Need access to linguistic resources (WordNet, TreeBank)
- Working on academic research
- Prototyping ideas
- Need maximum flexibility
Use spaCy when:
- Building production systems
- Need fast, accurate NER
- Processing large volumes of text
- Want beautiful visualizations
- Need dependency parsing
- Building information extraction pipelines
Use BERTopic when:
- Discovering topics in document collections
- Working with short texts (tweets, reviews)
- Need interpretable, coherent topics
- Want to avoid specifying number of topics
- Have access to GPU resources
- Analyzing topic evolution over time
Combining Packages
These packages can be used together effectively:
Document: Apple Inc. announced new products in Cupertino yesterday.
Entities: [('Apple Inc.', 'ORG'), ('Cupertino', 'GPE'), ('yesterday', 'DATE')]
Document: Google is developing AI technology in Mountain View.
Entities: [('Google', 'ORG'), ('AI', 'ORG'), ('Mountain View', 'GPE')]
Document: Microsoft released a new version of Windows in Seattle.
Entities: [('Microsoft', 'ORG'), ('Windows', 'NORP'), ('Seattle', 'GPE')]
Processed documents:
1. Apple Inc. announce new product Cupertino yesterday
2. Google develop AI technology Mountain View
3. Microsoft release new version Windows SeattleSummary
- NLTK: Comprehensive, educational, flexible but slower
- spaCy: Fast, production-ready, excellent for NER and pipelines
- BERTopic: Modern topic modeling with transformer embeddings
Choose based on your specific needs: learning (NLTK), production (spaCy), or topic discovery (BERTopic). Often, combining these tools yields the best results!