This project demonstrates the implementation of multiple Natural Language Processing (NLP) techniques using Python in a Jupyter Notebook. The goal is to process raw text data (news articles, product reviews, or social media posts) and apply various NLP tasks while comparing different approaches.
- Regex-based text cleaning (removal of URLs, hashtags, emails)
- Tokenization-based cleaning
- Comparison of both methods
- Implemented using NLTK
- Compared accuracy and efficiency
- Implemented using spaCy
- Compared rule-based vs ML-based approaches
- POS tagging using NLTK
- N-gram models (Unigram, Bigram, Trigram)
- Next word prediction analysis
- Dictionary-based approach using WordNet
- Compared with ML-based methods
- Implemented Word2Vec using Gensim
- Discussed GloVe comparison
- Used embeddings for text representation
- Context-Free Grammar (CFG)
- CKY Parsing
- Dependency Parsing (Subject–Verb–Object extraction)
- Comparison with constituency parsing
- Basic heuristic-based pronoun resolution
- Compared with deep learning approaches
| Task | Method 1 | Method 2 | Conclusion |
|---|---|---|---|
| Text Cleaning | Regex | Tokenization | Tokenization is more flexible |
| Stemming vs Lemmatization | Stemming | Lemmatization | Lemmatization is more accurate |
| NER | Rule-based | ML-based | ML-based performs better |
| N-grams | Unigram | Bigram/Trigram | Higher n improves context |
| Parsing | CKY | Earley | CKY is efficient but rigid |
- Python 🐍
- Jupyter Notebook 📓
- NLTK
- spaCy
- Gensim
- Regular Expressions (re)
📁 NLP_Assignment
├── 📄 NLP_Assignment.ipynb
├── 📄 README.md
- Clone the repository:
git clone https://github.com/your-username/NLP_Assignment.git- Install dependencies:
pip install nltk spacy gensim pandas
python -m spacy download en_core_web_sm- Run the notebook:
jupyter notebook- Lemmatization provides better linguistic accuracy than stemming.
- ML-based NER significantly outperforms rule-based approaches.
- Trigram models improve prediction accuracy but increase complexity.
- Dependency parsing is more useful for extracting relationships than constituency parsing.
This project covers fundamental and advanced NLP techniques and highlights the trade-offs between different approaches in terms of accuracy, efficiency, and complexity.
Sneha Sarraf SRM Institute of Science and Technology