TopicScope — Interactive LDA Topic Discovery Dashboard

A local, free, multipage Streamlit app that runs the full topic-modelling pipeline (load → preprocess → document-term matrix → LDA → interpret → evaluate → export) on 20 Newsgroups or your own documents, and compares scikit-learn variational LDA with Tomotopy Gibbs-sampling LDA.

Features

Nine pages: Overview, Dataset Explorer, Preprocessing Lab, Document-Term Matrix, LDA Training, Topic Explorer (editable topic names), Document Explorer, Evaluation Report (UMass coherence, method comparison, optional topic-count sweep), Export Centre (CSV/JSON/NPZ/PNG/model files/report/ZIP).

Stack

Python, Streamlit, pandas, NumPy/SciPy, scikit-learn, NLTK, Tomotopy, Plotly, Matplotlib, pytest. No paid API, cloud service or GPU.

Folder structure

app.py (navigation) · pages/1..9_*.py (UI) · src/ (data_loader, preprocessing, vectorization, lda_models, topic_extraction, evaluation, visualization, report_generator, export_utils, styles, state) · tests/ · data/sample (a small synthetic demo CSV) · data/cache (downloaded dataset, git-ignored) · models/, reports/ (generated, git-ignored).

Inputs

Built-in 20 Newsgroups (2,000 default / 3,000 / 5,000 / full), CSV (choose the text column), multiple TXT files (one file = one document). Original labels of 20 Newsgroups are kept as metadata only and are never given to either model.

LDA in brief

Each topic k is a distribution P(word | topic); each document d has a mixture P(topic | document). Priors: alpha (document-topic) and eta (topic-word, often called beta). Defaults here: alpha=0.1, eta=0.01 for both methods.

• Variational inference (scikit-learn) optimises a deterministic approximation to the posterior; results depend on the seed/initialisation.
• Gibbs sampling (Tomotopy) repeatedly resamples each word's topic from its conditional distribution; Tomotopy also re-estimates alpha every optim_interval iterations by default (set 0 to disable). Use one worker for reproducible results.

Both methods use the same document subset, tokens and vocabulary (Tomotopy receives the vocabulary-filtered tokens). Unavoidable differences: different algorithms, vocabulary ordering, and Tomotopy's alpha optimisation.

Evaluation

UMass coherence (own, tested implementation; mean over word pairs, closer to 0 is better) and topic diversity. Gensim c_v is not used. Likelihood/perplexity are model-specific and are not compared across libraries. Coherence does not prove a topic is meaningful.

Privacy

Everything runs locally; uploaded documents are never sent anywhere. Only the first 20 Newsgroups download (and optional NLTK download) uses the internet; afterwards the cached copy in data/cache works offline.

Setup (Windows PowerShell)

cd TopicScope
python -m venv .venv
.\.venv\Scripts\Activate.ps1      # if blocked: Set-ExecutionPolicy -Scope Process RemoteSigned
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m streamlit run app.py
python -m pytest -q               # run the tests

Optional (once): in the Preprocessing Lab use the NLTK download button, or python -c "import nltk; nltk.download('stopwords'); nltk.download('wordnet')".

Tested versions

Python 3.12.3 on Linux x86_64 with the pinned versions in requirements.txt. Not yet tested on Windows.

Known limitations

• scikit-learn gives no per-iteration progress; Tomotopy progress is real.
• Variational LDA is seed-sensitive (see tests/test_lda_models.py).
• English-only default preprocessing; full 20 Newsgroups needs more RAM than 4 GB may comfortably allow.
• Saved scikit-learn models are pickles — only load files you created.

Troubleshooting

• Tomotopy fails to install: use a Python version with a published wheel (check https://pypi.org/project/tomotopy/#files), e.g. create the venv with py -3.12 -m venv .venv; the rest of the app still works with the scikit-learn method.
• Dataset download fails: check your connection; the first run needs internet.
• Missing NLTK data: the app falls back to scikit-learn stopwords and skips lemmatization.