Uncover
Hidden Structure
in Text Corpora

This almost always means min_topic_size is too large relative to your corpus. HDBSCAN requires at least min_topic_size documents to form a cluster — if your corpus has 50 documents and min_topic_size is 20, only themes with 20+ documents will appear as topics; everything else becomes outlier noise (-1). Try reducing min_topic_size to 5–10 for small corpora. Also check that your ngrok/Colab URL is active — a disconnected backend will return an empty response silently.

BERTopic uses deep neural embeddings (BERT) to understand the meaning of words in context. It is more accurate, especially for complex or ambiguous language, but requires a GPU and significant computation time. It works best on corpora of 500+ documents.

LDA treats each document as a bag of words — word order and context are ignored. It is much faster, runs on any machine, and works well on corpora as small as 50 documents. For most research use cases, LDA is the practical starting point and BERTopic is used to validate or deepen the findings.

It depends on the method. LDA works from around 50 documents, though 200+ produces more stable results. BERTopic needs at least 100–200 documents and performs best above 500. TF-IDF and Sentiment Analysis work on any number of documents — even a single file. Word Cloud follows LDA's requirements since it runs LDA internally. As a general rule: more documents → more reliable topics.

This is the most common preprocessing problem. Domain-specific high-frequency words pass through the stopword filter because they are not in the standard English stopword list. The fix is to add these words to the Custom Stopwords field (comma-separated) and rerun. Also try enabling Lemmatization and lowering Max Doc Frequency to automatically exclude corpus-wide terms.

Use the Coherence Score tool — it automatically tests a range of K values and identifies the one producing the most semantically coherent topics. As a starting heuristic, try K ≈ √(N/2) where N is the number of documents (e.g. 200 documents → start around K = 10). Always inspect the actual topic words after selecting K — a coherence-optimal K that produces uninterpretable topics is less useful than a slightly lower K with clearer themes.

VADER was designed for social media and informal English. It struggles with sarcasm, formal academic writing, domain-specific language, and non-English text. If your corpus uses specialised vocabulary or implied sentiment, switch to RoBERTa (requires Colab). For non-English text, neither model will perform reliably.

TXT is fastest — plain text requires no parsing. XLSX is medium speed. PDF is slowest — the binary structure is parsed page by page, and scanned/image-based PDFs return empty text. Use the ⚡ to TXT button next to any uploaded PDF to convert it client-side before running analysis.

ngrok URLs are session-based — they expire when the Colab runtime is idle or restarted. Each time you rerun the notebook, a new URL is generated. Click ⚙ Advanced, paste the new URL from your Colab output, and click Test Connection. The Render backend for LDA/VADER is always-on but may take 30–60 seconds to wake from idle.