Calculating Topic Diversity in BERTopic using OCTIS

[ovi97]

Hello @MaartenGr. Thank you for making BERTopic so flexible to make Topic Modelling to fun to use.

Anyways I have several implementations of BERTopic using different Sentence Transformers, Dimensionality Reduction, and Clustering techniques.

However, I would love to evaluate these models numerically using different metrics. I have been able to calculate the Coherent scores using Gensim. But I want to calculate Topic Diversity, Pairwise Jaccard Similarity, and other diversity and similarity metrics using the OCTIS library.

However, when I checked their implementation here(https://colab.research.google.com/github/MIND-Lab/OCTIS/blob/master/examples/OCTIS_LDA_training_only.ipynb#scrollTo=18Ayd5ZaBrSp),

these measures need a dictionary of

1)'test-topic-document-matrix' ---> this is an array
2) 'topic-document-matrix' ---> this is an array
3) 'topic-word-matrix' ---> this is an array
4) topic -----> this is a list of words in each topic

I can get the parameters in 4, but can not get the matrices in the first 3.

What do these arrays mean and how can they be computed in BERTopic. Thanks

[MaartenGr]

For the paper accompanying the code, there is a repository available from which you can use some code to evaluate BERTopic based on topic diversity. You can find that repository here. Without having look at it too much in-depth, I believe you only need a single parameter for calculating topic diversity as you can find in their code and documentation here. This would make it much easier for you to calculate topic diversity and prevents having to create those arrays you refer to.

[ovi97]

Thanks, I have been able to compute these values. Also, is it the same procedure when computing the pairwise Jaccard similarity index?

[MaartenGr]

I have not tried that one out before but from what I can see it is the same procedure. To be sure, I would highly advise reading through the code and docstrings here for a description on what is necessary.

[tawfiqam]

Hello @MaartenGr, I am trying to reproduce this code.

However, I am wondering if there is a way to simply apply the OCTIS evaluations directly on a number of models that I have created. The models I created are pegged at different cluster sizes. I am hoping to get topic diversity (and depending on the size of the model) cohesion score (it requires way too much RAM for larger models).

[MaartenGr]

To do that, I would advise either reading through how I approached it in the repository that you refer to or go through the documentation of OCTIS. Essentially, you would only need to remove the training step in the BERTopic evaluation repo and instead use the models that you created.

[tawfiqam]

Got it. Thanks!

[vinnyricciardi]

I found a nice way to do this in an Octis discussion here

in short:

from octis.evaluation_metrics.diversity_metrics import TopicDiversity

model_output = {'topics': [['word_0', 'word_1', 'word_2'], ['word_3', 'word_4', 'word_5']]}
metric = TopicDiversity(topk=3) # Initialize metric
topic_diversity_score = metric.score(model_output) # Compute score of the metric

[chrissydonaldson]

What is the best way to calculate topic diversity when my n_gram_range is set to something other than (1,1), for example bigrams?

I don't think it makes sense to do this sort of thing: topic_model.get_topic(i)[:topk] as I'll get a list of phrases per topic not a list of words per topic.