Evaluating LLMs (Part 1)- Knowing when we don't know through ensemble based uncertainty quantification

Motivation

Large language models are used in all kinds of settings, from healthcare to robotics. Given its ubiquity, how much we can trust the output of the LLM becomes an important question. The following proposition motivated by intuition aims to quantify the uncertainty of an LLM under the setting of limited training compute.

Problem setting

Given a large language model $p_{\theta}$ , some prompt/question $x_{0}$ and answer $y_{0}=f_{\theta}(x_{0})$ we want to gauge the uncertainty of our answer $y_{0}$ . Here, we consider the predictive (total) uncertainty of our answer $p_{\theta}(y_{0}|x_{0})$ , which combines both model and data uncertainty.

Background and previous methods

There is substantial research on Bayesian Neural Networks for uncertainty quantification; but it is difficult to establish an appropiate prior distribution on the parameters. Moreover, most methods are intractable. In the BNN setting, to estimate predictive uncertainty we can calculate the entropy of the average distribution over multiple parameters: $H[\frac{1}{M}\sum_{m=1}^{M}p_{\theta_{m}}(y|x,\theta_{m})]$ . Alternately, uncertainty quantification has been approached using deep networks, either directly from one network or from an ensemble of networks. We note that one problem that arises from gauging certainty solely using the softmax outputs of the model is that deep networks give overconfident softmax outputs.

(Lakshminarayanan et al. 2017) show good performance in uncertainty quantification using an ensemble of deep networks that are (optionally) adversarially trained through random sampling. To quantify predictive uncertainty, they utilize a proper scoring rule on the ensemble distribution.

Def: A scoring rule $S(p_{\theta},q)$ measures the predictive uncertainty of some distribution $p_{\theta}(y|x)$ -where the real distribution is defined by $q(y|x)$ - by considering the distribution's calibration. A proper scoring rule satistifes the property that $S(p_{\theta},q)\leq S(q,q)$ for all distributions $p_{\theta}$ .

Notably, familiar objective functions such as negative log likelihood and softmax cross entropy are proper scoring rules.

The ensemble distribution for an ensemble of randomly trained models $(\theta_{1},...,\theta_{M})$ is defined as the average $\hat{p}(y|x)=\frac{1}{M}\sum_{m=1}^{M}p_{\theta_{m}}(y|x,\theta_{m})$ and the proper scoring rule, negative log likelihood, is applied on such ensemble distribution to quantify the predictive uncertainty.

(Sun et al. 2022) use an ensemble to quantify the uncertainty of large language models. Instead of using a scoring rule, they quantify uncertainty through ensemble disagreement as measured by the standard deviation of the likelihood of the mode output (defined as the mode of the outputs generated by each model in the ensemble), where the variation of the the likelihood is across the models in the ensemble. The ensemble models are independently fine-tuned GPT2 models with random seeds.

(Lin et al. 2023) quantify the uncertainty of black box large language models-i.e no further training and no access to probability distribution-by generating multiple outputs from a single model and single input and measuring uncertainty through the pairwise similarity scores of the different outputs.

(Wang et al. 2022) show improved performance in chain-of-thought reasoning in language models through self consistency. They consider a diverse set of reasoning paths, where each reasoning path $r_{i}$ can lead to a different answer $a_{i}$ , and choose the most consistent answer from the set of answers $\{a_{i}\}$ . The most consistent answer is defined by a majority vote over the answers, $argmax_{a}\sum_{i=1}^{m}\boldsymbol{1}\{a_{i}=a\}$ . (Note: The authors lack details on how to calculate the majority vote.)

Proposed Method

Inspired by ensemble-based uncertainty quantification and working with LLM training limitations, we consider a single model $p_{\theta}$ . Following the self consistency procedure in (Wang et al 2022), we get a diverse set of multiple reasoning paths and answers $(r_{i},a_{i})$ and also the most consistent answer $a$ . Then, rather than finetuning or independently training multiple models, we consider an ensemble of distributions $p_{\theta}(a|x_{0},r_{i})$ defined by the various reasoning paths $r_{i}$ and the question/prompt in question $x_{0}$ .

Following (Sun et al. 2022) we could calculate uncertainty by measuring ensemble disagreement through the standard deviation of the likelihood $\{p_{\theta}(a|x_{0},r_{i})\}$ across the reasoning paths. The intuition is that a confidently correct answer should have a similar high likelihood even when under different reasoning paths. Conversely in the case where the model generates various divergent answers $a_{i}$ through different reasoning paths, we do not expect a similar high likelihood across the different reasoning paths for $a$ . It could be that we get a similar low likelihood under the different reasoning paths, so we should check the mean of the likelihood in addition to its standard deviation.

Following (Lakshminarayanan et al. 2017) we can also calculate predictive uncertainty through the proper scoring rule of negative log likelihood on the "ensemble" distribution $\hat{p}(a|x_{0})=\frac{1}{M}\sum_{m=1}^{M}p_{\theta}(a|x_{0},r_{i})$ to estimate the uncertainty of our most consistent answer. The intuition is that (Lakshminarayanan et al.) consider an ensemble of different models trained through randomly sampled data, which incorporates variety stemming from the stochasticity of the training process. An ensemble of probability distributions defined by different reasoning paths incorporates variety stemming from the stochasticity that arises from sampling different reasoning paths in the self-consistency method.

Citation

@article{jung2024llmuq,
  title   = "Knowing when we don't know through ensemble based uncertainty quantification",
  author  = "Jung, Justin",
  journal = "Deep Exploration",
  year    = "2024",
  month   = "Jan",
  url     = "https://deep-exploration.vercel.app/blog/llm-uncertainty-quantification"
}

References

B. Lakshminarayanan, A. Pritzel, and C. Blundell, "Simple and scalable predictive uncertainty estimation using deep ensembles," in Advances in neural information processing systems, 2017, pp. 6402--6413.
M. Sun, W. Yan, P. Abbeel, and I. Mordatch, "Quantifying Uncertainty in Foundation Models via Ensembles," in NeurIPS 2022 Workshop on Robustness in Sequence Modeling, 2022. [Online]. Available: https://openreview.net/forum?id=LpBlkATV24M
X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, "Self-Consistency Improves Chain of Thought Reasoning in Language Models," arXiv preprint arXiv:2203.11171, 2023. [Online]. Available: https://arxiv.org/abs/2203.11171
J. Gawlikowski, C. R. N. Tassi, M. Ali, J. Lee, M. Humt, J. Feng, A. M. Kruspe, R. Triebel, P. Jung, R. Roscher, M. Shahzad, W. Yang, R. Bamler, and X. X. Zhu, "A Survey of Uncertainty in Deep Neural Networks," CoRR, vol. abs/2107.03342, 2021. [Online]. Available: https://arxiv.org/abs/2107.03342
M. Abdar, F. Pourpanah, S. Hussain, D. Rezazadegan, L. Liu, M. Ghavamzadeh, P. W. Fieguth, X. Cao, A. Khosravi, U. R. Acharya, V. Makarenkov, and S. Nahavandi, "A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges," CoRR, vol. abs/2011.06225, 2020. [Online]. Available: https://arxiv.org/abs/2011.06225
Z. Lin, S. Trivedi, and J. Sun, "Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models," arXiv preprint arXiv:2305.19187, 2023. [Online]. Available: https://arxiv.org/abs/2305.19187