Background/Aims: To date, some large language models (LLMs), such as Chat Generative Pre-trained Transformer 4-omni (ChatGPT-4o), can process images through visual transformer patching. An LLM analysis was conducted to assess the ability of ChatGPT-4o to assign the Mayo endoscopic score (MES). Methods: A selection set of high-quality endoscopic frames was identified to compare 4 input models to select the most performant, confirmed in an extended set of 304 frames. Concordance with the evaluation by expert endoscopists was assessed. Results: Only one of the pre-tested models demonstrated significant concordance (κ=0.232, 95% confidence interval [CI]=0.167 to 0.296, P=0.003; ρ=0.36, P=0.011), with a mean bias of –0.26±1.192 (95% CI, –2.596 to 2.076). This was confirmed in the extended set (κ=0.260, 95% CI=0.195 to 0.324, P<0.001; ρ=0.288, P<0.001). The absolute concordance for the selected model was 44% and 45.3% in the selection and extended sets, respectively. For the identification of moderately-to-severely active disease, a sensitivity of 73% (95% CI, 60% to 82%), specificity of 60% (95% CI, 54% to 66%), positive predictive value of 32% (95% CI, 25% to 40%), and negative predictive value of 90% (95% CI, 84% to 93%) were identified. Conclusions: ChatGPT-4o shows a mild potential in evaluating MES in endoscopic frames, but further refinements are mandatory. (Intest Res 2026;24:724-734)

Evaluation of the Mayo endoscopic score in ulcerative colitis using a multimodal large language model: a human-blinded accuracy study

Gravina, Antonietta Gerarda;Pellegrino, Raffaele
;
Imperio, Giuseppe;Izzo, Michele;De Costanzo, Ilaria;Landa, Fabio;Federico, Alessandro
2026

Abstract

Background/Aims: To date, some large language models (LLMs), such as Chat Generative Pre-trained Transformer 4-omni (ChatGPT-4o), can process images through visual transformer patching. An LLM analysis was conducted to assess the ability of ChatGPT-4o to assign the Mayo endoscopic score (MES). Methods: A selection set of high-quality endoscopic frames was identified to compare 4 input models to select the most performant, confirmed in an extended set of 304 frames. Concordance with the evaluation by expert endoscopists was assessed. Results: Only one of the pre-tested models demonstrated significant concordance (κ=0.232, 95% confidence interval [CI]=0.167 to 0.296, P=0.003; ρ=0.36, P=0.011), with a mean bias of –0.26±1.192 (95% CI, –2.596 to 2.076). This was confirmed in the extended set (κ=0.260, 95% CI=0.195 to 0.324, P<0.001; ρ=0.288, P<0.001). The absolute concordance for the selected model was 44% and 45.3% in the selection and extended sets, respectively. For the identification of moderately-to-severely active disease, a sensitivity of 73% (95% CI, 60% to 82%), specificity of 60% (95% CI, 54% to 66%), positive predictive value of 32% (95% CI, 25% to 40%), and negative predictive value of 90% (95% CI, 84% to 93%) were identified. Conclusions: ChatGPT-4o shows a mild potential in evaluating MES in endoscopic frames, but further refinements are mandatory. (Intest Res 2026;24:724-734)
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11591/616106
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
social impact