A Bi-level representation learning model for medical visual question answering

Yong Li, Shaopei Long, Zhenguo Yang, Heng Weng, Kun Zeng, Zhenhua Huang, Fu Lee Wang, Tianyong Hao

Research output: Contribution to journalArticlepeer-review

8 Citations (Scopus)


Medical Visual Question Answering (VQA) targets at answering questions related to given medical images and it contains tremendous potential in healthcare services. However, researches on medical VQA are still facing challenges, particularly on how to learn a fine-grained multimodal semantic representation from relatively small volume of data resources for answer prediction. Moreover, the long-tailed distribution labels of medical VQA data frequently result in poor performance of models. To this end, we propose a novel bi-level representation learning model with two reasoning modules to learn bi-level representations for the medical VQA task. One is sentence-level reasoning to learn sentence-level semantic representations from multimodal input. The other is token-level reasoning that employs an attention mechanism to generate a multimodal contextual vector by fusing image features and word embeddings. The contextual vector is used to filter irrelevant semantic representations from sentence-level reasoning to generate a fine-grained multimodal representation. Furthermore, a label-distribution-smooth margin loss is proposed to minimize generalization error bound of long-tailed distribution datasets by modifying margin bound of different labels in training set. Based on standard VQA-Rad dataset and PathVQA dataset, the proposed model achieves 0.7605 and 0.5434 on accuracy, 0.7741 and 0.5288 on F1-score, respectively, outperforming a set of state-of-the-art baseline models.

Original languageEnglish
Article number104183
JournalJournal of Biomedical Informatics
Publication statusPublished - Oct 2022


  • Label-distribution-smooth margin loss
  • Medical visual question answering
  • Sentence-level reasoning
  • Token-level reasoning


Dive into the research topics of 'A Bi-level representation learning model for medical visual question answering'. Together they form a unique fingerprint.

Cite this