Publications
2026
2026
- EMNLPUnderstanding the Role of Prompt Template in Knowledge Distillation for Safety AlignmentAnjila Budathoki , Manish Dhakal, Benjamin M. Ampel , and 1 more authorIn EMNLP (Findings) , 2026
@inproceedings{budathoki2026understanding, title = {Understanding the Role of Prompt Template in Knowledge Distillation for Safety Alignment}, author = {Budathoki, Anjila and Dhakal, Manish and Ampel, Benjamin M. and Ding, Yi}, booktitle = {EMNLP (Findings)}, year = {2026}, } - DeCaFFederated LoRA Adaptation of BiomedCLIP Across Four International Chest X-Ray CohortsSanjaya Poudel , Nirajan Kunwor , Manish Dhakal, and 2 more authorsIn Distributed, Collaborative, and Federated Learning (DeCaF), MICCAI Workshop , 2026
Federated learning (FL) lets institutions train a shared model without exchanging data, and Low-Rank Adaptation (LoRA) makes this practical at scale by communicating only compact low-rank updates. Biomedical imaging is a compelling setting for this combination: patient data are archived behind privacy regulations, and institutions differ widely in scanners, protocols, and compute. Such heterogeneity raises the question of how federated LoRA updates should be aggregated, increasingly pressing as multimodal vision-language models become central to medical image analysis. We benchmark federated Parameter-efficient fine-tuning (PEFT) of BiomedCLIP for chest radiograph classification across four public cohorts on three continents (USA, Vietnam, Spain). Federated LoRA adaptation improves shared-class AUC on all four cohorts over the unadapted BiomedCLIP backbone (mean 0.687 to 0.802), showing that the gains come from federated adaptation rather than from the pretrained model’s zero-shot ability. Relative to isolated single-cohort training, federation improves the weaker cohorts while largely preserving the strongest and approaches a centralized reference (0.812) that pools all data. The singular value decomposition (SVD)-based product-space aggregation introduced by FlexLoRA is essential to this gain (naive factor averaging drops mean AUC by 0.097), whereas a drift-correcting optimizer (FedProx) shows no benefit over FedAvg in our single-seed runs, consistent with LoRA’s low-rank updates already limiting client drift. Biomedical vision-language models can thus be adapted collaboratively across heterogeneous, geographically distributed institutions without centralizing data.
@inproceedings{poudel2026federated, title = {Federated LoRA Adaptation of BiomedCLIP Across Four International Chest X-Ray Cohorts}, author = {Poudel, Sanjaya and Kunwor, Nirajan and Dhakal, Manish and Jha, Debesh and Gaire, Sunil Kumar}, booktitle = {Distributed, Collaborative, and Federated Learning (DeCaF), MICCAI Workshop}, year = {2026}, } - CVPR-WParameter Efficient Fine-tuning for Domain-specific Gastrointestinal Disease RecognitionSanjaya Poudel , Nikita Kunwor , Raj Simkhada , and 3 more authorsIn Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, PHAROS-AIF-MIH , Jun 2026
Despite recent advancements in the field of medical image analysis with the use of pretrained foundation models, the issue of distribution shifts between cross-source images largely remains adamant. To circumvent that issue, investigators generally train a separate model for each source. However, this method becomes expensive when we fully fine-tune pretrained large models for a single dataset, as we must store multiple copies of those models. Thus, in this work, we propose using a low-rank adaptation (LoRA) module for fine-tuning downstream classification tasks. LoRAs learn lightweight task-specific low-rank matrices that perturb pretrained weights to optimize those downstream tasks. For gastrointestinal tract diseases, they exhibit significantly better results than end-to-end finetuning with improved parameter efficiency. The code is available at https://github.com/sanjay931/peft-gi-recognition
@inproceedings{poudel2026parameter, title = {Parameter Efficient Fine-tuning for Domain-specific Gastrointestinal Disease Recognition}, author = {Poudel, Sanjaya and Kunwor, Nikita and Simkhada, Raj and Munir, Mustafa and Dhakal, Manish and Poudel, Khem}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, PHAROS-AIF-MIH}, month = jun, year = {2026}, pages = {6313--6318}, } - arXivDistillLens: Symmetric Knowledge Distillation Through Logit LensManish Dhakal, Uthman Jinadu , Anjila Budathoki , and 2 more authorsarXiv preprint arXiv:2602.13567, Feb 2026
Standard Knowledge Distillation (KD) compresses Large Language Models (LLMs) by optimizing final outputs, yet it typically treats the teacher’s intermediate layer’s thought process as a black box. While feature-based distillation attempts to bridge this gap, existing methods (e.g., MSE and asymmetric KL divergence) ignore the rich uncertainty profiles required for the final output. In this paper, we introduce DistillLens, a framework that symmetrically aligns the evolving thought processes of student and teacher models. By projecting intermediate hidden states into the vocabulary space via the Logit Lens, we enforce structural alignment using a symmetric divergence objective. Our analysis proves that this constraint imposes a dual-sided penalty, preventing both overconfidence and underconfidence while preserving the high-entropy information conduits essential for final deduction. Extensive experiments on GPT-2 and Llama architectures demonstrate that DistillLens consistently outperforms standard KD and feature-transfer baselines on diverse instruction-following benchmarks.
@article{dhakal2026distilllens, title = {DistillLens: Symmetric Knowledge Distillation Through Logit Lens}, author = {Dhakal, Manish and Jinadu, Uthman and Budathoki, Anjila and Sunderraman, Raj and Ding, Yi}, journal = {arXiv preprint arXiv:2602.13567}, month = feb, year = {2026}, } - WACVGFT: Graph Feature Tuning for Efficient Point Cloud AnalysisManish Dhakal, Venkat R. Dasari , Raj Sunderraman , and 1 more authorIn Proceedings of the Winter Conference on Applications of Computer Vision (WACV) , Mar 2026
Parameter-efficient fine-tuning (PEFT) significantly reduces computational and memory costs by updating only a small subset of the model’s parameters, enabling faster adaptation to new tasks with minimal loss in performance. Previous studies have introduced PEFTs tailored for point cloud data, as general approaches are suboptimal. To further reduce the number of trainable parameters, we propose a point-cloud-specific PEFT, termed Graph Features Tuning (GFT), which learns a dynamic graph from initial tokenized inputs of the transformer using a lightweight graph convolution network and passes these graph features to deeper layers via skip connections and efficient cross-attention modules. Extensive experiments on object classification and segmentation tasks show that GFT operates in the same domain, rivalling existing methods, while reducing the trainable parameters.
@inproceedings{dhakal2026gft, title = {GFT: Graph Feature Tuning for Efficient Point Cloud Analysis}, author = {Dhakal, Manish and Dasari, Venkat R. and Sunderraman, Raj and Ding, Yi}, booktitle = {Proceedings of the Winter Conference on Applications of Computer Vision (WACV)}, month = mar, year = {2026}, }
2025
2025
- IJCNLP-AACLCan a Unimodal Language Agent Provide Preferences to Tune a Multimodal Vision-Language Model?Sazia Tabasum Mim , Jack Morris , Manish Dhakal, and 3 more authorsIn International Joint Conference on Natural Language Processing & Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) , Dec 2025
To explore a more scalable path for adding multimodal capabilities to existing LLMs, this paper addresses a fundamental question: Can a unimodal LLM, relying solely on text, reason about its own informational needs and provide effective feedback to optimize a multimodal model? To answer this, we propose a method that enables a language agent to give feedback to a vision-language model (VLM) to adapt text generation to the agent’s preferences. Our results from different experiments affirm this hypothesis, showing that LLM preference feedback significantly enhances VLM descriptions. Using our proposed method, we find that the VLM can supply multimodal scene descriptions to help the LLM better understand multimodal context, leading to improvements of maximum 13% in absolute accuracy compared to the baseline multimodal approach. Furthermore, a human study validated our AI-driven feedback, showing a 64.6% preference alignment rate between the LLM’s choices and human judgments. Extensive experiments provide insights on how and why the method works and its limitations.
@inproceedings{mim2025can, title = {Can a Unimodal Language Agent Provide Preferences to Tune a Multimodal Vision-Language Model?}, author = {Mim, Sazia Tabasum and Morris, Jack and Dhakal, Manish and Xiu, Yanming and Gorlatova, Maria and Ding, Yi}, booktitle = {International Joint Conference on Natural Language Processing {\&} Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL)}, month = dec, year = {2025}, } - arXivAdversarial Robustness Analysis of Vision-Language Models in Medical Image SegmentationAnjila Budathoki , and Manish DhakalarXiv preprint arXiv:2505.02971, Dec 2025
@article{budathoki2025adversarial, title = {Adversarial Robustness Analysis of Vision-Language Models in Medical Image Segmentation}, author = {Budathoki, Anjila and Dhakal, Manish}, journal = {arXiv preprint arXiv:2505.02971}, year = {2025}, }
2024
2024
- ACCVTuneVLSeg: Prompt Tuning Benchmark for Vision-Language Segmentation ModelsIn Proceedings of the Asian Conference on Computer Vision (ACCV) , Dec 2024
Vision-Language Models (VLMs) have shown impressive performance in vision tasks, but adapting them to new domains often requires expensive fine-tuning. Prompt tuning techniques, including textual, visual, and multimodal prompting, offer efficient alternatives by leveraging learnable prompts. However, their application to Vision-Language Segmentation Models (VLSMs) and evaluation under significant domain shifts remain unexplored. This work presents an open-source benchmarking framework, TuneVLSeg, to integrate various unimodal and multimodal prompt tuning techniques into VLSMs, making prompt tuning usable for downstream segmentation datasets with any number of classes. We test various prompt tuning on 8 diverse medical datasets, including 3 radiology datasets (breast tumor, echocardiograph, chest X-ray pathologies) and 5 non-radiology datasets (polyp, ulcer, skin cancer), and two natural domain segmentation datasets. Our study found that textual prompt tuning struggles under significant domain shifts, from natural-domain images to medical data. Furthermore, visual prompt tuning, with fewer hyperparameters than multimodal prompt tuning, often achieves performance competitive to multimodal approaches, making it a valuable first attempt. Our work advances the understanding and applicability of different prompt-tuning techniques for robust domain-specific segmentation. The source code is available at https://github.com/naamiinepal/tunevlseg.
@inproceedings{Adhikari_2024_ACCV, author = {Adhikari, Rabin and Thapaliya, Safal and Dhakal, Manish and Khanal, Bishesh}, title = {TuneVLSeg: Prompt Tuning Benchmark for Vision-Language Segmentation Models}, booktitle = {Proceedings of the Asian Conference on Computer Vision (ACCV)}, month = dec, year = {2024}, pages = {126-144}, } - MICCAIVLSM-Adapter: Finetuning Vision-Language Segmentation Efficiently with Lightweight BlocksIn International Conference on Medical Image Computing and Computer-Assisted Intervention , Dec 2024
Foundation Vision-Language Models (VLMs) trained using large-scale open-domain images and text pairs have recently been adapted to develop Vision-Language Segmentation Models (VLSMs) that allow providing text prompts during inference to guide image segmentation. If robust and powerful VLSMs can be built for medical images, it could aid medical professionals in many clinical tasks where they must spend substantial time delineating the target structure of interest. VLSMs for medical images resort to fine-tuning base VLM or VLSM pretrained on open-domain natural image datasets due to fewer annotated medical image datasets; this fine-tuning is resource-consuming and expensive as it usually requires updating all or a significant fraction of the pretrained parameters. Recently, lightweight blocks called adapters have been proposed in VLMs that keep the pretrained model frozen and only train adapters during fine-tuning, substantially reducing the computing resources required. We introduce a novel adapter, VLSM-Adapter, that can fine-tune pretrained vision-language segmentation models using transformer encoders. Our experiments in widely used CLIP-based segmentation models show that with only 3 million trainable parameters, the VLSM-Adapter outperforms state-of-the-art and is comparable to the upper bound end-to-end fine-tuning.
@inproceedings{dhakal2024vlsm, title = {VLSM-Adapter: Finetuning Vision-Language Segmentation Efficiently with Lightweight Blocks}, author = {Dhakal, Manish and Adhikari, Rabin and Thapaliya, Safal and Khanal, Bishesh}, booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention}, pages = {712--722}, year = {2024}, organization = {Springer}, } - MIDLExploring Transfer Learning in Medical Image Segmentation using Vision-Language ModelsIn Medical Imaging with Deep Learning , Dec 2024Equal contribution by the first five authors
Medical image segmentation with deep learning is an important and widely studied topic because segmentation enables quantifying target structure size and shape that can help in disease diagnosis, prognosis, surgery planning, and understanding. Recent advances in the foundation Vision-Language Models (VLMs) and their adaptation to segmentation tasks in natural images with Vision-Language Segmentation Models (VLSMs) have opened up a unique opportunity to build potentially powerful segmentation models for medical images that enable providing helpful information via language prompt as input, leverage the extensive range of other medical imaging datasets by pooled dataset training, adapt to new classes, and be robust against out-of-distribution data with human-in-the-loop prompting during inference. Although transfer learning from natural to medical images for imageonly segmentation models has been studied, no studies have analyzed how the joint representation of vision-language transfers to medical images in segmentation problems and understand gaps in leveraging their full potential. We present the first benchmark study on transfer learning of VLSMs to 2D medical images with thoughtfully collected 11 existing 2D medical image datasets of diverse modalities with carefully presented 9 types of language prompts from 14 attributes. Our results indicate that VLSMs trained in natural image-text pairs transfer reasonably to the medical domain in zero-shot settings when prompted appropriately for non-radiology photographic modalities; when finetuned, they obtain comparable performance to conventional architectures, even in X-rays and ultrasound modalities. However, the additional benefit of language prompts during finetuning may be limited, with image features playing a more dominant role; they can better handle training on pooled datasets combining diverse modalities and are potentially more robust to domain shift than the conventional segmentation models. The code and prompts are released at https://github.com/naamiinepal/medvlsm
@inproceedings{poudel2023exploring, title = {Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Models}, author = {Poudel, Kanchan and Dhakal, Manish and Bhandari, Prasiddha and Adhikari, Rabin and Thapaliya, Safal and Khanal, Bishesh}, booktitle = {Medical Imaging with Deep Learning}, year = {2024}, note = {Equal contribution by the first five authors}, archiveprefix = {arXiv}, primaryclass = {cs.CV}, url = {https://openreview.net/forum?id=sN3sDKkGeN}, doi = {https://doi.org/10.48550/arXiv.2308.07706}, }
2023
2023
- MICCAISynthetic Boost: Leveraging Synthetic Data for Enhanced Vision-Language Segmentation in EchocardiographyIn International Workshop on Advances in Simplifying Medical Ultrasound , Dec 2023Equal contribution by the first three authors
Accurate segmentation is essential for echocardiography-based assessment of cardiovascular diseases (CVDs). However, the variability among sonographers and the inherent challenges of ultrasound images hinder precise segmentation. By leveraging the joint representation of image and text modalities, Vision-Language Segmentation Models (VLSMs) can incorporate rich contextual information, potentially aiding in accurate and explainable segmentation. However, the lack of readily available data in echocardiography hampers the training of VLSMs. In this study, we explore using synthetic datasets from Semantic Diffusion Models (SDMs) to enhance VLSMs for echocardiography segmentation. We evaluate results for two popular VLSMs (CLIPSeg and CRIS) using seven different kinds of language prompts derived from several attributes, automatically extracted from echocardiography images, segmentation masks, and their metadata. Our results show improved metrics and faster convergence when pretraining VLSMs on SDM-generated synthetic images before finetuning on real images. The code and prompts are released at https://github.com/naamiinepal/synthetic-boost
@inproceedings{adhikari2023synthetic, title = {Synthetic Boost: Leveraging Synthetic Data for Enhanced Vision-Language Segmentation in Echocardiography}, author = {Adhikari, Rabin and Dhakal, Manish and Thapaliya, Safal and Poudel, Kanchan and Bhandari, Prasiddha and Khanal, Bishesh}, booktitle = {International Workshop on Advances in Simplifying Medical Ultrasound}, pages = {89--99}, year = {2023}, note = {Equal contribution by the first three authors}, organization = {Springer}, }
2022
2022
- IEEEAutomatic speech recognition for the Nepali language using CNN, bidirectional LSTM and ResNetIn 2022 International Conference on Inventive Computation Technologies (ICICT) , Dec 2022
This paper presents an end-to-end deep learning model for Automatic Speech Recognition (ASR) that transcribes Nepali speech to text. The model was trained and tested on the OpenSLR (audio, text) dataset. The majority of the audio dataset have silent gaps at both ends which are clipped during dataset preprocessing for a more uniform mapping of audio frames and their corresponding texts. Mel Frequency Cepstral Coe cients (MFCCs) are used as audio features to feed into the model. The model having Bidirectional LSTM paired with ResNet and one-dimensional CNN produces the best results for this dataset out of all the models (neural networks with variations of LSTM, GRU, CNN, and ResNet) that have been trained so far. This novel model uses Connectionist Temporal Classification (CTC) function for loss calculation during training and CTC beam search decoding for predicting characters as the most likely sequence of Nepali text. On the test dataset, the character error rate (CER) of 17.06 percent has been achieved. The source code is available at: https://github.com/manishdhakal/ASR-Nepali-using-CNN-BiLSTM-ResNet.
@inproceedings{dhakal2022automatic, title = {Automatic speech recognition for the Nepali language using CNN, bidirectional LSTM and ResNet}, author = {Dhakal, Manish and Chhetri, Arman and Gupta, Aman Kumar and Lamichhane, Prabin and Pandey, Suraj and Shakya, Subarna}, booktitle = {2022 International Conference on Inventive Computation Technologies (ICICT)}, pages = {515--521}, year = {2022}, organization = {IEEE}, doi = {https://doi.org/10.1109/ICICT54344.2022.9850832}, }