Exemplar selection for in-context learning under complex scenarios

Abstract

Large Language Models (LLMs) exhibit remarkable few-shot capabilities, where the selec- tion of exemplars for In-Context Learning (ICL) has been shown to significantly impact downstream performance. However, prior research largely focuses on either input-side se- mantic similarity or directly leveraging unmodified sentence embeddings, assuming they already encode the necessary task information, which may be inadequate for tasks like semantic parsing requiring deeper language understanding. In contrast, we introduce a two-fold strategy that enriches LLM representations with deeper linguistic properties and structural signals tailored for semantic parsing. First, we use tree-edit-distance–based similarity to form contrastive pairs, finetuning a BERT model that integrates both seman- tic and structural information of the target parse. Second, we perform an intermediate- layer intervention by injecting linguistic property information into the hidden states, subsequently letting higher layers further refine these enhanced representations. This injection process builds upon recent findings that LLMs often internalize more latent knowledge than is reflected by their outputs, and that suitable representation-level in- terventions can better activate such knowledge for specific tasks. Empirical results show our augmented representations more effectively capture both the semantic and structural similarity pertinent to semantic parsing, outperforming conventional exemplar selection baselines. Our approach thus highlights the value of integrating structural signals and explicit hidden-state interventions in creating more linguistically informed embeddings, ultimately benefiting in-context learning for semantic parsing tasks requiring rich lan- guage understanding.