Overview

As LLMs become central to software development workflows, understanding what they actually learn about code and how reliably they perform is critical. Our research builds benchmarks, training methods, and evaluation frameworks that move beyond surface-level accuracy to probe deeper code understanding.

We study semantic reasoning, self-consistency, execution awareness, and the impact of data contamination on benchmark validity.

Key Directions

• Semantic Reasoning: Benchmarks and models that test whether LLMs understand code semantics, not just syntax, including execution-aware pre-training.
• Self-Consistency: Evaluating whether code LLMs produce internally consistent outputs across equivalent formulations of the same problem.
• Dynamic Evaluation: Combating data contamination with dynamic benchmarking approaches that generate fresh evaluation instances.
• Code Representations: Contrastive learning and pre-training strategies that build better representations of source code.

Impact

SemCoder introduced comprehensive semantics reasoning into code LLM training, achieving state-of-the-art results at NeurIPS 2024. IdentityChain established self-consistency as a key evaluation dimension for code models. Our work on data contamination has influenced how the community designs and trusts code benchmarks.

Contributors

Selected Publications