Reyhan Jabbarvand Behrouz

Beyond Crafted Benchmarks, Using LLMs to Automate Software Engineering Tasks in Real-World Projects

Large Language Models (LLMs) are the new automation technology to explore for software development and maintenance. State-of-the-art LLMs and techniques leveraging them for code-related tasks are mostly evaluated on crafted benchmarks (mostly not reflective of the complexity of real-world projects) and under very simple tasks (even if the studied programs are real-world projects, the tasks are very simple). In this presentation, I will first talk about the challenges of using LLMs for (1) whole-repository code translation and (2) code synthesis for repairing test flakiness in real-world projects. After discussing such challenges, I will talk about how neuro-symbolic approaches that combine the generalizability of LLMs and the soundness of program analysis can overcome these challenges.