Insights from my Software Engineering Radio interview with Itamar Friedman
Introduction
In a recent episode of the Software Engineering Radio podcast, I interviewed Itamar Friedman, CEO and Co-founder of Qodo (a company previously called CodiumAI). Our conversation focused on automated testing with generative artificial intelligence (AI), exploring how large language models (LLMs) can augment existing test suites and increase code coverage. Itamar and I discussed the design, implementation, and use of Qodo-Cover (previously called Cover-Agent), an open-source tool that uses LLMs to automatically enhance a provided test suite.
Insights
I’m thankful that Itamar took the time for this interview! Here are some key insights from it:
What testing tasks does Qodo-Cover automatically perform?
“Qodo-Cover can try to generate all types of testing, but the sweet spot that we saw is mostly around component testing. If you provide an initial few tests that cover one or more components and you run Qodo-Cover, it’ll try to generate many more. It takes the first few tests that were given as part of the test suite and exploits these to inspire it to generate more.”
Why is code coverage still a valuable metric despite its limitations?
“Many would say that code coverage is a proxy metric. I think almost 100% people would say that. But I think that also some percentage of developers and managers would say that code coverage is actually a vanity metric. Practically what we’re trying to say is that did we check that the code work exactly as expected? Okay and now how do you check that?”
What is a flaky test case? How does Qodo-Cover handle flaky test cases?
“If you have a test and you’re running it five times and it doesn’t give you the same result each time, especially if sometimes it’s passed, sometimes it fails, that’s the most obvious flakiness, then it’s flaky. It means how can you trust that test if it passed four times the same test, you know, just run it five times and pass four times and fail one that’s a flaky test. LLMs, you can try to prompt it not to generate the flaky test. I think the prompting to start with will try to do that and because there are tests smells that can be smelled for example if there is an obvious API call an obvious one.”
What advice do you have for developers adopting AI-powered development tools?
“About dev tools, I think developers love writing code, they love building and now they can, you know, use code completion to do even a bit more, probably save time and go drink more coffee. I don’t think it’s really helpful for generating more features. But actually let’s think about the real bottlenecks. I don’t think it’s writing more lines of code. It’s about code quality, it’s about code reviewing, it’s about testing, etc. So I think that when you want to start integrating, you know, different AI dev tools to improve your productivity, think what are your ‘today bottlenecks’ and what are your ‘today problems’?”
Listen
If you’re interested in learning more about automated testing with generative AI and how tools like Qodo-Cover can enhance your development workflow, I highly recommend listening to Itamar Friedman on Automated Testing with Generative AI from Software Engineering Radio! You can find it on your favorite podcast player, Apple Podcasts, Spotify, YouTube, or you can listen to it with this handy podcast player.