AI and Development

The autocomplete grew a personality

Half the internet spent this spring announcing that programming is finished, and the other half announcing that the whole thing is a parlour trick. I don't think either group has had the tools switched on in a real project for very long.

So instead of guessing, I want to walk through the four places this stuff actually touches my day: writing code, reviewing it, testing it and rewriting it. Some of it has earned a keyboard shortcut. A surprising amount of it is a demo with a pricing page attached.

One thing up front, and it connects to what I wrote about idiomatic programming: a machine that produces code nobody on your team would write by hand has not saved you anything. It has moved the work from typing to reading, and reading is the expensive part.

Completion, and what it is good at

The current generation of tooling starts with Codex, the model OpenAI trained on public code and shipped in 2021. It is the engine that GitHub Copilot was built on. If you went looking for the Codex API this spring, you found a tombstone: OpenAI deprecated it in March with three days’ notice and pointed everyone at GPT-3.5 and GPT-4 instead. Three days. That says more about building on somebody else’s model endpoint than any benchmark could, and it has nothing to do with whether the model was any good.

What survived is the editor integration, and that part I genuinely like. Copilot is at its best on the code I resent writing: the fifth case in a reducer, a test fixture with twelve fields, the regular expression I would otherwise have gone looking for. It reads the open buffers, notices the pattern I set up two files ago, and finishes the shape.

It is much worse at anything it has not seen a thousand times before. It will invent a method on a library that has no such method, give it the right naming convention and a completely plausible signature, and offer it with total confidence. Wrong code that looks like your code is more expensive to review than wrong code that looks wrong. That is the actual tax, and nobody puts it on the landing page.

Worth having a look at:

• GitHub Copilot suggests completions inline as you type, using the surrounding files as context. It’s the one I’d start with. Their site has the current plans.
• Tabnine does the same job and will run models locally, which matters a lot if your employer has feelings about source code leaving the building. tabnine.com
• A ChatGPT tab is a different tool entirely. No editor context, so no completions, but far better for “explain this stack trace” or “what is this build error actually telling me”.

Review is where half the category is a rebrand

Snyk Code is the interesting one here, and it has a Swiss origin story. DeepCode came out of the Secure, Reliable and Intelligent Systems lab at ETH Zurich, learned issue patterns from a very large pile of open source repositories, and got bought by Snyk in 2020. It now ships as Snyk Code inside their platform. Nice to see something out of Zurich end up in that many editors.

The rest of the category is thinner than the marketing suggests. Code Climate is a maintainability and static analysis platform. DeepSource is a static analyser with automatic fixes. Both are useful, and neither is doing anything a linter author would recognise as machine learning. Putting “AI” on the box sells seats.

• Snyk Code (formerly DeepCode) flags security and reliability issues from learned patterns rather than hand-written rules. snyk.io
• Code Climate grades maintainability and tracks churn against complexity. codeclimate.com
• DeepSource analyses on every push and can open the fix as a pull request. deepsource.io

The honest pitch for all three is that they raise the floor. They catch the null dereference at two in the morning that a tired reviewer waves through. They will not tell you that the abstraction is wrong, and that is still the review comment that matters most.

Tests are where it surprised me

I expected test generation to be the weakest of the four and it is the one that changed my mind.

Diffblue Cover reads Java bytecode, uses reinforcement learning to choose inputs that reach each path through a method, then writes JUnit tests with the mocks and assertions in place. The important detail is that it executes what it writes, so the output compiles and runs. That is a very different bargain from asking a language model for a test file and discovering the problem three commits later.

Testim goes at the other end of the pipeline. The real cost of an end-to-end suite is not writing it, it is the selectors breaking every time somebody restructures a component, and Testim uses a model to pick locators that survive that. It’s a neighbour of the problem I looked at in screenshot testing, where the suite tells you something changed without telling you whether it matters.

One correction while I’m here, because I keep seeing it repeated. DeepMind’s AlphaCode, from early 2022, is not a bug-finding system. It solves competitive programming problems by sampling an enormous number of candidate programs and filtering them against the example tests, and it placed roughly in the middle of the Codeforces field. Genuinely impressive research. It is not something you can point at your repository, and citing it as automated QA is just wrong.

Refactoring is still hand work

Facebook published Aroma back in 2019 and it gets described badly almost everywhere, including by me until I went and read the paper. It indexes a large corpus of code, takes a partial snippet as input, finds method bodies that are structurally similar, and clusters them into a handful of recommendations. It is code-to-code search. It does not remove redundant operations or improve your memory management, whatever the summaries say.

Sourcery is the closest thing to a daily-use refactoring assistant I’ve found, and it’s Python only. It sits in the editor and suggests the tidier form of what you just wrote, which is pleasant and occasionally slightly humbling.

But refactoring worth the name is not about rewriting a loop as a comprehension. It’s about moving a boundary, splitting a module that grew two responsibilities, deciding that a prop should have been context all along. That work needs the argument you had in the planning meeting, and none of these tools were in the room.

What stays switched on

Copilot stays on, because the boilerplate tax is real and it pays that for me. Snyk Code and DeepSource stay in CI, because a second opinion that never gets tired is worth the money. Test generation I would reach for on a Java service, not on a React frontend.

What I don’t do is let a suggestion into a commit I haven’t read line by line. That rule has not changed and I can’t see it changing.

This article is in part written using AI technologies 🤣

‘Till next time!