Will ChatGPT (and others) really replace developers?

In recent months, ChatGPT and generative AI tools have captured the attention of many, especially in the technology world.

A recurring question is: will ChatGPT replace programmers?

In our team, we decided to test these tools to understand if they can really represent a threat to the developer’s profession.

The result? A mix of surprises, disappointments, and some interesting insights.

Here’s what we discovered.

The first thing we wanted to test was ChatGPT’s ability to generate useful code. We asked it to write scripts and functions for some of our internal projects, but the results were disappointing.

The code produced was often incomplete, with errors or simply unsuitable for our needs. In the best case, ChatGPT generated educational examples, similar to those we might use in a training course to illustrate a concept without revealing the complete solution.

For example, we asked ChatGPT to create a function to process data from a specific API. The code it produced was theoretically correct, but lacked optimization and did not account for some project constraints. In the end, we spent more time correcting the code generated by ChatGPT than we would have spent writing it from scratch.

ChatGPT is not yet able to replace an experienced programmer, but it could be useful for those learning to program and needing basic examples.

One of the areas where ChatGPT showed some potential is the generation of regular expressions.

We asked it to create regex for standard and custom problems, and the results were often correct.

However, we are not sure if ChatGPT is better than traditional regex generation tools.

To test its limits, we decided to make some more “creative” requests, and here’s what happened:

We asked ChatGPT for the last digit of pi. With confidence, it answered that it was 9. Then, to prove it, it showed us an example of pi with a dozen digits… that ended with a 6;

We asked ChatGPT to write fantasy stories and imaginary interviews with historical characters. The results were grammatically correct and easy to read, but boring and superficial, like written by a bored student or a clickbait copywriter;

We asked ChatGPT for instructions to start a vintage motorcycle. It provided almost correct instructions, but completely overlooked crucial details to avoid getting hurt. A colleague asked for similar instructions to land a glider, but ChatGPT thought it best to give maximum power to the engines to reach the runway;

It is good at generating theoretically correct content, but often lacks depth, precision, and practical sense.

The only area where ChatGPT excelled was recipe generation. We asked for a recipe for lentil and potato stew, and the result was to provide a decent recipe.

Moreover, when asking for a printable format, it provided exactly what we needed: ingredients, quantities, and steps for the preparation.

It handled contradictory requests well, such as a vegan stew with chicken, refusing politely to produce an impossible recipe.

It could be a useful assistant in the kitchen, but not yet ready to replace an experienced chef.

One of the most interesting questions is whether ChatGPT can become a valid assistant for programmers, helping us solve the boring and repetitive parts of our work. Unfortunately, our experience suggests that we are not there yet.

ChatGPT cannot understand business context, budget constraints, timelines, or technological preferences. Moreover, it cannot interview users and stakeholders to design solutions aligned with the business vision.

When should we start worrying?

The real question is: when will ChatGPT – or similar tools – become a threat to programmers?

According to us, we should start worrying when AI is able to:

Understand business context: including budget, times, infrastructure, and regulations;

Design complex solutions: considering functional and non-functional requirements;

Collaborate with users and stakeholders: to define objectives and priorities;

Solve creative problems: going beyond simple code or text generation;

It is likely that future versions of ChatGPT will be more advanced and capable of handling more complex tasks. However, even in that case, we believe that programmers will have a crucial role in guiding, supervising, and optimizing AI work.

Instead of fearing AI, we should see it as an opportunity to improve our work and focus on more creative and strategic activities.

ChatGPT is an impressive tool, but not yet able to replace programmers.

For now, it remains a useful assistant for simple and repetitive tasks, but lacks the depth, creativity, and context understanding necessary for complex problems.