A new study shows that the ChatGPT AI bot is not effective at helping developers write code. A study conducted by researchers at Purdue University indicates that more than half of the answers provided by robots to questions related to the program are incorrect.
The researchers analyzed 517 questions from the developer platform Stack Overflow and compared ChatGPT answers with human answers. The results showed that ChatGPT bot errors are widespread, with the percentage of incorrect answers reaching about 52% of the total answers.
These errors include misunderstanding of programming concepts, errors in the accuracy of information, and logical errors in programming code and technical terminology.
In addition to these drawbacks, the study also criticized ChatGPT for its tendency to provide longer and more complex answers than necessary, which led to developers being confused and distracted.
However, a small survey of 12 programmers found that a third of them preferred clear, structured ChatGPT answers.
The findings are important because programming errors can ultimately lead to major problems affecting an entire department or organization, and these errors can lead to system or application errors that require costly and time-consuming corrective interventions on the part of programmers.
In the study summary, the researchers stressed the need to “exercise caution and vigilance when using ChatGPT answers in programming tasks,” as errors often appear in the answers generated by ChatGPT.
It is recommended to use other tools such as GitHub Copilot, which has proven to be more effective in helping programmers, as well as using human review or code analysis tools to validate the code provided by ChatGPT. These steps help reduce error rates and improve the quality of the program's operation.