The research from Purdue University, first spotted by news outlet Futurism, was presented earlier this month at the Computer-Human Interaction Conference in Hawaii and looked at 517 programming questions on Stack Overflow that were then fed to ChatGPT.

“Our analysis shows that 52% of ChatGPT answers contain incorrect information and 77% are verbose,” the new study explained. “Nonetheless, our user study participants still preferred ChatGPT answers 35% of the time due to their comprehensiveness and well-articulated language style.”

Disturbingly, programmers in the study didn’t always catch the mistakes being produced by the AI chatbot.

“However, they also overlooked the misinformation in the ChatGPT answers 39% of the time,” according to the study. “This implies the need to counter misinformation in ChatGPT answers to programming questions and raise awareness of the risks associated with seemingly correct answers.”

  • tsonfeir@lemmy.world
    link
    fedilink
    English
    arrow-up
    11
    arrow-down
    9
    ·
    5 months ago

    If you ask the wrong questions you get the wrong results. If you don’t check the response for accuracy, you get invalid answers.

    It’s just a tool. Don’t use it wrong because you’re lazy.

    • StereoTrespasser@lemmy.world
      link
      fedilink
      English
      arrow-up
      1
      arrow-down
      3
      ·
      5 months ago

      Lemmy is trying really, really hard to convince you that coding is going to be a viable career in 5 years.

      • tsonfeir@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        arrow-down
        1
        ·
        5 months ago

        Lemmy is trying real hard to convince you that AI is going to do everyone’s job in 5 years—including yours