An exploratory analysis of Community-based Question-Answering Platforms and GPT-3-driven Generative AI: Is it the end of online community-based learning?
Authors:
Mohammed Mehedi Hasan,
Mahady Hasan,
Mamun Bin Ibne Reaz,
Jannat Un Nayeem Iqra
Abstract:
Context: The advent of Large Language Model-driven tools like ChatGPT offers software engineers an interactive alternative to community question-answering (CQA) platforms like Stack Overflow. While Stack Overflow provides benefits from the accumulated crowd-sourced knowledge, it often suffers from unpleasant comments, reactions, and long waiting times.
Objective: In this study, we assess the eff…
▽ More
Context: The advent of Large Language Model-driven tools like ChatGPT offers software engineers an interactive alternative to community question-answering (CQA) platforms like Stack Overflow. While Stack Overflow provides benefits from the accumulated crowd-sourced knowledge, it often suffers from unpleasant comments, reactions, and long waiting times.
Objective: In this study, we assess the efficacy of ChatGPT in providing solutions to software engineering questions by analyzing its performance specifically against human solutions.
Method: We empirically analyze 2564 Python and JavaScript questions from StackOverflow that were asked between January 2022 and December 2022. We parse the questions and answers from Stack Overflow, then collect the answers to the same questions from ChatGPT through API, and employ four textual and four cognitive metrics to compare the answers generated by ChatGPT with the accepted answers provided by human subject matter experts to find out the potential reasons for which future knowledge seekers may prefer ChatGPT over CQA platforms. We also measure the accuracy of the answers provided by ChatGPT. We also measure user interaction on StackOverflow over the past two years using three metrics to determine how ChatGPT affects it.
Results: Our analysis indicates that ChatGPT's responses are 66% shorter and share 35% more words with the questions, showing a 25% increase in positive sentiment compared to human responses. ChatGPT's answers' accuracy rate is between 71 to 75%, with a variation in response characteristics between JavaScript and Python. Additionally, our findings suggest a recent 38% decrease in comment interactions on Stack Overflow, indicating a shift in community engagement patterns. A supplementary survey with 14 Python and JavaScript professionals validated these findings.
△ Less
Submitted 30 September, 2024; v1 submitted 25 September, 2024;
originally announced September 2024.
