The American Science Development Association (AAAS) has decided to check whether TATGPT can write short scientific notes in the Scipak style or not such a special Briffs for journalists, often prepared by the Eurekalent Science and Service Magazine.
Experiment It lasted a year: from December 2023 to December 2024, journalists have launched a model or two difficult scientific studies every week.
The mission is very simple-re-release them to make it convenient for colleagues: minimum terms, a clear structure and clear context. Therefore, chatgpt has handled 64 works.
Result? In general, the model has learned how to copy the forms of Scipak Stamps, but did not achieve content. According to the author of the study, journalist Aaas Abigail Aizenstadt, TATGPT's documents are too simple and often incorrect. I had to double -check everything, so it took more time to write a note from the beginning.
The numbers also tell themselves. When the editors are asked, such a calendar may be lost among the real Briffs of Scipak, the average rank is only 2.26 points in the number 5. And only one work in the whole year receives the highest score from journalists.
Typically, chatgpt confused relationships and causal relationships, forgetting to mention important limitations for research (for example, soft transmission devices operate very slowly) and sometimes disperse too much to praise the breakthroughs of Islam and novelty.
The authors of the experiment have summarized: So far, chatgpt has not complied with Scipak standards. But they did not completely put the cross in the idea. AAAS Note that you can return to tests if the model is updated seriously. And in August, we reminded, just GPT-5 outIn it, unfortunately, Found the gap.