Robin Crockett (Academic Integrity Lead – University of Northampton) has run a small scale study investigating two AI detectors with a range of AI created assignments and has shared some of the initial results.<\/p>\n\n\n\n
He used ChatGPT to generate 25 nominal 1000-word essays: five subjects, five different versions of each subject. For each subject, he instructed ChatGPT to vary the sentence length as follows: ‘default’ (i.e. I didn’t give it an instruction re. sentence length), ‘use long sentences’, ‘use short sentences’, ‘use complex sentences’, ‘use simple sentences’.<\/p>\n\n\n\n
The table below shows the amount of the assignment which was detected as using AI in two different products: Turnitin and Copyleaks<\/p>\n\n\n\n X = “Unavailable as submission failed to meet requirements”.<\/p>\n\n\n\n 0% -> complete false negative.<\/p>\n\n\n\n Robin noted:<\/p>\n\n\n\n Turnitin highlights\/returns a percentage of ‘qualifying’ text that it sees as AI-generated, but no probability of AI-ness.<\/p>\n\n\n\n Copyleaks highlights sections of text it sees as AI-generated, each section tagged with the probability of AI-ness, but doesn’t state the overall proportion of the text it sees as AI-generated (hence his estimates).<\/p>\n\n\n\n <\/td> Essay 1\u00a0<\/strong><\/td> Essay 2\u00a0<\/strong><\/td> Essay 3\u00a0<\/strong><\/td> Essay 4<\/strong>\u00a0<\/td> Essay 5\u00a0<\/strong><\/td><\/tr> Turnitin<\/strong>\u00a0<\/td> <\/td> <\/td> <\/td> <\/td> <\/td><\/tr> Default <\/td> 100% AI <\/td> 100% AI <\/td> 76% AI <\/td> 100% AI <\/td> 64% AI <\/td><\/tr> Long <\/td> 0% AI <\/td> 26% AI <\/td> 59% AI <\/td> 67% AI <\/td> 51% AI <\/td><\/tr> Short <\/td> 0% AI <\/td> X <\/td> 31% AI <\/td> 82% AI <\/td> 27% AI <\/td><\/tr> Complex <\/td> 33% AI <\/td> 15% AI <\/td> 0% AI <\/td> 63% AI <\/td> 0% AI <\/td><\/tr> Simple <\/td> 100% AI <\/td> 0% AI <\/td> 100% AI <\/td> 100% AI <\/td> 71% AI <\/td><\/tr> Copyleaks<\/strong>\u00a0<\/td> <\/td> <\/td> <\/td> <\/td> <\/td><\/tr> Default <\/td> 100% AI at p=80.6% <\/td> 100% AI at p=83.5% <\/td> 100% AI at p=88.5% <\/td> 100% AI at p=81.3% <\/td> 100% AI at p=85.4% <\/td><\/tr> Long <\/td> ~80% AI at p=65-75% <\/td> 100% AI at p=81.5% <\/td> ~95% AI at p=75-85% <\/td> 100% AI at p=79.1% <\/td> 100% AI at p=80.6% <\/td><\/tr> Short <\/td> ~70% AI at p=66-72% <\/td> 100% AI at p=76.9% <\/td> 100% AI at p=87.3% <\/td> ~85% AI at p=77-79% <\/td> 100% AI at p=78.4% <\/td><\/tr> Complex <\/td> 100% AI at p=72.9% <\/td> 100% AI at p=81.0% <\/td> ~90% AI at p=62-73% <\/td> 100% AI at p=77.7% <\/td> 0% AI <\/td><\/tr> Simple <\/td> 100% AI at p=83.6% <\/td> ~90% AI at p=73-81% <\/td> 100% AI at p=95.2% <\/td> ~90% AI at p=76-82% <\/td> 100% AI at p=84.9% <\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n