Introduction
Further to the post of 26/09/2023, Robin Crockett (Academic Integrity Lead – University of Northampton) has added to his small-scale study by adding a second AI ‘writer’ and a third AI-text detector. The full report is available to view by clicking on the graphic below.
Robin Crockett (Academic Integrity Lead – University of Northampton) has run a small scale study investigating two AI detectors with a range of AI created assignments and has shared some of the initial results.
He used ChatGPT to generate 25 nominal 1000-word essays: five subjects, five different versions of each subject. For each subject, he instructed ChatGPT to vary the sentence length as follows: ‘default’ (i.e. I didn’t give it an instruction re. sentence length), ‘use long sentences’, ‘use short sentences’, ‘use complex sentences’, ‘use simple sentences’.
The table below shows the amount of the assignment which was detected as using AI in two different products: Turnitin and Copyleaks
Essay 1 | Essay 2 | Essay 3 | Essay 4 | Essay 5 | |
Turnitin | |||||
Default | 100% AI | 100% AI | 76% AI | 100% AI | 64% AI |
Long | 0% AI | 26% AI | 59% AI | 67% AI | 51% AI |
Short | 0% AI | X | 31% AI | 82% AI | 27% AI |
Complex | 33% AI | 15% AI | 0% AI | 63% AI | 0% AI |
Simple | 100% AI | 0% AI | 100% AI | 100% AI | 71% AI |
Copyleaks | |||||
Default | 100% AI at p=80.6% | 100% AI at p=83.5% | 100% AI at p=88.5% | 100% AI at p=81.3% | 100% AI at p=85.4% |
Long | ~80% AI at p=65-75% | 100% AI at p=81.5% | ~95% AI at p=75-85% | 100% AI at p=79.1% | 100% AI at p=80.6% |
Short | ~70% AI at p=66-72% | 100% AI at p=76.9% | 100% AI at p=87.3% | ~85% AI at p=77-79% | 100% AI at p=78.4% |
Complex | 100% AI at p=72.9% | 100% AI at p=81.0% | ~90% AI at p=62-73% | 100% AI at p=77.7% | 0% AI |
Simple | 100% AI at p=83.6% | ~90% AI at p=73-81% | 100% AI at p=95.2% | ~90% AI at p=76-82% | 100% AI at p=84.9% |
X = “Unavailable as submission failed to meet requirements”.
0% -> complete false negative.
Robin noted:
Turnitin highlights/returns a percentage of ‘qualifying’ text that it sees as AI-generated, but no probability of AI-ness.
Copyleaks highlights sections of text it sees as AI-generated, each section tagged with the probability of AI-ness, but doesn’t state the overall proportion of the text it sees as AI-generated (hence his estimates).
Additional reading: Jisc blog on AI detection
Recent Posts
- Staff GenAI Survey Report 2024
- Blackboard Upgrade – May 2024
- Learning Technology Team Newsletter – Semester 2, 2023/24
- Getting started with AI: A guide to using the Jisc Discovery Tool’s new AI question set.
- Blackboard Upgrade – April 2024
- Exploring the Role of GenAI Text to Enhance Academic Writing: A Conversation with Learning Development Tutor Anne-Marie Langford.
- Interview with the University’s Digital Skills Ambassador
- Blackboard Upgrade – March 2024
- Case study: GenAI in BA Fashion, Textiles, Footwear & Accesories 2024
- Exploring the Educational Potential of Generative Artificial Intelligence: Insights from David Meechan
Tags
ABL Practitioner Stories Academic Skills Accessibility Active Blended Learning (ABL) ADE AI Artificial Intelligence Assessment Design Assessment Tools Blackboard Blackboard Learn Blackboard Upgrade Blended Learning Blogs CAIeRO Collaborate Collaboration Distance Learning Feedback FHES Flipped Learning iNorthampton iPad Kaltura Learner Experience MALT Mobile Newsletter NILE NILE Ultra Outside the box Panopto Presentations Quality Reflection SHED Submitting and Grading Electronically (SaGE) Turnitin Ultra Ultra Upgrade Update Updates Video Waterside XerteArchives
Site Admin