Robin Crockett (Academic Integrity Lead – University of Northampton) has run a small scale study investigating two AI detectors with a range of AI created assignments and has shared some of the initial results.
He used ChatGPT to generate 25 nominal 1000-word essays: five subjects, five different versions of each subject. For each subject, he instructed ChatGPT to vary the sentence length as follows: ‘default’ (i.e. I didn’t give it an instruction re. sentence length), ‘use long sentences’, ‘use short sentences’, ‘use complex sentences’, ‘use simple sentences’.
The table below shows the amount of the assignment which was detected as using AI in two different products: Turnitin and Copyleaks
Essay 1
Essay 2
Essay 3
Essay 4
Essay 5
Turnitin
Default
100% AI
100% AI
76% AI
100% AI
64% AI
Long
0% AI
26% AI
59% AI
67% AI
51% AI
Short
0% AI
X
31% AI
82% AI
27% AI
Complex
33% AI
15% AI
0% AI
63% AI
0% AI
Simple
100% AI
0% AI
100% AI
100% AI
71% AI
Copyleaks
Default
100% AI at p=80.6%
100% AI at p=83.5%
100% AI at p=88.5%
100% AI at p=81.3%
100% AI at p=85.4%
Long
~80% AI at p=65-75%
100% AI at p=81.5%
~95% AI at p=75-85%
100% AI at p=79.1%
100% AI at p=80.6%
Short
~70% AI at p=66-72%
100% AI at p=76.9%
100% AI at p=87.3%
~85% AI at p=77-79%
100% AI at p=78.4%
Complex
100% AI at p=72.9%
100% AI at p=81.0%
~90% AI at p=62-73%
100% AI at p=77.7%
0% AI
Simple
100% AI at p=83.6%
~90% AI at p=73-81%
100% AI at p=95.2%
~90% AI at p=76-82%
100% AI at p=84.9%
X = “Unavailable as submission failed to meet requirements”.
0% -> complete false negative.
Robin noted:
Turnitin highlights/returns a percentage of ‘qualifying’ text that it sees as AI-generated, but no probability of AI-ness.
Copyleaks highlights sections of text it sees as AI-generated, each section tagged with the probability of AI-ness, but doesn’t state the overall proportion of the text it sees as AI-generated (hence his estimates).
The new features in Blackboard’s September upgrade will be available from the morning of Friday 8th September. This month’s upgrade includes the following new features to Ultra courses:
Email non-submitters: anonymous marking improvement to Turnitin in Ultra courses
Currently, when marking students’ anonymous Turnitin submissions, it is not possible to precisely determine who has and who has not submitted, which makes it difficult for staff to contact and support non-submitters.
With Turnitin assignments, the student progress indicator in Ultra courses displays one of three states, the current meaning of which is:
Unopened (student has not submitted)
Started (student has opened the assignment, and may or may not have submitted)
Completed (student has opened the assignment, and may or may not have submitted)
At present, when a student opens a Turnitin assignment, the student progress indicator automatically changes from ‘Unopened’ to ‘Started’. (It is not possible for a student to manually change the student progress indicator from ‘Unopened’ to ‘Started’; this can only be done by opening the assignment.) However, in order to display ‘Completed’, each student who submitted the assignment has to manually change the progress tracking state from ‘Started’ to ‘Completed’. This means that while staff know that a student whose student progress indicator shows ‘Unopened’ has definitely not submitted, it is not possible to determine whether students whose student progress indicator shows ‘Started’ have submitted the assignment or only opened the submission but not submitted. Additionally, as a student can change the status from ‘Started’ to ‘Completed’ without submitting an assignment, the ‘Completed’ indicator does not provide a sufficient guarantee that a submission was actually made.
Following the September update, the way that the student progress indicator works will change, and will allow staff to know with certainty which students have and have not submitted. These progress states will effectively become locked to the submission status in Turnitin and will not be modifiable by students. Therefore, after the September update the meaning of the statuses will be:
Unopened (student has not submitted)
Started (student has opened the assignment, but has not submitted)
Completed (student has submitted)
For staff, this is a considerable improvement over Turnitin’s ‘Email non-submitters’ feature (which was withdrawn by Turnitin for assignment submission points set up after February 2022), as staff will now be able to see which students have submitted and which have not, and will be able to filter and sort the Gradebook and quickly send a message to all non-submitters. And, crucially, this will all be possible to do while not disclosing the identities of the individual authors when marking their work in Turnitin.
Following the September upgrade, staff will continue to see anonymised Turnitin submissions as normal in the Turnitin assignment inbox:
• View of the Turnitin assignment inbox with anonymous marking enabled
However, when looking at the progress summary for the assignment in the Ultra course, staff will be able to see the submission status for each student, and can use filtering and sorting to quickly select and message non-submitters. When using this feature to select and simultaneously message multiple students, each student will receive a private message which will not disclose the identities of the other recipients. As well as being able to read the message in NILE, students are also automatically emailed a copy of the message. Following the changes to University policy regarding student email addresses, from the 13th of September onward, copies of announcements and messages emailed from NILE courses will go to students UON email addresses (i.e., those ending @my.northampton.ac.uk) and not to their personal email addresses, e.g., addresses ending @gmail.com, @yahoo.com, @qq.com, etc.
• View of the Ultra progress summary for a Turnitin assignment with anonymous marking enabled
Progress tracking – automatically enabled in Ultra courses
Progress tracking is a helpful feature for students, as it lets them view their progress in Ultra courses. Progress tracking also provides useful insights for staff about how content in their Ultra courses is being used, and how students are getting on in their courses. When considered alongside other information, progress tracking can provide clues to staff about which students might be struggling and might benefit from additional support.
At present, progress tracking is default off in Ultra courses and needs to be switched on manually in each course. In the days following the September upgrade, progress tracking will be automatically and permanently enabled in all new and existing Ultra courses.
Progress tracking in Ultra courses is a considerable improvement over what was available in Original courses, both in its scope and ease of access. Plus, unlike Original courses, which only provided data about students who had accessed the course using a laptop or desktop computer, the data provided about student progress in Ultra courses takes account of access via laptops/desktops, mobile devices using a mobile browser, and mobile devices using the Blackboard Learn app.
Did you put together a great NILE Ultra course in 2022/23? Do you know someone who did? Or, have you recently put together a great NILE Ultra course for 2023/24? If so, please consider making a nomination for the next round of Ultra Course Awards. Nominations are open until the 31st of December, 2023, and winners of Ultra Course Awards for their 2022/2023 and 2023/2024 courses will be formally announced at the University of Northampton Learning and Teaching Awards 2024: Ultra Course Awards 22/23 & 23/24 – Make a nomination
If your students are looking for some information about logging in to NILE, finding their way around, and understanding a bit more about how NILE works, you’ll be pleased to know that we’ve refreshed the student section of the Learning Technology Team website, which now includes an online induction to NILE for students.
And, of course, do feel free to add a link to this page in your NILE courses.
More information
As ever, please get in touch with your learning technologist if you would like any more information about the new features available in this month’s upgrade: Who is my learning technologist?
The new features in Blackboard’s August upgrade will be available from the morning of Friday 4th August. This month’s upgrade includes the following new features to Ultra courses:
Ability for staff to add images to learning modules
Following feedback from staff about the need to make Ultra courses more visually engaging, the August upgrade will allow staff to add images to learning modules.
• Blackboard Ultra course with images added to learning modules.
Prior to the August upgrade, although courses which contained unread messages were flagged, they were not sorted to the top of the list in the messages page. Following feedback from staff and students, courses with unread messages will now be automatically placed at the top of the list.
• Messages page with courses with unread messages sorted to the top of the list.
The Learning Technology Team would like to say a very big thank you and many congratulations to the four members of academic staff who received a NILE Ultra Course Award at the University’s Learning and Teaching Awards ceremony on Thursday 6th July.
This year’s winners were:
Helen Caldwell & Joanne Barrow – EDUM074: International Perspectives on Education
Alison Power – MID1028: Scholarly Practice in Midwifery
Jodie Score – SLS1019: Introduction to Microbiology
The Ultra Course Awards recognise staff who have created excellent Ultra courses for their students, and each course submitted for an award was reviewed by a panel to ensure that it met the award criteria, which are that the course:
Is clearly laid out and structured via the use of learning modules or folders at the top level, and uses, where appropriate, sub-folders to organise content within the top-level learning modules or folders
Contains a variety of content, including activities for students to take part in
Nominations for the next round of Ultra Courses Awards will open in August 2023, and the winners will be announced at the 2024 UON Learning and Teaching Conference.
The new features in Blackboard’s July upgrade will be available from the morning of Friday 7th July. This month’s upgrade includes the following new features to Ultra courses:
Advanced scoring options in multiple choice and multiple option test questions
Currently, when setting up multiple choice and multiple option test questions staff can select from the following options:
All or nothing – students must select all the correct answer choices to receive credit. If a student selects one or more incorrect answer choices, they earn no points;
Allow partial credit – students receive partial credit if they correctly answer part of the question;
Subtract points for incorrect answers – students will have points subtracted for incorrect answer choices, although the overall question score will not be less than zero.
• Multiple choice/option test question options prior to the July upgrade
Following the July upgrade staff will have more control on the way that scores for each question are compiled, and will be able to:
Define a positive percentage value to award credit for a correct answer;
Enter a negative percentage value (up to a maximum of -100% ) to subtract credit for an incorrect answer;
Allow a negative overall score for the question.
In the following example, it is possible to score from a maximum of 10 points (by selecting only the first and third answers) to a minimum of -30 points (by selecting only the second, fourth, fifth, and sixth options). A student selecting all six options would score -20 points.
• Multiple choice/option test question options following the July upgrade
Complete/incomplete grade schema and formative assessment option
Following the July upgrade, staff will be able to use a complete/incomplete schema when assessing student work that does not require a grade to be given. The complete/incomplete schema still requires an assessment to be given a point value, but any amount of points given, including zero, will result in the assessment grade awarded being ‘complete’.
• Ultra assignment with complete/incomplete schema selected
When a value is provided, the mark shows as a tick to show that the assessment has been completed.
• Gradebook view of an assessment with a complete/incomplete schema.
The July upgrade also includes the option to mark assessments as formative. When this option is selected, the word ‘Formative’ is displayed alongside the assignment in the gradebook. Staff have the option to display ‘Formative’ to students also.
• Assignment settings panel with complete/incomplete schema and formative assessment selected
The complete/incomplete schema and the ability to note assessments as formative can be used independently or together, and can be used with both Turnitin and Blackboard assessments.
The following screenshot shows the student view of the gradebook where two assignments have marked as formative and where the complete/incomplete schema has been used.
After the July upgrade, when adding an image to an Ultra document staff will be able to resize the media using the corner handles that appear when the document is being edited and the image is selected.
Blackboard are currently researching the next round of gradebook enhancements for Ultra courses and are keen to hear directly from academic staff who have used the Ultra gradebook.
If you would like to give feedback directly to Blackboard about your experience of using the gradebook, and to put forward suggestion for gradebook improvements, you can access the survey below.
Kaltura is now available again, following essential server maintenance that took place between 12:00 BST on Monday the 3rd of July and 12:00 BST on the Wednesday the 5th of July, 2023.
Important changes to be aware of and actions to take
• New location for video.northampton.ac.uk Students and staff are no longer able to log in to video.northampton.ac.uk to access Kaltura MediaSpace. The new address is: mymedia.northampton.ac.uk
• Action needed to reconnect the Kaltura Capture recorder Please note that action is required by all students and staff to relink their Kaltura Capture recorder. See the following FAQ for more information: How do I connect Kaltura Capture to MyMedia?
• Videos temporarily not displaying the correct creation date Content uploaded to Kaltura prior to 12:00 BST on the 3rd of July will not currently display the correct creation date. However, all content creation dates will be restored in due course. Content uploaded to Kaltura on or after the 5th of July will display the correct content creation date.
Kaltura – Essential Maintenance Notification – 3rd to 5th July 2023
JUNE 19, 2023
Essential maintenance will be taking place to the Kaltura system between 12:00 BST on Monday the 3rd of July and 12:00 BST on the Wednesday the 5th of July, 2023.
During the maintenance window some aspects of the Kaltura system will be unavailable.
Services not affected:
Kaltura media content embedded in NILE courses and elsewhere will continue to be available and will play as normal. This includes Kaltura videos which have already been submitted by students for assessment.
Services affected for staff
Staff will not be able to upload new content into Kaltura through NILE, at video.northampton.ac.uk, via Kaltura Capture, or when using the KMSGo app. Media previously uploaded into Kaltura will not be able to be embedded into NILE courses and elsewhere.
Services affected for students
Students will not be able to upload new content into Kaltura through NILE, at video.northampton.ac.uk, via Kaltura Capture, or when using the KMSGo app. Media previously uploaded to Kaltura will not be able to be added to assignments as part of the assignment submission process.
Recommended actions:
Kaltura content that needs to be available in NILE courses during the maintenance window should be added before 12:00 BST on the 3rd of July.
Video assignments with submission dates during the maintenance window should be moved to fall after 12:00 BST on the 5th of July.
NILE courses for the 23/24 academic year have been created and staff are now able to find and enrol themselves onto their 23/24 NILE courses. All NILE courses created for the 23/24 academic year and onward are now Ultra courses.
• To find out more about the expected standards and requirements for NILE courses, please see the NILE Design Standards.
• For practical design ideas, and to see what a well-designed Ultra course can look like, please see the Learning Technology Team’s Ultra demonstration course, CRIT101, which is available for all staff to view in NILE under the Organisations tab.
• For additional guidance about creating effective Ultra courses, staff are encouraged to book a NILE Ultra Course Development Workshop with their learning technologist.
The new features in Blackboard’s June upgrade will be available from the morning of Friday 2nd June. This month’s upgrade includes the following new features to Ultra courses:
Prior or to the June upgrade, Ultra tests could only be created directly in Blackboard. After the upgrade, staff will be able to author tests offline in Excel or in a plain text editor (e.g., Notepad for Windows or TextEdit for Mac) and create tests questions in bulk in Ultra courses by importing the file. For more information about how to do this, please see: Blackboard Help – Upload Questions
• Blackboard Ultra test setup screen showing option to upload questions from file
Journals – submission status filters available when using assessed journals
Following the June upgrade, staff will be able to use filters in the Marks & Participation view in journals. These filters will allow staff to quickly view their students’ journal entries based on the status of their submissions.
• Journal Marks & Participation view with Marking Status options highlighted
