An evaluation framework for GenAI LLMs in the tertiary sector
Mari Cruz Garcia Vallejo
Digital Education Consultant
Digital Education Consultant
SFHEA, SCMALT, MSc, MEng, ITIL, Councillor
General Information and Social Responsibility
Provider Details
- Provider
- LLM used, purpose and potential use
- On what devices can it be used?
- LLM used, purpose and potential use
- On what devices can it be used?
Social justice and Ethical Considerations
- Provider's legal entity location
- Labour legislation and tax jurisdiction
- Data server and computing infrastructure locations
- Natural resource consumption and impact on local communities
- Human resources used for LLM training
- Adherence to specific ethical frameworks
- Labour legislation and tax jurisdiction
- Data server and computing infrastructure locations
- Natural resource consumption and impact on local communities
- Human resources used for LLM training
- Adherence to specific ethical frameworks
Accessibility and Data Protection
Accessibility Features
- User interface compatibility with screen readers and assistive technologies
- Compliance with Web Content Accessibility Guidelines (WCAG)
- Adherence to national legislation on software/web page accessibility
- Compliance with Web Content Accessibility Guidelines (WCAG)
- Adherence to national legislation on software/web page accessibility
Privacy Policies
- Availability of privacy policy and terms of use
- Information on data protection regulations
- Types of personal information stored and purpose
- Storage location of personal information
- Necessity of sensitive personal information for access
- Information on data protection regulations
- Types of personal information stored and purpose
- Storage location of personal information
- Necessity of sensitive personal information for access
Data Processing and Security
- How LLM processes user-provided data and information
- Storage of user queries
- Third-party access to user data
- Protection measures for data integrity and security
- Storage of user queries
- Third-party access to user data
- Protection measures for data integrity and security
Pricing and Result Management
Subscription and Payment Plans
- Types of subscription and payment plans offered by the provider
- Availability of a free version for users
- Availability of a free version for users
Query Management
- Maximum number of queries per day/account
- Access to previous queries
- Ability to search, catalogue, and archive previous queries
- Export/download options and formats
- Time limit for accessing previous queries
- Sharing capabilities with other users and/or on the Internet
- Access to previous queries
- Ability to search, catalogue, and archive previous queries
- Export/download options and formats
- Time limit for accessing previous queries
- Sharing capabilities with other users and/or on the Internet
User Interface and Input/Output Formats
1
Navigation and Ease of Use
On a scale of 1 to 10, how easy is it to navigate the application's interface to provide instructions to the LLM?
2
Input Formats
What types of formats does the LLM use for receiving instructions: text, audio, scanned images, etc.? Can files be attached as part of the instructions in the prompt?
3
Input Limitations
Is there a maximum character limit for text instructions provided to the chatbot?
4
Output Formats
What types of formats (text, images, video) does the LLM allow for the output?
Evaluation of LLM's Limitations and Veracity
1
Accuracy and Consistency
Did you find inconsistencies and/or hallucinations in the results provided by the LLM? What are these inconsistencies like? How would you rate the veracity of the results obtained from an academic/scientific knowledge perspective that the LLM's response provides?
2
Source Citation and Bias
Is the LLM able to cite the bibliographic sources it uses? Is there any cultural or anthropological bias in the answer/results generated by the LLM?
3
Language Adaptability and Limitations
Is the LLM capable of changing language style, semantics and syntax according to the instructions provided? Is there any limits in the number of characters/words provided in the answer?
4
Creativity Assessment
On a scale of 1 to 10, how would you rate the originality and creativity of the results obtained? What is the basis of your score?
Effectiveness and Intellectual Property
1
Prompting Techniques
Is the LLM able to identify errors committed and the result through prompting? How many interactions were necessary for the LLM to produce an acceptable result? Was it necessary to provide additional data or information to the LLM, such as examples, to improve the result?
2
Customization
Can the LLM be customized to adopt a certain perspective or persona by personalizing parameters such as information sources, text results, multimedia results, and initial configuration for a particular context or scenario?
3
Copyright Compliance
Does the provider offer information about copyrighted data and information used to train the language model? Does the provider offer information on how its LLM complies with current legislation (international and national) on copyright/intellectual property?
4
Content Generation and Rights
Does the LLM allow modifying copyrighted multimedia content as part of the query/task? Does the LLM provide copyrighted elements as part of the result in content generation? Who retains copyright in the production of multimedia content by the LLM? Does the LLM provider inform whether it retains any copyright on the result generated by the LLM?
Institutional Integration
Policy Compatibility
Are the terms of use and privacy policies of the LLM compatible with institutional policy on the use of new computer systems and software?
Pedagogical Alignment
Is the use of the LLM for pedagogical purposes compatible with institutional policies such as recommendations on AI use, pedagogical model, and academic regulations?
System Integration
Can the LLM be integrated with other systems and computer platforms (LMS, Microsoft 365, etc.)? Can the use of the LLMs be scaled up within the organisation to staff and students? What costs would be involved in scaling its use?
Organizational Support
What teams/deparments the organization could provide technical and/or pedagogical assistance for the use of the LLM?
