ICT302 End-of-Course Assessment

Instruction To Students

1. This End-of-Course Assessment paper comprises 5 pages (including the cover page).
2. You are to include the following particulars in your submission: Course Code, Title of the ECA, SUSS PI No., Your Name, and Submission Date.
3. Late submission will be subjected to the marks deduction scheme. Please refer to the Student Handbook for details.

Important Note

ECA Submission Deadline: Saturday, 04 April 2026, 12 noon

ECA Submission Guidelines

Please Read This Information before You Start Working on your ECA.

This ECA carries 70% of the course marks and is a compulsory component. It is to be done individually and not collaboratively with other students.

Submission  

You are to submit the ECA assignment in exactly the same manner as your tutor-marked assignments (TMA), i.e. using Canvas. Submission in any other manner like hardcopy or any other means will not be accepted.

Electronic transmission is not immediate. It is possible that the network traffic may be particularly heavy on the cut-off date and connections to the system cannot be guaranteed. Hence, you are advised to submit your assignment the day before the cutoff date in order to make sure that the submission is accepted and in good time.

Once you have submitted your ECA assignment, the status is displayed on the computer screen. You will only receive a successful assignment submission message if you had applied for the e-mail notification option.

ECA Marks Deduction Scheme

Please note the following:

1. Submission Cut-off Time – Unless otherwise advised, the cut-off time for ECA submission will be at 12:00 noon on the day of the deadline. All submission timings will be based on the time recorded by Canvas.
2. Start Time for Deduction – Students are given a grace period of 12hours. Hence calculation of late submissions of ECAs will begin at 00:00 hrs the following day (this applies even if it is a holiday or weekend) after the deadline.
3. How the Scheme Works – From 00:00 hrs the following day after the deadline, 10 marks will be deducted for each 24-hour block. Submissions that are subject to more than 50 marks deduction will be assigned zero mark. For examples on how the scheme works, please refer to Section 5.2 Para 1.7.3 of the Student Handbook.

Any extra files, missing appendices or corrections received after the cut-off date will also not be considered in the grading of your ECA assignment.

Plagiarism and Collusion

Plagiarism and collusion are forms of cheating and are not acceptable in any form of a student’s work, including this ECA assignment. You can avoid plagiarism by giving appropriate references when you use some other people’s ideas, words or pictures (including diagrams). Refer to the American Psychological Association (APA) Manual if you need reminding about quoting and referencing.  You can avoid collusion by ensuring that your submission is based on your own individual effort.

The electronic submission of your ECA assignment will be screened through a plagiarism detecting software. For more information about plagiarism and cheating, you should refer to the Student Handbook. SUSS takes a tough stance against plagiarism and collusion. Serious cases will normally result in the student being referred to SUSS’s Student Disciplinary Group. For other cases, significant marking penalties or expulsion from the course will be imposed.  (Full marks: 100)

Section A(25 marks)

Answer all questions in this section.

Word Embedding Evaluation

Question 1

This question focuses on the evaluation of word embeddings using standardized tests. You are expected to demonstrate the underlying theory of embeddings and their evaluation.

Question 1a

Using the analogy test set

(https://github.com/stanfordnlp/GloVe/tree/master/eval/question-data ) to evaluate two off-the-shelf embedding models. They are the GloVe 50d model (https://nlp.stanford.edu/data/glove.6B.zip ) vs the Fasttext 200d model (https://dl.fbaipublicfiles.com/fasttext/vectors-english/wiki-news-300d-1M.vec.zip).

Report the categorised as well as the overall accuracy. There are marks assigned to the process (method marks) as well as the actual results.

(12 marks)

Question 1b

Analyse the differences between these TWO (2) embeddings in terms of performance, why is that the case?

(5 marks)

Question 1c

Instead of reporting the accuracy in Question 1a, report the cosine similarity. What are the differences between accuracy vs cosine similarity? There are marks assigned to the process (method marks) as well as the actual results.

(8 marks)

AI ethics chatbot.

(75 marks)

Answer all questions in this section.

You are tasked to design a chatbot that specialises in QA on the topic of AI ethics.

Question 2

This question expects you to evaluate the potential weaknesses of an AI system. The task is to evaluate whether ChatGPT can satisfactorily answer any queries regarding

the Safety Testing of LLM-Based Applications. (https://www.imda.gov.sg/- /media/imda/files/about/emerging-tech-and-research/artificial-intelligence/largelanguage-model-starter-kit.pdf)

Question 2a

Logically if the off-the-shelf models like ChatGPT can do it without modification, that would be the most convenient scenario. Design an evaluation benchmark to test if the off-the-shelf models can perform satisfactorily. The benchmark should include at least 20 diverse question and answer pairs regarding the latest AI ethic guidelines. Of the 20+ question and answer pairs, the question should be manually created and not be found in the handbook, the answer can be lifted from the handbook. Provide this evaluation benchmark in the dictionary format.

(15 marks)

Question 2b

Report the accuracy of the ChatGPT model (in this case we are using GPT-5) using the evaluation benchmark which you have designed in Question 2a. What is your evaluation criterion, i.e. how do you decide what is right or wrong? Please comment on its objectivity.

(6 marks)

Question 2c

Analyse the different types of errors made (by the GPT-5 model). Are they expected and what could be the cause for them?

(4 marks)

Question 3

In this question, you are going to evaluate the potential of large language models (LLMs) in solving a specific problem. Now that there is the evaluation benchmark, implement RAG to improve the performance of the GPT-5 model.

Question 3a

For the retrieval model, suggest a method to improve the retrieval result and compare it to the RAG implementation taught in class (you should make use of the evaluation benchmark designed in Question 2a). Report the performance of the different methods and justify their differences.

(20 marks)

Question 3b

What differences do you observe between RAG and the off-the-shelf model?

(5 marks)

Question 4

Question 4a

Discuss ONE (1) additional feature or modification to the existing pipeline to improve this system and explain the reason behind this improvement.

(5 marks)

Question 4b

Implement your suggestion and report the result, evaluate your proposal’s effectiveness.

(15 marks)

Question 5

In this question you will examine the ethical aspects of the system.

Question 5a

Which part of the pipeline will be susceptible to privacy or security risks? Analyse the reason and potential mitigations.

 (5 marks)

—– END OF ECA PAPER —–