TechFest Hackathon 1st Place Winners
|
|
TechFest Hackathon 1st Place Winners |
or, the story of how I won a PS5 at a Hackathon
In February of 2024 I took part in the SCSE TechFest 2024 Hackathon at NTU, and the theme was ‘Applications of A.I. in everyday life’ focusing on AI Generated Content. Link to DevPost.
Being really passionate about video games and their unique way of storytelling, I knew exactly what I wanted to do: make a Dungeons & Dragons powered by AI.
Tech Stack
Language Model:
|
|
At the heart of the project is the language model, for which we chose Meta’s llama2. Setting up the model was very easy thanks to Ollama, which allows us to run these models locally with ease, and query them with APIs. |
The story does not end there however, as the model had to be fine-tuned with a lot of prompt engineering to get the model to behave like a DND Dungeon Master and characters in the game. We configured the AI by giving them personalities, teaching them the context and scenario; and giving it some hand-crafted example messages for it to learn from.
Image Model:
For the image model, I used Stable Diffusion (and some models based on Stable Diffusion.) I ran these models using ComfyUI which allows you to create these complicated workflows to generate images to your liking, and then prompt using APIs.
Above is the ComfyUI Interface, below is a showcase of some of the background we generated using our ComfyUI workflows.
One innovation we made was making use of LoRAs to give different expressions to the character portraits. By keeping the prompt and seed the same, but changing the LoRA for each different expression we were able to generate convincing expressions for the characters.
BONUS: Using the correct emotion
We used a simple sentiment analysis model to classify the dialogues into different emotions, in order to pick the correct expression to display for the character.
Frontend:
We made the frontend using an existing frontend, SillyTavern. We adapted this frontend to better suit our needs of displaying images. It was built around the common APIs for querying language models, so it was pretty easy to set up!
Below is a showcase of the UI:
a
You can find our project on DevPost here! Do also check out this YouTube video below that we made about the project, although this was at a stage before we integrated the SillyTavern GUI.
