PIANO AR RHYTHM GAME
The goal for this personal project was to generate an idea for an AR application and create a visual mock-up of what the final project would look like in action.
My concept for this project was based on my love for playing the piano. I taught myself to play the piano as a kid, but never properly learned to play from sheet music. I instead learned from watching YouTube videos of people playing and memorize the patterns in order to play the song I wanted to play. Knowing that I'm not the only person who learns piano this way, this game is designed for people like me who want to learn to play piano songs without having to ever look at the sheet music for it.
The Goal
In order to achieve this, the game needed to work over a real piano. Allowing the player to play on their real piano and all the UI would be seen through an AR headset. It had to have real time feedback to the music, with rewards for doing well and visual recognition of missed notes. As well as a list of songs with multiple categories of genres. And, of course, a report card showing how well the player did on the song.
Research & Inspiration
As rhythm games have been made for a while now, I looked to how others have made their games for inspiration. Below is a few of images I drew inspiration from in creating this AR mock-up.
AR User Interface Inspiration
Initial Sketches
Design Photo Mock-Ups
Mock 1
Upon review of this first round of designs, there arose some usability issues. The buttons are hard to see because of their lower opacity. As this game design is meant for a type of AR headset, these buttons need to be visible regardless of lighting and background the user may be in. This was something I addressed in the next phases of design.
The game play design was also missing some important things. A difference in notes per left and right hand as well as between black and white notes. The bars for the notes are also missing a way to distinguish which notes are which as they approach the keyboard. These design issues also needed to be addressed in the next design phase.
Mock 2
This iteration of designs got closer to the mark with the opening menu and the song selection being laid out more linearly to help walk the user through the process more intuitively, instead of radial menus. Adding the square black drop behind all the menus solved the issue of visibility to handle any background.
For the piano playing UI the skinnier and lighter colored bars for the black keys worked better to distinguish them from the white notes. But the majority of the piano/game-play portion is still needing a better solution. The added note names on the bars cluttered the playing space instead of clearing up which note was which. The floating combo bar still feels out of place floating in the air, and needs to be more grounded and visually connected to the playing space.
Final Design and Motion Mock-Up
This last round of design took a good stride in problem solving from the last two iterations. The playing portion of the design improved through the use of octave and measure lines. These lines helped to section off the notes and better distinguish them from each other. The measure lines also help to pace the user and give them a better sense of rhythm. As for the combo bar, it's new location on the playing field and minimal design fit much better with the rest of the UI.
The pointer interface is something I had to do some research on. I found that Microsoft's Hololense AR headset uses a system where the user pinches to select with their hand to click things, while their finger simply moves the cursor around. This system seemed intuitive to me so I adopted it for my design for this game.
Motion design for the game was meant to be minimal to match the UI design. The piano playing design is split for right and left hand for color coordination. The combo counter has a slight bounce to give positive feedback for each correct note, and also bounces along with the rhythm. It turns red and shakes when hitting a wrong note, but quickly starts bouncing again with the correct notes, as to not distract the player too much and encourages them to keep playing despite the missed notes.
Conclusion
The final design shows the capability of AR headsets for want to be piano players. This AR rhythm game allows players to pick a song and just start learning how to play, with options to practice at slower tempos to perfect their playing. The simplistic and jazzy design make it a cool and relaxed way to learn songs on the piano without ever having to know how to read sheet music.