AT2 - Development of Interface Application

Jes Lewincamp [497495]

1.  Overview

PianoAR is an educational resource that simplifies the process of chord memorisation and recognition for piano beginners. Using AR, chord basics are taught in a digestible, interactive, visual experience rather than through theory dense memorisation.

2.  Description of the application

Application Overview 

This application uses AR to help teach the memorisation of basic chord shapes to pianists. The app overlays a 3D piano model on top of the image target, with each key able to behave individually. Upon selection from dropdown, chord shapes are visually displayed to the user to allow for practice and memorisation.

 

The significance of the application in the context of education and piano tutoring

For some, the barrier of entry to learn piano is high. The time and costs associated with finding and hiring an appropriate piano tutor, and to then learn largely contributes to this. The traditional method of teaching requires time dedication for lessons which generally have duration between 30-60min. There is little flexibility in this making it even less accessible for those with rigid, full timetables. If the user wishes to only learn casually or to a basic level, little work is required to achieve this, but these barriers still remain largely the same.

The designed software provides an ideal solution with increased accessible whenever and for whoever, at a significantly more affordable price.

Approaching music theory can be an overwhelming experience due to the shear mass of information that lays beneath playing music. 

What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention, and a need to allocate that attention efficiently among the overabundance of information sources that might consume it.  (Herbert, et al., 1971)

By condensing the learning of chord shapes down to its most basic, efficient form, the information shown is minimised allowing attention to be maximised. This streamlines the process of learning and makes it incredibly approachable for beginners who struggle with focus, motivation, and the difficulty of learning piano. Visual cues and UI are clear and minimal to achieve this.

 

The significance of AR interface technology in this context

Virtual piano teachers and chord learning guides exist however, they largely rely on theory or diagrams to convey chord shapes. When utilising the traditional mouse and keyboard interface, the motions of piano cannot be accurately replicated. These programs provide a disjointed experience where they can only teach the visual learning of shapes and not the physical practice.

The utilised interface in this software allows for an integrated experience where chord shape recognition and muscle memorisation occur simultaneously. The user physically responds to the visual cues displayed.

 

3.  Interaction Design 

The user can select one of two modes which dictates possible interactions:

Chord

This mode assists the user with chord memorisation. A chord is selected from the drop-down UI and the corresponding keys are highlighted on the 3D model. This visual cue indicates the correct chord shape which the user can then press on the image target to familiarise the shape. Audio is played to help familiarisation and emulate real-world piano. User can press each key on the phone touchscreen to hear individual notes that comprise the chord.

Keyboard

This mode allows for full freedom by displaying the entire keyboard at once for the user to play and interact with. The user can press keys on the touchscreen to play note.

Storyboard

 

4.  Technical Development

Specific interface and technologies utilised

Augmentation over physical keyboard – visualisation of 3D modelled keyboard on top of existing keyboard. Each key can be altered (toggled on/off) to signpost notes in chord. Allows for accurate piano input.

This app runs on android phones for increased accessibility and transportability. This app can be used whenever the urge strikes the user whether they are at home or out. One hand for phone, one for practice.

The use of Augmented Reality (AR) as a solution sufficiently fulfills the requirements. In this application, AR provides an interactive experience with “human-like” input that can mimic the real-world motions of playing piano (Furness III, 2001). Integrated experience where visual shape recognition and physical practice can occur simultaneously.

 

How to engage with available interaction

The user can press the phone touch screen to interact with both the UI and the displayed keyboard keys.

Pressing keys

• Triggers sound

Using UI

• Toggles modes
• Toggles displayed chord (if in chord mode)

 

5.  Descriptions of 3D Models

3D models utilised

Keyboard: two octave keyboard spread with keys separated for individual functionality.

6. Limitations and Improvements

In the development of this app, three main compromises were made due to technical limitations which in turn limited the functionality of the app.

Android App

 Due to some errors within Unity, the app was unable to successfully present the UI in the mobile builds. Hence, this app was instead built for Vuforia targeting desktop webcam. Ideally, this would have been resolved as the accessibility and  functionality of the app are improved using android phone.

Virtual Buttons

Ideally, the user would be able to interact directly with image target to trigger sound bettering the tactile experience and consolidation of chord shapes. Using Vuforia, this would be achieved using virtual buttons, however, the functionality of these is temperamental and would make for a clunky experience. Touchscreen buttons were instead implemented for prototype.

Image Target

Ideally, any pre-existing piano could be used as the image target for this program and overlay would be displayed on actual keys. This could be achieved using a QR code/Patterned sticker on one key (Middle C) as the image target. By then using standard piano key width, the overlay could extend from this one note. For this prototype, a printout piano image target was utilised.

Other potential improvements could include using AR headset to allow for two hands free for piano use. Whilst this would increase user experience there would be downsides in accessibility.

 

7.  References

No AI was used for this project.

24 key Piano sound samples sourced from (Lauser, 2018)

 

Bibliography

Furness III, T. A., 2001. Toward tightly-coupled human interfaces. Seattle: University of Washington.

Herbert, S. A., Deutsch, K. W., Shubik, M. & Daddario, E. Q., 1971. Computers, Commuications and the Public Interest. Designing Organizations for an Information-Rich Wolrd, pp. 40-41.

Lauser, P., 2018. Internet Archives. [Online] Available at: https://archive.org/details/24-piano-keys [Accessed 14 August 2023].