In from the margins-enabling people with disabilities to learn and create music

People with the most severe physical disabilities, for example who are only able to move their eyes, are at most risk of being left in the margins of society. They can communicate via 'eyegaze' text systems, but the challenge is to design an analogous music system to enable them to learn about, explore, and create music, so as to communicate and connect with others in a more universal way. The combination of eyegaze and music is a new area, so the design methodology relies on incorporating eyegaze features into an existing music system designed for switch-control. The design of this usercentred system has evolved over 20 years, with the continual involvement of disabled users and teachers. The resulting prototype eyegaze music system is designed to facilitate exploration and creative work without continual help to operate it, and was successfully tested by eyegaze users. This is the genesis of an instrument to enable anyone even with the most severe disability to participate and share in creative activities, and connect them more with society.


Introduction
This article focuses on people who have great difficulty taking part in the kind of music-making and learning which are normal opportunities or activities for most people in society.It focuses on potential students who are at the limit of physical disability, and cannot control any part of the body except their eye direction (often termed 'locked in').Such people can too easily be hidden in the margins of mainstream human culture, and giving them tools to enable them to explore and learn about music can help them back.Creating a musical work can allow them to connect with people who hear it, and let them communicate on an equal level with other people, and help them bring their voice 'in from the margins'.
If people can control their eye movement, then 'eyegaze' systems built into a computer screen can detect this and move a cursor to match where they are looking.The screen can display a set of graphic items (eg 'buttons'), and they can select one by 'dwelling' on it -ie looking at it for a set time.Such systems are now commonly used to enable people to type words and communicate ('AAC'3 ) using only eye movement.
Having been asked to develop new software features to provide a way for such people to create or learn music using eye-control, the designer therefore had to ask the question 'How does such a system need to work to enable people to learn, to explore and work with music independently using eyecontrol?'.This article explores the issues in designing such a system, to enable people using eyegaze to participate in music education, and describes some initial outcomes with eyegaze users.
We first briefly look at existing eyegaze AAC systems, then at the issues around designing a similar system for music.We also find that there is almost no available body of expertise or knowledge for building an eyegaze music system.Thus, we need to examine other options, specifically whether we can utilise the experience which has derived from the development of the accessible music software 'E-Scape'.This is focused on enabling disabled switch-users to work with, and explore music makingboth by composing or by live performing.If need be, a student can operate everything using just one physical movement to operate a single switch.E-Scape has been created via user-centred design over almost 20 years of observation of, and consultation with, users and teachers, during which a number of specialist features have been added and refined, and much knowledge of user interface and operational issues gained by the designer.We then look at how these features could be adapted to enable a viable working prototype to be developed as the first stage of eyegaze music system development, enabling the first users to operate it successfully and provide essential feedback.

Art didactics
We first need to consider how a severely disabled student can be integrated and brought into music education, and given a chance to learn.Music is a subject where interaction forms an integral part, with people playing and listening together.Those learning theories which emphasise the importance of participation and interaction in the learning process, and of the relational aspect of teaching, are therefore of particular relevance to music education.From this perspective, one can say that learning, communication and participation are core concepts that have a close relationship with each other (Skogdal 2014b).
In the area of didactics, one aspect is a consideration of the ways a teacher and student (and their peers) can relate within the learning process.These can span a range of teaching models, from traditional 'showing and telling' at one extreme, to unguided 'learning through experiencing', or 'learning by exploration' at the other.
The former involves an expert teacher showing and telling information, with the student then going away (a typical higher-education 'lecture').This is the extreme; the model would usually also be accompanied by some exercises set by the teacher, for the student to practice and hone the topic they have been shown, and test if they have absorbed it.Music in particular has an extensive history of teaching methods of this kind which focus on the 'master' filling up the student with his or her knowledge.This is a tradition which has excluded many people from music education (Ruud 1996).There are still many teachers working with conventional instruments who find it hard to embrace or adapt to the newer explorative teaching methods.However, it must be said that in recent years teaching has slowly started changing through adapting a wider variety of methods, and adopting plans and regulations aiming towards a more inclusive educational system.
In contrast, 'learning by exploration' (we can use the term 'explorative learning') focuses on the student learning about a topic by exploring and working within it.The idea of learning by participating in activity is over a century old."Learn to do by knowing and to know by doing" (Dewey and McClellan 1889).
Students can think they know something, but until they test it by trying it out in action they can't rely on it as 'real knowledge'.The activities will usually not be just random 'play' -a teacher can advise and gently guide students during their exploration.Most importantly, the teacher can plan the set of activities (or even specific tasks) with some educational goals in mind.But the focus is on students trying things out themselves, and 'learning by doing'.
At any learning stage, there will be a range of tasks of varying difficulty.Students will always be able do some easier things independently without help, and other more advanced things when assisted by someone with more knowledge (Daniels 1995).If only given tasks at the easier end of the range, then students are able to work independently without any help, but they learn nothing new.It is only when stretched to try harder tasks which do require some help that they can develop their knowledge and skills.Vygotsky calls this range the 'zone of proximal development', and a good teacher should plan activities that encompass it (Karpov & Haywood 1998).Students may at first rely on help from others, but can gradually evolve towards working more independently as they gain knowledge, and a good teacher should gradually withdraw support as a student achieves success (Hammond & Gibbons 2001).
The aim ideally is for the student to achieve results, and create something while working and learning.To have 'results' is especially important for a disabled student, as the working effort is far more laborious and physically tiring, so achievement and self-satisfaction are important to maintain motivation.Concerts and performances with disabled musicians have shown the importance not only of the process, but also the need to challenge people to do a good job, even when they have other difficulties in their daily life (Svendsen 2014).
Whatever their starting point, everyone has the potential to evolve their experience and increase their knowledge (Imsen 2005).With support, learners are able to extend their current level of knowledge (Cole 2001).This support for development and learning is sometimes termed scaffolding (Rasmussen 2001).Some have criticised this term, as the metaphor might be interpreted as the teacher telling students what they should learn (Stone 1998).But a correctly judged supportive scaffolding can show what the student is able to achieve in the future, "What they can ask" (Vygotsky et al. 1978).Having tried doing something with support, the student now knows what is possible, and what to ask for help with if the support is reduced.
However, to participate in any learning activities, all students need appropriate 'tools' (whether a pen, a computer, or a flute) which they can physically operate.For example, when learning to compose or play a musical instrument, a big part of the process is exploring and practising with composing tool or instrument, and to do this students need to be able to work independently.
Another important facet of learning is participation in a group.As well as working with a teacher, Dewey considers that communication and activity together with others is important in order to transfer and share knowledge.(Dewey 2010).Learning in the company of others, especially interaction with peers in a social context, is thought to bring many educational benefits, as well as enhancing children's self-esteem and motivating them to tackle new tasks.(Damon 1984).
So the concept of students learning more by working with peers who have more knowledge than they could alone is well-established.Peers can be 'scaffolding' support for a learner, but for the latter to really be a 'peer' the relationship needs to be mutual -ie sometimes the disabled student should be able to support their peers, not always the other way round.This will only be possible if a disabled student's music system can provide them with the necessary support.
Until recently there haven't really been any tools or systems available which can enable people who can only move their eyes to work with music.Because participation hasn't been thought of as a possibility, these people haven't been taking part in music education or musical activities, and no one has considered how they could be included.But new opportunities developing from music technology and computer access technology are gradually changing the perception and reality of what is possible, and enable us to now envisage the provision of appropriate tools for these students via a computer-based music system.
The system should ideally aim to provide some of the support 'scaffolding' that disabled students need, so they can operate on a more equal level and can interact with teachers and peers in a far more normal manner.The teacher is not then having to continually work for the student (for example to write things down for them, or operate a computer), but can collaborate with them in discussing or performing the activity.The system should provide consistent support, but the aim is to gradually reduce the level as the student learns more.Thus, the system should also aim to promote "... a methodology that emphasises customising learning" (Skogdal 2014a).This can provide a flexible framework which enables a teacher to not only set tasks and activity environments of varying difficulty, but also tailor the level of support, eg to 'remove scaffolding' at points.These are ambitious aims -for the system to not only provide support for disabled students to participate at all in music eduction, but also enable explorative learning by varying the level of support and challenge to maximise each student's learning opportunities.

Design methodology
We now consider the issue of researching how to provide a system which provides such 'scaffolding' support for an eyegaze student, and also empowers them to work it themselves.As the evolution of eyecontrolled music systems is at a very early stage, little or no literature or experience exists.Thus, the basis of the research and development of this eyegaze music system has, of necessity, been almost entirely built on experience from existing users and developers of eyegaze AAC systems, the design of a proven switch-controlled music system, plus prototype testing with eyegaze users and people working with them.Volume 4, No 2 (2015) Even literature about 'assistive music technology'4 in general is very limited.Hence, during the process it has been especially important to keep in touch with people working within, and experienced in the same field -to root the work in practise on the ground, and to discuss observations and design ideas.These often comprise scattered individuals or small groups in various niches, (eg York, Tromsø, Glasgow, Manchester, Rotterdam, Cork, Dublin).
Observation and feedback, both from users and teachers working with them is especially vital for eyegaze, as very few designers can replicate the experience of operating an eyegaze system, and each user is different.It is very hard to design for eyegaze users, and determine the best ways of doing operations or placing items on the screen, without observing users who are genuinely dependent on using their eyes.
This kind of design based on user involvement and testing is often termed 'user-centred design' -the designer analyses user needs and abilities, foresees how a user will use the system, and designs features accordingly.However, it is hard for designers to intuitively understand what any user experiences -let alone a disabled one -so testing in the real world is vital to assess the validity of the designer's assumptions.
It must be made clear that we are here talking about 'user-centred' design, not 'user-led' or 'userdriven'.There are some criticisms of 'user-led' design, for abdicating responsibility and expecting users to know what they need.Although users can well flag up problems, they may not always know the best solution."Design is about addressing user needs, not just listening to user demands."(Kitson 2011).In user-centred design then, designers should not just listen to users' ideas, although these are important, but also observe how they work and conceive improvements to help with specific issues which are noted.
Being user-centered means understanding a problem and the users, analyzing user behavior and listening to their wants, then translating this into needs that drive a creative solution to the problem.Being user-led or user-driven, on the other hand, means responding to user feedback without applying the filters of analysis and translation.(Kitson 2011).
So we both want, and need, to base research on the involvement, and observation, of eyegaze users who have the ability and motivation to work with music.However, this does require an initial prototype system being in place for people to try out.As suitable eyegaze users willing or wanting to try out the system are geographically widespread, and have limited time and energy, it was especially important that the first system to be designed and built -before any user contact -was as usable and capable as possible, otherwise the process would take a long time to progress.
The question is then of how we create a prototype eyegaze music system good enough to trial by users, when there is little or no experience or literature available on the subject.We have plenty of knowledge of workable eyegaze AAC systems, with practical proof in the shape of competing commercial systems, which have broadly similar user interfaces and features, and can be seen to be effective for people to use.
What we do have in the music area, is research done over many years to develop a disabled-access music system called 'E-Scape', which is now proven in use.So can we use this information to make an eyegaze music system?In the following sections, we will describe existing eyegaze systems for textentry, the differences in requirements to support working with music, and some relevant features of E-Scape which assist switch-users.
If we can identify the requirements for an eyegaze-controlled music system, and the differences from, and similarities to switch-control, we can determine the changes needed, or additional features which should be applied to E-Scape to 'convert it' for eyegaze use.We could thus effectively merge features of the two types of system to have the best chance of having a viable first prototype for eyegaze user trials.

Eyegaze systems for communication
Over the last 15 years, eyegaze systems have become more sophisticated, usable, and inexpensive, and it should now be feasible (in the developed world) for anyone in this situation to have one and learn to use it.
The first stage of the research process involved examining existing AAC eyegaze systems for entering text, which will inform the design of the equivalent music system.It should be noted that when using such a system to write text, the task is relatively simple -we are not doing complex layouts and tables, mainly just writing words.The main 'top level' screen typically displays text items which can be groups of letters, 'numbers', or 'punctuation'.If users look at and 'dwell' on a group, they are next presented with a further screen which displays items in that group (eg individual letters or numbers) from which they can select one.This is then added to a word or sentence they are building up.Various additional features help reduce the amount of work needed, for example: (a) if you enter a full-stop (period), the system automatically inserts a space, and makes the next letter capitalised; (b) after you have finished a word, a space will automatically be inserted before the next word; (c) most importantly, there is a 'prediction' system, as used in smart-phones -a dictionary of words, so that after typing just 3 or 4 letters, you are presented with a list of words beginning with those letters, from which you can choose.Thus typing a 10 letter word could only actually involve 3 or 4 selections rather than 10.Some systems also 'understand' grammar, and in some cases can predict and provide a list of words which could naturally follow the previous one.
But even without these aids, compared with music (see below) the operations the user may want to do are relatively limited; the majority of users will just be entering text, ie choosing letters, numbers, words or punctuation in turn.Any editing which they might do will still be at a relatively simple level, Volume 4, No 2 (2015) such as moving through the text selecting words, deleting, copying, moving, or pasting them, or perhaps choosing text style or size.

Working with music
Eyegaze AAC systems, as described above, are in widespread use and work effectively for the job they do.What are the challenges in transferring their design to work with music?
Let us briefly look at the kinds of activities a student might need to do to explore music-making.Firstly, the structure of music is far more complex than text, and the operations needed to be done to enter and edit it are also more varied and involved than word processing.For example, a piece of music will typically have many tracks ('parts'), each usually having a different instrument.Each track will contain many notes, each one of which has to be entered, and has various aspects, eg a position in time, pitch, length and loudness.Notes can overlap other notes or start with them (ie a chord), or be grouped in other ways.It should also be obvious that we can't use text-style 'prediction' without curtailing a composer's freedom to structure the music any way they want -there is not usually one 'correct' next note!There are then many ways we can edit notes and tracks.As for letters and words in a text editor, we need to select one or more notes as a block, then copy, delete or paste them.But in addition we need to be able to do many other operations, for example: change notes' position in time, their pitch, length, or loudness; choose instrument sounds; add, move or delete parts.We also want to be able to play back the music we have put in, sometimes while in the process of editing, in various ways and from various time positions.

The E-Scape switch-accessible music system
We now present some of the history and rationale behind the research and development of the switchaccessible music system 'E-Scape'.This has been built over many years using methods informed by 'emancipatory research', and has been the starting point for this eyegaze music project and the inspiration for its methodology.Emancipatory research involves researchers being focused on helping disabled people to empower themselves."The placing of control in the hands of the researched, not the researcher" (Oliver 1992)."Researchers must put their knowledge and skills at the disposal of disabled people" (Barnes 1992).
User participation has been the guideline for this research work over the last 20 or more years, developing a system which aims to empower people with various disabilities, and enable them to learn and work with music.Because every user is different, the design of the system has been (and still is) continually evolving to cater for individual needs, and enable as many people as possible to create music independently.
Although people with disabilities haven't been directly involved in the detailed design work, they have been involved in every other part of the process.The concept has all the time been to develop a system which can adapt music-making (or in other words adapt society) to the disabled person, rather than the person having to adapt to how most people compose or perform music.The aim has been, and is, to improve the life and opportunities of people with severe disabilities.
Thus, E-Scape has developed as a necessarily user-focused or 'user-centred' system.We say 'necessarily', because the target users of the system have very limited scope to adapt to the way a system works; the system needs to adapt to the way they need to work.Disabled users simply don't possess much 'ability overhead', where the user copes with sub-optimal features or inefficiencies in a system.The fact that the developer has been in constant touch with users throughout the development cycle has not only enabled adaptations to be tailored for each specific user, but also delivered to them on a fast rotation to enable fresh feedback to be received.This development is thus a prime example of usercentred 'emancipatory' methodology.
Testing and observation has been undertaken over a 20 year period (and is still ongoing), with many dozens of end users, from 18 months old to late middle-aged adults, from several different countries.Information has been collected both from direct feedback from users and teachers, and from observations of people using the system in different situations and stages of development.This degree of user participation, and continual development means the system is now quite refined for a range of disabled users.The number and diversity of testers has also resulted in a wide knowledge of the variation of adaptions which are needed to provide an optimum system for people with disabilities.However, the research has at no stage hitherto included eyegaze users.

Adapting E-Scape for eyegaze music
We now present the research results and the evolution of the prototype 'E-Scape eyegaze' system.We can see that E-Scape's user-centred design with extensive observation over time of a wide variety of users, as well as advice from teachers and helpers has led to the present switch-operated system which is now well-proven.The question was then as to how to design an eyegaze-operated music system, with the limited information and experience available.As described earlier, for such a complex system to evolve we have to rely on observation of people using it, but for this process to work the initial prototype needs to be capable enough for people to use it to do something useful and interesting in the first instance.
So, how do the systems compare, and what changes will need to be made to E-Scape?An examination of both systems showed that many of its specialist features that support independent use with switches could be transferred to facilitate eyegaze use, and required little or no structural change.So by adding a few new features, plus some adaptation of existing ones, we might produce a prototype system with enough functionality to be tested by real eyegaze users.
Key features we know are needed in eyegaze AAC systems include linked screens of buttons, one of which can be selected by dwelling the cursor in it.Each button performs an action, eg typing a letter into the text, or opening another screen of buttons.There are various settings such as variable dwell time, changeable colours and text size, etc.
Happily, E-Scape's top-level user interface is already quite similar; it has a structure of menus which appear one at a time, with each item performing a musical action or opening another linked menu.This matches the hierarchy of screens of buttons used in eyegaze AAC systems.So the menus simply needed expanding to fill the screen, plus making the layout of menu items more flexible, with variable numbers of columns and rows of items.NB.E-Scape menu items can already speak, for use by non-readers.E-Scape's menus also have a large 'header' section with a question, prompt or other information, which can provide a place where the user can look without making anything happen (ie no 'dwell' function there).Each user required different sizes and positions for this rest area -for example some needed it on top, some on the right.Volume 4, No 2 (2015) Some users need larger buttons to be able to dwell on them successfully, so have fewer options at once.There are many other settings, allowing the system to be tailored to suit each user best.
As described above, the structure and order of context-dependent menus following each other guides the user, and also reduces the number of scanning and selection operations.This goes some way to compensate for the fact that there is no possibility of 'text prediction' to save work.Such features are important for eyegaze users, who typically find working with eye direction very fatiguing, arguably more so than many switch-users; it is far harder to keep your eyes pointing somewhere particular to 'rest', than just relaxing and not touching your switch.The accuracy you need to look at and maintain your direction in order to 'dwell' and select something is also hard to achieve without tiring.Some additional features were needed when performing an editing procedure in the music ('score') window when no menu is open, for example stepping through a series of notes.In this situation we need a way to trigger each action like a switch-user would.Happily, the score window already has a large area at the bottom right where information is shown.This can be used as a button to dwell on (or just move to) to act as a switch for 'next action', eg to select the next note.The bottom left 'play button' can also be used as a 'stop' button during operations.All that was needed were some label changes, and options to make each button taller or wider, to suit the user.Various key features from E-Scape were also able to be used almost unchanged to help reduce the physical and cognitive work-load for the eyegaze user.These include menus, guidance, customisation, and auditioning.

Menus
Each menu (ie screen of buttons) fills the screen, and only presents the choices which are appropriate in that situation, depending on the activity being undertaken.For example, after opening the main (toplevel) menu, the user might choose 'Edit', which then opens the Edit menu, with various options to edit the selected notes.But if no notes were selected, the same menu looks rather different.Menus have a large 'rest' panel, which can be looked at without causing anything to happen.This panel can contain a 'question' or instruction, which may be phrased in verbose or conversational language.The question can also be altered to suit different circumstances and/or contain information about the current situation, or stage of the operation.(This exact structure was present in the switch-operated system).Volume 4, No 2 (2015) Figure 7. Verbose menu asking a question after putting in a note All this has been found to reduce confusion in users, and enable guidance or explanation to be given when needed.

Guidance
Much design effort has gone into guiding the user through activities, via a linked series of operations.This design has evolved from watching people at various stages of experience using the system, and noting what they find hard to do, and/or understand.
The aim is to guide users through operations and reduce the number of dwell selections needed, both to reduce physical effort and lead them through the process.Users still need to make decisions at each stage as to what they want to do, but the laboriousness is reduced.During an operation, they are presented with a series of context-dependent menus within a framework aimed at completing the task.After each choice or stage of the procedure, they are automatically presented with the next appropriate menu.A menu choice might perform a musical operation, after which (if appropriate) it might automatically return to the previous menu, or re-show the same menu.Menus may alter their contents depending on the previous action, and only show the options which are relevant.
A good example of this is copying and pasting.We are used to doing this in documents, and the same can be done in music with notes being copied then pasted somewhere else.However, observation of users illustrated that the process was problematic, as it does involve a number of stages.We need to describe these in a little detail, as this issue is important to grasp.
First we must select the set of notes to copy.Next we have to go back to the 'Copy' menu, and select the 'Copy' button, but nothing then seems to happen, which confused users -remember that most will have no experience of editing text, even if they have been writing with AAC.Thus we had to explain that the notes have been copied 'into a hidden storage place'.Then we want to paste the notes, but before we can do this we need to choose the location we want to paste at.Again we must get to the 'Copy' menu, then select the 'Move cursor' button, then get to the position we want.Finally, we need to go once more to the 'Copy' menu, and select 'Paste'.
Using eyegaze (or indeed switches), each of these stages can involve a long-winded process, and it was noted that users -especially those newer to music-making -could easily get lost as to where they were, and what the next step was.They often needed continual verbal guidance and prompting of what to do next.So could the system itself help them with similar guidance to make the operation simpler, easier to operate and less confusing?An 'automated' procedure for copying was designed5 , which guides users through the process by prompting them at each stage: First they are asked to select the notes to copy, and are guided through that process.After this, the notes are automatically 'copied' (no menu operation needed), and users are then presented with a final menu with options to choose where to paste the notes.When they get to the position they want and finally choose 'OK', the notes are automatically pasted in.
At each stage they are given a menu to confirm they are happy, or want to go back or forward one step (users quite often made this mistake), or start again.This menu keeps reappearing until they choose 'OK', and means users have the chance to correct a mistake within the process, and don't have to go back to the top-level menu until they've finished.This is just one example of a 'guided procedure', developed with user participation over a period of time.Not all are as involved as this, but observations have shown that users are helped by using such procedures.People are able to operate the system themselves who might otherwise 'get lost' doing it step by step as described earlier.
But equally important is that such procedures drastically reduce the number of selection ('dwell') operations needed.It must be remembered that operating with eyegaze is tiring and requires concentration, and that reducing laboriousness can help people who couldn't otherwise work physically for long enough, enabling them to explore more freely and achieve and learn more.

Customisation
A teacher is able to 'customise' each menu to remove items not relevant to the musical activity being undertaken, or reduce the level of support 'scaffolding'.In addition, by removing or adding choices the teacher can effectively guide students through a different set of possibilities, to tailor their pathway of exploration or introduce new topics or terminology.Thus different activity sessions can have different themes depending on the options made available or not.

Music auditioning
During some note editing operations (eg after each change of pitch or position), music in the area you are editing can be automatically played back for you.This greatly reduces physical effort -in most music systems, playing back a specific fragment of music around a note after you have edited it is often surprisingly laborious (Anderson 2002).Some editing operations also repeat, with feedback provided each time the note changes.
An example of how this greatly helps users (especially beginners) is when they are adding a new note to their piece.Eyegaze-users need to do this by 'Step Entry' -they choose the time where the note will start and its length, then choose the pitch.For beginners, the easiest way to choose the pitch is 'by ear' using another example of a 'guided procedure': A new note is added and initially has the same pitch as the previous note, but can then repeatedly change by one step, each time playing back how the music now sounds.Students can stop the process when they hear the pitch they want.Hopefully, as they learn more about music, students can choose a pitch and octave for a new note using menus ('C' , 'D#' etc).Another example is that in some menus (eg for choosing an instrument, or note length), as each button is looked at, a small fragment of music is played to illustrate how it would sound with that option.
All this means that a teacher needs to be less involved in the minutiae of operating the system; students can work at their own pace, and the teacher can work more as a guide or adviser on the musical decisions to be taken.Switch-users in London for example, who had only been able to play simple tunes prepared by a teacher, were able to have weekly composition lessons with a tutor, and produce a CD of pieces they have composed themselves.

The E-Scape eyegaze prototype in use
As described above, the relevant E-Scape features were already suitable in general ('eyegaze ready'), so were able to be modified relatively easily for eyegaze users.The two key changes were that (i) menus fill the screen to provide a grid of buttons which can then be selected by 'dwelling'; (ii) during editing and playing operations in the score screen, the switch actions for 'next' and 'finish' are performed by looking into the two areas at the bottom of the screen.
Before testing with real eyegaze users, we needed to make sure it worked well enough, so trials were carried out by the designer and colleagues, using the 'camera mouse' application, which tracks the face to move the cursor in a similar way to eyegaze.Tests confirmed that all the previously switch-controlled musical operations could be carried out just by moving the cursor, ie by moving the head.

First user trials -Richard Bennett
During this time we also visited our first user, Richard, at the Royal Hospital for Neuro-disability, Putney.This eyegaze music project actually came about because of him -the author received an enquiry in 2013 from the hospital to see if we could help one of their 'locked in' patients, Richard6 , who was very keen to learn and play music.He was happy to commission new work, if nothing was available.Nothing was.Several months of development work ensued, as described above, and the first visit to Richard was more about sorting out his equipment for music-making, and testing what he could do.
Nevertheless, Richard was able to select items on screen and enter some notes, but concentrating on the screen and where he was looking proved very tiring.His helper suggested it would be useful if she had quick access using the eyegaze touch-screen -if she could see what Richard was trying to select, she could press an item immediately for him without needing to wait for the dwell time.This facility was added for the next session, and indeed proved most useful.This is a good example of user-centred (or even in this case 'user-led'!) design.It is also useful for a teacher to be able to show a student something quickly, using keyboard shortcuts, eg 'what it would sound like if we make this note higher, or louder, or longer'.The principle should be that a specialist system should be as usable as possible for all users.
But people all felt confident that the system was going to work for him, and a further session was scheduled.So in March 2014 the first real trial of E-Scape eyegaze took place, and Richard was quite quickly able to enter and edit a melody successfully.The prototype, backed by all the previous years of developments, was deemed a success -it was usable, and feedback was very positive.By the end of the session Richard was very tired, but we mentioned that he could also play music live, and would he like to try it?Suddenly Richard was less tired!E-Scape has several ways of performing music live using the mouse, trackpad or touchscreen, and as there is an option to 'not press the mouse button', it should also work straight away to perform live using eyegaze.There is no 'dwell' function -you play something as soon as you look at it, which does take some practice, but there are various options to help.
Richard tried two ways of playing.Firstly he tried playing phrases by looking at them, and when he looked at a different phrase the previous one was set to cut off which made for a cleaner performance.When we zoomed in enough so there were not too many phrases on screen, and had spaces between Volume 4, No 2 (2015) them, he was able to pick out and play different phrases quite well, allowing for having had no practice at all.Next, he tried playing individual notes by looking along a track.Any note or chord which crosses the place he looks at will play, but will stop as he looks 'off it' (into some empty space).In both cases, Richard's vertical eye position was ignored, giving him an easier task -only looking left to right.
A video https://www.youtube.com/watch?v=BE8BMa97mAY&feature=youtu.beshows Richard trying out performing like this -remember this is his very first time.In this example, Richard is currently playing the dark blue notes in the middle, by looking approximately in the centre of the screen.Richard was quite excited, and played and had fun for another 20 minutes; then he really was exhausted.
There are many further developments planned in the performing area, for example to let players change track themselves -which changes which set of notes they are playing.One way would be to dwell in the bottom right panel, to change to the next track down.

First user trials -Bram Harrison
Bram's aim was to compose his own jingles for his radio show7 , and the BBC Performing Arts Fund sponsored a project to find a system for him to compose music using eyegaze with an expert tutor / technologist.Attempting to use his eyegaze system to operate a variety of mainstream music software had not worked for him -it was just too difficult for him to control.The author was asked to help at a late stage, and was able to provide the system developed for Richard, along with improvements resulting from his initial trial feedback earlier in the year.In summer 2014, a system was delivered to Bram, with his tutor setting up and reporting observations and ideas.
Despite some issues with his computer, when the system was working both Bram and the tutor felt it to be a success, as he could actually work it to compose a number of short pieces, and was later able to work on his own.In the video https://vimeo.com/112689731we see how Bram navigates the menus to choose an instrument and note length.He then enters notes and changes their pitch, creates chords, and plays back the music he's written so far.Seeing him working shows how physically demanding it is.Several features proved their worth in both sessions: The ability to 'customise' each menu proved invaluable -some menus had too many items, and gave users too many choices, or simply made each button too small to easily dwell on.The automatic auditioning and repeated procedures while editing also proved most useful to reduce work.
The feedback and additional ideas from Bram's session -both from him and his tutor -were very useful, and led to some additional improvements and features to improve usability.This is an example where an expert in the area can provide useful ideas to the designer, in a way a normal user might not.This feedback resulted in many improvements, but three major ones were 'auto-repeat' of operations, a 'pause dwell' button in menus, and splitting some menus into two pages.

Auto-repeat
When doing repeating editing operations where the music plays back each time something changes (such as moving or transposing a note), users still needed to dwell on the bottom panel every time they wanted to change by one step.So, for example, if they wanted to change by 7 steps, this added up to a lot of effort.
The new 'auto-repeat' option makes such successive repeated actions happen automatically, ie there is no need for users to dwell somewhere to make it keep happening.They can just 'rest' (ie can look at most of the music screen without triggering anything), and listen to the music as it changes, only needing to take action (ie look at the 'STOP' button') when they want to finish.Users are then presented with a menu with options to start again, or go on or back one step.

Pause dwell
A standard feature of eyegaze AAC systems is a 'pause' button, which when triggered will de-activate the 'dwell' function temporarily.All buttons then do nothing if looked at, apart from the 'unpause' button so the user can turn dwell back on.Pausing dwell enables the user to look round the grid of buttons without triggering something to happen, and can also relax for a moment.This feature was already planned for E-Scape phase 2 development, but its importance was highlighted by observing users navigating a menu screen which gives musical feedback for each item.
One example is the menu to choose an instrument: as soon as an item is looked at, an excerpt of the music the user is working on is played, using that instrument.But if the eyegaze user has a short dwell time set, then there won't be time to hear all the music before the button activates.Thus the 'pause dwell' button was very much needed, and consequently brought well up the priority list.

Split menus
Some menus needed all the choices to be shown, so could not be 'customised', but still had too many items on them.They were therefore split into 'pages', with a button to swap between them.For example, the menu for choosing an instrument had 16 groups of sounds, and thus 18 items on screen, which made them too small to dwell on.This menu was therefore split into two banks which can be swapped between, thereby making it usable.Volume 4, No 2 (2015)

Future work
From observations of what users struggle with, and what they most commonly want to do, the aim in future is to increase creative possibilities while further reducing laboriousness, for example: (i) Designing better ways of selecting notes or positions within the score display, eg by dwelling in the approximate position, then zooming in and dwelling more accurately, with the option of a final menu to go backwards or forward a tiny step.(ii) Covering up irrelevant items on the score display to make it clearer what users are currently doing.For example, when selecting notes in one track, other tracks could be covered up and/or made smaller.(iii) Having a library of phrases or motifs, which can be auditioned then selected to insert and edit.This means that users will have musical material already available to be worked with, rather than having to start with single notes.(iv) Creating a 'compositional variations' generator, which lets users create new sets of phases from an original one, according to simple rules they can select or alter.(v) Increasing and refining the range of possibilities for performing using eyegaze.
Examples of performing developments under way, or planned in this area include: Making it easier to play through a melody from left to right, by ignoring any movement leftwards, until the user has played (looked at) the last note; Enabling eyegaze users themselves to edit buttons in the performing screen; Playing phrases by looking at them in any music track, not just in one track; Dwelling on a phrase to 'cue it' while another one is playing, so the phrase will play as soon as the previous one finishes.
Two further improvements to general eyegaze functionality were also suggested by Rolltalk 8 in Oslo.Firstly, a circular 'dwell timer' display should show in the centre of each button, rather than a moving bar.It would get smaller to show the passing of time, which helps users keep their eyes on a button and not 'slip off'.Secondly, if a user's gaze slips into the next button for just an instant then comes back again, then the dwell timer in the first button should resume from where it was (rather than starting again).The system is thus recognising that the user's intention was to stay on the first button.But if the user really does want to select the second button, and dwells on it for more than a certain (user-settable) time, then it will become active, and its dwell timer jumps into action (to include the time already spent there).

Conclusion
One of the main aims in designing this music system is to expand the range of students who can have an 'instrument' to work with music.Thus, a key focus has always been to support exploratory learning by enabling eyegaze users to operate it themselves.Several features in E-Scape eyegaze were designed to assist in this, and although there is plenty of scope for further refinement, two features in particular have proved their worth in trials so far.Firstly, user-guidance through certain procedures makes many editing and note-entry procedures simpler and less laborious to carry out.Although students still need to make decisions, the system understands the activity they are undertaking and presents a structured pathway through appropriate sets of choices for that activity.
Secondly, automatic auditory feedback lets students immediately hear the audible result of changes they have just made, and enables them to start working and composing almost immediately 'by ear', just by listening and making choices by 'trial and error'.Thus, a beginner without any music theory knowledge can explore and experiment with music, and achieve results and motivation right away.
Although dozens of further additions and improvements can be envisaged (and many are planned), the people who have worked with it, and observed its use so far feel that the goal of unaided use has been broadly achieved.Richard and Bram were able to start working and exploring very quickly, and a student in Ireland is now using the system to compose music.Although short of time, Bram managed to create some musical phrases to use -and was able to 'own' more of his radio show.Richard has now engaged a regular tutor and developed his interest in film music; he can load pieces and explore them.He is now planning to try more performing.
So the system can guide (via 'support scaffolding') beginner students to provide interest and achievement.Although the ideal is always for students to learn more, the fact that they can start and achieve something without such knowledge greatly helps motivation and enthusiasm.Thus, E-Scape eyegaze seems to be an effective tool for what might be termed 'emancipatory learning', where the disabled student is 'emancipated' from relying on a helper or teacher to do things for them.Students with any degree of physical disability can thus operate and make decisions themselves within a guided framework, without needing continual instructions to operate the system.The helper can join in, advise or help on occasion, but they do not need to continually do everything to operate the student's music system.However, the focus on students working and learning by doing in no way makes a teacher redundant.
A teacher is still needed, as most eyegaze students will have little experience of operating computers at anything more than a basic level, but more importantly won't have had the chance to 'play around' with music at an earlier age (eg plucking a guitar, hitting a chime bar, touching piano keys, clapping or singing) which non-disabled children usually enjoy.In addition, the act of composing is not intuitivethe system can certainly help to guide users through some musical procedures, but students still need to decide at a higher level what procedure they want to do.How do they create musical building blocks?How do they then develop a piece from those?Students will need the help of a teacher to guide and set up activities for them to explore, or work through.So, teachers are still very much wanted and needed, but they are able to focus on discussion of musical concepts or tasks with the student, and also plan didactic activities.If creating a piece of new music, the teacher is more able to take the role of guiding and suggesting, not 'telling' -there is less of a 'right answer' than if, say, doing a theory test.What the teacher does not need to do is operate the computer; the student is more in change of that themself; they are in control, and can (if they wish!) ignore the teacher's suggestions.This has been observed quite frequently, with the disabled student making a certain musical decision 'against advice', often with interesting results.The student is far more 'master of their own destiny' and has a demonstrable pride and ownership of their artistic creations.E-Scape eyegaze has the advantage that it is effectively a 'new instrument', with less tradition of 'conventional' teaching methods.So what the teacher can and -according to the learning theories adduced earlier -should do, is customise the explorative learning environment to engender optimum learning outcomes.Again, E-Scape eyegaze has two key features which facilitate this.
Firstly, a teacher is able to tailor the 'exploratory framework' used by students, by selecting or limiting the options available to them when they are working.This lets students explore and make their own choices, but makes them operate within an area that the teacher has planned.As students progress, the available options can be changed in various ways, eg to focus the activities on a different topic, or to expand the range of exploration possible.
Secondly, a teacher can customise the 'support scaffolding' provided to the student, by selectively or progressively removing some of the guidance features.This may involve removing some or all of the 'guided' options from menus leaving only the more basic operations.With less guidance on offer, students then have to think harder about what they are doing, and in what order -but conversely have more freedom.The teacher can also change other support options, for example to use more music terminology, or alter some procedures to move the student gradually away from 'trial by ear' to more considered choices from menu selections using musical knowledge gained.Thus, the system can hopefully enable students to progress in learning as they work or explore, while still achieving creative results.But the fact that the system can be used without help means there are opportunities for the student to be given homework, or practice something out of lesson time, or work with friends to make music.The student can play and contribute on more equal terms with their peers, or could even support them.As was stated at the beginning, this kind of interaction is a key part of both music making, and learning.So, it is to be hoped that the development of access to music-making via eye-control can open up possibilities for people presently living their lives on the margins of society.They need to be able to participate in music education in a way that they can truly take part, and in due course create and present their own music.Recently, there are signs that eye-control systems are being investigated by a wider range of people who want to try new ways of interacting with computers, so eyegaze users may end up guiding other people, and at the least working and playing with them as peers.
It is exciting to experience the musical expression of people whose voices are rarely heard, and might change the way we look at people who have hitherto had few possibilities to take part in music or express themselves creatively.

Figure 1 .
Figure 1.A typical eyegaze AAC screen to enter text

Figure 2 .
Figure 2. 'Add notes' menu in eyegaze mode (with header on right)

Figure 3 .
Figure 3. 'Add notes' menu, header at top, with just 4 items

Figure 6 .
Figure 6.Edit menu -if no notes are selected

Figure 10 .
Figure 10.Richard entering music for the first time via E-Scape Eyegaze

Figure 11
Figure 11.E-Scape score being performed live by Richard using eyegaze

Figure 12 .
Figure 12.Bram Harrison wants to write his own music links for his radio show

Figure13.
Figure13.Bram using eyegaze to operate E-Scape and write a melody

Figure 14 .
Figure 14.Film of Bram at an early stage, composing with E-Scape

Figure 15 .
Figure 15.Confirm or continue menu after stopping an operation

Figure 16 .
Figure 16.Choosing an instrument category in 'bank A'