|
|
|
|
A recent development in the signing books field is that of sign language animations, or signing avatars. Instead of a human person signing a story, a 3D animation (puppet, manikin, robot) is used to render the sign language on screen. This development may well have as large an impact on the ‘signing books’ field, as the development of ‘synthesised speech’ had for talking books. The objectives of ‘synthesised sign language’ can be compared to those of ‘synthesised speech’: a computer converts printed text into spoken text in real time, and without human intervention. In the case of ‘synthesised sign language’ however, the problem is more complex. It is not only a matter of converting printed text into another medium, but also into another language. The software therefore has to be a combination of translation software, and conversion software. To complicate matters even more, most sign languages have been described only partially. In many countries, sign language dictionaries are only now being developed, while research of the syntax and pragmatics of many national sign languages is only just starting.
Nevertheless, 3D animations are now being developed for two applications: sign language training programmes and books in sign language in the USA, and for adding sign language to mainstream television broadcasts in the UK.
The USA system is being developed by Seamless Solutions Incorporated (http://www.seamless-solutions.com/html/sign_language/ . Several models have been developed (called ‘avatars’) including a signing frog.
|
|
|
|
|
8-1 |
8-2 |
8-3 |
Examples 8-1 to 8-3 shows a dictionary application, a boy avatar signing a story, and a frog avatar .
Early ’99, the dictionary and story programs were evaluated at the Florida School for the Deaf and Blind, with deaf students between 7 and 15 years, and with teachers of the deaf. According to the report on the website, the students all found the Signing Avatar characters to be very appealing. Facial expression seemed to provide an important part of this appeal. The children liked the signing frog best. Again according to the report on the website, none of the students had difficulty understanding the signing. The great majority preferred ASL to Signed English. Neither the students nor the teachers found the signing being robotic or unnatural in any way.
A recent project by Seamless Solutions (FACE, Facial Animation for Communication Enhancement) involved the inclusion of facial characteristics in an avatar in real time. An “Animation Engine” was developed to interactively control the facial expressions, in synchronisation with signing and optionally, voice.
A new project, starting September 1999 will be the development of an Automated PC-based Speech to Sign Language Interpreter.
The attraction of animated sign language is not (only) the fact that it enables frogs to sign. The animations are software generated, which allows for efficient storage and transmission, and flexibility in use. Once a sign specification is stored, it can be rendered on screen by different avatars, in all colours and sizes. The major driving force behind animated sign language, however, is computer generated translation from written text or speech, into sign language.
In the UK, ITC (Independent Television Commission) has commissioned Televirtual to develop Simon, the virtual signer (8-4) to translate printed text – television captions – into sign language, to solve some of the problems of adding sign language to television programmes.
(Copyright
Televirtual)
To quote from the report by the development company Televirtual (http://www.televirtual.com):
“The provision of live signers is extremely expensive for the broadcaster, and signing services also require additional studio facilities and personnel. In-vision signing (LP: a superimposed signer that all viewers see) is often not liked by the larger community, who find it distracting, and for this additional reason broadcasters are often reluctant to extend signing across their range of services. In the UK, growing recognition of the needs of the deaf community has resulted in legislation requiring broadcasters to provide signing to accompany an increasing proportion of their programmes. (..) It can be seen that a system which could generate deaf signing at the viewer end of the broadcast chain would resolve most of these issues. It would be significantly cheaper for the broadcasters to produce; it takes significantly less of the DTT digital multiplex capacity to transmit than would a second MPEG-2 video signal of a quarter-screen signer; it would automatically provide signing for all subtitled programmes; it would not force in-vision signing upon hearing viewers , and – well executed – it would meet the needs of the deaf community itself.”
“The
software consists of two basic modules: linguistic translation from printed
English into sign language, and virtual human animation. The animation software
allows Simon to sign in real-time. A dictionary of signed words enables the
system to look up the accompanying physical movement, facial expressions and
body positions, which are stored as motion-capture date (not images or video) an
a hard-disk. The system can call these physical moves (in any order) and
interpolate between them to create smooth, natural looking signing sequences.
The motion-capture data – which includes hand, face and body information –
is applied to a highly detailed 3D graphic model of a virtual human. This model
includes very realistic and accurate hand representations, developed within the
project. Natural skin textures are applied to the hands and face of the model to
create the maximum impression of subjective reality. (..)
Different
graphic models – younger or ‘cartoon’ in nature will be developed for
younger deaf TV audiences and can be switched in at the viewer’s choice. (..)
The motion-capture data is acquired from expert signers, using three forms of advanced motion-capture (8-5). Cyberglove VR datagloves to capture hand movements, an Ascension Motion Star wireless magnetic body suit is used to acquire upper torso, arm and head positions, and an optical Facetrak device to capture facial expression and lip positions.”(quoted from the Televirtual website report, October 1999).
8-5
(Copyright Televirtual)
At the moment, the Simon system can translate printed texts into Signed English, but not into BSL. A 1999 postscript on the Televirtual website describes the future developments:
“Early in 1999 there has been debate about the date upon which it is reasonable for digital terrestrial television operators to begin the broadcasting of signing services. Naturally this will depend upon the availability of set-top boxes carrying the necessary technology and on the agreement of any international standards for the transmission of signals. It is also expected that the deaf community will prefer the use of British Sign Language (BSL) rather than Sign-Supported English (SSE), which is grammatically close to English.
To enable services to begin as early as possible, and to minimise the complexity of processing required in the set-top box, the companies developing Simon have recently proposed an evolutionary approach to its implementation. In the long-term this will lead to the same desirable system described above and is thus a ‘future proof’ approach – an important consideration because the digital television standards being defined today are expected to be in use for decades to come.
In step 1 of the evolutionary approach, the body movements and facial expressions of a human signer are captured as he or she signs an interpretation of a television programme using BSL. The low quantity of compressed motion-capture data representing the movements is transmitted with the digital television signal to the set-top box where a virtual Simon mimics the moves exactly.
When a television programme is available with subtitles (but the broadcaster has chosen not to sign it with a human signer), step 2 recognises that the linguistic translator system can be employed in the studio, to produce an SSE broadcast data stream automatically. Thus Simon automatically produces SSE. Although this may be less desirable than BSL, it does mean that some form of signing will be available for all programmes which are subtitled (ie the majority of terrestrial programmes).
Step 3 in the evolution represents a time in the future when subtitles-to-BSL linguistic conversion can be performed reliably in the studio. Although BSL grammar is complex, advances in linguistic processing are rapid, so implementation of this stage will only be a few years away. With this, all programmes which carry subtitles can be automatically signed in BSL.
A further development which will also be included in the evolutionary progress is a ‘tagged’ version of the subtitle text, where hidden ‘tags’ are used indicate which words are more important that others in the context of the phrase being subtitled. This can not only allow for the variable-rate signing discussed above, but can also allow the display of subtitles which have been subjected to a greater or lesser degree of elision.” (Televirtual, 1999)
Visicast is a EU project that grew out of the "Simon project". It started in January 2000. More information can be found at the project's website: http://www.visicast.co.uk.
MultiReader is also a EU project, that will start in January 2001. Among other things, the MultiReader project will investigate the use of sign language animations, to increase the accessibility of printed and multimedia documents for Deaf readers. For more information about this aspect of the MultiReader project, contact Liesbeth Pyfers.
