Lucent Technologies INTUITY CONVERSANT System Version 6.0 Guide

							Testing and Using a Voice Response Application Design 
Page 5-1 Overview 
5
INTUITY CONVERSANT System Version 6.0
Application Design Guidelines  
585-310-670  Issue 1.0
December 1996
5
5Testing and Using a Voice Response
Application Design
Overview
This c hap ter p rovid es g uid elines to test and  use a voic e resp onse ap p lic ation 
design.
Purpose
The p urp ose of this c hap ter is to p rovid e the information needed  to test and  use 
your voice resp onse ap p lication d esig n.  Testing  can p rovid e valuab le 
information that c an b e used  to imp rove the ap p lic ation d esig n, and  make it more 
useful to your callers. 
						

							INTUITY CONVERSANT System Version 6.0
Application Design Guidelines  
585-310-670  Issue 1.0
December 1996
Testing and Using a Voice Response Application Design 
Page 5-2 Testing the Application 
5
Testing the Application
There are several step s to g etting  your ap p lic ation in servic e.  No matter how 
c arefully you plan your app lic ation, d o not exp ec t your first d esig n to b e p erfec t.  
The most suc c essful ap plic ation d esig ners arrive at their d esig ns throug h 
rep eated  testing.  The key to rep eated  testing  is to test your app lic ation a numb er 
of times, and  use the d ata c ollec ted  d uring a test to refine the ap p lic ation d esign 
b efore testing  ag ain.  Rep eated  d esig n and  testing  help s d etermine if your 
ap plic ation follows the g uid elines in Chap ter 4, ‘‘Desig ning  a Voic e Resp onse 
Ap p lication.’’
Rep eated  d esign testing  involves a c yc le of ac tivities you d o until your 
ap plic ation d esig n meets your usability g oals.  This sec tion desc rib es the step s 
in this c yc le of ac tivities.
Plan a Preliminary Application Design
Planning your application design is described in Chap ter 3, ‘‘Planning a Voic e 
Resp onse Ap p lic ation.’’
Set Usability Goals
Usab ility g oals allow you to measure whether or not your ap p lic ation will serve 
your callers well.  Different services will require different usability goals.  
Rememb er to make your g oals measurab le, so that your testing  will g ive d ata to 
help  you d ec id e if the ap p lic ation meets the g oals.  Here are some examp les of 
usability goals:
nAt least 95% of test c allers are ab le to c omp lete the transac tion in less 
than 45 sec ond s with no errors.
nAt least 90% of test c allers are ab le to rec over from all errors, thereb y 
avoid ing  transfer to an attend ant.
nAt least 98% of test c allers desc rib e the app lic ation’s ease of use as very 
g ood  or exc ellent.
Test the Preliminary Application
When you have a func tioning  app lic ation, it is ready to b e tested .  Use a small 
g roup  of rep resentative c allers to ac c omp lish a set of typ ic al tasks.  It is 
imp ortant to make sure that the p eop le tested  are rep resentative of the c alling  
p opulation.  If your ap p lic ation is d esig ned for c ustomers, d o not test it with your 
c olleagues as they may have knowled g e about the app lic ation that c allers d o 
not.  If you know that a larg e p rop ortion of your c allers is over the ag e of 65, use 
test c allers that are over the ag e of 65. 
						

							INTUITY CONVERSANT System Version 6.0
Application Design Guidelines  
585-310-670  Issue 1.0
December 1996
Testing and Using a Voice Response Application Design 
Page 5-3 Testing the Application 
5
Collect Data During the Test
You want to c ollec t two typ es of d ata in your testing :
nPerformanc e measurements — Two imp ortant p erformanc e measures are 
time (how muc h time c allers used  to c omp lete tasks) and  errors (the 
numb er and  typ es of errors mad e, and  where in the ap p lic ation they 
occurred).
nOp inion feed b ac k — Imp ortant op inion measures inc lud e c aller 
jud g ments ab out how understand ab le the p romp ts are, the ease of eac h 
task, and  the overall level of satisfac tion with the ap p lic ation.
It is very imp ortant to c ollec t 
both typ es of d ata, sinc e eac h g ives you d ifferent 
information ab out the ap plic ation.  The d ata will b e used  to refine and  mod ify the 
ap plic ation d esig n to meet your usability g oals.
Usability Testing
Testing  the user interac tion with the ap p lic ation using typ ic al c allers is c alled  
usab ility testing .  This testing  will help  you d etermine how easily the intend ed  
aud ienc e c an use the ap p lic ation to p erform sp ec ific  tasks.  You c an adap t 
usab ility testing  methods to the time, resourc es, and  fac ilities availab le.
Testing in a Controlled Environment
The more struc tured  usab ility testing  method s involve b ring ing  rep resentative 
c allers into a c ontrolled, laboratory environment.  Give c allers a list of tasks to 
p erform b y c alling  into the ap p lic ation.  Ob serve c allers interac ting  with the 
ap plic ation, vid eotap e or rec ord  them if p ossib le, so you c an refer to the tap es 
later and  take p erformanc e measurements (suc h as c ounting  the numb er of 
errors and  measuring  the time taken to c omp lete the assig ned  tasks).  After the 
c allers have used the preliminary application, use a questionnaire to gather their 
p referenc es and  reac tions to the ap p lic ation.  See Fi g u re  5 - 1
 for a samp le 
q uestionnaire.
You may also want to g ather feed b ac k b y sp eaking p ersonally with eac h c aller.  
Be sure to ask eac h an id entic al set of q uestions. 
						

							INTUITY CONVERSANT System Version 6.0
Application Design Guidelines  
585-310-670  Issue 1.0
December 1996
Testing and Using a Voice Response Application Design 
Page 5-4 Testing the Application 
5
Figure 5-1. Sample QuestionnaireName (op tional):                                                                                                    
Gend er:   Male   Female
What is your native lang uag e?                                                                                
Overall, I rate the voic e d iffic ult to use easy to use
response servic e as  1       2        3       4       5
The rec ognition ac c urac y is p oor exc ellent
 1       2        3       4       5 
The p romp ts p resent too little information too muc h information
 1       2        3       4       5
Did  you exp erienc e any p rob lems using  the voic e resp onse servic e?  If yes, p lease 
d esc rib e.                                                                                                               
                                                                                                                             
What d o you like b est ab out the voic e resp onse servic e?                                          
                                                                                                                             
What d o like least ab out the voic e resp onse servic e?                                                
                                                                                                                              
						

							INTUITY CONVERSANT System Version 6.0
Application Design Guidelines  
585-310-670  Issue 1.0
December 1996
Testing and Using a Voice Response Application Design 
Page 5-5 Testing the Application 
5
Testing in an Uncontrolled Environment
The less struc tured  usab ility testing method s involve similar tasks and  
measurements, b ut d o not req uire that rep resentative c allers b e b roug ht into a 
c ontrolled  environment.  You c an g ive the c allers a list of tasks to p erform b y 
c alling  into the ap p lic ation, b ut allow them to c all from their homes or offic es.  
Measure p erformanc e b y listening  in to the transac tions as they happ en, or 
rec ord  them for later measurement.  Use a q uestionnaire to g ather their feed b ac k 
or c all them and  interview them instead.
When you observe c allers interac ting  with the ap p lic ation d uring usability testing, 
p ay attention to everything  that oc c urs d uring  the c all.  You c an ob serve b y 
sitting with c allers as they c all in to the app lic ation, or b y listening  on a telep hone 
line b y using a c hannel monitor ap p lic ation availab le from your Luc ent 
rep resentative.
Observe Callers During Testing
This sec tion d esc rib es the typ es of ob servations to make with c allers.
When Callers Become Confused
You c an assume a c aller is c onfused  if you hear extraneous sp eec h (suc h as 
“ What?”  or “ I d o not understand ” ), resp onse d elays, or silenc e instead  of a 
response.  Often, you will know c allers are c onfused  b y ob serving  their fac ial 
exp ressions or b y listening  to the tone of their voic e.
Information you g ather from ob serving  c allers interac ting  with the ap p lic ation c an 
help  you d ec id e how to modify the ap p lic ation to inc rease its usab ility.  For 
examp le, you mig ht d isc over that c allers sound  unsure or d elay resp ond ing  at a 
p romp t, “ What is your c ustomer numb er?”   To remed y the p romp t, you c ould  
p romp t instead  with, “ Your eig ht-d ig it c ustomer numb er is p rinted  on the b ac k of 
your c atalog .  Do you have a c ustomer numb er?”   Callers who say no c an b e 
transferred  to an attend ant, while c allers who say yes c an b e p romp ted  next with 
“ What is your c ustomer numb er?”
When Callers Say Non-Key Words
For example, you mig ht ob serve that several c allers g ave non-key word  
responses to the q uestion “ What d epartment d o you want?”   In this c ase, you 
mig ht want to list the d ep artments after you have p romp ted  onc e and  no key 
word was recognized.  The application could say “What department do you 
want?  Please say sales, servic e, c red it, or offic e.” 
						

							INTUITY CONVERSANT System Version 6.0
Application Design Guidelines  
585-310-670  Issue 1.0
December 1996
Testing and Using a Voice Response Application Design 
Page 5-6 Testing the Application 
5
Recognition Errors
If your ap p lic ation uses sp eec h rec og nition, listen to what the c aller says, and  
what the rec ognizer d etermines that the c aller says.  Chang e your ap plic ation, if 
nec essary, so that c allers g ive you the resp onses you want at the rig ht time.  For 
examp le, if you d o not want c allers to barg e-in during  the p rompt b ut they tend  to 
b arg e-in, you mig ht announc e at the b eginning  of the transac tion that c allers 
should  wait until the end of the p romp t b efore respond ing , or use the word “ now”  
in your p romp ts when the system is read y to ac c ep t inp ut.
If you are using  FlexWord  speec h rec og nition, sub stitution errors (that is, where 
the rec og nizer mistakes one word  for another) are extremely imp ortant.  If many 
c allers often mistake one op tion for another, c hang e one of the menu op tions so 
the rec og nizer d oes not c onfuse the two op tions.  Expec t to c hang e to your 
voc ab ulary b efore you are finished  with your p reliminary ap p lic ation d esig n.
System Performance
Pay attention to how the system op erates.  Is the resp onse time too slow?  If the 
server stop s working  or other system p rob lems oc c ur, d oes the ap plic ation 
hand le the c all in a g rac eful manner?
Refine the Preliminary Application
Based on Usability Goals
After c ollec ting performance measurements and caller feedbac k, study the 
information to see if your usab ility goals are ac c omp lished .  If not, let the d ata 
show you how to refine your ap p lic ation.  If you ob serve c aller p rob lems with the 
ap plic ation, use both p erformanc e and  feed b ac k d ata to d etermine the p rob lem 
areas in the app lic ation.  Refine your ap p lic ation b ased  on your find ing s.
Test the Application Again
Next, test your modified  ap p lic ation with another g roup  of rep resentative c allers.  
Use the same test and  g ather the same d ata.  Comp are the sec ond test results to 
the first, to d etermine if the ap p lic ation refinements had  a p ositive effec t on the 
p erformanc e measurements. 
						

							INTUITY CONVERSANT System Version 6.0
Application Design Guidelines  
585-310-670  Issue 1.0
December 1996
Testing and Using a Voice Response Application Design 
Page 5-7 Testing the Application 
5
Continue Testing Until the Usability Goals
are Accomplished
If your usab ility goals are not yet ac c omp lished , refine the ap p lic ation ag ain, 
b ased on the sec ond  test results.  If you make refinements, b e sure to test the 
ap plic ation ag ain with rep resentative c allers to d etermine if the c hang es move 
you toward s ac c omp lishing  your usab ility g oals.
When you are c onfid ent in your ap p lic ation, test the entire ap p lic ation.  This 
c omp lete test may inc lud e ac c essing  information from a datab ase, host 
c omp uter interfac es, in-servic e and  out-of-servic e hours, and  other asp ec ts of 
the app lic ation.  You c ould  p ut your ap p lic ation in servic e for a sub set of c allers 
to use for specific transactions.  Collec t both performance and opinion data.
Use the Tested and Refined Application
in Service
When your usab ility g oals are ac c omp lished , p lac e the ap plic ation in servic e for 
all of your intend ed  c allers.  You may want to use a sing le c hannel at a time, to 
see how the system performs und er load .  It is imp ortant to ob serve the 
ac c eptanc e of your ap p lic ation, and  how suc c essful your c allers are using  it.
It is time c onsuming  to listen to c allers interac ting with your ap p lic ation, b ut you 
will c ollec t valuab le d ata ab out your c allers and  the ap p lic ation.  This information 
c ould  influenc e your d ec isions ab out the p resent ap p lic ation, or ones that you 
will write later.  For examp le, you may have assumed  that only a few of your 
c allers would  have non-US Eng lish ac c ents, when in fac t many of them do.  In 
this c ase, you p rob ab ly d o not want to write a sp eec h-rec og nition-only 
ap plic ation.  Rather, you would  enc ourag e c allers to use touc h-tone or d ial p ulse 
input.
Even when your app lic ation is availab le and  serving  your c ustomers, it is a good  
id ea to c ontinue ob serving  the ap p lic ation and  system p erformanc e.  Listen to 
rand om c alls to d etermine how c allers are interac ting  with the ap p lic ation.  Ask 
your c ustomers for their op inions on how to imp rove the voic e resp onse servic es 
you offer. 
						

							INTUITY CONVERSANT System Version 6.0
Application Design Guidelines  
585-310-670  Issue 1.0
December 1996
Testing and Using a Voice Response Application Design 
Page 5-8 Testing the Application 
5 
						

							Index 
Page IN-1  
INTUITY CONVERSANT System Version 6.0
Application Design Guidelines  
585-310-670   Issue 1.0
December 1996
IN
Index
A
ac curac y
dial pulse recognition,2-3, 4-36
FlexWord speech rec ognition,2-9imp rovement
dial pulse recognition,2-3, 4-36
speech recognition,2-10
measurement,2-10Text-to-Speech,2-12touc h-tone recog nition,2-3
WholeWord  sp eec h rec og nition,2-6announc ements
confirmation
,4-7d efinition,1-5design guidelines,4-7
feedback,4-7p rerec ord ed speech,4-46Text-to-Speech,4-45
application
bilingual,2-5, 4-40design guidelines,4-40
enhanc ed  b asic  sp eec h,4-44
language g ate,4-40
d efinition,1-2, 1-5
design,4-1principles,4-2
testing,5-2
diagrams,3-5
errors,4-23, 5-6multiling ual,4-45p ac e guidelines,4-21
p ut in servic e,5-2testing,5-2, 5-6touc h-tone and  sp eec h rec og nition
modify ap plication
,4-32
separate application,4-27
sing le ap plication,4-30
voic e resp onse ap p lic ations,1-2
B
barge-in,4-34
c aller errors,4-23d efinition,1-5design guidelines,4-34
dial pulse recognition,4-36d iscourag ing,4-35enc ourag ing,4-35
FlexWord speech rec ognition,2-9prompt wording,4-34Text-to-Speech,2-12 
						

							INTUITY CONVERSANT System Version 6.0
Application Design Guidelines  
585-310-670    Issue 1.0
December 1996
Index 
Page IN-2  
barge-in, (Con’t)WholeWord  sp eec h rec og nition,2-6
with error messages,4-34yes/no q uestions,4-19b ilingual ap plications,2-5, 4-40
design guidelines,4-40enhanc ed b asic sp eech,4-44language gate,4-40
touc h-tone recog nition,4-40WholeWord  sp eec h rec og nition,2-5, 4-40
C
caller
attrib utes,4-2
confusion,5-5d efinition,1-5feedback,5-3, 5-5
observations,5-5short-term memory,4-3c hec ksums,4-16
c hoosing word s for word lists, see word  c hoice
class detectors
d efinition
,4-47
examp le,4-48more information,4-48c ollecting  d ata, see d ata collection
Confirm ac tion step
,4-8see also confirmation
confirmation
announc ements
,4-7digit input,4-17digit sequences,4-17
sing le d igits,4-17
when need ed,4-8when not need ed,4-10connected digit recognition
d efinition
,1-5entry,4-15custom
grammars
,4-16sp eech
definition,1-6
D
data collection
c aller opinion
,5-3performance,5-3d ata interfac e p rocess (DIP),4-16, 4-48, 4-49
database
lookup s,4-16