Sounds such as co-workers chatting nearby or a dripping faucet help us maintain awareness of and respond to our surroundings. Without a tool that communicates ambient sounds in a non-auditory manner, maintaining this awareness is difficult for people who are deaf. We present an iterative investigation of peripheral, visual displays of ambient sounds. Our major contributions are: (1) a rich understanding of what ambient sounds are useful to people who are deaf, (2) a set of visual and functional requirements for a peripheral sound display, based on feedback from people who are deaf, (3) lab-based evaluations investigating the characteristics of four prototypes, and (4) a set of design guidelines for successful ambient audio displays, based on a comparison of four implemented prototypes and user feedback. Our work provides valuable information about the sound awareness needs of the deaf and can help to inform further design of such applications.