Public version of the slideshow I used during my presentation about adaptive music in video games and other interactive media at #UXMonday event, organized by http://asociaceux.cz in Prague, March 2, 2015.
3. @adam_sporka Adaptive Music in Games UX Monday 3
Purpose of Sound in UIs
• Interaction modality (input and output)
• Alerts
• Widget sounds
• Presentation of data
• Mood setters
• Sound branding
4. @adam_sporka Adaptive Music in Games UX Monday 4
Describing Sound and Music
Signal
Sound
field
Tone
/ Stroke
Sequence
Concord
Composition
Environment
Reverberation
Vibration
Distribution
Spectrum
Volume
Periodicity
Pitch
Length
Volume Accord
Interval
Harmony
Rhythm
Tempo
Phrase
Loop
Soundtrack
Intention
Abstraction level
Playlist
Genre
science art
5. @adam_sporka Adaptive Music in Games UX Monday 5
Fuzzy Terminology
• What it does:
– Adaptive music
– Responsive music
– Reactive music
– Interactive music
• How it is done:
– Procedural music
– Algorithmic music
6. @adam_sporka Adaptive Music in Games UX Monday 6
Science: Data Sonification
• Synthesis of sound
controlled by
input data
• ICAD 2004 contest
– http://www.icad.org/websiteV
2.0/Conferences/ICAD2004/c
oncert.htm
8. @adam_sporka Adaptive Music in Games UX Monday 8
Sonification of Rowing sounds
• https://www.youtube.com/watch?v=YxWNw9fIizk
• Audible comparison of trainee’s sound and
reference
• Audible feedback while people is rowing
9. @adam_sporka Adaptive Music in Games UX Monday 9
Outline
• Purpose of music in games
• Static music soundtracks
– Music design
• Dynamic music soundtracks
– Design, production, …
• Music production technology
– How the actual music is made
11. @adam_sporka Adaptive Music in Games UX Monday 11
Purpose of Music
• Tradition
– Present in most of the released titles
– Why not in yours?
• Affirmation of the genre
– “Yes, this is an 8bit retro.”
– “Indeed, cowboys in spaceships.”
– “Yay, manga.”
12. @adam_sporka Adaptive Music in Games UX Monday 12
Purpose of Music
• Presentation of emotion
– Easygoing, happy-go-lucky, …
– Depressing, sad, …
• Setting the expectations
– Level difficulty
• Status monitoring
– Changes of music over time indicate changes in
game
13. @adam_sporka Adaptive Music in Games UX Monday 13
Purpose of Music
• Essence of the gameplay
– Dance Dance Revolution (1998)
– Guitar Hero (2005)
– Pugs Luv Beats (2011)
• http://www.youtube.com/watch?v=V0i18_--8Yc
– Not covered by this talk
14. @adam_sporka Adaptive Music in Games UX Monday 14
Game Music Components
• Theme Music
– Often synchronized with visual events on the
screen
– Cinematic experience
• Underscore
– In-game music
– User interface background
15. @adam_sporka Adaptive Music in Games UX Monday 15
Game Music Components
• Stingers
– Specific music triggered by an event
– Transition music
– Level Notification, “Game Over” Screen, “Game Won”
Screen
– Fallout New Vegas:
Pulling out / holstering Magnum
• http://youtu.be/sWLpSpZR6J4?t=41s
16. @adam_sporka Adaptive Music in Games UX Monday 16
Soundtrack
• Diegetic sounds
– Sounds in story space
– On-screen or off-screen
– Voices of characters, sounds of objects, music coming
from objects
• Non-diegetic sounds
– Not implied to be present in the action
– Narrator, sounds for dramatic effect, underscore
17. @adam_sporka Adaptive Music in Games UX Monday 17
Sound in Games
– Voices, diegetic sound effects
– Diegetic ambience
– Non-diegetic ambience
• “Sound design pads”
– Underscore
ABSTRACT
CONCRETE
18. @adam_sporka Adaptive Music in Games UX Monday 18
Game Music Characteristics
• Theme music is feature
– Recognizable theme
– Elaborate
– Genre announcer
– Everyone will hear this
• Underscore is background
– Mood setter
– Does not distract
– Can be listened to throughout the gameplay
19. @adam_sporka Adaptive Music in Games UX Monday 19
Game Music Characteristics
• Silence
– “Dosage” of music
• Used to emphasize music
– Keep the ambient sounds present
– Complete silence “something wrong”
• “Are my speakers on? Will I hear anything at all when
I’ll really have to?”
20. @adam_sporka Adaptive Music in Games UX Monday 20
Game Music Interactivity
• Static soundtrack
– Predefined and unchanging
– Loops
– Assigned to specific screens / levels
• Dynamic (adaptive) soundtrack
– More complex control of the music playback
– Engine “aware” of the state of the game
22. @adam_sporka Adaptive Music in Games UX Monday 22
Music Design
• Genre?
• Situations in game?
• Mechanics of music?
23.
24.
25.
26.
27.
28. @adam_sporka Adaptive Music in Games UX Monday 28
Music Design
• Multiple contexts –
typical for open-world RPGs
– Landscape exploration
– Combat
– Dialogs
– …
• Considerations of importance of music
scenes
• Is “combat” always more important than “location”?
31. @adam_sporka Adaptive Music in Games UX Monday 31
Static Soundtrack
• Early and simpler games
• Level-based games
• Pengon (1984, Atari 800XL)
– http://www.youtube.com/watch?v=MDhLxRLvwvY
– One music loop
– Game over stinger
32. @adam_sporka Adaptive Music in Games UX Monday 32
Static Soundtrack
• Supaplex (1991, Digital Integration, MS-DOS)
– Played throughout the game
– http://www.youtube.com/watch?v=yknubWX2KYI
• Goonies (1985, Datasoft, Atari 800XL)
– Two loops, one used for title, then alternating with each
level
– https://www.youtube.com/watch?v=FKh5b8jcwLk
• Jazz Jackrabbit (1994)
– Separate tracks for levels
– http://youtu.be/b16upFloYak?t=29s
34. @adam_sporka Adaptive Music in Games UX Monday 34
Case Study: Nimble Quest
• 2013; NimbleBit; iOS, Android
• Arcade / RPG
• Early 2000s pixel art graphics
35. @adam_sporka Adaptive Music in Games UX Monday 35
Case Study: Nimble Quest
• Music by Whitaker Blackall
• Contents:
– Title / menu
– Loop per level
• Genre:
– Contemporary chiptune
– http://wtrebella.bandcamp.com/album/nimble-quest
36. @adam_sporka Adaptive Music in Games UX Monday 36
Case Study: Nimble Quest
• Typical length of a loop: 2’
• Unobtrusive
• Reflects the increasing difficulty of the levels
– L1 http://www.youtube.com/watch?v=6mh7qJgnpJI
– L2 https://www.youtube.com/watch?v=9HaOTX-m0SU
– L4 http://www.youtube.com/watch?v=YKIDg8VkF0A
37. @adam_sporka Adaptive Music in Games UX Monday 37
Case Study: Nimble Quest
• Sounds good on small speakers
– Chiptune sounds
– Lot of square waves / triangles
• Better estimation of pitch by the human
low- as well as high-pitched tones
• General remark:
• PC gamers often don’t spend money on a high-quality
speakers
38. @adam_sporka Adaptive Music in Games UX Monday 38
Case Study: Quido (2013)
• Experimental arcade game
– Developed at DCGI/CTU
• Showcasing non-standard haptic modalities
– Presented on Designblok 2013
– http://ulab.cz/naviterier/quido/
• Genre:
– Contemporary chiptune
– https://quido.bandcamp.com/
39. @adam_sporka Adaptive Music in Games UX Monday 39
Quido: Technology used
• Reaper
• VSTs
– SuperWave (leads / chords / bass)
– EXD-80 (beats)
– Some reverb, some volume compressor
40. @adam_sporka Adaptive Music in Games UX Monday 40
Quido: Music
• Theme music
– Exposition of the main theme
– Happy, easygoing
– Not used in the end
• Explicit request from the exhibition organizer: Silence
between the games
41. @adam_sporka Adaptive Music in Games UX Monday 41
Quido: Music
• Level 1: Theme Park
– A player will be likely to be receiving instructions
from a fellow player
– Expected playtime: 3—4 minutes
– Not “dense”
– Diverse in terms of the structure
– Performance of the main theme
42. @adam_sporka Adaptive Music in Games UX Monday 42
Quido: Music
• Level 2: City
– Variation of the theme.
– Structure: [A B A Bbr Avoid]
– Expected playtime: 2 minutes
– Continuous music
43. @adam_sporka Adaptive Music in Games UX Monday 43
Quido: Music
• Level 3: Sky (Heaven)
– Variation of the theme. Barroque.
– Continuous music
– Expected playtime: 2 minutes
44. @adam_sporka Adaptive Music in Games UX Monday 44
Quido: Music
• Level completed
– Joyous jingle
• Advancing to the next level
– Expectation
48. @adam_sporka Adaptive Music in Games UX Monday 48
Adaptive Music Styles
• Resequencing
– Selecting what to play next
• Reorchestration
– Selecting which instruments to play
• Modulation
– Modifying the timbres of notes
– Related to Data Sonification
49. @adam_sporka Adaptive Music in Games UX Monday 49
The Space Game
• Game loops:
– Menu / Idle
– The enemy is approaching
– The enemy opens fire
• <demo>
– http://www.candystand.com/play/the-space-game
50. @adam_sporka Adaptive Music in Games UX Monday 50
Reorchestration
• Multiple tracks
– Turned on/off
– Faded in and out
• Synchronous
<Demo: honza_test_02.sqc>
52. @adam_sporka Adaptive Music in Games UX Monday 52
Reorchestration
• Fallout: New Vegas
– 2010, Obsidian Entertainment, Bethesda
Softworks
– Multiple renderings of the same track
• “low”, “mid”, “high”
– <demo; NewCaliforniaRepublic>
54. @adam_sporka Adaptive Music in Games UX Monday 54
Resequencing
• Slices chosen according to game state
• Precise timing of transitions
• <Demo: chiptune.sqc>
55. @adam_sporka Adaptive Music in Games UX Monday 55
Resequencing and Reorchestration
• Example: Monkey Island 2
– 1991, LucasArts
– http://www.youtube.com/watch?v=Nsc5nTrCzm
w#t=322
– iMUSE system
• Parallel tracks for different locations
• Transition patterns to return back
56. @adam_sporka Adaptive Music in Games UX Monday 56
Modulation
• Analog variables in the game
– Health
– Completion
– …
• Parameters of sound synthesis
– Volume
– Low-pass filter cut-off
– …
57. @adam_sporka Adaptive Music in Games UX Monday 57
Modulation of Synthesis
• Mapping
game state sound synthesis
• <Demo: Angel choir gone wrong>
61. @adam_sporka Adaptive Music in Games UX Monday 61
Music Composition
• Movies
– Music traditionally created aside from the entire
project
– Story is defined
– Timing is known
• “Horizontal” production
62. @adam_sporka Adaptive Music in Games UX Monday 62
Music Composition
• Computer Games
– Games are not (very) predictable
– “Horizontal” production
• Advantages
– Music per se works well (classic composition techniques)
– No limitations on instrumentations etc.
• Problems
– Repetition
– Abrupt changes
– Failure to respond
63. @adam_sporka Adaptive Music in Games UX Monday 63
Music Composition
• Computer Games
– “Vertical” production
• Advantages
– Good support of the moment
• Problems
– Difficult to express tensions, developments
– Music forms get disrupted
– Discontiuity
– Can be too descriptive
64. @adam_sporka Adaptive Music in Games UX Monday 64
Music Production
• Input
– Artistic intent
– Gameplay description
• List of supported locations
• List of supported situations
– Available technology
• Platform, available storage, …
– Legal considerations
• Licensing of the game
65. @adam_sporka Adaptive Music in Games UX Monday 65
Music Production
• Orchestration
– What instruments are used
• Structure
– Duration, planning
• Production
– Composing
• Mixdown / Mastering
– “Locking in” the music
– Optimizing the volumes, polishing
66. @adam_sporka Adaptive Music in Games UX Monday 66
Music Production
• Practical problems
– Multiple stakeholders
– Tons of files (as with any content development)
– Hiring composers
– Finding instrumentalists