1 00:00:00,000 --> 00:00:05,368 [SQUEAKING] [RUSTLING] [CLICKING] 2 00:00:11,407 --> 00:00:12,490 CHARLENE: Hello, everyone. 3 00:00:12,490 --> 00:00:13,443 My name is Charlene. 4 00:00:13,443 --> 00:00:14,610 THERESE: My name is Therese. 5 00:00:14,610 --> 00:00:16,735 CHARLENE: And we're here to present INMUSE, a music 6 00:00:16,735 --> 00:00:18,330 capturing game. 7 00:00:18,330 --> 00:00:21,360 So you're going for a walk, and you put on Bose Air glasses 8 00:00:21,360 --> 00:00:24,235 and turn on your phone, and open up the INMUSE app. 9 00:00:24,235 --> 00:00:25,860 As you walk with your Bose Air glasses, 10 00:00:25,860 --> 00:00:28,650 you will hear a melody to your right. 11 00:00:28,650 --> 00:00:31,110 You turn towards the melody to get a better hearing of it. 12 00:00:31,110 --> 00:00:33,630 And if you like it, you double-tap on your glasses 13 00:00:33,630 --> 00:00:36,050 to capture the melody. 14 00:00:36,050 --> 00:00:38,300 If you feel the melody is not something that you like, 15 00:00:38,300 --> 00:00:39,900 you can shake your head and let it go. 16 00:00:39,900 --> 00:00:41,550 And by the end of the walk, you will 17 00:00:41,550 --> 00:00:44,220 have to capture a series of melodies 18 00:00:44,220 --> 00:00:47,910 and compose your own little soundtrack that you experienced 19 00:00:47,910 --> 00:00:49,470 during your walk. 20 00:00:49,470 --> 00:00:53,070 So here's a little video of a [INAUDIBLE].. 21 00:00:53,070 --> 00:00:54,000 [LAUGHTER] 22 00:00:54,000 --> 00:00:57,177 AUDIENCE: [INAUDIBLE] 23 00:00:57,177 --> 00:00:58,010 AUDIENCE: All right. 24 00:00:58,010 --> 00:00:58,790 [LAUGHTER] 25 00:00:58,790 --> 00:01:02,276 CHARLENE: [INAUDIBLE] 26 00:01:02,276 --> 00:01:05,748 [STEADY BEAT PLAYING] 27 00:01:14,676 --> 00:01:18,148 [MUSIC PLAYING] 28 00:02:00,308 --> 00:02:01,870 AUDIENCE: Yeah, that was great. 29 00:02:01,870 --> 00:02:04,640 CHARLENE: Great, so now we can do the demo. 30 00:02:04,640 --> 00:02:07,690 So the demo is set up where if you take 10 steps 31 00:02:07,690 --> 00:02:09,820 and it's in a very high sensitivity for the step 32 00:02:09,820 --> 00:02:14,140 counter because we're doing this indoor to trigger a sound, 33 00:02:14,140 --> 00:02:16,480 and it's ready for whoever want to try it out. 34 00:02:16,480 --> 00:02:26,020 AUDIENCE: [INAUDIBLE] 35 00:02:26,020 --> 00:02:27,640 CHARLENE: We have a very simple UI 36 00:02:27,640 --> 00:02:30,670 to indicate how many melodies you encounter, 37 00:02:30,670 --> 00:02:32,470 how many steps you took, and then 38 00:02:32,470 --> 00:02:33,970 how many melodies captured. 39 00:02:33,970 --> 00:02:36,610 The whole experience is aimed to be phones down, 40 00:02:36,610 --> 00:02:38,260 heads up, and just for you to walk. 41 00:02:38,260 --> 00:02:40,910 You don't really want to be distracted, first of all. 42 00:02:40,910 --> 00:02:43,090 It's really just to enhance and to kind of capture 43 00:02:43,090 --> 00:02:44,230 the moment as you walk. 44 00:02:44,230 --> 00:02:47,611 So let me turn and get on the app. 45 00:02:47,611 --> 00:02:48,667 There you go. 46 00:02:48,667 --> 00:02:51,250 You should connect, and then you can just slip the phone right 47 00:02:51,250 --> 00:02:51,875 in your pocket. 48 00:02:56,550 --> 00:02:58,730 AUDIENCE: So this is not head direction dependent, 49 00:02:58,730 --> 00:03:00,510 meaning one direction or the other-- 50 00:03:00,510 --> 00:03:01,760 CHARLENE: That doesn't matter. 51 00:03:01,760 --> 00:03:02,240 AUDIENCE: Doesn't matter. 52 00:03:02,240 --> 00:03:03,282 CHARLENE: Doesn't matter. 53 00:03:03,282 --> 00:03:05,285 So the audio is spatialized. 54 00:03:05,285 --> 00:03:07,910 We experiment a couple of times with spatialized music in order 55 00:03:07,910 --> 00:03:11,390 for the user can distinguish between the new sound 56 00:03:11,390 --> 00:03:13,580 and the sound that they captured. 57 00:03:13,580 --> 00:03:16,340 But during our testing, we did find out it 58 00:03:16,340 --> 00:03:18,465 does make the walking experience more uncomfortable 59 00:03:18,465 --> 00:03:20,673 because they would like to face the direction they're 60 00:03:20,673 --> 00:03:23,030 walking instead of keep turning their head around. 61 00:03:23,030 --> 00:03:27,380 So spatialization in the moving-- 62 00:03:27,380 --> 00:03:30,410 when they're moving isn't as reasonable for the user 63 00:03:30,410 --> 00:03:31,610 to do as a request. 64 00:03:31,610 --> 00:03:32,780 So we cut that out. 65 00:03:35,945 --> 00:03:38,070 THERESE: Yeah, and one thing that the more and more 66 00:03:38,070 --> 00:03:41,610 we worked on this game we wanted to focus on 67 00:03:41,610 --> 00:03:45,660 is distinguish it from just playing your normal music 68 00:03:45,660 --> 00:03:48,593 and have it more of a meditative and almost reflective 69 00:03:48,593 --> 00:03:50,760 experience, and really make sure that you're picking 70 00:03:50,760 --> 00:03:52,200 the sounds that speak to you. 71 00:03:52,200 --> 00:03:54,250 So that way at the end, when you have this song, 72 00:03:54,250 --> 00:03:56,850 it's kind of connected to a memory or whatever 73 00:03:56,850 --> 00:03:59,340 specific feeling that you're having at the time. 74 00:03:59,340 --> 00:04:00,780 CHARLENE: One big caveat-- 75 00:04:00,780 --> 00:04:03,900 all the music that's in the app is something that we composed, 76 00:04:03,900 --> 00:04:05,610 and we're not composers. 77 00:04:05,610 --> 00:04:08,530 So the music itself isn't-- 78 00:04:08,530 --> 00:04:10,530 it's like the one we pick out that's open source 79 00:04:10,530 --> 00:04:13,230 online and the one that we just dropped our own little drum 80 00:04:13,230 --> 00:04:13,950 set drum beat. 81 00:04:13,950 --> 00:04:16,470 So they're not a wide set of it. 82 00:04:16,470 --> 00:04:19,560 So it's not super descriptive of everyone's experience 83 00:04:19,560 --> 00:04:20,235 at that time. 84 00:04:20,235 --> 00:04:23,718 So don't hate on the music too much. 85 00:04:23,718 --> 00:04:25,212 [LAUGHTER] 86 00:04:25,212 --> 00:04:26,208 AUDIENCE: Sounded good. 87 00:04:26,208 --> 00:04:29,250 CHARLENE: Oh, good. 88 00:04:29,250 --> 00:04:33,180 So implementation evolution-- so throughout our user 89 00:04:33,180 --> 00:04:36,545 testing, the how to experience-- is it working well? 90 00:04:36,545 --> 00:04:37,170 AUDIENCE: Cool. 91 00:04:37,170 --> 00:04:38,087 CHARLENE: Yeah, great. 92 00:04:38,087 --> 00:04:40,170 So feel free if you want to try it later on, 93 00:04:40,170 --> 00:04:42,270 come on and try it. 94 00:04:42,270 --> 00:04:44,820 Originally, we had several implementation 95 00:04:44,820 --> 00:04:50,780 in terms of having spatialized music 96 00:04:50,780 --> 00:04:52,880 where we describe having the new audio coming 97 00:04:52,880 --> 00:04:54,350 in different direction. 98 00:04:54,350 --> 00:04:56,667 The previous iteration when we first started the idea 99 00:04:56,667 --> 00:04:58,250 is that in order to capture the music, 100 00:04:58,250 --> 00:05:01,220 we wanted to actually face the direction of the music. 101 00:05:01,220 --> 00:05:03,987 You'd kind of be looking at it with the frame 102 00:05:03,987 --> 00:05:06,320 because you're hearing it in the direction of the sound. 103 00:05:06,320 --> 00:05:07,745 AUDIENCE: Should I turn it off? 104 00:05:07,745 --> 00:05:09,650 CHARLENE: Yeah. 105 00:05:09,650 --> 00:05:12,680 But then as we user test, we realized again having-- 106 00:05:12,680 --> 00:05:16,160 while moving forward and dictate what a user's head direction be 107 00:05:16,160 --> 00:05:17,930 is not a reasonable thing to ask. 108 00:05:17,930 --> 00:05:19,718 So we cut that part down. 109 00:05:19,718 --> 00:05:21,260 The user just had the option later on 110 00:05:21,260 --> 00:05:23,720 to change how many steps they want to take. 111 00:05:23,720 --> 00:05:25,512 There's a range between low amount of step, 112 00:05:25,512 --> 00:05:27,262 depending on the environment [INAUDIBLE],, 113 00:05:27,262 --> 00:05:29,730 to the higher amount of step for the music to be generated. 114 00:05:29,730 --> 00:05:31,063 So they've gone for a long walk. 115 00:05:31,063 --> 00:05:32,897 They don't want to be bombarded with a bunch 116 00:05:32,897 --> 00:05:33,980 of new music coming up. 117 00:05:33,980 --> 00:05:36,920 They select a longer generation. 118 00:05:36,920 --> 00:05:39,238 If they are indoor, for example, if you just 119 00:05:39,238 --> 00:05:41,030 want to take a short walk around the office 120 00:05:41,030 --> 00:05:43,100 and it's snowing outside, then you 121 00:05:43,100 --> 00:05:49,600 do a short iteration, short generation. 122 00:05:49,600 --> 00:05:52,410 THERESE: Yeah, so one of our main research questions 123 00:05:52,410 --> 00:05:58,080 during this process was how we can create a mind space using 124 00:05:58,080 --> 00:06:04,350 only audio to make the player more meditative and reflective. 125 00:06:04,350 --> 00:06:08,310 So at first-- so we drew a lot of inspiration 126 00:06:08,310 --> 00:06:09,610 from a lot of different places. 127 00:06:09,610 --> 00:06:12,750 So I think, obviously, first is Pokémon GO. 128 00:06:12,750 --> 00:06:16,627 So originally, we were thinking have a phone AR component, 129 00:06:16,627 --> 00:06:17,460 and you walk around. 130 00:06:17,460 --> 00:06:20,040 And in addition to hearing the sounds, 131 00:06:20,040 --> 00:06:22,170 also have a little sprite character 132 00:06:22,170 --> 00:06:24,150 where you would put your phone there and then 133 00:06:24,150 --> 00:06:25,770 capture it on the phone. 134 00:06:25,770 --> 00:06:29,040 But we found that really drew a person out of their own head 135 00:06:29,040 --> 00:06:32,100 and put them in more of a virtual world, which 136 00:06:32,100 --> 00:06:35,640 was kind of the opposite of what we wanted. 137 00:06:35,640 --> 00:06:38,820 Yeah, so then another thing that we started thinking about was 138 00:06:38,820 --> 00:06:41,490 meditation, and how when people meditate, 139 00:06:41,490 --> 00:06:45,970 a very common way to do that is to count your breathing. 140 00:06:45,970 --> 00:06:49,590 So one way that we wanted to implement that with audio 141 00:06:49,590 --> 00:06:52,110 is have a consistent cadence. 142 00:06:52,110 --> 00:06:54,420 So you kind of match your steps to it, 143 00:06:54,420 --> 00:06:58,410 and it's that rhythmic, meditative state. 144 00:06:58,410 --> 00:07:00,930 And we found that it really helped out the users 145 00:07:00,930 --> 00:07:03,143 to get into that mindset. 146 00:07:03,143 --> 00:07:04,560 And then during some play testing, 147 00:07:04,560 --> 00:07:06,248 we found in the beginning, people 148 00:07:06,248 --> 00:07:07,290 were really stressed out. 149 00:07:07,290 --> 00:07:09,750 And it's a lot of things going on, new sounds, 150 00:07:09,750 --> 00:07:11,500 a lot of decisions that they have to make. 151 00:07:11,500 --> 00:07:13,583 And people were saying that it was really intense. 152 00:07:13,583 --> 00:07:15,090 But the more that they played it, 153 00:07:15,090 --> 00:07:18,360 it became almost second nature, and players 154 00:07:18,360 --> 00:07:20,070 were really able to get into that mindset 155 00:07:20,070 --> 00:07:21,540 as soon as they put on the glasses, 156 00:07:21,540 --> 00:07:23,430 and wanted to make these decisions 157 00:07:23,430 --> 00:07:25,390 and make their own creation. 158 00:07:25,390 --> 00:07:27,380 CHARLENE: So the user that we've being asking 159 00:07:27,380 --> 00:07:29,880 around have been people who are really into composing music, 160 00:07:29,880 --> 00:07:32,130 and people who are more interested in just having 161 00:07:32,130 --> 00:07:33,690 music in their life in general. 162 00:07:33,690 --> 00:07:36,330 And as we work with the instructor, 163 00:07:36,330 --> 00:07:38,850 there was a differentiation between composing music 164 00:07:38,850 --> 00:07:41,430 and almost like the experience of birdwatching, 165 00:07:41,430 --> 00:07:43,500 where you have spur of the moment encounter, 166 00:07:43,500 --> 00:07:46,050 and you can choose to capture it or let it go. 167 00:07:46,050 --> 00:07:48,870 And we want to emphasize the feeling of you 168 00:07:48,870 --> 00:07:51,150 can save the music, but you don't want to-- 169 00:07:51,150 --> 00:07:53,250 very spur of the moment, instead of keep 170 00:07:53,250 --> 00:07:56,340 tweaking it and premeditating over the experience, 171 00:07:56,340 --> 00:07:59,460 and recomposing it over and over again. 172 00:07:59,460 --> 00:08:02,750 THERESE: Yeah, so that leads me to our next question, 173 00:08:02,750 --> 00:08:03,980 which was about control. 174 00:08:03,980 --> 00:08:07,690 So how does the level of control that you give the user 175 00:08:07,690 --> 00:08:09,700 affect the user experience? 176 00:08:09,700 --> 00:08:12,670 So as you were saying, we had a lot 177 00:08:12,670 --> 00:08:16,240 of different musical backgrounds of play testers. 178 00:08:16,240 --> 00:08:18,145 So that can be frustrating for people who 179 00:08:18,145 --> 00:08:20,290 don't know a lot about music. 180 00:08:20,290 --> 00:08:24,810 Or it was just kind of changing the experience in a way 181 00:08:24,810 --> 00:08:26,640 that we didn't want it. 182 00:08:26,640 --> 00:08:30,720 So also, if you give the user the option 183 00:08:30,720 --> 00:08:32,400 of as soon as you get a sound, you 184 00:08:32,400 --> 00:08:35,190 can change all these different things about it, 185 00:08:35,190 --> 00:08:37,950 it made walks much longer, and drew people out 186 00:08:37,950 --> 00:08:38,700 of the experience. 187 00:08:38,700 --> 00:08:40,533 Because you don't want to wait until the end 188 00:08:40,533 --> 00:08:43,299 to be like, oh, that thing that I caught like 10 minutes ago, 189 00:08:43,299 --> 00:08:44,260 what can I do with it? 190 00:08:44,260 --> 00:08:47,910 So it was kind of derailing the experience. 191 00:08:47,910 --> 00:08:52,500 And presenting it more as a kind of birdwatching experience 192 00:08:52,500 --> 00:08:56,850 really helped align everyone's expectations so that everyone 193 00:08:56,850 --> 00:09:00,360 would enjoy it, and kind of dealing with what you get 194 00:09:00,360 --> 00:09:04,200 and making the most of what you get also 195 00:09:04,200 --> 00:09:08,550 supported our goal of having it as a meditative experience, 196 00:09:08,550 --> 00:09:10,500 and kind of analogous to life. 197 00:09:10,500 --> 00:09:14,078 You just make the most of what you get, and have 198 00:09:14,078 --> 00:09:15,370 something beautiful in the end. 199 00:09:15,370 --> 00:09:18,180 So it was really exciting. 200 00:09:18,180 --> 00:09:20,840 CHARLENE: So the future iteration of what we want to do 201 00:09:20,840 --> 00:09:22,760 is more-- 202 00:09:22,760 --> 00:09:25,880 right now, the current set of music is at a particular BPM. 203 00:09:25,880 --> 00:09:28,520 But sometimes on a slow day, you want a slower melody. 204 00:09:28,520 --> 00:09:31,430 Or if you're on a fast day, you want a higher, pumpy one. 205 00:09:31,430 --> 00:09:33,770 We would like to have the option for the user to choose 206 00:09:33,770 --> 00:09:35,180 that in the beginning. 207 00:09:35,180 --> 00:09:38,160 Ideally, it's a input that's more intuitive. 208 00:09:38,160 --> 00:09:41,655 So it's to have them input-- so it's hard to-- 209 00:09:41,655 --> 00:09:44,030 for someone who is not musically inclined, if I tell you, 210 00:09:44,030 --> 00:09:47,210 oh, put it into BPM of your choice, 211 00:09:47,210 --> 00:09:48,980 they're not going to be able to have 212 00:09:48,980 --> 00:09:52,882 an idea of 140 BPM versus 120 BPM and how that feels. 213 00:09:52,882 --> 00:09:55,340 Instead, we want to be able to incorporate more the sensors 214 00:09:55,340 --> 00:09:58,500 to address the gyroscope or accelerometer to track 215 00:09:58,500 --> 00:10:00,433 your head nod to the beat that they want. 216 00:10:00,433 --> 00:10:01,850 So if they want a faster beat one, 217 00:10:01,850 --> 00:10:03,650 they could shake their head a little bit faster. 218 00:10:03,650 --> 00:10:06,150 If they want a slow beat one, it'd be a mellower head shake. 219 00:10:06,150 --> 00:10:07,790 And from the tracking of the beat, 220 00:10:07,790 --> 00:10:10,160 you pre-select what music you want 221 00:10:10,160 --> 00:10:12,720 to be able to offer to the user. 222 00:10:12,720 --> 00:10:19,070 So in the future, we want to be able to passively track 223 00:10:19,070 --> 00:10:24,740 users' emotional state with the physiological sensor that's 224 00:10:24,740 --> 00:10:30,860 embedded onto the AR frame, and this way have a more continuous 225 00:10:30,860 --> 00:10:32,715 experience. 226 00:10:32,715 --> 00:10:34,340 THERESE: Yeah, and then right now, it's 227 00:10:34,340 --> 00:10:35,828 a very personalized experience. 228 00:10:35,828 --> 00:10:38,120 And one thing that we wanted to implement in the future 229 00:10:38,120 --> 00:10:40,070 is start to build a community around that. 230 00:10:40,070 --> 00:10:43,340 So right now, at the end of your walk you have this song, 231 00:10:43,340 --> 00:10:45,260 and you can just save it for yourself. 232 00:10:45,260 --> 00:10:47,810 But in the future, we'd like to have it 233 00:10:47,810 --> 00:10:49,640 as an open platform for a community 234 00:10:49,640 --> 00:10:51,740 of other users on the app. 235 00:10:51,740 --> 00:10:55,490 And then maybe someone has this really nice beat. 236 00:10:55,490 --> 00:10:57,230 And then someone else could-- 237 00:10:57,230 --> 00:11:01,580 they could collaborate and add vocals and make songs, or just 238 00:11:01,580 --> 00:11:03,800 share with your friends. 239 00:11:03,800 --> 00:11:06,740 CHARLENE: So the whole idea started out with the feeling 240 00:11:06,740 --> 00:11:10,100 that music can capture emotion or a certain moment. 241 00:11:10,100 --> 00:11:12,620 And we want to be able to incorporate that 242 00:11:12,620 --> 00:11:14,120 into everyone's life. 243 00:11:14,120 --> 00:11:16,780 And that's the goal of INMUSE, hopefully. 244 00:11:16,780 --> 00:11:19,250 Yeah, and that's it. 245 00:11:19,250 --> 00:11:22,600 [APPLAUSE]