WEBVTT 00:00:00.000 --> 00:00:01.467 align:middle line:90% [SQUEAKING] 00:00:01.467 --> 00:00:03.423 align:middle line:90% [RUSTLING] 00:00:03.423 --> 00:00:06.846 align:middle line:90% [CLICKING] 00:00:06.846 --> 00:00:11.277 align:middle line:90% 00:00:11.277 --> 00:00:13.610 align:middle line:84% MICHAEL SCOTT ASATO CUTHBERT: Hello, computational music 00:00:13.610 --> 00:00:14.660 align:middle line:90% theorists. 00:00:14.660 --> 00:00:19.850 align:middle line:84% Here's a short video on what some of the problems in MIR, 00:00:19.850 --> 00:00:23.060 align:middle line:84% or Music Information Retrieval, are and how they can be 00:00:23.060 --> 00:00:26.220 align:middle line:84% connected to music theory and musicology, 00:00:26.220 --> 00:00:31.160 align:middle line:84% which is the main topic of this class, 21m.383. 00:00:31.160 --> 00:00:36.060 align:middle line:84% These slides owe a lot to Avi Pfeffer, professor at Harvard, 00:00:36.060 --> 00:00:40.190 align:middle line:84% who inspired them with his lectures. 00:00:40.190 --> 00:00:42.710 align:middle line:84% The main question that we'll be working on 00:00:42.710 --> 00:00:49.070 align:middle line:84% is, how do we move from sound that's made in the real world 00:00:49.070 --> 00:00:53.630 align:middle line:84% to a score that's printed out or displayed on the screen? 00:00:53.630 --> 00:00:56.400 align:middle line:84% What kinds of technologies are needed, 00:00:56.400 --> 00:00:59.390 align:middle line:84% and what kinds of problems need to be solved in order to do this 00:00:59.390 --> 00:01:00.600 align:middle line:90% really well? 00:01:00.600 --> 00:01:03.400 align:middle line:84% Well, first, when we sing or play an instrument, 00:01:03.400 --> 00:01:06.190 align:middle line:84% we're making actual sound waves in the real world, 00:01:06.190 --> 00:01:08.640 align:middle line:84% so we need to solve the problem of how 00:01:08.640 --> 00:01:13.120 align:middle line:84% to most accurately transform sound into a signal. 00:01:13.120 --> 00:01:16.810 align:middle line:84% And this involves microphones and other devices, 00:01:16.810 --> 00:01:19.890 align:middle line:84% and it's kind of a pure engineering problem 00:01:19.890 --> 00:01:22.840 align:middle line:84% that we're not going to be focusing on in this class. 00:01:22.840 --> 00:01:25.120 align:middle line:84% But it's very complex and very valuable, 00:01:25.120 --> 00:01:27.570 align:middle line:84% and we offer some courses in the music department 00:01:27.570 --> 00:01:30.540 align:middle line:84% that will help you do this better. 00:01:30.540 --> 00:01:34.680 align:middle line:84% But once we've got the sound from the real world 00:01:34.680 --> 00:01:38.130 align:middle line:84% into an electrical signal, we're working 00:01:38.130 --> 00:01:41.290 align:middle line:84% in the time domain, as we call it, 00:01:41.290 --> 00:01:46.600 align:middle line:84% and that is the amplitude of sound at a single moment-- 00:01:46.600 --> 00:01:48.940 align:middle line:84% forward, backwards, positive, or negative. 00:01:48.940 --> 00:01:54.300 align:middle line:84% And we usually record in something close to CD quality 00:01:54.300 --> 00:01:56.470 align:middle line:90% or 44,000 times per second. 00:01:56.470 --> 00:02:00.720 align:middle line:84% We're measuring the amplitude as a 16-bit signed integer. 00:02:00.720 --> 00:02:05.330 align:middle line:84% And so the sound, at this point, looks something like this 00:02:05.330 --> 00:02:08.110 align:middle line:84% if this was, I don't know, an attack 00:02:08.110 --> 00:02:09.979 align:middle line:84% of an instrument that's decaying, 00:02:09.979 --> 00:02:14.140 align:middle line:84% maybe a piano or something of that sort. 00:02:14.140 --> 00:02:18.280 align:middle line:84% The problem here is, how do we convert from the time domain 00:02:18.280 --> 00:02:24.130 align:middle line:84% to frequency, to being able to say, OK, all these impulses add 00:02:24.130 --> 00:02:27.910 align:middle line:90% up to a particular frequency? 00:02:27.910 --> 00:02:31.060 align:middle line:84% How do we decode it from the signal 00:02:31.060 --> 00:02:33.790 align:middle line:84% and understand that, hey, in this sound, 00:02:33.790 --> 00:02:36.250 align:middle line:84% there's two principal frequencies that 00:02:36.250 --> 00:02:38.030 align:middle line:90% are being heard? 00:02:38.030 --> 00:02:41.360 align:middle line:84% And to do this, we use Fourier transforms, 00:02:41.360 --> 00:02:44.410 align:middle line:84% and the fast Fourier transform algorithm 00:02:44.410 --> 00:02:48.100 align:middle line:84% is something that is very good to know in order 00:02:48.100 --> 00:02:49.310 align:middle line:90% to work in this field. 00:02:49.310 --> 00:02:51.850 align:middle line:84% So now we have frequencies, and we 00:02:51.850 --> 00:02:57.200 align:middle line:84% want to translate them to pitch and pitch connections. 00:02:57.200 --> 00:03:00.650 align:middle line:84% And that's not just thinking about, 00:03:00.650 --> 00:03:02.990 align:middle line:84% what is A, 440 hertz or something, 00:03:02.990 --> 00:03:05.960 align:middle line:84% but which of those frequencies that pop up are 00:03:05.960 --> 00:03:08.910 align:middle line:84% prolonged to make a particular pitch? 00:03:08.910 --> 00:03:12.500 align:middle line:84% Because even while I'm speaking, there are particular frequencies 00:03:12.500 --> 00:03:15.780 align:middle line:84% that are popping up, but they don't necessarily make pitches. 00:03:15.780 --> 00:03:21.020 align:middle line:84% The attack on a flute isn't necessarily the pitch 00:03:21.020 --> 00:03:21.600 align:middle line:90% that we want. 00:03:21.600 --> 00:03:24.320 align:middle line:84% So which ones are connected for this? 00:03:24.320 --> 00:03:26.960 align:middle line:84% And once we have these particular frequency pitch 00:03:26.960 --> 00:03:32.010 align:middle line:84% connections, we want to move from there to pitch notes. 00:03:32.010 --> 00:03:35.210 align:middle line:84% As we saw with the piano example, 00:03:35.210 --> 00:03:36.930 align:middle line:84% there were two frequencies there. 00:03:36.930 --> 00:03:38.960 align:middle line:84% Does that mean two pitches, or is 00:03:38.960 --> 00:03:41.700 align:middle line:84% one of them a reverberation in the room? 00:03:41.700 --> 00:03:44.400 align:middle line:84% Or if you're listening to another instrument, 00:03:44.400 --> 00:03:46.440 align:middle line:90% maybe you hear seven pitches. 00:03:46.440 --> 00:03:49.350 align:middle line:84% Six of them are higher and much quieter, 00:03:49.350 --> 00:03:52.130 align:middle line:84% and you might think, oh, that's the overtone series 00:03:52.130 --> 00:03:53.370 align:middle line:90% of that instrument. 00:03:53.370 --> 00:03:56.900 align:middle line:84% And so going from pitch connections to pitch notes 00:03:56.900 --> 00:03:59.970 align:middle line:84% involves an estimation of timbre. 00:03:59.970 --> 00:04:01.830 align:middle line:84% What is the timbre of the instrument? 00:04:01.830 --> 00:04:04.740 align:middle line:84% What is the overtone series that one expects 00:04:04.740 --> 00:04:06.190 align:middle line:90% and who we remove them. 00:04:06.190 --> 00:04:08.925 align:middle line:84% And from there, do we end up still with multiple pitches 00:04:08.925 --> 00:04:11.940 align:middle line:90% in a chord? 00:04:11.940 --> 00:04:15.060 align:middle line:84% Now that we know the musical pitches 00:04:15.060 --> 00:04:16.890 align:middle line:84% that we're interested in, we want 00:04:16.890 --> 00:04:20.010 align:middle line:84% to go from there to understanding 00:04:20.010 --> 00:04:24.100 align:middle line:84% the duration of these pitches and the tempo of the piece. 00:04:24.100 --> 00:04:26.940 align:middle line:84% And so there are particular algorithms 00:04:26.940 --> 00:04:29.710 align:middle line:84% that we can use to estimate the duration. 00:04:29.710 --> 00:04:32.850 align:middle line:84% I mean, one thing is you just look at milliseconds and things 00:04:32.850 --> 00:04:34.350 align:middle line:84% and then say, OK, well, that's going 00:04:34.350 --> 00:04:35.830 align:middle line:90% to be the most common note. 00:04:35.830 --> 00:04:38.080 align:middle line:84% We'll call it a quarter note and so on. 00:04:38.080 --> 00:04:41.820 align:middle line:84% And from there, we can estimate the tempo. 00:04:41.820 --> 00:04:44.440 align:middle line:84% Or we can do it in the other way. 00:04:44.440 --> 00:04:47.910 align:middle line:84% We can take all these pitches and try 00:04:47.910 --> 00:04:51.990 align:middle line:84% to estimate a tempo based on, what are the recurring 00:04:51.990 --> 00:04:54.460 align:middle line:90% onsets that seem to be beats? 00:04:54.460 --> 00:04:58.750 align:middle line:84% And from there, go and figure out the duration-- quarter note, 00:04:58.750 --> 00:05:01.740 align:middle line:84% half note, three beats long, something like that 00:05:01.740 --> 00:05:03.810 align:middle line:90% for each of the notes. 00:05:03.810 --> 00:05:08.560 align:middle line:84% So which direction do MIR people go? 00:05:08.560 --> 00:05:12.470 align:middle line:84% It turns out that we usually create a sort of feedback loop. 00:05:12.470 --> 00:05:16.030 align:middle line:84% So we try to do both at once, and estimations of tempo 00:05:16.030 --> 00:05:20.630 align:middle line:84% lead to durations, which then can be rejected or accepted, 00:05:20.630 --> 00:05:25.300 align:middle line:84% and estimations of the duration lead to estimations of tempo. 00:05:25.300 --> 00:05:27.670 align:middle line:84% And so the technologies involved are 00:05:27.670 --> 00:05:30.350 align:middle line:84% beat tracking and self-similarity detection 00:05:30.350 --> 00:05:32.140 align:middle line:90% and others. 00:05:32.140 --> 00:05:36.470 align:middle line:84% From the pitches, the duration, and the tempo, 00:05:36.470 --> 00:05:42.470 align:middle line:84% we can create what we call the full notes or just the notes. 00:05:42.470 --> 00:05:47.920 align:middle line:84% That is something from a pitch and a duration combined. 00:05:47.920 --> 00:05:50.230 align:middle line:84% From all these full notes, we can 00:05:50.230 --> 00:05:53.990 align:middle line:84% construct what we call a virtual score, 00:05:53.990 --> 00:05:56.740 align:middle line:84% and that is the score as represented 00:05:56.740 --> 00:06:01.310 align:middle line:84% purely as collections of notes and other events. 00:06:01.310 --> 00:06:04.900 align:middle line:84% We might be estimating, at this point, crescendos 00:06:04.900 --> 00:06:06.350 align:middle line:84% and decrescendos, and maybe we're 00:06:06.350 --> 00:06:07.860 align:middle line:84% hearing certain notes are slurred, 00:06:07.860 --> 00:06:12.180 align:middle line:84% and we're keeping these all as objects inside memory. 00:06:12.180 --> 00:06:14.720 align:middle line:84% And so this involves the technology 00:06:14.720 --> 00:06:17.000 align:middle line:84% of music representation, which we will be getting 00:06:17.000 --> 00:06:18.690 align:middle line:90% to very soon in this class. 00:06:18.690 --> 00:06:22.010 align:middle line:84% And so how might we represent a score, maybe as 00:06:22.010 --> 00:06:24.670 align:middle line:84% a set of parts with measures and notes in between? 00:06:24.670 --> 00:06:27.800 align:middle line:90% 00:06:27.800 --> 00:06:32.460 align:middle line:84% From the virtual score, we need to figure out, OK, 00:06:32.460 --> 00:06:35.700 align:middle line:84% how is this going to look on the page or on the screen? 00:06:35.700 --> 00:06:39.260 align:middle line:84% And so we perform layout estimation, 00:06:39.260 --> 00:06:42.920 align:middle line:84% and we create a laid-out score where it's not just 00:06:42.920 --> 00:06:45.800 align:middle line:84% notes inside of an array or a list or something 00:06:45.800 --> 00:06:50.150 align:middle line:84% like that but actual positions on a page. 00:06:50.150 --> 00:06:52.477 align:middle line:84% And this involves some technologies 00:06:52.477 --> 00:06:54.060 align:middle line:84% that we'll be getting to in the class. 00:06:54.060 --> 00:06:58.156 align:middle line:84% Here's an example of very poor layout estimation. 00:06:58.156 --> 00:07:01.880 align:middle line:90% 00:07:01.880 --> 00:07:05.000 align:middle line:84% From the laid-out score, we choose 00:07:05.000 --> 00:07:07.880 align:middle line:84% what the glyphs that would be appropriate are 00:07:07.880 --> 00:07:11.430 align:middle line:84% and where are they going to be placed on the page? 00:07:11.430 --> 00:07:13.970 align:middle line:84% A sample of some of the things that might 00:07:13.970 --> 00:07:17.180 align:middle line:90% be involved in doing that. 00:07:17.180 --> 00:07:19.620 align:middle line:84% And then once we have the laid-out score, 00:07:19.620 --> 00:07:21.330 align:middle line:84% the glyph selection, and placement, 00:07:21.330 --> 00:07:25.070 align:middle line:84% we move to pixel rendering, individual 00:07:25.070 --> 00:07:28.970 align:middle line:84% dots no longer connected to any semantic meaning but just, 00:07:28.970 --> 00:07:33.230 align:middle line:84% is this x, y coordinate going to be black, white, or something 00:07:33.230 --> 00:07:36.770 align:middle line:90% in between or something else? 00:07:36.770 --> 00:07:40.490 align:middle line:84% And from there, we have particular technologies 00:07:40.490 --> 00:07:42.890 align:middle line:84% that we're going to need, such as screen 00:07:42.890 --> 00:07:47.300 align:middle line:84% drawing, old-school cathode ray tubes, LED display, things 00:07:47.300 --> 00:07:51.020 align:middle line:84% like that, or how to actually make a printer distribute ink 00:07:51.020 --> 00:07:51.780 align:middle line:90% on a page. 00:07:51.780 --> 00:07:53.780 align:middle line:84% And these, again, are engineering problems 00:07:53.780 --> 00:07:56.060 align:middle line:84% that are beyond the scope of this class but still 00:07:56.060 --> 00:07:58.010 align:middle line:90% very, very important. 00:07:58.010 --> 00:08:01.980 align:middle line:84% So let's look, again, a summary of what we have here. 00:08:01.980 --> 00:08:04.400 align:middle line:84% We move from the time domain to frequency, 00:08:04.400 --> 00:08:06.840 align:middle line:90% frequency to pitch connections. 00:08:06.840 --> 00:08:10.300 align:middle line:84% We estimate the duration and the tempo, creating full notes. 00:08:10.300 --> 00:08:13.240 align:middle line:84% Create a virtual score or logical score. 00:08:13.240 --> 00:08:15.600 align:middle line:84% Some people separate these two out. 00:08:15.600 --> 00:08:19.740 align:middle line:84% Then we try to make the layout and to the output. 00:08:19.740 --> 00:08:24.790 align:middle line:84% At each of these points in the process, 00:08:24.790 --> 00:08:28.570 align:middle line:84% we can involve music theory or musicology, 00:08:28.570 --> 00:08:32.909 align:middle line:84% and we can learn aspects about music theory or musicology 00:08:32.909 --> 00:08:35.919 align:middle line:84% from the data that we're getting at this point. 00:08:35.919 --> 00:08:39.552 align:middle line:84% So, for instance, the duration/tempo level, 00:08:39.552 --> 00:08:41.760 align:middle line:84% we can start to ask, well, what are the most commonly 00:08:41.760 --> 00:08:43.419 align:middle line:90% used rhythms in this piece? 00:08:43.419 --> 00:08:46.740 align:middle line:84% And from there, hey, let's create a backing drum track 00:08:46.740 --> 00:08:50.280 align:middle line:84% to accompany the soloist because we figured out 00:08:50.280 --> 00:08:53.220 align:middle line:84% what kinds of durations, what kinds of rhythms 00:08:53.220 --> 00:08:56.940 align:middle line:84% connect to this with a database, so on. 00:08:56.940 --> 00:09:02.880 align:middle line:84% Or at the frequency to pitch connections level, 00:09:02.880 --> 00:09:04.770 align:middle line:84% we can now understand, based on which 00:09:04.770 --> 00:09:07.150 align:middle line:84% overtones were removed, what instruments were being used? 00:09:07.150 --> 00:09:09.370 align:middle line:84% Are they playing in their natural ranges? 00:09:09.370 --> 00:09:11.940 align:middle line:84% And from there, we might estimate 00:09:11.940 --> 00:09:15.690 align:middle line:84% the year of this recording based on particular sounds 00:09:15.690 --> 00:09:18.180 align:middle line:84% and sound artifacts or the year of the piece 00:09:18.180 --> 00:09:21.900 align:middle line:84% based on instruments that are used and so on. 00:09:21.900 --> 00:09:25.860 align:middle line:84% So these are places where music theory and musicology can 00:09:25.860 --> 00:09:30.090 align:middle line:90% get data from this MIR process. 00:09:30.090 --> 00:09:33.790 align:middle line:84% But there's another side that's been much less talked about 00:09:33.790 --> 00:09:35.820 align:middle line:84% and it's a place of great potential 00:09:35.820 --> 00:09:37.920 align:middle line:84% that I think people in this class 00:09:37.920 --> 00:09:39.810 align:middle line:84% will be able to contribute to, and that 00:09:39.810 --> 00:09:42.690 align:middle line:84% is going from music theory or musicology 00:09:42.690 --> 00:09:46.230 align:middle line:84% back and using things that we know 00:09:46.230 --> 00:09:49.500 align:middle line:84% about theory or history of music or music studies 00:09:49.500 --> 00:09:53.280 align:middle line:84% to enhance the quality of the algorithms being 00:09:53.280 --> 00:09:54.730 align:middle line:90% used in these places. 00:09:54.730 --> 00:09:57.960 align:middle line:84% For instance, at the pitch connection level, 00:09:57.960 --> 00:10:00.810 align:middle line:84% we can say, well, let's look at the metadata, 00:10:00.810 --> 00:10:03.100 align:middle line:84% and it suggests that this is a piece by Mozart. 00:10:03.100 --> 00:10:04.800 align:middle line:90% He died in 1791. 00:10:04.800 --> 00:10:06.720 align:middle line:84% So let's look at those instruments. 00:10:06.720 --> 00:10:12.050 align:middle line:84% Now, the frequency, overtone estimation was saying, OK, 00:10:12.050 --> 00:10:14.890 align:middle line:84% the most likely instrument is electric guitar. 00:10:14.890 --> 00:10:16.850 align:middle line:90% This is very likely wrong. 00:10:16.850 --> 00:10:17.750 align:middle line:90% I mean, I don't know. 00:10:17.750 --> 00:10:21.500 align:middle line:84% Maybe it's a modern arrangement, but it's pretty likely wrong. 00:10:21.500 --> 00:10:24.800 align:middle line:84% Why don't you try this overtone profile of a fortepiano, 00:10:24.800 --> 00:10:26.360 align:middle line:90% an early piano instead? 00:10:26.360 --> 00:10:30.710 align:middle line:84% And wow, you find out, OK, that was the second-best score 00:10:30.710 --> 00:10:33.340 align:middle line:84% but probably the best one when taking into account 00:10:33.340 --> 00:10:34.905 align:middle line:90% the metadata on the piece. 00:10:34.905 --> 00:10:37.660 align:middle line:90% 00:10:37.660 --> 00:10:41.320 align:middle line:84% Or we can use music theory and musicology 00:10:41.320 --> 00:10:43.160 align:middle line:84% at the frequency-detection level. 00:10:43.160 --> 00:10:44.980 align:middle line:84% So we might say, hey, all of the notes 00:10:44.980 --> 00:10:47.990 align:middle line:84% you've been detecting around this note have been in E major. 00:10:47.990 --> 00:10:53.930 align:middle line:84% And then you hear an A sharp, and that's a little fishy. 00:10:53.930 --> 00:10:55.630 align:middle line:84% Oh, hey, there's a runner-up note 00:10:55.630 --> 00:11:00.640 align:middle line:84% of A natural that was 95% as likely. 00:11:00.640 --> 00:11:05.860 align:middle line:84% From a music-theory standpoint, I think that, more likely, 00:11:05.860 --> 00:11:07.640 align:middle line:90% we've detected the wrong note. 00:11:07.640 --> 00:11:10.150 align:middle line:84% So these are ways that music theory and musicology 00:11:10.150 --> 00:11:13.580 align:middle line:90% can enhance this process. 00:11:13.580 --> 00:11:18.680 align:middle line:84% Most exciting at all, though, is when parts of the process 00:11:18.680 --> 00:11:23.000 align:middle line:84% feed into music theory and musicology algorithms 00:11:23.000 --> 00:11:24.120 align:middle line:90% and are enhanced by it. 00:11:24.120 --> 00:11:26.730 align:middle line:84% So we say, oh, hey, given these frequencies, 00:11:26.730 --> 00:11:30.170 align:middle line:84% pitches, durations, we get the style of the piece, 00:11:30.170 --> 00:11:34.250 align:middle line:84% and we suggest that it's from the swing era. 00:11:34.250 --> 00:11:41.350 align:middle line:84% And then we can help later parts of the process by saying, 00:11:41.350 --> 00:11:43.350 align:middle line:84% if the style of the piece is from the swing era, 00:11:43.350 --> 00:11:45.990 align:middle line:84% when you do your virtual score, your logical score, 00:11:45.990 --> 00:11:49.910 align:middle line:84% probably all those triplets, those "yum, ba bum, ba bum, 00:11:49.910 --> 00:11:53.550 align:middle line:84% ba bum" should be written down and represented as eighth notes. 00:11:53.550 --> 00:11:55.105 align:middle line:84% That's what, conceptually, they are. 00:11:55.105 --> 00:11:57.620 align:middle line:90% 00:11:57.620 --> 00:12:00.570 align:middle line:84% But we can not just move forward. 00:12:00.570 --> 00:12:02.460 align:middle line:84% We can, at the virtual score level, 00:12:02.460 --> 00:12:05.490 align:middle line:84% maybe it's beginning to lay out the score, 00:12:05.490 --> 00:12:08.760 align:middle line:84% and they're putting everything nicely into 3/4. 00:12:08.760 --> 00:12:12.250 align:middle line:84% And the musicology or music-theory algorithm says, 00:12:12.250 --> 00:12:17.520 align:middle line:84% well, these rhythms are highly unusual for 3/4 time. 00:12:17.520 --> 00:12:21.720 align:middle line:84% Let's go back and rerun some of the previous steps with that 00:12:21.720 --> 00:12:25.440 align:middle line:84% notion that that was highly improbable and try to make 00:12:25.440 --> 00:12:28.060 align:middle line:84% the score again so we can move backwards in time, 00:12:28.060 --> 00:12:32.650 align:middle line:84% redo a particular part of the process, rerun it, 00:12:32.650 --> 00:12:36.210 align:middle line:84% and then maybe come up with a different result. 00:12:36.210 --> 00:12:38.580 align:middle line:90% We'll put this in 6/8. 00:12:38.580 --> 00:12:41.520 align:middle line:84% So here are some of the ways that music theory and musicology 00:12:41.520 --> 00:12:47.550 align:middle line:84% can feed back into the MIR sound-to-score process. 00:12:47.550 --> 00:12:50.940 align:middle line:84% Next time, sometime soon, I hope to say 00:12:50.940 --> 00:12:53.010 align:middle line:84% how we can go the opposite direction, how 00:12:53.010 --> 00:12:56.710 align:middle line:84% we can take a score and automatically perform it. 00:12:56.710 --> 00:12:58.200 align:middle line:84% And even though we're going to be 00:12:58.200 --> 00:12:59.803 align:middle line:84% going in the opposite direction, it 00:12:59.803 --> 00:13:02.220 align:middle line:84% turns out that there's a totally different set of problems 00:13:02.220 --> 00:13:04.280 align:middle line:90% that we have to solve. 00:13:04.280 --> 00:13:14.000 align:middle line:90%