WEBVTT 00:00:00.000 --> 00:00:01.464 align:middle line:90% [SQUEAKING] 00:00:01.464 --> 00:00:02.440 align:middle line:90% [CLICKING] 00:00:02.440 --> 00:00:03.904 align:middle line:90% [RUSTLING] 00:00:03.904 --> 00:00:06.832 align:middle line:90% [CLICKING] 00:00:06.832 --> 00:00:09.577 align:middle line:90% 00:00:09.577 --> 00:00:11.660 align:middle line:84% MICHAEL SCOTT ASATO CUTHBERT: Hello, computational 00:00:11.660 --> 00:00:13.370 align:middle line:90% music researchers. 00:00:13.370 --> 00:00:17.810 align:middle line:84% We're going to continue our study of corpora and what we can 00:00:17.810 --> 00:00:22.970 align:middle line:84% do with the corpus of music by talking about some research that 00:00:22.970 --> 00:00:28.910 align:middle line:84% I did with Sophia H. Sun, who was a visiting student at MIT 00:00:28.910 --> 00:00:36.770 align:middle line:84% in 2016-2017 and graduated in 2018 from Wellesley College. 00:00:36.770 --> 00:00:40.290 align:middle line:84% We've published our work as a motion painting, lyric, affect, 00:00:40.290 --> 00:00:44.900 align:middle line:84% and musical relationships in a large lead sheet corpus. 00:00:44.900 --> 00:00:47.360 align:middle line:84% So this is work that's about looking 00:00:47.360 --> 00:00:51.890 align:middle line:84% at a large musical corpus and also looking at words 00:00:51.890 --> 00:00:55.760 align:middle line:84% and how music and words interact. 00:00:55.760 --> 00:00:58.040 align:middle line:84% Words and musical moments are generally 00:00:58.040 --> 00:01:00.420 align:middle line:84% both felt to have intrinsic emotions, 00:01:00.420 --> 00:01:03.110 align:middle line:84% but while the emotional context of text 00:01:03.110 --> 00:01:05.990 align:middle line:84% is determined at least in part from the semantics that 00:01:05.990 --> 00:01:09.120 align:middle line:84% is the meaning of individual words and sentences, 00:01:09.120 --> 00:01:12.200 align:middle line:84% the emotional content of individual musical parameters 00:01:12.200 --> 00:01:15.020 align:middle line:90% is far less well defined. 00:01:15.020 --> 00:01:17.630 align:middle line:84% This lack of definition is especially 00:01:17.630 --> 00:01:20.030 align:middle line:84% acute in written musical scores that 00:01:20.030 --> 00:01:22.550 align:middle line:84% are separated from the aural and visual components 00:01:22.550 --> 00:01:25.640 align:middle line:84% of particular performances or recordings. 00:01:25.640 --> 00:01:28.340 align:middle line:84% So a major question for music perception 00:01:28.340 --> 00:01:32.780 align:middle line:84% or the psychology of music is do particular musical features 00:01:32.780 --> 00:01:35.220 align:middle line:84% such as consonance and dissonance 00:01:35.220 --> 00:01:37.410 align:middle line:90% beat placement and pitch height. 00:01:37.410 --> 00:01:40.620 align:middle line:84% When they're separated from their performative elements, 00:01:40.620 --> 00:01:43.730 align:middle line:84% do they still carry particular emotional meanings 00:01:43.730 --> 00:01:47.930 align:middle line:90% such as exhilaration or sadness? 00:01:47.930 --> 00:01:51.930 align:middle line:84% It might not be possible to answer such a question directly, 00:01:51.930 --> 00:01:54.380 align:middle line:84% but we can approach the answer indirectly 00:01:54.380 --> 00:01:57.380 align:middle line:84% by asking are there musical features that 00:01:57.380 --> 00:01:59.690 align:middle line:84% are used in conjunction with texts 00:01:59.690 --> 00:02:02.270 align:middle line:84% and are associated with particular sentiments 00:02:02.270 --> 00:02:05.930 align:middle line:84% or emotions with such regularity that we might believe 00:02:05.930 --> 00:02:08.930 align:middle line:84% that the musical features themselves are used 00:02:08.930 --> 00:02:12.510 align:middle line:84% to strengthen the effect of these affects and emotions 00:02:12.510 --> 00:02:18.050 align:middle line:84% or do they indeed carry that affect or emotion themselves. 00:02:18.050 --> 00:02:20.330 align:middle line:84% The problem with making such connections 00:02:20.330 --> 00:02:22.430 align:middle line:84% with the traditional musical analysis 00:02:22.430 --> 00:02:24.650 align:middle line:84% is that there's no perfect correlation 00:02:24.650 --> 00:02:29.810 align:middle line:84% between any musical feature and any emotional state. 00:02:29.810 --> 00:02:32.990 align:middle line:84% We might think at least in Western common practice 00:02:32.990 --> 00:02:36.200 align:middle line:84% tradition that words connoting joy 00:02:36.200 --> 00:02:40.850 align:middle line:84% would tend to be higher in pitch than the notes around them 00:02:40.850 --> 00:02:42.450 align:middle line:90% or the notes preceding it. 00:02:42.450 --> 00:02:46.970 align:middle line:84% And it's easy to find examples to confirm this theory. 00:02:46.970 --> 00:02:49.490 align:middle line:84% In Schubert's "Gretchen am Spinnrade" 00:02:49.490 --> 00:02:51.600 align:middle line:84% or "Gretchen at the Spinning Wheel," 00:02:51.600 --> 00:02:55.950 align:middle line:84% one of his leads every single time, the word Herz, 00:02:55.950 --> 00:03:01.050 align:middle line:84% which is the German for heart, appears or lacheln, smile, 00:03:01.050 --> 00:03:05.060 align:middle line:84% or Zauber, magic, Der Kuss, kiss, 00:03:05.060 --> 00:03:08.960 align:middle line:84% it's always at the same pitch height or higher than 00:03:08.960 --> 00:03:12.500 align:middle line:84% the preceding note and always in the upper register 00:03:12.500 --> 00:03:15.380 align:middle line:90% for the singer. 00:03:15.380 --> 00:03:17.655 align:middle line:90% [MUSIC PLAYING] 00:03:17.655 --> 00:03:20.210 align:middle line:90% SINGER: [NON-ENGLISH SINGING] 00:03:20.210 --> 00:03:28.794 align:middle line:90% 00:03:28.794 --> 00:03:33.470 align:middle line:84% PROFESSOR: So it might sound like this is an easy connection, 00:03:33.470 --> 00:03:36.970 align:middle line:84% but then a little bit of Elvis enters into your head. 00:03:36.970 --> 00:03:43.445 align:middle line:84% ELVIS PRESLEY: (SINGING) But I can't help 00:03:43.445 --> 00:03:52.665 align:middle line:90% Falling in love with you 00:03:52.665 --> 00:03:55.440 align:middle line:84% PROFESSOR: In this song by Elvis, 00:03:55.440 --> 00:03:58.310 align:middle line:84% originally written by George Weiss and others, 00:03:58.310 --> 00:04:01.940 align:middle line:84% called "Can't Help Falling in Love," 00:04:01.940 --> 00:04:07.580 align:middle line:84% we always hear the words love and darling below the previous 00:04:07.580 --> 00:04:08.310 align:middle line:90% note. 00:04:08.310 --> 00:04:12.410 align:middle line:84% Even in an expression as magicalistic or full 00:04:12.410 --> 00:04:16.070 align:middle line:84% of word painting as falling, even that 00:04:16.070 --> 00:04:19.760 align:middle line:90% is set with a rising motion. 00:04:19.760 --> 00:04:25.250 align:middle line:84% So this says that individual pieces, some here and some 00:04:25.250 --> 00:04:27.260 align:middle line:84% there, might have clear relationships 00:04:27.260 --> 00:04:30.710 align:middle line:84% among text and emotion and musical features. 00:04:30.710 --> 00:04:35.090 align:middle line:84% But across a large corpus, the aggregated experience 00:04:35.090 --> 00:04:40.070 align:middle line:84% of listening to common practice or folk or popular music 00:04:40.070 --> 00:04:42.710 align:middle line:90% is far more muddled. 00:04:42.710 --> 00:04:45.990 align:middle line:84% To clarify this, we can turn to corpus studies, 00:04:45.990 --> 00:04:48.410 align:middle line:84% especially computational analysis 00:04:48.410 --> 00:04:52.250 align:middle line:84% of large repertories of musical encoded data sets 00:04:52.250 --> 00:04:55.850 align:middle line:84% whose sheer volume makes it impossible for human researchers 00:04:55.850 --> 00:04:58.880 align:middle line:84% to analyze all these pieces manually. 00:04:58.880 --> 00:05:01.790 align:middle line:84% With computational methods for extracting connections 00:05:01.790 --> 00:05:04.100 align:middle line:84% among music, text, and emotion, we 00:05:04.100 --> 00:05:06.830 align:middle line:84% can find statistically significant trends 00:05:06.830 --> 00:05:10.040 align:middle line:84% in music affect relationships that stand out 00:05:10.040 --> 00:05:14.150 align:middle line:84% from a large corpus even if the magnitude of the trend itself 00:05:14.150 --> 00:05:15.055 align:middle line:90% is quite small. 00:05:15.055 --> 00:05:17.600 align:middle line:90% 00:05:17.600 --> 00:05:20.510 align:middle line:84% Sophia Sun and I used a repository 00:05:20.510 --> 00:05:26.570 align:middle line:84% of lead sheets, that is melodies with lyrics and chord symbols. 00:05:26.570 --> 00:05:31.230 align:middle line:84% There were about 1,900 songs, 1,895 in total, 00:05:31.230 --> 00:05:35.750 align:middle line:84% that were in English language, that the encodings seem to be 00:05:35.750 --> 00:05:39.380 align:middle line:84% high quality and that there were no duplicates. 00:05:39.380 --> 00:05:41.840 align:middle line:84% This is a repository called Wikifonia 00:05:41.840 --> 00:05:44.430 align:middle line:84% that used to be freely available on the internet, 00:05:44.430 --> 00:05:47.660 align:middle line:84% and unfortunately their license expired and now 00:05:47.660 --> 00:05:52.945 align:middle line:84% is only available to those who had downloaded it beforehand. 00:05:52.945 --> 00:05:58.570 align:middle line:84% In these 1,900 songs, there's about 200,000 notes and a little 00:05:58.570 --> 00:06:02.360 align:middle line:84% bit less than that in terms of the words and 70,000 chords. 00:06:02.360 --> 00:06:06.040 align:middle line:84% So this is a big repository to look at. 00:06:06.040 --> 00:06:11.570 align:middle line:84% We also used a second corpus but this one not a musical corpus. 00:06:11.570 --> 00:06:16.600 align:middle line:84% This is an open crowdsourced database of 14,000 common 00:06:16.600 --> 00:06:20.080 align:middle line:84% English words that was made Mohammad and Turney 00:06:20.080 --> 00:06:22.850 align:middle line:84% of the National Research Council of Canada, 00:06:22.850 --> 00:06:26.590 align:middle line:84% and each of these words has been coded according to the affects 00:06:26.590 --> 00:06:28.240 align:middle line:90% that they have. 00:06:28.240 --> 00:06:33.010 align:middle line:84% For instance, words with the affect of anticipation 00:06:33.010 --> 00:06:38.150 align:middle line:84% include skewed or recreational or zeal, holiness, intimately, 00:06:38.150 --> 00:06:41.950 align:middle line:84% harbinger, birthday, something I think we all anticipate, 00:06:41.950 --> 00:06:43.330 align:middle line:90% or plan. 00:06:43.330 --> 00:06:46.870 align:middle line:84% And we can also look at words that have that affect 00:06:46.870 --> 00:06:48.350 align:middle line:90% and that affect alone. 00:06:48.350 --> 00:06:52.190 align:middle line:84% So, for instance, if we look at disgust, 00:06:52.190 --> 00:06:57.250 align:middle line:84% we see slime, dung, barf, diarrhea, gratuitous heartworm, 00:06:57.250 --> 00:07:02.050 align:middle line:84% things that only have this particular affect. 00:07:02.050 --> 00:07:06.170 align:middle line:84% Together with a small list of what we call stopwords-- 00:07:06.170 --> 00:07:09.770 align:middle line:84% these are words that have no affect and appear too commonly 00:07:09.770 --> 00:07:15.320 align:middle line:90% such as the, and, us, she, he-- 00:07:15.320 --> 00:07:18.490 align:middle line:84% about 60% of all the words in the songs 00:07:18.490 --> 00:07:21.130 align:middle line:84% could be classified according to affect 00:07:21.130 --> 00:07:25.900 align:middle line:84% or not having affect in the stop list neutral. 00:07:25.900 --> 00:07:31.280 align:middle line:84% And there are at least 2,000 instances of each affect 00:07:31.280 --> 00:07:35.150 align:middle line:84% and some with over 12,000 that appeared in this corpus. 00:07:35.150 --> 00:07:37.670 align:middle line:84% So there were-- in the repository, 00:07:37.670 --> 00:07:41.805 align:middle line:84% there were 2,853 songs that had anger in them. 00:07:41.805 --> 00:07:44.530 align:middle line:90% 00:07:44.530 --> 00:07:50.830 align:middle line:84% Also in the corpus were 25,000 words that were coded as being 00:07:50.830 --> 00:07:52.900 align:middle line:90% neutral. 00:07:52.900 --> 00:07:55.700 align:middle line:84% And then of those plus stopwords, 00:07:55.700 --> 00:07:59.230 align:middle line:84% there were about 12,000 words that we can be pretty sure to be 00:07:59.230 --> 00:08:00.140 align:middle line:90% neutral. 00:08:00.140 --> 00:08:05.440 align:middle line:84% And then 100-- sorry, 123,000 that could be believed to be 00:08:05.440 --> 00:08:06.200 align:middle line:90% neutral-- 00:08:06.200 --> 00:08:12.520 align:middle line:84% and then 183,000 is all of the comprehensive words that did not 00:08:12.520 --> 00:08:14.200 align:middle line:90% fill foot-- 00:08:14.200 --> 00:08:16.485 align:middle line:84% fit into one of these 10 affects. 00:08:16.485 --> 00:08:19.945 align:middle line:90% 00:08:19.945 --> 00:08:23.830 align:middle line:84% Sun and I use music21 to automatically label 00:08:23.830 --> 00:08:28.600 align:middle line:84% every instance of the word in the data set alongside one value 00:08:28.600 --> 00:08:30.470 align:middle line:84% that we are measuring at the time. 00:08:30.470 --> 00:08:34.000 align:middle line:84% So, for instance, this is the song "Greensleeves," 00:08:34.000 --> 00:08:38.030 align:middle line:84% also known at Christmas as "What Child Is this," 00:08:38.030 --> 00:08:45.220 align:middle line:84% and in this excerpt, every word except for the word alas was 00:08:45.220 --> 00:08:48.700 align:middle line:84% identified by the composer-- by the computer as being somewhere 00:08:48.700 --> 00:08:49.940 align:middle line:90% in the data set. 00:08:49.940 --> 00:08:52.870 align:middle line:84% But only two words in this excerpt 00:08:52.870 --> 00:08:56.750 align:middle line:84% had a particular affect that had been identified. 00:08:56.750 --> 00:09:00.250 align:middle line:84% They affect-- identified love as joy 00:09:00.250 --> 00:09:06.040 align:middle line:84% and positive and wrong as negative. 00:09:06.040 --> 00:09:10.850 align:middle line:84% So then we looked at particular values that we could measure. 00:09:10.850 --> 00:09:14.380 align:middle line:84% For instance, the one that I talked about earlier about 00:09:14.380 --> 00:09:15.620 align:middle line:90% pitch height-- 00:09:15.620 --> 00:09:19.450 align:middle line:84% so we're going to look for each word looking 00:09:19.450 --> 00:09:22.540 align:middle line:84% at the first syllable, how many semitones 00:09:22.540 --> 00:09:26.870 align:middle line:84% above or below the previous note does that word appear? 00:09:26.870 --> 00:09:31.300 align:middle line:84% So my is two semitones or a whole step above alas. 00:09:31.300 --> 00:09:35.050 align:middle line:90% Love is two semitones above my. 00:09:35.050 --> 00:09:40.960 align:middle line:84% Going forward, wrong is four semitones below me. 00:09:40.960 --> 00:09:45.070 align:middle line:84% With measurements like this, we can test the first theory. 00:09:45.070 --> 00:09:49.390 align:middle line:84% Do positive words rise in pitch and negative words 00:09:49.390 --> 00:09:50.490 align:middle line:90% descend in pitch? 00:09:50.490 --> 00:09:54.610 align:middle line:90% 00:09:54.610 --> 00:10:02.450 align:middle line:84% Here is our graph of the results for this particular measurement, 00:10:02.450 --> 00:10:04.690 align:middle line:84% what we call the relative pitch measurement. 00:10:04.690 --> 00:10:09.280 align:middle line:84% And what we can do on the left are the 10 affects or emotions 00:10:09.280 --> 00:10:12.200 align:middle line:84% that we were looking at-- anger, anticipation, disgust, fear, 00:10:12.200 --> 00:10:15.440 align:middle line:84% joy, negative, positive, sadness, surprise, and trust-- 00:10:15.440 --> 00:10:21.250 align:middle line:84% and we can look at words with that connotation 00:10:21.250 --> 00:10:24.760 align:middle line:84% compared to the neutral words that 00:10:24.760 --> 00:10:28.300 align:middle line:84% had been identified in the National Research Council Canada 00:10:28.300 --> 00:10:29.510 align:middle line:90% data set. 00:10:29.510 --> 00:10:32.240 align:middle line:84% And we can see that, for instance, 00:10:32.240 --> 00:10:36.490 align:middle line:84% looking across at the top line to the highlighted-- 00:10:36.490 --> 00:10:42.200 align:middle line:84% to the highlighted figure that anger was a little bit higher-- 00:10:42.200 --> 00:10:44.750 align:middle line:84% well, there's a plus sign, so it was higher 00:10:44.750 --> 00:10:46.050 align:middle line:90% than the neutral thing. 00:10:46.050 --> 00:10:49.920 align:middle line:90% But the 0.89 is the p value. 00:10:49.920 --> 00:10:51.890 align:middle line:90% That is our confidence. 00:10:51.890 --> 00:10:55.770 align:middle line:84% How often would such a value happen by chance? 00:10:55.770 --> 00:11:02.010 align:middle line:84% And you subtract from 1 so 1 minus 0.89 whatever. 00:11:02.010 --> 00:11:04.880 align:middle line:84% It's about 10% chance it would happen 00:11:04.880 --> 00:11:09.140 align:middle line:84% by chance that all the words for anger would be higher. 00:11:09.140 --> 00:11:14.190 align:middle line:84% And usually we don't think of p values that are below 0.95 00:11:14.190 --> 00:11:19.980 align:middle line:84% or 95% chance or 5% chance of happening accidentally. 00:11:19.980 --> 00:11:23.400 align:middle line:84% We don't think of those as having meaning. 00:11:23.400 --> 00:11:27.020 align:middle line:84% In fact, when you do what we call multiple comparison 00:11:27.020 --> 00:11:29.730 align:middle line:84% sets where we're looking at a lot of things at once, 00:11:29.730 --> 00:11:33.975 align:middle line:84% we really want p values that are much better than 0.05, 00:11:33.975 --> 00:11:38.660 align:middle line:90% 0.01, 0.001, and so on. 00:11:38.660 --> 00:11:43.360 align:middle line:84% And so in this chart, anger doesn't have a particular color 00:11:43.360 --> 00:11:45.650 align:middle line:90% on the NRC Mutual Set. 00:11:45.650 --> 00:11:50.000 align:middle line:84% There's no statistically significant value there, 00:11:50.000 --> 00:11:53.860 align:middle line:84% nor does anticipation, disgust or fear. 00:11:53.860 --> 00:11:58.910 align:middle line:84% But joy on the other hand is labeled with a darker blue, 00:11:58.910 --> 00:12:02.080 align:middle line:84% meaning that this is statistically significant. 00:12:02.080 --> 00:12:05.390 align:middle line:84% The three stars show how statistically significant. 00:12:05.390 --> 00:12:09.620 align:middle line:84% So the words with joy are higher than neutral words. 00:12:09.620 --> 00:12:12.500 align:middle line:84% Negative words didn't have any sort of p value, 00:12:12.500 --> 00:12:16.370 align:middle line:84% but positive words had a slight positive correlation. 00:12:16.370 --> 00:12:18.560 align:middle line:84% They're a little bit higher, but if we go down, 00:12:18.560 --> 00:12:21.980 align:middle line:90% surprise was a strong thing. 00:12:21.980 --> 00:12:27.040 align:middle line:84% So words with surprise were higher than other words. 00:12:27.040 --> 00:12:29.770 align:middle line:84% We can look at other parts, and, for instance, 00:12:29.770 --> 00:12:33.380 align:middle line:84% on the right side this triangular chart, 00:12:33.380 --> 00:12:37.850 align:middle line:84% we can look at how each affect correlates with another affect. 00:12:37.850 --> 00:12:41.320 align:middle line:84% So we can see that surprise is slightly statistically 00:12:41.320 --> 00:12:44.300 align:middle line:84% significantly higher than words about anger 00:12:44.300 --> 00:12:47.200 align:middle line:84% or slightly statistically significantly higher 00:12:47.200 --> 00:12:50.110 align:middle line:90% than words about negative. 00:12:50.110 --> 00:12:55.630 align:middle line:84% The column just to the right of the yellow highlighted chart 00:12:55.630 --> 00:12:58.630 align:middle line:84% are the stop words compared to the, a, 00:12:58.630 --> 00:13:02.980 align:middle line:84% and, and every single emotional word 00:13:02.980 --> 00:13:06.070 align:middle line:84% is higher in pitch than a stop word. 00:13:06.070 --> 00:13:08.530 align:middle line:90% Just think about when I say the. 00:13:08.530 --> 00:13:09.680 align:middle line:90% It's pretty low. 00:13:09.680 --> 00:13:13.730 align:middle line:84% We don't tend to put this kind of high emphasis on stop words. 00:13:13.730 --> 00:13:17.480 align:middle line:84% So I'm going to show some of the things that came up 00:13:17.480 --> 00:13:20.900 align:middle line:84% comparing these affects to neutral words 00:13:20.900 --> 00:13:26.380 align:middle line:84% because this seems to be the most fun and interesting part 00:13:26.380 --> 00:13:27.800 align:middle line:90% of the research. 00:13:27.800 --> 00:13:30.610 align:middle line:84% So all of this, this whole chart, 00:13:30.610 --> 00:13:37.870 align:middle line:84% will get reduced to a single column in this chart. 00:13:37.870 --> 00:13:40.210 align:middle line:84% And so we can see the relative pitch 00:13:40.210 --> 00:13:45.040 align:middle line:84% that joy, positive, and surprise are higher. 00:13:45.040 --> 00:13:48.170 align:middle line:90% Here is the key for the chart. 00:13:48.170 --> 00:13:51.950 align:middle line:84% So I don't show how much higher or how much lower in pitch 00:13:51.950 --> 00:13:54.260 align:middle line:84% or how much more positive or negative, 00:13:54.260 --> 00:13:57.320 align:middle line:84% just the ma-- just the direction in terms of the colors. 00:13:57.320 --> 00:14:00.320 align:middle line:84% So blue colors tend to be higher in some way, 00:14:00.320 --> 00:14:03.350 align:middle line:84% and mine-- and brown colors are lower. 00:14:03.350 --> 00:14:07.330 align:middle line:84% And then the intensity of the shade shows the p value. 00:14:07.330 --> 00:14:13.300 align:middle line:84% So the first things are shaded happen only 1% chance that 00:14:13.300 --> 00:14:17.680 align:middle line:84% they're happening by chance, and then the next shade is 1/10 00:14:17.680 --> 00:14:22.280 align:middle line:84% of 1% Then going down 1/100 to 1%, 1,000th of 1%, 00:14:22.280 --> 00:14:24.650 align:middle line:90% 1/10,000 of 1%. 00:14:24.650 --> 00:14:28.340 align:middle line:84% And you can see on the right as 10 to the negative 5, 00:14:28.340 --> 00:14:31.870 align:middle line:84% negative 6, negative 7, you'll see some of the charts were 00:14:31.870 --> 00:14:35.650 align:middle line:84% really showing things that are absolutely positively not 00:14:35.650 --> 00:14:40.890 align:middle line:84% by chance, things with 10 to the negative 200ths or more. 00:14:40.890 --> 00:14:44.860 align:middle line:90% 00:14:44.860 --> 00:14:47.570 align:middle line:84% One of the things that we thought was really interesting, 00:14:47.570 --> 00:14:50.480 align:middle line:84% we looked at the beat strength of every note. 00:14:50.480 --> 00:14:53.960 align:middle line:84% So we compared to the prevailing time signature, 00:14:53.960 --> 00:14:57.190 align:middle line:84% which is quite often 4/4, and we were able to figure out 00:14:57.190 --> 00:14:59.480 align:middle line:84% what's on the beat and what's off the beat. 00:14:59.480 --> 00:15:02.830 align:middle line:84% And one of the things we can see looking at the blue parts that 00:15:02.830 --> 00:15:05.830 align:middle line:84% words connoting joy and positive emotions 00:15:05.830 --> 00:15:09.700 align:middle line:84% are really much more common to happen on the beat, similarly 00:15:09.700 --> 00:15:11.450 align:middle line:90% with surprise and trust. 00:15:11.450 --> 00:15:15.100 align:middle line:84% But looking at things that happen off the beat 00:15:15.100 --> 00:15:19.930 align:middle line:84% or on weak beats, that's where you put the words for anger 00:15:19.930 --> 00:15:24.520 align:middle line:84% or syncopate, disgust, or fear or negative emotions 00:15:24.520 --> 00:15:28.533 align:middle line:90% tend to happen on weaker beats. 00:15:28.533 --> 00:15:31.750 align:middle line:84% Similarly, we can look at the length of notes 00:15:31.750 --> 00:15:37.030 align:middle line:84% that almost nothing has a positive correlation that 00:15:37.030 --> 00:15:39.740 align:middle line:84% is to they say that is longer than you would expect. 00:15:39.740 --> 00:15:43.720 align:middle line:84% But these negative emotions-- anger, disgust, fear, 00:15:43.720 --> 00:15:46.330 align:middle line:90% and just the general negative-- 00:15:46.330 --> 00:15:53.440 align:middle line:84% tend to be on shortened notes so that we're really just spitting 00:15:53.440 --> 00:15:58.270 align:middle line:90% out these short emotions. 00:15:58.270 --> 00:16:03.250 align:middle line:84% Intriguingly, when we look at the overall pitch, the pitch 00:16:03.250 --> 00:16:06.070 align:middle line:84% of a note compared to the average pitch 00:16:06.070 --> 00:16:09.200 align:middle line:84% of the entire song, there's basically no correlation. 00:16:09.200 --> 00:16:16.990 align:middle line:84% We don't tend to sing happy things high and sad things low. 00:16:16.990 --> 00:16:23.050 align:middle line:84% Only a little bit with positive happy, joy, surprise-ly things 00:16:23.050 --> 00:16:24.790 align:middle line:84% do we set them a little bit higher 00:16:24.790 --> 00:16:28.150 align:middle line:90% than what was happening before. 00:16:28.150 --> 00:16:31.245 align:middle line:90% Did look at consonants. 00:16:31.245 --> 00:16:35.970 align:middle line:84% What notes with-- what words are set to notes that 00:16:35.970 --> 00:16:38.538 align:middle line:90% are part of a consonant chord? 00:16:38.538 --> 00:16:40.080 align:middle line:84% Remember these are lead sheets, so we 00:16:40.080 --> 00:16:44.110 align:middle line:84% have the melody, the words, and also the underlying chords. 00:16:44.110 --> 00:16:47.770 align:middle line:84% We can see some of these negative emotions, anger 00:16:47.770 --> 00:16:53.430 align:middle line:84% but especially sadness and some are on more dissonant thing 00:16:53.430 --> 00:16:53.970 align:middle line:90% notes-- 00:16:53.970 --> 00:16:57.880 align:middle line:84% than the dissonant chords than the prevailing piece 00:16:57.880 --> 00:17:01.890 align:middle line:84% but also somewhat surprising joy. 00:17:01.890 --> 00:17:03.540 align:middle line:84% Maybe that's the way you emphasize 00:17:03.540 --> 00:17:09.990 align:middle line:84% joyful words or surprising words that you put them on slightly 00:17:09.990 --> 00:17:12.869 align:middle line:90% more dissonant words. 00:17:12.869 --> 00:17:17.190 align:middle line:84% Looked at two ways of classifying the mode of a piece, 00:17:17.190 --> 00:17:24.660 align:middle line:84% so major or minor, and we looked at do certain emotions appear 00:17:24.660 --> 00:17:27.430 align:middle line:84% more in songs that are in minor versus in major? 00:17:27.430 --> 00:17:30.715 align:middle line:84% And this is something that I at least had always been taught 00:17:30.715 --> 00:17:32.340 align:middle line:84% and I think many of us have been taught 00:17:32.340 --> 00:17:36.090 align:middle line:90% that minor pieces are sad. 00:17:36.090 --> 00:17:38.670 align:middle line:84% And maybe minor pieces are sad, but the words 00:17:38.670 --> 00:17:41.740 align:middle line:84% aren't any sadder than major mode pieces. 00:17:41.740 --> 00:17:45.630 align:middle line:84% There's no distinction between major and minor 00:17:45.630 --> 00:17:48.855 align:middle line:90% when we look at the whole piece. 00:17:48.855 --> 00:17:50.520 align:middle line:90% So instead we used-- 00:17:50.520 --> 00:17:53.580 align:middle line:84% oh, and by the way, we will be looking later in this semester 00:17:53.580 --> 00:17:57.840 align:middle line:84% and how we can computationally measure the key of a piece 00:17:57.840 --> 00:17:59.340 align:middle line:90% when we get to Caroline-- 00:17:59.340 --> 00:18:02.400 align:middle line:90% Carol Krumhansl's work. 00:18:02.400 --> 00:18:06.220 align:middle line:84% So we thought maybe it's not the whole piece, 00:18:06.220 --> 00:18:09.930 align:middle line:84% but maybe that these sad emotions appear 00:18:09.930 --> 00:18:14.370 align:middle line:84% in a minor section or piece using a lot of minor chords 00:18:14.370 --> 00:18:16.090 align:middle line:90% within a larger piece. 00:18:16.090 --> 00:18:20.550 align:middle line:84% So we looked at the local mode, just the mode of the four 00:18:20.550 --> 00:18:22.890 align:middle line:90% measures on either side of-- 00:18:22.890 --> 00:18:26.080 align:middle line:84% I think it was four measures-- on either side of a word. 00:18:26.080 --> 00:18:29.410 align:middle line:84% And there we found a very small correlation, 00:18:29.410 --> 00:18:32.670 align:middle line:84% but it was exactly the opposite of what we expected. 00:18:32.670 --> 00:18:37.890 align:middle line:84% In a local section, we use brown to represent minor in this case 00:18:37.890 --> 00:18:39.720 align:middle line:90% and blue to represent major. 00:18:39.720 --> 00:18:43.950 align:middle line:84% Joy and positive emotions tend to appear a little bit more 00:18:43.950 --> 00:18:46.263 align:middle line:90% in locally minor pieces. 00:18:46.263 --> 00:18:50.550 align:middle line:90% 00:18:50.550 --> 00:18:56.040 align:middle line:84% What it might be is that there's not this connotation of sadness 00:18:56.040 --> 00:19:00.750 align:middle line:84% or negative or anger with minor, but about not knowing 00:19:00.750 --> 00:19:02.610 align:middle line:90% where you are. 00:19:02.610 --> 00:19:06.450 align:middle line:84% So we looked at the tonal certainty. 00:19:06.450 --> 00:19:08.980 align:middle line:84% How clear is it what key we're in? 00:19:08.980 --> 00:19:13.060 align:middle line:84% And here we found some correlations that were negative. 00:19:13.060 --> 00:19:15.930 align:middle line:84% That is to say, anger and negative words 00:19:15.930 --> 00:19:21.810 align:middle line:84% tend to appear a little bit more in tonally uncertain passages. 00:19:21.810 --> 00:19:24.270 align:middle line:84% Now this isn't necessarily a guide to composers 00:19:24.270 --> 00:19:27.940 align:middle line:84% for what to do but what composers have done, 00:19:27.940 --> 00:19:30.383 align:middle line:84% what people have done in the past. 00:19:30.383 --> 00:19:33.150 align:middle line:90% 00:19:33.150 --> 00:19:37.650 align:middle line:84% So what did we learn by doing all of this? 00:19:37.650 --> 00:19:39.810 align:middle line:84% Words connoting joy and surprise are 00:19:39.810 --> 00:19:42.010 align:middle line:84% higher in pitch than their surrounding context, 00:19:42.010 --> 00:19:45.000 align:middle line:84% but negative connotations such as anger or disgust 00:19:45.000 --> 00:19:49.710 align:middle line:84% are not lower in pitch than in neutral were. 00:19:49.710 --> 00:19:53.350 align:middle line:84% Positive words are not highlighted with consonances. 00:19:53.350 --> 00:19:56.250 align:middle line:84% If anything, it's slightly the opposite. 00:19:56.250 --> 00:19:59.940 align:middle line:84% In one measure of dissonance, however, words connoting sadness 00:19:59.940 --> 00:20:02.670 align:middle line:84% are more common, but negative words 00:20:02.670 --> 00:20:07.290 align:middle line:84% appear more often in passages of tonal uncertainty. 00:20:07.290 --> 00:20:11.070 align:middle line:84% Finally, mode has almost no correlation 00:20:11.070 --> 00:20:13.770 align:middle line:84% with the emotion of words whether we're 00:20:13.770 --> 00:20:15.270 align:middle line:84% talking about the mode of the piece 00:20:15.270 --> 00:20:17.160 align:middle line:84% or whether the chord at that moment 00:20:17.160 --> 00:20:20.220 align:middle line:84% is major or minor though surprise and anticipation do 00:20:20.220 --> 00:20:21.785 align:middle line:90% get minor chords more often. 00:20:21.785 --> 00:20:24.840 align:middle line:90% 00:20:24.840 --> 00:20:32.145 align:middle line:84% It may be that this connection of minor with sad 00:20:32.145 --> 00:20:36.040 align:middle line:84% is one of these cases of a purely musical connection. 00:20:36.040 --> 00:20:38.190 align:middle line:84% That is to say we've learned to hear 00:20:38.190 --> 00:20:43.540 align:middle line:84% minor music as sad regardless of what the words are about. 00:20:43.540 --> 00:20:45.960 align:middle line:84% And this is something that, as a music historian 00:20:45.960 --> 00:20:49.930 align:middle line:84% I know hasn't been true all throughout music's history. 00:20:49.930 --> 00:20:53.880 align:middle line:84% There were times when a pure major chord was considered 00:20:53.880 --> 00:20:59.730 align:middle line:84% harsh or a little bit sad, and that's not the case today. 00:20:59.730 --> 00:21:02.790 align:middle line:84% But maybe when people do consistently 00:21:02.790 --> 00:21:06.750 align:middle line:84% report in music psychology surveys that they hear sadness 00:21:06.750 --> 00:21:10.680 align:middle line:84% in minor pieces that there's something happening that's not 00:21:10.680 --> 00:21:12.430 align:middle line:90% fully explained in the words. 00:21:12.430 --> 00:21:15.100 align:middle line:84% So there's always more research to happen. 00:21:15.100 --> 00:21:18.000 align:middle line:84% And I just want to thank you for allowing 00:21:18.000 --> 00:21:22.670 align:middle line:84% me to share what Sophia Sun and myself have found. 00:21:22.670 --> 00:21:30.000 align:middle line:90%