Binaural Model-Based Source Separation and Localization

When listening in noisy and reverberant environments, human listeners are able to focus on a particular sound of interest while ignoring interfering sounds. Computer listeners, however, can only perform highly constrained versions of this task. While automatic speech recognition systems and hearing aids work well in quiet conditions, source separation is necessary for them to be able to function in these challenging situations.

This dissertation introduces a system that separates more than two sound sources from reverberant, binaural mixtures based on the sources’ locations. Each source is modelled probabilistically using information about its interaural time and level differences at every frequency, with parameters learned using an expectation maximization (EM) algorithm. The system is therefore called Model-based EM Source Separation and Localization (MESSL). This EM algorithm alternates between refining its estimates of the model parameters (location) for each source and refining its estimates of the regions of the spectrogram dominated by each source. In addition to successfully separating sources, the algorithm estimates model parameters from a mixture that have direct psychoacoustic relevance and can usually only be measured for isolated sources. One of the key features enabling this separation is a novel probabilistic localization model that can be evaluated at individual time-frequency points and over arbitrarily-shaped regions of the spectrogram.

The localization performance of the systems introduced here is comparable to that of humans in both anechoic and reverberant conditions, with a 40% lower mean absolute error than four comparable algorithms. When target and masker sources are mixed at similar levels, MESSL’s separations have signal-to-distortion ratios 2.0 dB higher than four comparable separation algorithms and estimated speech quality 0.19 mean opinion score units higher. When target and masker sources are mixed anechoically at very different levels, MESSL’s performance is comparable to humans’, but in similar reverberant mixtures it only achieves 20–25% of human performance. While MESSL successfully rejects enough of the direct-path portion of the masking source in reverberant mixtures to improve energy-based signal-to-noise ratio results, it has difficulty rejecting enough reverberation to improve automatic speech recognition results significantly. This problem is shared by other comparable separation systems.

Download it as a single pdf (7.6 MB)

Download separate chapters as pdfs (some of the internal links don’t work):

• Front matter (142 KB)
• Chapter 1: Introduction (450 KB)
• Chapter 2: Literature review (979 KB)
• Chapter 3: Statistics of interaural parameters (2.5 MB)
• Chapter 4: Localization (1.8 MB)
• Chapter 5: Separation (1.9 MB)
• Chapter 6: Evaluation (335 KB)
• Chapter 7: Conclusion (74 KB)
• Bibliography (210 KB)

My project was called Bowie S-S-S-Similarities. I got all of David Bowie’s music (thanks, Brian!) and lined it up from beginning to end. Then I ran my autotaggers from Musically Intelligent Machines over every 10-second clip in it. Then I measured the similarity of the tags applied to every 10-second clip with every other 10-second clip, generating a big (5000×5000) similarity matrix. I originally wanted to use some sort of google maps interface to browse through this big matrix, but I couldn’t find one, so I wrote a python wrapper to let me browse through it and listen to the clips. Bowie seemed like a good artist to pick for this because he’s put out a lot of albums and they’ve been quite varied.

Some fun observations from playing around with it:

• The album Low was different from everything else, being one of Bowie’s ambient collaborations with Brian Eno.
• The track “Kingdom Come” from the album Scary Monsters is the most self-similar track and is also the most similar to Bowie’s other tracks.
• Using a similarity based only on genre tags makes a pretty different similarity matrix than using a similarity based on tags related to vocals.

The audience liked the hack and the presentation enough to get me a copy of Rock Band. Check out a picture of me and Frank, who also won a copy of Rock Band for his hack PartyLister, on Frank’s post.

After getting home, I read Kurt’s blog post about his hack, which used Microsoft’s Seadragon to zoom and pan around a giant visualization of artist similarity. I thought that was pretty awesome, and as it was what I had wanted to do for my hack, I made a Seadragon viewer for the Bowie similarity matrix. I got it working most of the way with the seadragon.py script to chop up the big image into a pyramid of tiles and the Seadragon AJAX code to display it. But I couldn’t get that to work on the blog here, so I used the online Seadragon builder.

Note that the seadragon version is using more music than my original hack (9000 clips), which should be all of Bowie’s studio albums. I’d like to add the ability to see which clips you’re mousing over and to play them, but I haven’t had time. I would also like to get the image to be in color, but I haven’t been able to get the Python Imaging Library to behave with numpy’s colormaps. And finally, it would be nice to make the similarity steerable so that you can seamlessly switch between different types of similarity or even different weights on different tags.

We introduce the Ideal Interaural Parameter Mask as an upper bound on the performance of mask-based source separation algorithms that are based on the differences between signals from two microphones or ears. With two additions to our Model-based EM Source Separation and Localization system, its performance approaches that of the IIPM upper bound to within 0.9~dB. These additions battle the effects of reverberation by absorbing reverberant energy and by forcing the ILD estimate to be larger than it might otherwise be. An oracle reliability measure was also added, in the hope that estimating parameters from more reliable regions of the spectrogram would improve separation, but it was not consistently useful.

I finally cleared my photo backlog, which went back to December, and uploaded them to flickr. There are more than will fit in the sidebar, so check out the rest of my concert photos and random phone photos there.

I also added a LibraryThing widget, which shows the last N books I’ve read. I haven’t been doing much extracurricular reading in the last month or two, but I haven’t told you about the ones before that, so they’re new to you.

I’ve been using citeulike for a while, and it’s great for scraping IEEE and JASA abstracts. It can import bibtex files, but they’re harder to get linked in to pdfs and you might end up with a lot of duplicates if you’re not careful. On Neeraj’s recommendation, I’ve been trying out mendeley and it’s a bit cooler. It can read in a directory of pdfs and figure out to some extent what they are. This is most useful with popular papers, because they have some sort of fingerprinter that recognizes the same pdf from multiple users and matches up the metadata. That way, only one person has to correct each entry and others can benefit. I’m not sure if it’s been able to recognize when a pdf is the same as a bibtex entry, but it might be able to. They also seem to have a very responsive feedback system using UserVoice. And of course there’s always google scholar to actually find these papers.

But, I think these apps could be a lot more useful. Instead of just linking to the papers that cite a paper, a lot could be gained by keeping track of the “anchor text” that does the linking. This means not only noting that paper X cites paper Y, but that paper X describes paper Y in this way, uses this information from it, cites it in this context, etc.

The first thing that this would enable would be the annotation of a paper’s bibliography with the relevant parts of its text. These are all of the outgoing links from a paper. By analyzing the paper, all of the [22]s or the (Mojo, 1987)s could be associated with the right entry in the bibliography. It would give the references some much-needed context. It would also show which references in a bibliography were actually discussed and which were just mentioned in passing. This could be an application by itself. And while we’re linking the references to the bibliography, it could put in some hyperlinks, like the hyperref package in latex does, but after the fact.

The second thing that it would enable would be the annotation of a paper with all of the things that other papers have said about it. These would be all of the incoming links from other papers. It would give you some context on a new paper that you had just come across in addition to what the authors wrote in the abstract. If you wanted to be really fancy, each reader could have trusted sources of these incoming links. These opinions are like little mini reviews or summaries that have already been published, no need to solicit readers’ opinions. Instead of the first few sentences on a “papers that cite X” page on google scholar, you’d get a page of summaries, reviews, and extracts.

Both of these features would make it much easier to get introduced to a new field or to write a more balanced review of a familiar or semi-familiar field. I know it’s tough to match up bibliography entries with references and with papers themselves, and that there are some user interface issues to work out here, but it shouldn’t be that hard. Maybe crowdsourcing could help if necessary. Hopefully all of this would help allay that niggling fear of missing an important paper.

While I enjoyed it, the book seemed a bit padded at times. There were tangential tables that took up multiple pages and the epilogue, an account of the lucky occurrences throughout Gladwell’s family tree, was a nice anecdote but didn’t really bring any more support to the thesis. A couple of the citations came from wikipedia. Certainly not worth the $29 list price, or even the $20 discount price, I’d recommend waiting for the paperback edition or a secondhand copy. There were, however, a number of choice Gladwellian factoids, which I will relate.

In the first half of the book, there were some interesting anecdotes about the lucky breaks that Bill Gates and Bill Joy got on their ways to the top, but the most interesting idea was the fact that there are certain birth months and years that are better than others for success in various fields. The first example of this is in sports, where a national birthday cutoff for kids leads to a disproportionate number of the best adult athletes being born just after that cutoff. The explanation is that the kids who are the oldest for their age group are the biggest and best and they get put on traveling teams, get more practice, play more, and get better coaching, which eventually leads to a significant advantage when they grow up. In English football, the cutoff is September 1st and, at one point in the mid 90s, the Premier League had twice as many players born in the three months after the cutoff than the three before it (nature article). This is apparently also true in elementary school where older 4th graders score better on math tests than younger classmates. This advantage even continues through college, where “students belonging to the relatively youngest group in their class are under-represented by about 11.6 percent.”

Gladwell also describes two cases where birth year gave people legs up. The first was in the software industry. The titans of which were disproportionately born in 1954 and 1955. Gladwell’s argument is that these people were 20 or 21 in 1975 when the Altaire 8800, the first personally-attainable computer, was released. The second is in the lawyers specializing in hostile takeovers in the 1970s, who were disproportionately born during the great depression in the 1930s when birth rates dropped significantly. This gave them better access to schools, colleges, and law schools, which had just been expanded to accommodate the previous generation.

The second half of the book focused on the measurable effects of differences between cultures. In an interesting, but less convincing argument, Gladwell claims that rice cultivation in paddies leads to more entrepreneurial farmers, while wheat and corn cultivation lead to stronger feudal hierarchies. Apparently rice cultivation is quite a tricky endeavor and yields are increased by leveling the ground in the paddy, maintaining the correct water level, using the right combination of rice strains, weeding thoroughly, and fertilizing properly. Rice landlords charged fixed rent, allowing rice farmers to profit from larger harvests while wheat landlords payed fixed wages regardless of yield.

Chinese rice farmers were able to grow rice all year round, harvesting and planting new seedlings two or three times a year. French peasants, on the other hand, planted in the spring, harvested in the fall, and hibernated through the winter. Rice paddies, furthermore, are enriched by nutrients in the irrigation and can be used continuously. Wheat and corn fields, on the other hand, are exhausted by agriculture and need to lie fallow every few years to recover. Gladwell suggests that this difference in farming practices led to opposing cultural analogies for human mental growth, and to differences national school schedules: the American school year is on average 180 days long, while the Japanese school year is 243 days long.

His slightly more convincing argument in this section was about plane crashes. Apparently, plane crashes are generally caused by the compounding of a number of small errors, a condition that is best mitigated by sharing responsibilities between the captain and the first officer. In cultures that have a great deal of respect for authority, such as Korea, the deference the first officers showed to captains tended to cause more crashes. Between 1988 and 1998, Korean Airlines lost 4.79 planes in accidents for every million departures. Compare that to United Airlines, which in the same period lost 0.27 planes in accidents for every million departures. By training its first officers to be more assertive when they noticed a problem, Korean Air has gotten these numbers in line with other carriers. An IBM psychology Geert Hofstede surveyed employees around the globe and used their answers to assemble a set of dimensions for measuring how cultures differ from each other, now known as Hofstede’s dimensions. Korea is apparently second from the top of Hofstede’s list in deference towards authority.

Overall it was a fun read, but I think I’m a bigger fan of Gladwell’s shorter writings.

It was always clear to me that if you can get new ground truth data, you can do cool new things, but it took a while for it to sink in that you really can’t develop a system to solve a problem without having the problem already solved, in some sense. Of course, the power of machine learning comes from being able extrapolate results from a small subset of labeled data to an infinite amount of as-yet-unlabeled data. I can develop music classifiers (and pick the one that does best) using a small set of already-classified music and then use it to classify as much music as I want. The question is, is that as-yet-unlabeled data really that similar to the test set? When you have enough data to know the answer to that question, you probably have enough data to do pretty well with a basic classifier.

As an aside, I’m always highly doubtful of claims that computers can latch onto things beyond human perception. For watermarking, sure, it’s designed so that machines can perceive it, but people can’t. But when it comes to very human-grounded ideas like similarity, I think it is impossible to try to circumvent human “subjectivity”. There really is no objective measure of whether two sounds are similar besides the consistency in subjective ratings of human listeners. I think much of the trick of developing (provably) useful algorithms is defining problems that have objective solutions and then solving them objectively.