Spoken Language Processing

States of the Arts

noreply@blogger.com (Anonymous) — Tue, 25 Oct 2016 19:26:00 +0000

Fall 2016 has seen important improvements to the state of the art in both speech synthesis and speech recognition.

In September, Google DeepMind unveiled WaveNet, a speech synthesis system, that generates exceptionally natural sounding speech. In October, Microsoft Research announced that they had developed a speech recognition system that matches the word error rate of human transcribers.

A few observations.

Life moves pretty fast.

It’s a time of rapid progress in speech and spoken language processing. Microsoft, IBM and Baidu have all posted better and better speech recognition numbers in the last few years.

Deep Learning has the goods.

It’s very easy to be dismissive of “deep learning” as being over-hyped. However, both of these advances rely heavily on deep neural networks. So far, they continue to deliver on their promise.

Arxiv.

One of the first important ASR papers showing that DNNs can outperform traditional GMM acoustic models on a hard task (i.e. Switchboard) was presented at Interspeech 2011. This means the work was done at least 6 months earlier. Both of these advances are described not only by press releases and glossy webpages, but also technical papers [WaveNet paper, MSR paper]. Both were posted to arxiv. There’s no doubt that immediate, self-publication is flooding the scientific engine with oxygen. Progress is rapid because we’re still learning the limits of neural networks, but also groups are able to compete and learn from each other much more quickly than semi-annual conferences enable.

WaveNet is a new way of doing things.

WaveNet synthesizes speech in a novel way. The resulting waveform is generated one sample at a time, conditioned on the previous sample. This is essentially doing parametric speech synthesis without a vocoder. Not only does this approach work surprisingly well, it’s exciting in its newness as well. There’s other novelty in this work too (1. Using a classification approach to predict discretized mu-law values instead of predicting a continuous value 2. The dilating convolution layer.) but this work is most important for showing the promise of this approach to generating audio data.

The Microsoft ASR work is not a new way of doing things.

The work is of the highest quality, without a doubt. This paper represents a exeptionally well-engineered collection of effective speech recognition tools that have hit an interesting milestone. However, the individual pieces will seem familiar to anyone up to date on the current state-of-the-art. The major improvements come improvements to language modeling (an LSTM LM specifically), LACE units which MSR showed at Interspeech this year, lattice free MMI, and spatial smoothing (which is similar to Cambridge’s “stimulated training”). The Microsoft team has put these parts together more effectively than anyone else, and it’s an important achievement. But compared to the WaveNet development, it’s a more incremental step.

Interspeech 2014 Recap

noreply@blogger.com (Anonymous) — Mon, 13 Oct 2014 20:56:00 +0000

This year's Interspeech was in Singapore. Singapore is, in some ways, a very easy venue to travel to. It's a modern, cosmopolitan city. They speak English. It's tropical, but you're never more than a hundred meters from air conditioning. In other ways, it's so far. Over 20 hours each way. I like airplanes. They're as magical as any technology we've got. But 20 hours is a long time to sit still. Think about how many steps you take in 20 hours. How many different faces you see. Then reduce that to about 200 steps, and 10 people.

"Because we are all poets or babies in the middle of the night, struggling with being." - Martin Amis "London Fields"

Interspeech 2014 was a well run conference. The quality of papers was generally quite high. The venue easily handled the size of the event and the wifi was steady. It was difficult to find enough food at the Welcome Reception, but easy to find enough beer. The banquet was flawed -- segregating vegetarians is pretty rude -- but they all are, and at least there was enough food to go around, and everyone ate promptly. And of course, there was Mambo and Jambo. I'm not going to go into it here, but find someone who attended the opening ceremony and ask them to describe it. Then don't believe them, and ask someone else to do the same. It was "odd" at best. But be sure, I'll be attending the Dresden opening ceremony to see how they one-up it.

Deep Learning
A few years ago, DNNs invaded speech conferences. Deep Learning is still a significant buzz word, and a hot topic. But for, the better of everyone involved, the intensity has cooled. Now the interest in DNNs seems to have shifted into 1) understanding how they work, and how to best train them to a task and 2) Long Short-Term Memory units to model sequential data. The latter really broke out at this years conference. There were a number of papers finding that them to be an effective alternative to traditional recurrent nets trained with back-propagation through time.

BABEL
I've been involved with the IARPA-BABEL program, so my view is pretty biased on this front, but I felt like the presence of BABEL in this year's Interspeech was particularly large. The program's central task is performing keyword search on low-resource languages. It has an aggressive evaluation schedule with an increased number of *new* languages involved each year. There were at least two sessions devoted to keyword search, and papers evaluated by either BABEL-proper or the NIST OpenKWS challenge seemed to be all over the conference. (Searching the paper index suggests that there are between 30 and 40 BABEL papers, and another 10 or so OpenKWS papers.) It seems clear that this program has had a large impact on ASR and KWS research. 2014 was certainly the high-water mark here, as the program shrunk by 50% last year, but it's worth noting its effect on the field.

Some standout papers

I don't mean to suggest that these are "the best" papers, but they're ones that caught my eye for one reason or another.

Acoustic Modeling with Deep Neural Networks Using Raw Time Signal for LVCSR by Zoltán Tüske, Pavel Golik, Ralf Schlüter and Hermann Ney. Part of the promise of Deep Learning is the ability to "learn feature representations" directly from data. This is frequently touted as a description of what is happening in the first hidden layer of a deep net. So, the logic goes, do we need MFCC/PLP/etc. features, or can we do speech recognition directly on a raw acoustic signal? This is the first paper I'm aware of to affirmatively show that "yes, yes we can". It requires a good amount of training data, and rectified linear (ReLU) neurons work better for this, but 1) it works competitively with traditional features and 2) many of the first hidden layer neurons can be shown to be learning a filterbank. Very cool.

Backoff Inspired Features for Maximum Entropy Language Models by Fadi Biadsy, Keith Hall, Pedro Moreno and Brian Roark. In n-gram language modeling, when a sequence of words A, B, C have never been observed, its n-gram probability P(C|A,B) can be approximated by the probability P(C|B). But to be sensible about it, you've got to apply a backoff penalty. Discriminative language models can seamlessly incorporate backoff features, F(A,B,C), F(B,C), etc. and learn appropriate weights. The key insight in this paper is that when a discriminative model uses these backoff estimates, it incurs no penalty. It's essentially overestimating the probability of uncommon n-grams in the context of common (n-k)-grams. This paper seeks to fix this, and does.

Additional favorites some from my students:

Word Embeddings for Speech Recognition by Samy Bengio and Georg Heigold. Far and away the most popular poster at the conference. This is high on my list to read closely. It promises to learn a euclidean space into which word decoding can happen, so that words that sound similar are closer in space.

The Obligatory Contour Principle in African and European Varieties of French by Mathieu Avanzi, Guri Bordal and Gélase Nimbona. An investigation of prosodic differences in dialects of French. Very consistent with one of my student's dissertation projects.

Improving Spoken Document Retrieval by Unsupervised Language Model Adaptation Using Utterance-Based Web Search by Robert Herms, Marc Ritter, Thomas Wilhelm-Stein, Maximilian Eibl. A clever way of handling OOVs in spoken document retrieval. Essentially the idea is if I recognize W[i-2], W[i-1], UNK, W[i+1], W[i+2], go do a web search for the context, find some matching documents, and augment the language model with them, then re-decode. Kind of like distant supervision for spoken document IR.

Learning Small-Size DNN with Output-Distribution-Based Criteria by Jinyu Li, Rui Zhao, Jui-Ting Huang, Yifan Gong. How do you effectively train a small DNN without ruining its performance? This paper out of MSR suggests training a large DNN, then using its output distribution to train the small one. It'll take me some closer reading to fully understand why this works, but I'm intrigued.

Canonical Correlation Analysis and Local Fisher Discriminant Analysis based Multi-View Acoustic Feature Reduction for Physical Load Prediction by Heysem Kaya, Tuğçe Özkaptan, Albert Ali Salah, Sadık Fikret Gürgen. We tried a number of dimensionality reduction approaches for this year's Paralinguistic Challenge, but didn't get particularly good results. These guys did using CCA and LFDA. Looking forward to reading this one as a more general feature reduction approach when the number of features are larger than the number of training instances.

Things I didn't know before becoming a professor: #5 How to manage my time

noreply@blogger.com (Anonymous) — Tue, 12 Aug 2014 20:55:00 +0000

Grad school is an exercise in working independently.

What I mean by working independently is this: Your daily activities are barely supervised. Your long term activities are heavily scrutinized and harshly judged.

You must master this skill in order to finish a PhD. You must have a PhD to be a professor. So.... You'll be all set. Right?

This covers a lot of careers, not just academics, but being a professor provides an extremely lightly constrained schedule. On one hand, in this, grad school prepared me for the challenge of controlling my own time. On the other, it gave me a false sense of confidence. I was driving so well in the parking lot, and then I pulled into midtown traffic at rush hour. And I'm driving a go-kart.

In grad school you learn how to motivate yourself. You can read another paper, or you can watch another episode of The Sopranos. You can write up an experiment or you can play one more hour of Halo. Or you can just go to the beach. Once you're done with classes, there're no midterms coming up in a month. No papers due in two weeks. Just an uncomfortable meeting with your advisor, where you can make small talk and rehash previous discussions and results until the meeting is over and you think you've played it off. (You haven't, btw.) Most people figure this out. There is a wealth of advice about how to get off your rear and get work done. There are theories and apps and support groups and better blogs than this devoted to the subject. So I'm going to assume that even the most procrastinative of professors (my wife lovingly refers to me as an "epic time waster") have a set of tools at their disposal to handle the inevitable distractions and failings of motivation.

But here's where it goes haywire. The breadth and depth of things that need to be handled on a given day is far greater than what was expected in grad school. This isn't revelatory. Its to be expected; its much more responsibility. The diversity of responsibility is as surprising as anything else. Being a professor requires you to teach, write grants, mentor grad students, handle bureaucracies (internal to the university and externally), manage budgets, foster relationships with a broader scientific community, attend conferences, project meetings and site visits, and do some research. Its not too much; its very doable. But its much more faceted than grad student responsibilities ready you for. As a grad student, your responsibility is typically to do your research, and occasionally teach or take classes. There is relatively little necessary by way of prioritizing which of many projects and tasks need your attention.

I see the challenge in transitioning from student to professor like this: The motivational chops you honed during grad school have made your job as entertaining and rewarding as TV, video games and the beach. Now, you need to repurpose them so that the frustrating parts of the job are as entertaining and rewarding as fun parts.

Everyday I make two lists. One: things I want to do. Two: things I don't want to do. Things I want to do usually includes reading papers, and writing code, some errands, grant ideas, sometimes course planning. Things I don't want to do are typically bureaucratic. Plus I've got a rule that if I've put the same task on the "want" list for three days running, my attitude has outed itself and its actually a "don't want" kind of thing. Through the day, I go one for me, one for them. One from column a and one from column b. And everyday there are a couple of things that I don't want to do, that I've found some way to avoid doing.

I've tried a lot of approaches to bring order to the responsibilities that being a professor entails, the most basic lessons I've learned are these.

1. Write everything down. I've got a decent memory, but there's no way that I can keep my schedule and the status of each students project and other milestones and what I was thinking about yesterday at hand without a lot of notes.

2. Let go. Days are not long enough. When I go to bed, there is always something left undone. Make sure that it's not something truly important.

3. Do a little bit of soul-sucking work first thing in the morning every day. This keeps it from building up leading to dreaded days that are totally eaten up by it. Spending a full day on bureaucratic maintenance to dig out is something that (I sincerely hope) no grad student ever has to grapple with it. I treat this like the nuclear option; it's a clear sign that I've made a bunch of time management mistakes in the previous week/month.

4. Keep on top of email. Even when traveling or facing down a deadline. (See above.)

One thing I'm still trying to get my arms around is managing my time for priorities of different durations. Handling tasks for the day or week is pretty easy. Pegging against milestones like paper and grant deadlines, presentations at conferences or grant meetings, is pretty easy. It's harder for me too keep a longer perspective and give enough weight to the task that's not needed in 2 weeks, but one that might have impact in 2 months or 2 years. This medium- to long-view is harder to maintain and harder to integrate into daily priorities.

Get a little better every day.

Things I didn't know before becoming a professor: #4 Different types of collaboration

noreply@blogger.com (Anonymous) — Tue, 08 Jul 2014 05:28:00 +0000

There's a romantic and resilient fantasy of the scientist toiling away in isolation. He (sad to say, it's almost always he) emerges from the lab, bleary-eyed and triumphant and shares his genius with the world.

Da Vinci, Frankenstein, Edison, Einstein, Every Evil Genius, Zuckerberg

Our science narrative has expanded to pairs of geeks in garages

Hewlett & Packard, Gates & Allen, Jobs & Wozniak, Brin & Page

but hasn't yet embraced the fundamental reality of modern science: most work is done in collaboration. Most great big ideas are the product of many great small ideas.

This is why there's no satisfying answer to questions like "who invented the atomic bomb". Was it Fermi? Einstein? Oppenheimer? Of course Al Gore didn't invent The Internet, but who did?

As a graduate student, you are, at worst, Igor, the hunchbacked assistant doing all the work and getting none of the credit. At best, you are a near-equal collaborator with your advisor. But there's a sea of collaboration waiting once you're out of the nest and on to the next step.

Since becoming a professor I've noticed a handful of types of collaboration. From intimate to remote, they each have a role to play.

Peer Collaboration
This can be the most satisfying and most productive way to work with a partner or team. You sit in the same room, or with a chat window open. You talk through ideas, big and small. You tackle the parts of the work you're most suited to, or most excited by. You debug code together. You run experiments together. Try to make sense of your shared results. You iterate over writing papers together.

It can also be the most frustrating. A group only walks as fast as its slowest member. If you don't work well with your partners, for any reason, the stress is visible almost immediately. If the partnership is strong, you figure it out, and get back to work. If it's not, you'll know pretty quickly.

Mentor-Student Collaboration
There's a similar intimacy in this collaboration, but there are some very clear differences. First the pace of collaboration is typically slower. At slowest, you may only work with your student only once a week. Even if you're in regular email contact it's only a couple of times a day. Typically it's only during the most intense periods of an evaluation or paper deadline does the collaboration reach the level of immediacy and proximity of peer collaboration.

The division of labor here is more driven by the hierarchical nature of the relationship. The mentor typically has a broader vision of the work, while the student has a better grasp of the details. (Or if they don't, it's part of the responsibility and training to learn about the details.) The mentor has the responsibility of contextualizing the work, either into a broader research agenda that will include the students thesis as an offshoot, or into a broader project or publication. At the start of the relationship, the mentor is typically responsible for guiding the research, determining which research questions are most fruitful and

Over the course of graduate study, a good mentor-student collaboration will take on more characteristics of a peer collaboration, with the student ultimately initiating most of the research ideas. But there's always going to be a level of deference that the mentor receives.

-----

As a graduate student, I had a lot of experience with the previous two types of collaboration. I even had some exposure to the next three, but so much as to be able to identify the differences, or how they impact the work.

-----

Project Collaboration

On medium-to-large multifaceted projects there are frequently multiple PIs (principal investigators) each with independent research groups. Collaboration across sites towards a common goal is different and challenging. The PIs coordinate with some frequency. There are bureaucratic reasons for this -- mostly around budgets and report writing -- but coordination between students is much less common. Each of the individual research groups have their own broad responsibilities, timeframes and agendas. Each are balancing a number of projects, student milestones (qualifying exams, etc.), and specific personnel decisions.

The biggest difference in this kind of collaboration, in my experience, is the pace. The interaction between groups when it comes to research is infrequent. Sometimes just a few times a month. Another difference is in the level of detail. When the groups share their work, it is usually at a relatively high level -- far removed from unsuccessful approaches and interesting failures. There are opportunities to get advice at a high level about directions and plans, but close collaboration about specific decisions and approaches is much less common.

This is completely understandable. The nature of this collaboration demands it. These projects are big enough to demand a large number of people working on them. This task oriented partitioning into research groups is probably the best way to manage the effort.

However, there's the time when the rubber meets the road. This is usually in the form of deliverables, where you have to send a report or code to a funder, evaluations, where all of the moving parts that the groups have developed have to work together, or presentations at site visits or PI meetings, where you have to show that you're all working towards a common goal.

Often, these moments demand closer coordination and collaboration. Having made a few mistakes. The lesson I've learned is to stop and drill down. Take meetings offline to deal with discrete questions and issues. Get the students involved in these meetings. If at all possible, visit your partners. With your students. One hour face to face can take the place of a few days worth of emails.

Social Collaboration
Still another step more removed is the kind of collaboration you have with the scientific community which you're a part of. This is the sort of collaboration where you have a conversation with someone about an idea that one of you is chewing on. You bat the idea around for a little while, maybe hear about a technique you didn't know about or hadn't thought of, and then the conversation moves on. I've found that this kind of collaboration occurs with someone you know. Either you know each other's work, or you have a mutual colleague to bring you together.

Most of the time this type one-off micro-collaboration ends with a single conversation. Sometimes it results in shared data or tools. But given the right alignment of circumstances it can foster a more serious coming together. Usually just an extended conversation over email, or a phone/skype conversation, but if fate is really on your side, this can grow into a paper or a grant proposal. But by this point, you've switched categories and you're working much more closely.

I find that this collaboration is most available during visits to other institutions and at conferences. Frankly, this kind of collaboration is often the most valuable thing I take away from conferences.

"Public" Collaboration
If you're doing it right, people are going to know about your work. People will email you with questions and comments. You will get queries about how to use a tool you wrote, or (maybe more frequently) a bug report or feature request. Through this communication you are supporting other people's research. This rarely turns into a more substantial collaboration. But if it means that people will cite your work.

Here, the collaboration is somewhat lopsided. Essentially your prior effort of writing a paper or tool are supporting someone's current research. But this needs some additional support from your current self, either in clarification or amendment. The upside is that if the system is working, you can garner this support as much as you give it out.

Here, I find the challenge is to appropriately prioritize these interactions. More frequently than I'd like to admit, these are the emails that filter to the bottom of my inbox.

If you're reading this, and I still haven't gotten back to you, I'm sorry. I know these interactions are important. I'm working on it.

Things I didn't know before becoming a professor: #3 How to run a research group

noreply@blogger.com (Anonymous) — Wed, 21 May 2014 17:03:00 +0000

At this point, I've figured out my classes and secured a little bit of funding. Now it's time to staff up the group and get to the work of research.

Every researcher I know keeps a list of open questions and tasks that they want to get to next. This list is a best friend, and a worst enemy. This list is a hydra. Every time an element is taken off, two more appear in its place. To tame the beast, it takes a group effort.

It's no revelation to recognize that management is different from labor. I'm sure if I had an MBA instead of (in addition to?!) a PhD, some of these issues would be more intuitive. But I didn't, and most professors don't. We're self-taught.

When the research group is firing on all cylinders, it feels like my work is having a multiplicative effect, that I'm lifting more than I possibly could alone. When it's not, it's demoralizing. I try not to get sucked into this, but there are days when it feels like I could do more in a cave with a laptop, an internet connection and coffee.

These are some of the things about running a group that I didn't know before becoming a professor.

How to organize a research agenda

At my thesis proposal, Kathy McKeown asked me the hardest question: "What do you want to be famous for?" It totally blindsided me. I laughed uncomfortably, and looked to my other committee members for acknowledgement that this was an outrageous proposition. No help was coming. I have no idea what my answer was, but it's the only question I remember from my candidacy exam, proposal or defense.

A clear answer for this question, an organizing principle for the research that your lab will do, brings clarity and identity to the group. It represents a research agenda that guides the work. A clear research agenda serves a similar role as a mission statement and business plan. It communicates your goals internally and externally.

This also helps with recruitment -- if students and postdocs know exactly what kinds of questions you are interested in, you will attract people who are interested in these questions, and people who are capable of working on them. The same principle applies to students who are looking to do independent studies, undergraduate or master's theses with you.

I've generally under-appreciated the value of this focus. Personally, I've benefitted from broad research interests. I get a lot of satisfaction by working on a lot of different things. My most cited paper wasn't even cited in my dissertation. But I think I've also suffered from this generalist approach. I may have had greater impact on the questions I find most interesting had I limited my activities on the second tier. I may have made more progress in getting "famous" for something.

How to engender community and collaboration

Being a graduate student can be isolating. There are some classes. A weekly meeting or two that you need to show up for. But most of your work is alone.

How do you take a group of students who are organized only by you, the professor, and help them communicate, support each other, and become colleagues? I certainly had no idea.

Some things I've tried:

Taking everyone to lunch. this didn't work so well. it probably would work now, but at the time the students didn't know each other well enough and it was just pretty awkward.

Shared tasks. Inviting a group of students to work together on a short term project is the single best thing I've done to get students to trust each other and exercise their basic research skills. I use the interspeech paralinguistics challenges for these. I also invite other computational linguistics students to participate from the linguistics and cs programs. It's a really fun exercise and the students get a lot out of it.

Weekly lab meetings. One student gives a talk every week. It gives a safe space for students to practice giving presentations (and giving feedback on presentations), and it makes sure that everyone knows what everyone else is working on.

How to make and manage a budget

This is my least favorite part of the job. Making a budget is fine. It's just making puzzle pieces fit. But inevitably there is some wrinkle to sticking to it. Travel costs are more or less than budgeted. At CUNY tuition is highly variable based on what year a student is and how many courses they are taking.

And then there is interacting with admins, grants, financial officers and contract lawyers. These people are invaluable. When the relationship is good, your life is easy. When there are delays it can make sticking to the timing of a budget very very very difficult.

Having an eye for detail and the patience to use it really comes into play here.

Not everyone knows what I know

The next three observations are less about specific skills, and more about the personal effects of leading a group.

As a new professor with new graduate students, you know more than any other research group member. This isn't a "smartest person in the room" thing (though sometimes it is), but an experience thing. The professor has read more papers, seen more talks, (may have executed more experiments) and knows more about the topic.

The list of things that you want to work on next are based on your previous work, your thinking and your instincts.

In order for a student to pick up and work on a task, they need to get up to speed, and fast.

It is too easy to forget that the people working with me don't know what I know.

How to delegate

It became clear to me early on that not all CS professors still program. This struck me as a scary idea. First of all, I like to program, and I'm reasonably good at it. Secondly, it seems like distance between a professor and the nuts and bolts of research is a dangerous thing.

I was talking with Dan Ellis shortly after I had realized this. (Nevermind that Dan is a EE professor.) I asked him if he still coded himself. He had just finished a project over the summer and was happy with it and the process, but then he described it as "indulgent". why "indulgent"? because, according to Dan, he should be working on writing grants, papers and reports, executing the broad vision of the lab, while the students benefit from the programming experience. (Dan, if you read this, please forgive mistakes by paraphrase, and the fog of memory.)

The responsibility to delegate work to students is one that I haven't quite internalized. I still run a lot of experiments myself. I write a good deal of code still. And when we're under close deadlines, I have a tendency to tell students to walk away, and handle the final push myself. I'm not yet prepared to call this "indulgent", but it I'm ready to acknowledge it as a bad habit.

How to be a mentor

Internalizing the role of "mentor" is much more difficult than "teacher". Teaching is something that you do a few times a week, but it's a hat you take off. Mentoring, or advising graduate students, is an ongoing process.

I'm the most visible example of what a professor is for my students. How I do this job impacts how they will. (Just as my advisor is my first and best example that I draw from when advising students.)

My behavior as a mentor impacts how they'll behave as students. If I respond to emails at 1am, I will get emails at 1am. If I don't communicate what I expect from them, they won't know.

In addition to my advisor, I've had a number of important mentors. There are two lessons I've taken from those relationships.

Protect your advisees. There are plenty of external challenges -- defenses, negative paper reviews, job postings, evaluations -- that a student will participate in. They do most of the work. I feel like it's my job to make sure they're set up for success. (Only submit papers that you would accept. Don't volunteer them for too many responsibilities.) When things blow up, absorb the blow. Outwardly, take responsibility for mistakes, figure it out with the student internally. If the students feel safe, they'll be enabled to do good work.

Be a happy warrior. There are parts of the job that are difficult. It takes long hours and drive. And you're in this job for a reason. You want to discover, and share those discoveries. You want to contribute to Knowledge. As a student, I fed on the enthusiasm of my mentors. While it's easy and appropriate to feel a kinship with students, I feel a responsibility to set a tone that this is valuable, but also this is fun. And it is.

Things I didn't know before becoming a professor: #2 How to Write a Grant

noreply@blogger.com (Anonymous) — Mon, 05 May 2014 19:38:00 +0000

To get tenure in computer science, you have to secure external funding.

I don't know how important this is in other disciplines. I'm sure there are administrations that will say that external funding is only one of a number of criteria that determine tenure decisions. I'm sure they're right. But without funding, you're at a severe disadvantage. You can't work with grad students (as easily -- some programs provide student funding through other means). Travel to conferences -- where you present your work and solidify relationships with important people in the field (who may end up writing letters in support of your promotion) -- is expensive. And then there's the brass tacks. Depending on institution and some uninteresting details, the administration receives somewhere in the ballpark of 1/3 to 1/2 of the funds that you secure.

Before becoming a professor, I hadn't written a grant proposal. I had read a couple of successful proposals that I had worked under as a student, but I hadn't been a part of the preparation of any.

A little bit of context before I share what little I've learned about writing grants.

I've been pretty successful in securing funding over the last five years.

I don't expect this trend to continue forever.

I recently received an NSF CAREER award.

But my previous two proposals were not funded.

I'm hardly an expert, but I've gotten better since starting.

Here are a few things I didn't know

How many funding opportunities there are?
I knew about NSF and NIH. I knew about DARPA because I worked on a grant as a student. I didn't know how many programs NSF has (there's no way I know about all of them). I didn't know about the other DoD agencies that fund basic research. I still don't know about all of the foundations and industrial grants and awards that are available. I'm sure I'm still missing some, but each of IBM, Google, Microsoft offer programs to support faculty. On top of that, there are institutional awards -- CUNY has a handful of them, and if we do, I'm sure everyone else does. This doesn't mean that it's easy to find funding. But by keeping an open eye, asking around, and being lucky enough to be asked to join grant writing efforts, I've been surprised by how many opportunities are available (in this field).

How to tell the difference between a good proposal and a not-good-enough proposal
Friends and colleagues will offer to share successful grants with you. Take them up on it. But learning from only positive examples is challenging. It can be challenging to figure out how to translate structure and presentation cues from one grant to another. Also, it's not always clear what made these examples impress the reviewers. If you're so bold, ask these people to share the reviews along with the grant. I haven't tried this, and I'd have to trust a person quite a bit to share my highlighted flaws with them, but give it a shot.

Still better is getting to look at unsuccessful proposals and their associated reviews. Everyone you know has one or two of these kicking around. Funding rates are pretty low, say 20% or so, so most people will have a stack of failures for each success they have. But again, it takes moxie to ask someone for this, and a lot of trust to offer it. (I don't know if this happens ever, but I'd love to hear if other junior faculty were able to read unfunded proposals as they started writing themselves.)

The best way to see a lot of examples of grants, both positive and negative, and their reviews is to get on a grant reviewing panel. NSF Program Managers like to include junior faculty in their panels. Call one who funds work like yours, and have a conversation. I was invited to one in Spring 2010. It was an opportunity to closely read about 5-6 proposals, and more casually review another 12 or so. Seeing all the ways a basically good idea could not get funded was eye-opening. Generally, the quality is high, so the differentiation between successful and unsuccessful proposals can be the depth of their weaknesses rather than the height of their strengths.

That writing a grant is writing science fiction
It's believable, near-future science fiction, but still in the genre. You're describing something that doesn't exist, but will in the next few years.

This perspective has made the process of grant a lot more entertaining.

Science writing has a narrative arc. The introduction and motivation of your work should be exciting. The feeling I want to leave a reader with is somewhere between, "Of course that's what will happen", and "Wouldn't it be cool if that's what the world was like." Balancing these two extremes is the difference between the Scylla and Charybdis of incrementality ("boring") and overreaching ("i don't believe you can do this").

How to keep it together after getting a grant rejected
It's only natural to be offended, embarrassed, insulted, frustrated, and sad when a proposal that you spent months of your life on is not funded. The panel was clearly full of idiots who didn't understand your genius. There was some mistake, if only they had read more closely they would understand how important this work is.

Here's what rejection has taught me:

Take a deep breath.

Most proposals aren't funded.

The folks who review your grants are almost universally qualified to do so.

If a proposal is funded, you are not a genius. If a proposal is not funded, you are not an idiot.

The sooner you get over it, the better. Remember, it's the work that's being reviewed, those crammed 15 pages of science fiction, not your identity.

And there's a silver lining: almost always they will include a stack of written reviews. These reviews are a gold mine. Treat them as a genuinely helpful consolation prize. If you don't understand what they're saying, you can probably talk to a program manager about them.

When revising a proposal, I take all of the reviews, and cut out everything positive that they had to say. It's too easy to be comforted by that. What I really need to know is what didn't work. That's what needs work.

How to find writing advice
I still don't have a lot of confidence in my grant writing. But there are a lot of people who do, who will share their advice with you. Almost all institutions have grant writing workshops. NSF hosts workshops. There are posts like this, and this, and this, written by people who are truly qualified about the process to really steer you right. Writing with a group of people helps. Collaborative proposals come with their own logistical challenges, but it can be helpful for getting feedback on the writing.

Some graduate students write applications for fellowships, internships and other sources of funding. This is great practice, but this is different from the grant process. Maybe the best way for students to get exposure and experience at grant writing is through their advisor's proposals. I think some professors work with their students on their proposals. If the student is a strong enough writer, and far along in their dissertation, I could see this making a lot of sense for both the professor and student.

Things I didn't know before becoming a professor: #1 How to Teach

noreply@blogger.com (Anonymous) — Mon, 28 Apr 2014 04:02:00 +0000

In order to be a professor* you need a PhD. The two responsibilities you have as a professor are to do good research and to teach well.

To get a PhD, you need to do good research, so you're well equipped to handle this.

To get my PhD, I did not have to teach. I had TA'd a few times. But I had put in 10 years in college/grad school. I had taken countless classes, some with great teachers, some less great. So I figured no big deal; I can teach.

I've seen good movies. I've got a good story. I can make a good movie.

These are some things I didn't know about teaching before becoming a professor.

You are television.
I learned this from Michael Cirino in a very different context.

For the time that you are in front of students, you are television. You are putting on a show. If your students don't engage with you, your students won't learn anything beyond what they get in a book. That's not to say that you're entertainment, but you are performing.

My act isn't strong yet, but it's getting better.

Don't skimp on the basics.

In an early version of a Machine Learning class, I decided that I wanted a lecture on spectral clustering. It was a topic I was interested in, but didn't have a ton of experience with. I read a lot. I worked out math. By the time I had a lecture ready, I was feeling good about it. I went to class, delivered my A material....blank stares. It was a total dud.

It wasn't that the lecture was bad, but I hadn't earned it. Because I wanted to get to a topic that was exciting to me, I had rushed through or omitted a lot of background material. I had completely set myself up for failure. When I went back to figure out where I had went wrong, I realized it was months earlier, when I was planning the syllabus.

Every time I teach a new class, I tell myself that I'm going to have half a dozen or so lectures ready to go by the first day of class. Sometimes I hit this number, usually I don't. But I've come to realize that lectures are a lot easier if you have a clear structure ready for the class. Lately, I've become less concerned about having complete lectures prepared. Instead of writing a few complete lectures, I try to have a fairly detailed outline of every class. This helps me focus on the big picture.

When I have a clear direction for how the pieces fit together, the course is better. Even if that means I don't get to the most exciting material, or have an excuse to learn something new.

Preparing a lecture takes an unbelievable amount of time.
Truly unbelievable. In my first semester, it took 8-10 hours to prepare each 75 minute lecture. Add in writing assignments and exams, grading, office hours, and 2.5 hours in front of students. Thankfully, I was only teaching one class. But at CUNY a full load is 3 classes in a semester.

Most of your time is spent with students who are struggling.
"Do you learn from your students?" No. I teach them. "Are you inspired by your students?"

My PhD students are great. They bring some exciting ideas and papers, and there's a collaborative learning that happens there. They inspire me and drive me. Absolutely.

Students in class, I have a very limited and lopsided relationship with. I don't think I've ever had an office hours meeting with a student that took material from class and took it a step further. It's almost always something to the effect of going over material from the previous lecture or exercise in more detail. There's immense satisfaction from guiding someone to understanding challenging material, but it takes a lot of time.

Students cheat. A lot.
I have had a student cheat in almost every course I've taught. (This isn't unique to CUNY. Ask around.)

The cheating meeting is the most emotional human experience, I've had with anyone other than a family member or romantic partner.

The student usually cries.

The student usually tries to negotiate out of the repercussions.

Denial? Not so much. Most people own up to it pretty quickly. Forceful denial is a red flag for me. Be open to the possibility that you made a mistake. Maybe one party knew about the cheating and the other didn't. Maybe the similarities between two assignments really were random.

I've had over a dozen of these conversations. Here's my best advice: Be prepared. Be able to clearly explain how you know cheating happened. Be able to point to your syllabus and university student handbook about the penalties for cheating. Absolutely document everything you can about the exchange. Send the student an email after the fact, recapping the major points. Expect that the student will scramble for an out -- some way to lessen the impact -- don't let them. At this point, I have a loose script. It's almost as formulaic as a five-paragraph essay.

A. It's clear to me that you cheated on this assignment/exam.

B. Here's how I know. (it's about here where they usually admit to it.)

C. Because of this you will be getting a zero on the assignment/failing the class/getting expelled, and I will be send a letter with this information to the department. (Or whatever your policy is.)

Put it in writing. Get it in writing.
Almost all I knew about teaching I learned from a video (VHS!) I was shown while at Columbia. This was an impromptu sharing, it seemed like it was a tape that was passed around the CS department and shown by professors to their grad students. It was a lecture by John Kender called something like "How to Teach". It was fantastic. It's similar to this video on iTunes. Before you teach, watch it.

The most significant lesson I remember from this lecture was that a syllabus is a contract, an assignment is a contract, an exam is a contract. It is your responsibility to outline the terms of this contract as clearly as you can. This is what you will learn from this class. This is what to expect from this course, assignment, exam. If you do this, you will get this grade.

Most of the difficulty I have had with students and disputes can be traced back to not being rock solid in the language used in a syllabus, or on an assignment, or (and this was a surprise) not putting things that were discussed in a meeting, in writing.

If you make an arrangement outside your syllabus with a student around any element of your course, shoot them an email after the meeting recapping what was discussed. I didn't know this before teaching, but wish I had.

What now?
I've definitely become a better teacher over the last five years. I've made a lot of mistakes and I still do.

Each time I repeat a course, I toy with its structure. My boiler plate syllabus has gotten tighter.

The broader point is when I started, I was starting cold.

There must be ways, programs, seminars, etc. that aim to teach people how to teach. I was barely exposed to any as a graduate student; I know many of my peers weren't either. Very little was available before I was in front of students for the first time.

It's easy to gripe about a system that left me unprepared, but now i'm on the other side of the equation. I have graduate students, some of whom will go on to be professors. What can I (and my institution) do to make sure they have the skills to be good teachers when they land tenure-track jobs (which of course they all will).

Practice. Practice. Practice.
For me, becoming a better teacher has taken practice. I think that's the only way to learn how you are going to teach. And there aren't enough opportunities to practice this. (I've taught Algorithms 4 times, and Machine Learning 3. That may sound like a lot, but it's only 2 or 3 opportunities to revise structure, lectures and graded material.)

At CUNY, graduate students teach a lot**, so many of my students will have experience in front of students. The downside to this is 1) teaching takes a lot of time. This means that they're not focusing on their research. and 2) Often they're not given the responsibility/opportunity to design the class themselves. Instead they teach a section of a larger course with a fixed set of assignments and exams. This leaves students with plenty of experience lecturing, and leading discussions, but less experience with the mechanics of running a course (which is where your teaching lives or dies).

I think one solution might be to have graduate students prepare and teach mini-courses, complete with syllabus, and graded assignments. These should be short, maybe 6 or fewer meetings over a month or so. This keeps the workload more manageable compared to teaching a full course. But it would allow students to practice structuring material, and writing homeworks and exams. They shouldn't be offered during regular course periods, but in summer or between terms.

I think the best approach would be for this mini-course to be on the student's dissertation topic. First of all, they'll already know a ton about it. Second, if they go on to a tenure-track job, chances are they'll have an opportunity to reuse some of these lectures, either in a(nother) course of their own or at a conference tutorial. Third, lecturing on a topic, and fielding questions can bring to light all the things you don't know or are unsure of. But the practice would be useful even if it was on some other topic.

The biggest problem I see with this idea is getting the incentives right. To teach something like this takes a lot of work, and there's little reward. Moreover, there's little incentive for other students to take one of these mini-courses (and to do the homeworks/assignments). MIT has a thriving IAP program with a ton of activities and minicourses ranging from the technical (some for credit) to the slightly absurd to one of my favorite things. The IAP is well established in the MIT culture. Can something similar be started up elsewhere?

There's no way to do this through the university registrar without a lot of bureaucracy. However, if a department, or division, were to unofficially "bless" this kind of activity by 1) including a list of course offerings, 2) document who taught what when, and 3) conferring completion "certificates" (and 4) finding teaching space), the publicity of a program like this could encourage students to participate on both sides.

There are a lot of reasons that a program like this would fail to get off the ground, but if there were a mechanism for graduate students to get practice running courses in a relatively low-risk environment, I am confident that they would be better prepared for tenure-track positions.

I would have been.

* tenure-track
** maybe too much, but that's a different discussion

Things I didn't know before becoming a professor (and that i'm still not very good at)

noreply@blogger.com (Anonymous) — Thu, 24 Apr 2014 21:44:00 +0000

July 2009. I deposited my dissertation.

September 2009. I started a tenure-track position at CUNY.

Coming up on the close of my fifth year, I'm convinced of a simple proposition.

I was unprepared to be a professor.

This is not a reflection on the academic preparation I had, or that my weaknesses went unnoticed by the search committee that hired me. Neither do I think I've done a particularly bad job over the last five years. Rather, there is a disconnect between the skills that are required to become a professor and the skills that are needed to be a good professor.

To land a tenure-track position you must:

get a PhD
get a solid publication record
get good recommendations from important people
give a good job talk
be personable enough to not ruin your visit to campus
get lucky (there are fewer tenure-track positions than in the past)

(If you're very lucky, you've got some funding when you walk in the door. But this is a catch-22. It's really hard to get funding until you're already a professor.)

The only one of these that is non-negotiable is having a PhD.

In order to get a PhD* you must:

do good research.
survive on little money and less sleep

That's it.

Over the last five years, I've repeatedly found myself in situations where I have no idea how to do things that I am expected to do well. I was a good candidate for a tenure-track position, but a mediocre professor.

Here's an incomplete list of things I didn't know before becoming a professor (and that I'm still not very good** at).

how to teach
how to write a (successful) grant
how to head up a research group
that project collaboration is different from research collaboration
how to manage my time

In all new jobs, there are things that you have to learn how to do, skills that get developed through practice. But this isn't figuring out where the closest printer is, or how to fill out a TPS Report. Most of these skills are central to the job. There is a disconnect between the requirements to get a tenure-track job and the skills needed to do it well.

Over the next few weeks, I'll drill down on each of these. Hopefully, this will be some comfort to other pre-tenure faculty members, and a preview for graduate students. It's helpful to acknowledge these challenges and the gap between what we expect from graduate students and professors. In a perfect world, this points to opportunity for graduate programs (including mine) to provide more support to better prepare good graduate students to become good professors.

* specific requirements vary by institution

** I've gotten better... But mostly through missteps and course corrections.

Deep Thoughts on ICASSP 2013

noreply@blogger.com (Anonymous) — Wed, 05 Jun 2013 20:58:00 +0000

ICASSP 2013 is wrapping up today in Vancouver. Unfortunately, I missed the last day (and sessions on speech synthesis and prosody that I would have enjoyed). But a wedding on Saturday brought me back a day early.

I hadn't been to ICASSP before, mostly due to timing oddities and writing grants over summers rather than writing papers that would hit the deadline. It is a very large conference. About twice as large as Interspeech. But the scope is also much broader. Speech and Language work made up at most 30% of the work at the conference. And even this is generous, including machine learning, and other work on audio.

So take this recap with a grain of salt. I missed the last day of the conference, and my impressions are speech focused. (I think I've described all conference recaps as blind-men-and-the-elephant problems and this one is no exception.)

Deep Learning.
OK, I pointed out that Deep Neural Nets were a "hot topic" at last years Interspeech. It's hard to believe it's possible, but they're even hotter now. Geoffrey Hinton gave the first plenary talk. This was followed by an oral session called "Automatic Speech Recognition using Neural Networks", which was followed by a Special Session titled "New Types of Deep Neural Network Learning for Speech Recognition and Related Applications". The next morning, you could attend "Acoustic Modeling with Neural Networks". And this is just at the session level. Even more applications of multilayer neural networks were scattered around other oral and poster sessions. Some of these oral sessions were so crowded that people were standing along the walls and sitting in the aisles. Nothing else that I saw received nearly so much attention.

It's easy to view "deep" learning as a silver bullet -- the next great machine learning that will solve all of our problems. It's almost certainly not. However, a wide array of research groups are seeing similar impressive performance gains by using deep network models for a broad spectrum of spoken language processing tasks. This is especially true for acoustic modeling in speech recognition. Given this, deep learning shouldn't be ignored.

Hinton's coursera course is a solid place to start. (Though resist drinking the kool-aid. To my mind, perceptrons are bad approximations of neurons and worse approximations of the brain, and do little to advance our understanding of human intelligence.)

Another highlight
One paper which caught my attention for its simplicity came out of Google: "Language Model Verbalization for Automatic Speech Recognition". Essentially "verbalization" is defined as a sort of inverse text-normalization. In text normalization for speech synthesis we have to translate "10" to "TEN", and "7:11" to "SEVEN ELEVEN" or "ELEVEN PAST SEVEN". For ASR, the idea of verbalization is to convert decoding output of "SEVEN ELEVEN" into "7:11" or "7-11". Why bother? Well, Google (and everyone else) has big language models based on text data. You could run a text normalizer over all of this data, but the proposition here is to convert the ASR output into a form that looks more like the source material in your language model.

The Verbalizer solution to this problem is remarkably elegant. A traditional WFST decoder can be expressed as D = C • L • G, where C comes off the acoustic model mapping context dependent to independent phones, L is the pronunciation model and G the language model. The "Verbalized" WFST model includes a WFST V which maps ASR realizations like "SEVEN ELEVEN" to text-like realizations like "7-11" or "7:11" (and since it's a WFST it can do both simultaneously). The new decoder looks like D = C • L • V • G. No fuss, no muss. Except that you have to write Verbalizer rules by hand.

The paper focused on terms involving numbers, but the framework is very extensible. And it's great to see work coming out of Google that doesn't have Google-scale data as a prerequisite.

Meta-comment
The acceptance rate at this years ICASSP was 52%. This means that the ICASSP and Interspeech acceptance rates are identical for the first time. I know that Interspeech organizers have been working to lower the acceptance rate, while it sounds like there has been pressure to keep the size of ICASSP large, even at the expense of a higher acceptance rate. IEEE (the ICASSP parent organization) is a much larger bureaucracy than ISCA (Interspeech). There are clear expectations from IEEE about the expected revenue from hosting a conference, which translates to expectations on attendance and therefore the number of accepted papers regardless of the number of submissions.

Despite the near constant rain, I genuinely enjoyed Vancouver and ICASSP 2013. I'm looking forward to the next.

Reading and Reconnecting

noreply@blogger.com (Anonymous) — Mon, 22 Oct 2012 17:27:00 +0000

With travel finally settling down for me, but ramping up for my wife's book tour, I'm able to settle in to some long overdue reading, thinking and planning.

Also, after an exciting conversation with Dogan Can, during a trip to USC's SAIL lab, I'm trying to get more on top of sharing ideas, progress and information here.

First up: some drill-down reading from Paul Mineiro's blog post on Bagging!

Ensemble methods work too well for me to understand them so poorly, so:

How out-of-bag estimates can be used to get at generalization error (better than cross-validation can).
The relationship between the bias-variance tradeoff and ensemble methods from this lecture. This is a nicely framed discussion of ensemble methods that I hadn't seen before.

Lying Words: Predicting Deception From Linguistic Styles. This paper describes a common pattern of language use in deceptive story-telling: Less self-reference. More negative emotion words. Less cognitive complexity.

I'm looking forward to verifying these claims on some old deception data. And taking a look at debate transcripts through this lens.

Interspeech 2012 Recap

noreply@blogger.com (Anonymous) — Tue, 18 Sep 2012 02:49:00 +0000

Portland proved to be a great venue for this year's Interspeech. (Though people who attended ACL 2011 probably already could have guessed that.)

Setting up three simultaneous poster sessions in the parking garage may not sound like the mark of a good conference, but it was perfect. There was loads of space between each presenter. It allowed for all three sessions to be in the same place. And the folks at the Hilton did a great job of making it fairly unrecognizable as a parking lot. (In fact, Alejna Brugos didn't realize it until they were removing the carpets and "walls" on Thursday afternoon.)

Deep Neural Networks.
For "trends", there's really nothing hotter right now than Deep Neural Networks or Deep Belief Nets. This isn't an area that I do research in, but the story goes more or less this. Neural Networks with more than a few hidden layers don't train very well with back-propagation. Geoff Hinton and his group figured out how to overcome this limitation not too long ago. (I think this 2006 paper explains it, but I can't be 100% sure.) Then at ASRU 2012 and ICASSP 2011 and 2012, the folks at Microsoft showed that you can use Deep Neural Networks to generate *very* useful front end features. (Tara Sainath has a nice recap of ICASSP 2012 here.) Now, everyone wants a piece.

The field has expanded from Microsoft to include IBM and Stanford/Berkeley/Google and RWTH Aachen. Joining them with posters on Deep Neural Nets for ASR are Tsinghua, CMU, Karlsruhe, NTT, INESC-ID, UWashington, and Georgia Tech. At this point, there's no way to deny that this approach is receiving significant research attention. The results seem to be holding up. If only they didn't take so long to train...

Prominence Special Session.
I was particularly looking forward to the Special Session on Prominence. On balance I was happy about the session. It attracted work and discussion of prosody in a way that can sometimes feel diffuse and unfocused at a large conference like Interspeech.

I found this session to be surprising in a few ways.

It's been my understanding that "prominence" was used as a catch-all term to cover diverse prosodic phenomena including stress, emphasis, and pitch accenting. The first surprising element of this Session came in a review of the paper I submitted to it. The paper is on the use of automatically predicted pitch accents and intonational phrase boundaries to improve pronunciation modeling. The review, while generally positive, found the paper to not be appropriate for a prominence session because it explored the use of "pitch accents" rather than "prominence". I still haven't gotten a good explanation of the difference, and the reviews are blind.

A second surprise is that there seems to be a movement away from a phonological theory of prosody. Mark Hasegawa-Johnson and Jennifer Cole have been doing work over the last few years investigating how naive listeners perceive prominence. They've consistently found that listeners respond to different qualities sometimes at different thresholds when assessing prominence. I've found this line of research to be interesting and generally informative, but not a clear indictment of the theory that there perceptual and productive prosodic categories exist. The panel (which I was a part of) on balance seemed comfortable with the idea that prominence is a continuous rather than categorical phenomenon. This view was most directly expressed Denis Arnold who said approximately: focus can be categorical, stress can be categorical, while prominence is still continuous. I didn't understand this statement then, and still don't. But again, this may be due to a different definition of prominence than I use.

The last surprise comes from finding out that there is a direction of pursuing language universals in prominence and prosody more broadly. Petra Wagner and Fabio Tamburini (the session organizers) are planning a workshop to investigate this. In my experience, while the dimensions of prosodic variation may be used in multiple languages and some of these (e.g. increased intensity or duration) may be used to indicate prominence in all languages, it is extremely unlikely that either the communicative impacts of prosodic variation or its realization and perception are language-universal. From that perspective, I'm not quite clear about what this line of research hopes to accomplish, but I'm curious about where it ends up.

Dynamic Decoding.
It appears that every year, I find myself sitting in on an oral session on a topic that I know very little about. Last year it was the language identification session. This year it was Dynamic Decoding. I was most intrigued by this because I hadn't heard the term before. When I asked someone what it was, they said "I don't know, Viterbi?".

I'm not quite sure this is a good enough distillation of the topic, but the papers in this session were about how to make on-the-fly (or post-training) modifications to language or pronunciation models. This is a cool idea with clear practical importance -- how do you add words to a recognizer on a mobile device and have this appropriately incorporated into the LM and pronunciation model? These two papers have some interesting WFST based approaches on this task. I'll be curious to see learn more about this. Also, if anyone has a more precise definition of this research area, I'd love to hear it.

Finally, some comments on two of the keynotes.

There were four keynotes at this year's Interspeech, two were about interesting inter/multi-disciplinary questions about how speech processing intersects with music and animal vocalization, respectively.

Chin-Hui Lee: An Information-Extraction Approach to Speech Analysis and Processing
A third was delivered by this years ISCA medalist, Chin-Hui Lee. Prof. Lee's most famous accomplishment is MAP adaptation in acoustic modeling. This is a researcher who spent a career treating speech recognition as a pattern matching problem. This is a view embodied by the Fred Jelenik quote: "Every time I fire a linguist, the performance of the speech recognizer goes up". What struck me, is that despite this view, in a talk summarizing a successful career, Prof. Lee presented a view of speech recognition that says that linguistic knowledge and speech science should be incorporated into the task. This is an alternate perspective that has been investigated by a lot of talented researchers, including Hynek Hermansky, Jennifer Cole, Mark Hasegawa-Johnson, Alex Waibel, Hermann Ney (via speech-to-speech translation), Mari Ostendorf, Elizabeth Shriberg, Andreas Stolcke, Rene Beutler, Karen Livescu and many more (my apologies to anyone I missed).

I was struck by the evolution of perspective from someone who represents the statistical pattern matching approach to recognizing the potential importance of linguistic knowledge.

However, this talk was not so well received by some members of the audience for fairly obvious reasons. Firstly, it over-played the importance of Prof. Lee's own contributions. In a slide on "My contributions", virtually all major improvements to ASR over the last 20 years were mentioned including most styles of adaptation (including MAP), and virtually all major forms of discriminative training. Secondly, it failed to recognize that the linguistic inspired approach that he was advocating for the future had been extensively researched by other talented peers.

On balance, I found it a compelling message. In principle, it understandably rubbed some people the wrong way.

Michael Riley: Weighted Transducers in Speech and Language Processing
I should preface my comments about Michael Riley's keynote by saying that we worked together while I was interning at Google. I'm a fan. Michael has the rare quality of being the smartest guy in the room without letting anyone know until its genuinely useful.

The best part of this keynote was the history of the Weighted Finite State Transducer. This was a great story that takes place largely at Bell Labs in the 90s and features Fernando Pereira, Mehryar Mohri and, naturally, Michael Riley. This section was appropriately personal, while presenting this relevant recent history. The WFST is so ubiquitous in speech and NLP applications that it's easy to forget that it's has a human context.

Much of the rest of the keynote felt like a 3 hour tutorial compressed into 40 minutes. This involved showing algorithms, and example WFSTs and describing all of the things that they can be used for. While a successful demonstration of the breadth of application, it was presented at such a pace that it was difficult to get anything out of it, if you didn't know it already. I'd point the interested to the references found on the OpenFST page for more thorough tutorials that can be digested at your own pace.

Interspeech 2012 was successful and fun. Portland and the Hilton (and it's solid wifi) were excellent hosts. There was good work and as ever more than I could see. If you have great or favorite papers that I missed, please let me know!

Overview of Speech and Spoken Language Processing

noreply@blogger.com (Anonymous) — Wed, 29 Aug 2012 20:55:00 +0000

Here's the premise: I was invited to give a guest lecture in Advanced Natural Language Processing. The students will get one week out of 14 focusing on speech and spoken language processing. But it's early in the semester, so there's an opportunity to give a perspective about how speech fits in to the lessons that they'll be learning in more detail later in the semester.

Here's the question: how do you spend 75 minutes to provide a useful survey of speech and spoken language processing?

My answer, in powerpoint form, can be found here.

I spent about 2/3 or so of the material on speech recognition. I figured most students are fascinated by the idea of a machine being able to get words from speech, so let's go through the fundamentals of the technology behind it.

The remaining 1/3rd or so, I focus on the notion that speech recognition is not sufficient for speech understanding. This a lot of other information in speech that is either 1) unavailable in text, or 2) unavailable in ASR transcripts. The premise in this section is to convince students that speech isn't just a noisy string of unadorned words, but that there's a lot of information about structure, and intention that is available from the speech signal. What's more, we can use it in spoken language processing.

There are an outrageous amount of important concepts that get almost no attention here including but not limited to: Digital signal processing, human speech production and perception, speech synthesis, multimodal speech processing, speaker identification, language identification, building speech corpora, linguistic annotation, discourse and dialog, and conversational agents.

Would you do it differently? I'm curious what some other takes on this problem might look like.

AuToBI v1.3

noreply@blogger.com (Anonymous) — Thu, 16 Aug 2012 18:19:00 +0000

This release to AuToBI is a more traditional milestone release than v1.2 was. Trained models and a new .jar file will be available on the AuToBI site shortly.

There are improvements to performance that are thoroughly documented in a submission to IEEE SLT 2012. These improvements were achieved from two sources.

First, AuToBI uses importance weighting to improve classification performance on skewed distributions. I found this to be a more useful approach than the standard under- or over-sampling. This is discussed in a paper that will appear at Interspeech next month.

Second, inspired by features that Taniya Mishra, Vivek Sridhar and Aliaster Conkie developed at AT&T, I included some new features which had a big payoff. (They described these features in an upcoming Interspeech 2012 paper). One of the most significant was to calculate the area under a normalized intensity curve. This has a strong correlation with duration, but is more robust. You could make an argument that it approximates "loudness" by incorporating duration and intensity. This is a pretty poor psycholinguistic or perceptual argument so I wouldn't make it too strongly, but it could be part of the story.

Here is a recap of speaker-independent acoustic-only performance on the six ToBI classification tasks on BURNC speaker f2b.

Task	Version 1.2	Version 1.3
Pitch Accent Detection	81.01% F1:83.28	84.83% F1:86.58
Intermediate Phrase Detection	75.41% F1:43.15	77.97% F1:44.43
Intonational Phrase Detection	86.91% F1:74.50	90.36% F1:76.49
Pitch Accent Classification	18.46% Average Recall:18.97	16.33% Average Recall:21.06
Phrase Accent Classification	48.34% Average Recall:47.99	47.44% Average Recall:48.31
Phrase Accent/Boundary Tone Classification	73.18% Average Recall:25.92	74.47% Average Recall:26.02

There are also a number of improvements to AuToBI from a technical side and as a piece of code.

First of all, unit test coverage has increased from ~11% to ~73% between v1.2 and v1.3.

Second, there was a bug in the PitchExtractor code causing a pretty serious under prediction of unvoiced grames. (A big thanks to Victor Soto for finding this bug.)

Third, memory use is much lower by a more aggressive deletion of prediction attributes, and through a modification of how WavReader works.

I'd like to thank Victor Soto, Fabio Tesser, Samuel Sanchez, Jay Liang, Ian Kaplan, Erica Cooper and, as ever, Julia Hirschberg and anyone else who has been using AuToBI, for their patience and feedback.

I've been pretty lax about posting here. I'll try to get better about it in the coming academic year.

This fall is full of travel which will lead to a lot of ideas and not enough time to work on them.

AuToBI Version 1.2

noreply@blogger.com (Anonymous) — Wed, 11 Jan 2012 04:21:00 +0000

I hadn't really planned for this current improvement to AuToBI be a milestone release.

I'm about halfway through an effort to get test coverage up to 90-95% of lines and 100% of classes. I promise it'll get there eventually.

But in the mean time, I was playing with an improvement to how attributes are associated to data points. I knew this was a significant source of inefficiency, but didn't quite expect this much.

Here are memory usage graphs for training a Pitch Accent Detection model on the Boston University Radio News Corpus -- about 22k data points and 136 features. The first one is on my MacBookPro Laptop with 4G RAM (and a lot of other nonsense running).

The max memory usage of Version 1.1 was 1914Mb, with this improvement it tops out at 1049Mb. An improvement of about 45%. (You'll notice it also ends a little bit quicker too, but this is probably because of fewer or quicker garbage collection calls.)

I figured I'd check on a compute server too, one of the Speech Lab @ Queens College's Quad Core Intel Xeon Processor E5450 (3.0GHz,2X6ML2,1333) with 4Gb RAM.

Similar results here. Max memory usage of version 1.1 was 2343Mb and with the improvement 1392Mb. Improving by 40%. (And the speed improvement is here too.) I don't have a good explanation for why the linux version is taking more memory to run, but for now I'll assume it has something to do with the difference to the JVM.

There are some other bugfixes in this version, but this is the big reason to upgrade.

The version 1.2 is available from github
git clone git@github.com:AndrewRosenberg/AuToBI.git

English Pronunciation by G. Nolst Trenité

noreply@blogger.com (Anonymous) — Wed, 04 Jan 2012 00:58:00 +0000

This is a repost of a poem posted on spelling.wordpress.com that's been going around facebook today.

It's an incredibly elegant set of examples about why grapheme-to-phoneme (letter-to-sound) conversion is so difficult in English. (Maybe this should be a required regression test for any TTS frontend...)

Please enjoy.
English Pronunciation by G. Nolst Trenité (after the break)

Dearest creature in creation,
Study English pronunciation.
I will teach you in my verse
Sounds like corpse, corps, horse, and worse.
I will keep you, Suzy, busy,
Make your head with heat grow dizzy.
Tear in eye, your dress will tear.
So shall I! Oh hear my prayer.
Just compare heart, beard, and heard,
Dies and diet, lord and word,
Sword and sward, retain and Britain.
(Mind the latter, how it’s written.)
Now I surely will not plague you
With such words as plaque and ague.
But be careful how you speak:
Say break and steak, but bleak and streak;
Cloven, oven, how and low,
Script, receipt, show, poem, and toe.
Hear me say, devoid of trickery,
Daughter, laughter, and Terpsichore,
Typhoid, measles, topsails, aisles,
Exiles, similes, and reviles;
Scholar, vicar, and cigar,
Solar, mica, war and far;
One, anemone, Balmoral,
Kitchen, lichen, laundry, laurel;
Gertrude, German, wind and mind,
Scene, Melpomene, mankind.
Billet does not rhyme with ballet,
Bouquet, wallet, mallet, chalet.
Blood and flood are not like food,
Nor is mould like should and would.
Viscous, viscount, load and broad,
Toward, to forward, to reward.
And your pronunciation’s OK
When you correctly say croquet,
Rounded, wounded, grieve and sieve,
Friend and fiend, alive and live.
Ivy, privy, famous; clamour
And enamour rhyme with hammer.
River, rival, tomb, bomb, comb,
Doll and roll and some and home.
Stranger does not rhyme with anger,
Neither does devour with clangour.
Souls but foul, haunt but aunt,
Font, front, wont, want, grand, and grant,
Shoes, goes, does. Now first say finger,
And then singer, ginger, linger,
Real, zeal, mauve, gauze, gouge and gauge,
Marriage, foliage, mirage, and age.
Query does not rhyme with very,
Nor does fury sound like bury.
Dost, lost, post and doth, cloth, loth.
Job, nob, bosom, transom, oath.
Though the differences seem little,
We say actual but victual.
Refer does not rhyme with deafer.
Foeffer does, and zephyr, heifer.
Mint, pint, senate and sedate;
Dull, bull, and George ate late.
Scenic, Arabic, Pacific,
Science, conscience, scientific.
Liberty, library, heave and heaven,
Rachel, ache, moustache, eleven.
We say hallowed, but allowed,
People, leopard, towed, but vowed.
Mark the differences, moreover,
Between mover, cover, clover;
Leeches, breeches, wise, precise,
Chalice, but police and lice;
Camel, constable, unstable,
Principle, disciple, label.
Petal, panel, and canal,
Wait, surprise, plait, promise, pal.
Worm and storm, chaise, chaos, chair,
Senator, spectator, mayor.
Tour, but our and succour, four.
Gas, alas, and Arkansas.
Sea, idea, Korea, area,
Psalm, Maria, but malaria.
Youth, south, southern, cleanse and clean.
Doctrine, turpentine, marine.
Compare alien with Italian,
Dandelion and battalion.
Sally with ally, yea, ye,
Eye, I, ay, aye, whey, and key.
Say aver, but ever, fever,
Neither, leisure, skein, deceiver.
Heron, granary, canary.
Crevice and device and aerie.
Face, but preface, not efface.
Phlegm, phlegmatic, ass, glass, bass.
Large, but target, gin, give, verging,
Ought, out, joust and scour, scourging.
Ear, but earn and wear and tear
Do not rhyme with here but ere.
Seven is right, but so is even,
Hyphen, roughen, nephew Stephen,
Monkey, donkey, Turk and jerk,
Ask, grasp, wasp, and cork and work.
Pronunciation (think of Psyche!)
Is a paling stout and spikey?
Won’t it make you lose your wits,
Writing groats and saying grits?
It’s a dark abyss or tunnel:
Strewn with stones, stowed, solace, gunwale,
Islington and Isle of Wight,
Housewife, verdict and indict.
Finally, which rhymes with enough,
Though, through, plough, or dough, or cough?
Hiccough has the sound of cup.
My advice is to give up!!!

Evaluating multi-class classification performance

noreply@blogger.com (Anonymous) — Wed, 21 Dec 2011 18:35:00 +0000

Classification performance is generally easy to evaluate. Just use accuracy. It's the most intuitive evaluation measure you can think of.

$\frac{Correct Predictions}{Number of Data Points}$

We can also represent classification results as a contingency matrix A, with $A_{ij}$ for $i, j \in \vec{t} = \{t_1,\ldots, t_k\}$ where $k$ is the number of possible target values, and A_{ij} is the number of times that a data point from with true label $t_i$ was classified as label $t_j$. Then accuracy is defined as follows:

$Accuracy = \frac{\sum_i A_{ii}}{\sum_i \sum_j A_{ij}}$

This is great until the balance of classes gets out of whack.

This is most commonly seen in cases, where you're not really classifying but detecting something. Say, a classification task where 95% of data points are non-cancerous, and 5% are cancerous. A degenerate classifier which assigns the majority class label to all points will lead to 95% accuracy (seemingly very good) but is completely uninformative.

When the distribution of class labels is skewed, like this, accuracy becomes a poor evaluation measure. When there are only 2 labels, there are a variety of choices for evaluation that have been developed through investigation of detection problems, including F-measure, and ROC curves.

There are fewer available choices with more than 2 labels, and the community (at least in natural language processing and speech) has not yet settled on a consistent evaluation measure for these tasks.

Here are some options:

* Ian Read and Stephen Cox proposed the use of "Balanced Error Rate" in "Automatic Pitch Accent Prediction for Text-To-Speech Synthesis", Interspeech 2007.

\[
BER = 1 - \frac{1}{k}\sum_i\frac{A_{ii}}{\sum_j A_{ij}}
\]

This is one minus the average recall (correct predictions of a class / true instances) treating each class evenly, regardless of its class membership. It's worth noting that this measure -- average recall -- was also used in the Interspeech Challenges in 2009, 2010, and 2011. The organizers of this challenge, for reasons that escape explanation, chose to call the evaluation measure "unweighted accuracy" rather than "average recall" or "balanced error rate".

* In my thesis, I put a spin on this measure and defined, "Combined Error Rate". This was the average of the total false positive and false negative rates taken across each class.

\[
CER = \frac{\sum_i \frac{n_i}{n} * FP_i + \sum_i\frac{n_i}{n} * FN_i}{2}
\]

This was fairly ad hoc, and not all that well motivated.

* There is also micro and macro-averaged F-measure, which extend the idea of precision and recall from a single class to a set of classes.

* Mutual Information treats the class and hypothesis distributions as random multinomial variables, and calculates their, well,...mutual information. The downside to this technique is that perfectly bad predictions will have perfect MI, just as a set of perfectly good predictions will. This is a serious problem, making it significantly less useful.

With the exception of Mutual Information, each of these have the form, of examining the contingency table, and weighting the contribution of a correct prediction (or error) based on its true or hypothesized class membership.

This allows us to generalize (many of) these measures into something like

\[
Measure = \frac{1}{Z(A)}\sum_i \sum_j w(i,j) A_{ij}
\]

where w(i,j) is a function that determines how much a classification of a point of class $i$ as class $j$ contributes to the measure, and Z(A) is a normalizing function.

Essentially we're trying to determine how important it is for each point to be classified correctly, on the basis of its class membership, and how damaging an incorrect classification.

Information theory can give us a values for how important each point is. It seems as though w(i,j) := $\delta(i = j)*(-p_i \log p_i)$ would be a valuable evaluation measure. Where $\delta(i = j)$ is a Kronecker delta function, which is 1 if $i = j$ and 0 otherwise. This says that correct classifications contribute the amount of information in its true class to the measure function, while incorrect classifications contribute nothing. This instantiation of an accuracy-like measure weights correct classifications by the amount of information they transmit to the user.

I don't know of anyone who has done this, but it seems like someone should have. I'll keep looking in the literature until I find something, or I give up and write a little paper on this.

Interspeech 2011 Recap

noreply@blogger.com (Anonymous) — Tue, 06 Sep 2011 18:22:00 +0000

Interspeech 2011 was held in Florence Italy about a week ago, August 28-August 31. A vacation in Italy was too good to pass up on, so The Lady joined me, and we stayed until Labor Day.

I ended up sending a bulk of work to Interspeech, so spent more time than usual in sessions that I was presenting in rather than seeing a lot of papers.

Two interesting themes stood out to me this year. Not for nothing, but these represent some novel ideas about speech science through understanding dialog and the speech engineering.

Entrainment
Julia Hirschberg, my former advisor, received the ISCA medal for her years of work in speech. Her talk was on current work with Agustín Gravano and Ani Nenkova on entrainment. Entrainment is the phenomenon by which when people are speaking to each other, their speech becomes more similar. This can be realized in terms of the words that are used to describe a concept, as well as speaking rate, pitch, intensity. How this happens isn't totally understood, and measures of entrainment are still being developed. This research theme is still in its early phases, but I haven't seen an idea spread around a conference as quickly or as thoroughly as this did. There were questions and discussions all over the place (like Tom Mitchell's keynote about fMRI data and word meaning) about this phenomenon. The more engineering folks weren't as compelled by the utility of this in helping speech processing, but within the speech perception and production communities, and specifically the dialog and IVR folks, it was all the rage. It'll be something to see how this develops.

I-vectors
I-vectors are a topic that I need to spend some time learning. I hadn't heard of it before this conference, where there were no less than a dozen papers that used this approach. Essentially the idea is this: The location of mixture components in a GMM model are composed of (in at least one form) a UBM, a channel component, and a "interesting" component. This "interesting" component can be the contribution of a particular speaker, or a language/dialect, or anything else you're trying to model. Joint Factor Analysis is used to decompose the observation into these components in an unsupervised fashion. It's in this part where my understanding of the math is still limited. The crux is that the "interesting" component can be represented by a \[Dx\] transformation, where the dimensionality of x can be set by the user. In comparison to a supervector representation, where the dimensionality of the supervector is constrained to be equal to the number of parameters (or means) of the GMM, i-vectors can be significantly smaller leading to better estimation and smaller models. I'll be reading this tutorial by Howard Lei over the next few weeks to get up to speed.

There were few specific papers that stood out to me this conference. I'm intrigued by Functional Data Analysis as a way to model continuous time-value observations. Michele Gubian gave a tutorial on this that I sadly missed, and included it in at least one paper, Predicting Taiwan Mandarin tone shapes from their duration by Chierh Chung and Michele Gubian. This paper wasn't totally convincing in the utility of the technique, but there may be more appropriate applications.

It was a satisfying and inspiring conference, to be sure. I think I was more interested in talking to people than in papers in particular this time. If anyone has particular favorites that I missed, please use the comments to share or just email me.

Interspeech Tutorial

noreply@blogger.com (Anonymous) — Thu, 25 Aug 2011 02:34:00 +0000

I'm giving my first tutorial at Interspeech on Saturday in Florence.

It's titled "More than Words can Say: Prosodic Analysis Techniques and Applications".

The (very likely) final version of the slides are available here.

Now off to Italy. Interspeech Recap to come.

Because Research is a Creative Act.

noreply@blogger.com (Anonymous) — Thu, 12 May 2011 12:12:00 +0000

This quote by Ira Glass has been going around design and writing blogs and tumblrs. It's equally applicable to research.

Nobody tells this to people who are beginners, I wish someone told me. All of us who do creative work, we get into it because we have good taste. But there is this gap. For the first couple years you make stuff, it’s just not that good. It’s trying to be good, it has potential, but it’s not. But your taste, the thing that got you into the game, is still killer. And your taste is why your work disappoints you. A lot of people never get past this phase, they quit. Most people I know who do interesting, creative work went through years of this. We know our work doesn’t have this special thing that we want it to have. We all go through this. And if you are just starting out or you are still in this phase, you gotta know its normal and the most important thing you can do is do a lot of work. Put yourself on a deadline so that every week you will finish one story. It is only by going through a volume of work that you will close that gap, and your work will be as good as your ambitions. And I took longer to figure out how to do this than anyone I’ve ever met. It’s gonna take awhile. It’s normal to take awhile. You’ve just gotta fight your way through.

Do good work. But maybe more importantly, work.

I'll take "The Humans" for $1,000,000, Alex. (The obligatory Watson post.)

noreply@blogger.com (Anonymous) — Sat, 19 Feb 2011 06:42:00 +0000

Full disclosure: I worked one day a week for a year on improving Watson's speech synthesis. While I think the voice added flair and personality to the exhibition, everyone knows that question-answering is what Watson is all about. And I didn't have any hand in that.

Wednesday marked the end of the three day exhibition Jeopardy! match between Watson, IBM's question answering system, Ken Jennings and Brad Rutter. Watson decisively beat the human competitors.

There is no shortage of commentary on the implications of Watson's victory. One of my favorite pieces was by Ken Jennings himself. He's played more games of Jeopardy! than anyone else, and describes the experience in a very measured way.

One of his observations sticks out: "I expected Watson's bag of cognitive tricks to be fairly shallow, but I felt an uneasy sense of familiarity as its programmers briefed us before the big match: The computer's techniques for unraveling Jeopardy! clues sounded just like mine."

It is in this that Watson represents a step forward for Artificial Intelligence.

There is a human exceptionalism that most people have when discussing "intelligence". This exceptionalism leads to a dismissal of the success of Deep Blue's defeat of Kasparov as computation not intelligence. It leads to descriptions of Watson as a "big database". The same exceptionalism leads to the Turing Test as the final arbiter of intelligence.

Prediction: No machine will ever pass the Turing Test.

The test is explicitly defined to defend the exceptionalism of human intelligence. It all but explicitly says, "humans are intelligent. To be intelligent, you must be indistinguishable from a human."

Before this test is passed, machines will continue to perform tasks which we consider to demonstrate intelligence better than humans can. Chess: done. Quiz shows: done. Mathematical theorem proving: done. What's next? Perfect SAT scores. Perfect GRE scores? Writing a five paragraph essay? All conceivable in the next, say, five years. Performing basic research and writing an accepted peer-reviewed publication of its findings. Certainly further off, but not inconceivable. But none of these systems would pass the Turing Test; people would still ask, "can it make me coffee?" or "can it tell me why Ronnie and Sam are still together?" And yet, progress in artificial intelligence will be undeterred.

Ray Kurzweil's prediction of a self-aware system will almost certainly predate one that can convince us that it is human. Despite its ability to describe its "thought" process, and outperform humans on tasks that were formerly considered representative behavior, there will be those that won't consider this system "intelligent" until it passes the Turing Test.

Watson's success comes from some very impressive machine learning and natural language processing, not only in its ability to generate answers quickly, but in its collection of knowledge. The individual ml and nlp components used in Watson may individually be incremental improvements on previously existing approaches. Yet this doesn't mean that Watson is itself an incremental milestone. Watson bested a human at a task that we believed, until Wednesday night, required intelligence. But more impressively, it won in a way that was similar (in spirit if not in hardware) to the way humans play Jeopardy!

Requisite CYA: As always, this post doesn't represent anyone's views other than my own.

P == NP. The battle rages on.

noreply@blogger.com (Anonymous) — Thu, 20 Jan 2011 22:55:00 +0000

Last summer, Vinay Deolalikar claimed to have proved that P!=NP. This kicked up some terrific math and computer science dust, and a lot of people got excited, myself included. But this wasn't exactly a surprising result...most people suspect that P != NP after all, but a proof has been elusive.

Well, evening out the playing field, Vladimir Romanov claims to have come up with a polynomial time algorithm for 3-SAT -- one of the foundational NP-hard problems. Here's the paper describing the work. But what's more...there's source code for it. If he's right, then P==NP.

I fully expect that people will be picking this work apart pretty rigorously over the next few weeks, months and years. But if it holds up, it's a really big deal. Much bigger than P!=NP, most specifically because it implies that large factorization can be solved in polynomial time, which means all of our encryption is broken.

AuToBI Version 1.1

noreply@blogger.com (Anonymous) — Wed, 19 Jan 2011 00:34:00 +0000

I've made enough improvements to AuToBI to consider the toolkit a milestone more mature.

In addition to some uninteresting bug fixes, and refactoring, the version 1.1 is made up by 3 significant changes. As ever, the AuToBI homepage includes milestone releases, and the project itself is hosted on github.

1) Package restructuring.

The internal structure makes more sense now. Classes are divided into 'core', 'feature extractors', 'classifiers', 'feature sets', 'io' and 'utilities'. With over 100 classes, the code had outgrown a flat package structure. Unfortunately this restructuring changed the serialization signatures of class names. This means that old models won't work with the version 1.1 release. But I have new models trained on the Boston Directions Corpus and Boston Radio News Corpus which will be available from the AuToBI homepage shortly.

2) Implemented reference counting for Feature maintenance.

The feature extraction process allows a user to specify the features they want without explicitly going through all the intermediate steps to extract them. For example, extracting the mean speaker normalized pitch from the second half of a word requires pitch to be extracted, speaker normalization parameters to be calculated or retrieved, the pitch to be normalized, the second half of the word to be identified, and finally the mean calculated. In AuToBI each of these steps are treated as features that are required by the feature extraction of the user-desired feature. In version 1.0, AuToBI was able to identify which features were required, but never recognized when a feature wasn't needed any longer. Version 1.1 includes functionality that maintains a reference count for each feature based on how many features that still need to be extracted are going to need it -- speaker normalization parameters may be required by many requested features. This allows for tidier memory management, which should, in turn, allow AuToBI to operate on more material.

3) Reduced storage for acoustic contours.

In version 1.0 pitch and intensity contours were stored as lists of time-value pairs -- a storage class containing 2 doubles. Assuming 10ms samples, and 16bytes per sample, this is about 1600bytes per second. Not unacceptably inefficient, but definitely could be improved. The other approach would be to only store values, and let the time of each point be specified by a start time (t0) and step size (dt). This allows storage with 8bytes per sample plus 16bytes for the parameters. The problem with this approach is that pitch points are invalid when there is no periodic material detected. The time-value pair approach handled this simply and easily. The new solution uses an array of one-bit booleans for each sample in the contour, describing whether it was a 'valid' point or not. If the rate of periodic to aperiodic material is less than 32:1, the new approach will reduce the memory requirements. (As I'm writing this, I realize that this ratio could be explicitly tested during pitch extraction, selecting the most efficient storage solution at runtime. Keep an eye out in a new release.)

The changes are mostly inside baseball kind of things, but possibly useful for AuToBI users who get their hands dirty in the code -- whoever you are. For everyone else, the most obvious changes you'll notice is that version 1.0 models don't work any more, and a speedup of ~6%.

Personal Paper Writing Month - Recap

noreply@blogger.com (Anonymous) — Fri, 03 Dec 2010 22:30:00 +0000

Well, no one wanted to join with me for Paper Writing Month, so I can only report on my own progress for November.

I got one full draft written, and the experiments for two others are more or less done running.

I can point out a number of factors that kept me from being more productive, none of which make me disappointed in myself. In all say this was a good exercise, and one I'll do again. Maybe next time, I can find a buddy or two to join with me on it.

Now that I've written four (now five) sentences including a first-person personal pronoun, here's some more general thoughts about research.

Alex and Aki at Ideas in Food have a blog post about about Creativity in the Kitchen. They loosely break this down into Inspiration, Flexibility, Motivation, Adaptation and Refinement. Both these top-level categories and the specifics they address apply almost as well to creativity in research. So check it out and cross-pollenate a little.

Cross-validation with one model

noreply@blogger.com (Anonymous) — Thu, 11 Nov 2010 21:19:00 +0000

This is essentially a repost of Rob J Hyndman's blog post on the relevance of cross-validation for statisticians.

Within this very nice piece, Rob drops this bomb of mathematical knowledge:

It is not necessary to actually fit separate models when computing the CV statistic for linear models.

Say what?

Here is a broader excerpt and the method itself (after the jump).

While cross-validation can be computationally expensive in general, it is very easy and fast to compute LOOCV for linear models. A linear model can be written as

Then

and the fitted values can be calculated using

where is known as the “hat-matrix” because it is used to compute (“Y-hat”).

If the diagonal values of are denoted by , then the cross-validation statistic can be computed using

where is the residual obtained from fitting the model to all observations. See Christensen’s book Plane Answers to Complex Questions for a proof. Thus, it is not necessary to actually fit separate models when computing the CV statistic for linear models. This remarkable result allows cross-validation to be used while only fitting the model once to all available observations.

Very cool.

Semantically Related Term Challenge

noreply@blogger.com (Anonymous) — Sat, 06 Nov 2010 15:02:00 +0000

Joseph Turian over at MetaOptimize.com has posted a fun NLP challenge.

The task is to identify semantically related words from a shared corpus.

So you're thinking, sure, no problem. I'll look for common concurrences. Maybe I'll start with some seed pairs and do some bootstrapping. Or you do LSA, if you're into that sort of thing.

But here's the rub, there are a few million documents, so you've got to get clever if you're going to use LSA (cause that would require SVD of an impossibly large and sparse matrix).

As if that weren't challenging enough, these "documents" are only a word or two long, so the concurrences you find are going to be pretty sparse.

So, that's it. Have at it.