The Visual Linguist

Updates and such

2013-05-20T00:32:00.000-07:00

Things have been a bit busy lately, so I've had less time to devote to blogging (suggested topics always welcomed, FYI). I've been hard at work on several revisions of papers for journals, as well as prepping several more papers for submissions. I'm also gearing up to start running a new brainwave study here at UCSD, so that's exciting to have underway.

I've also been hard at work on a major redesign/relaunch of this website. The site is getting a new look, new organization, and even a new address. If you'd like to plan ahead for the switch, the new url will be: www.visuallanguagelab.com (currently a redirect to the present site, which will then be flipped on relaunch).

Finally, I'm happy to report that my upcoming book, The Visual Language of Comics, has now entered the production stage! It will be fun to see the proofs in a few weeks. For now though, it's exciting to see that my publisher has now created a webpage promoting the book, including a growing list of endorsements. Looks like there's even a page on amazon for it. Let the countdown until it's release in December begin!

CFP: Interdisciplinary approaches to visual narrative

2013-05-02T01:30:00.000-07:00

For those that might be interested, one of my projects is trying to organize a book summarizing important research on visual narrative. This book will be a companion volume to my book out later this year, The Visual Language of Comics. If you may be interested in contributing, here's a Call For Papers for it...

CFP: Interdisciplinary approaches to visual narrative

While there have been a growing number of books on comics in recent years, very few have addressed aspects of structure, particularly from theoretical, cognitive, or experimental points of view and outside the realm of literary or sociocultural theory. I am working to organize a compilation of important papers on the understanding of sequential images. Most of the chapters will be either 1) summary papers that provide extensive bibliographies that can provide an overview to students and a resource to other researchers, or 2) reprints of significant research that remain under-recognized or hard-to-find.

This Call for Papers asks for proposals for papers of two types of chapters focused particularly on research outside of English, presented for an English speaking audience:

1. Chapters that summarize, in English, advances in comic theory from non-English speaking researchers. Such chapters should be large in scope with extensive reference sections.

2. Translations into English of significant non-English comic theory (structural, cognitive, experimental, etc.) from important papers or book chapters.

Topics or chapters outside this scope may be considered, though best to contact me directly with inquiries. (Of interest may be: review papers of other types, historical development of comic “symbology”, empirically grounded discussions of differences between comics cross-culturally, etc.). Importantly, papers should be relevant not only to scholars of comics, but also to linguists, cognitive scientists, and psychologists.

Contributor Guidelines

1. Abstracts of 400-500 words accepted. Papers of 5000-9000 words, including notes and bibliography, accepted. Please also include a short biographical statement.

2. All documents should be submitted as Word or Word-compatible files. PDFs are also acceptable.

3. Submission deadline: ~~May 15, 2013~~. June 15, 2013

4. Materials should be sent to me via email (Neil Cohn)

Comics, games, and bad science

2013-04-28T16:54:00.001-07:00

I'm often very excited when I find new research on comics, especially when it's experimental. There is so little done that it's a treat to find something I didn't know about. Unfortunately, sometimes my excitement at new data turns sour when I see what was actually done...

I recently found this study (pdf) by Kinzer and colleagues that compares the comprehension and eye movements of readers for narratives in comics and video games. Their main goal is to help provide support for the use of comics and video games in educational contexts.

In this study, they presented sixth graders with either a video game version of a story or a comic created from the images of the video game. Overall, they find that participants understood the story in the video game version better than the comic version. They also found people spent more time engaged with the game than the comic.

I would take all of these findings with a grain of salt...

...because the stimuli appear to be extremely confounded because the comic versions of the story appear to be so badly created. Judging by the example in the text, the comic pages clearly were created by someone who had no real fluency in the visual language of comics. This is clear at a glance just by the example that they provide in the paper:

First off, the images make it extremely hard to tell what's going on. Second, almost all of the balloons are placed outside of their originating panels to the extent that they completely overlap in panels forward and backward in the text. I don't even need to know what's in the text to know this will be confusing to a reader. This is so "illegal" in the rules of page construction that it is almost painful.

If this is their example (what is probably the best example they have), I shudder to think what other pages in the experiment look like. Seriously, if I wanted to design an experiment that had "incomprehensible comics pages" as one type of stimuli, I'd use pages like these.

It's no wonder they found that their participants had poorer comprehension for the comic version—their stimuli are the equivalent of trying to test English comprehension while using broken English. It tells you next to nothing of interest.

There are two main points I'd like to make about this:

First, good experiments are hard to design, and having something worth saying must follow from having successfully designed an experiment that can give you good information. It pays to be critical as a creator and reader of scientific research (no matter what the topic).

Second, doing experiments using the visual language of comics is not trivial. Stimuli cannot be created by anyone, regardless of their fluency in comic creation. Just because you can throw together some images and words into panels on a page does not mean you've successfully created an example of "native" visual language. Believing otherwise does a disservice to yourself and to others who might read and cite your research.

Kinzer, C. K., Turkay, S., Hoffman, D. L., Gunbas, N., Chantes, P., Chaiwinij, A., & Dvorkin, T. (2012). Examining the Effects of Text and Images on Story Comprehension: An Eye-Tracking Study of Reading in a Video Game and Comic Book. In P. J. Dunston, S. K. Fullerton, C. C. Bates, K. Headley, & P. M. Stecker (Eds.), Literacy Research Association Yearbook 61 (pp. 259-275). LRA: Chicago, IL.

New paper: Navigating Comics

2013-04-17T11:12:00.000-07:00

I am happy to announce I have a new paper published in the journal Frontiers in Cognitive Science that outlines a theory of how people navigate page layouts. A previous version of this paper was posted here several years ago, though this is now the final version. You can download a pdf of the paper here: "Navigating Comics."

The paper describes an experiment I conducted that examined people's intuitions for moving through page layouts. I then incorporate the results into a larger theory of the structure of page layout. Here's the official abstract:

Like the sequence of words in written language, comic book page layouts direct images into a deliberate reading sequence. Conventional wisdom would expect that comic panels follow the order of text: left-to-right and down—a “Z-path”—though several layouts can violate this order, such as Gestalt groupings of panels that deny a Z-path of reading. To examine how layouts pressure readers to choose pathways deviating from the Z-path, we presented participants with comic pages empty of content, and asked them to number the panels in the order they would read them. Participants frequently used strategies departing from both the traditional Z-path and Gestalt groupings. These preferences reveal a system of constraints that organizes panels into hierarchic constituents, guiding readers through comic page layouts.

Cohn, Neil. 2013. Navigating Comics: An empirical and theoretical approach to strategies of reading comic page layouts. Frontiers in Cognitive Science. 4: 1-15.

Please help

2013-04-09T19:37:00.000-07:00

Just one more pleas for helping my colleague... www.nalinineedsyou.com. Please watch, repost, and help us save the life of an amazing woman:

Help Nalini Now

2013-03-31T23:07:00.001-07:00

This is not going to be your normal post on comics and visual language. This post is a plea for help from anyone who might read this blog and anyone compelled to help save someone's life. I summarize here much of what's said in a blog post from my colleague Sam Sommers.

From 2006 to 2012 I was a graduate student at Tufts University. One of the professors there was Nalini Ambady, who happened to be the advisor of many of my closest friends. You probably know Nalini's research in social psychology, though you may not have heard of her. Have you heard of Malcom Gladwell's book Blink, which talks about the "ability to extract an enormous amount of meaningful information from the very thinnest slice of experience"? The idea of "thin slices" comes from Nalini's work.

I am sad to say now that Nalini is battling with leukemia and is dire need of a bone marrow transplant. Because she is Indian, she has a very specific genetic marker that likely needs a match with another South Asian. Time is of the essence—she likely needs a match within 8 weeks. Her story is detailed much more at this website: http://www.helpnalininow.org

Helping save this woman's life is easy:

1) Register today as a potential bone marrow donor in the national registry. It’s easy: if you’re between the ages of 18-44 you can simply go here: http://marrow.org/Join/Join_Now/Join_Now.aspx?promo_code=MatchNalini

Make sure to enter the promo code “nalini” and your request for a cheek swab will be rushed to you and its processing expedited.

2) Everyone has the potential to save a life by registering on the site. In Nalini’s case, though, it’s particularly Indian donors who are likely to be a match. Accordingly, please forward this blog post or the http://www.helpnalininow.org website to any websites, email lists, or organizations with large South Asian memberships.

Please help. It takes just a few clicks of a mouse and time is running out. Your effort may help save a woman's life. Thank you.

New article: Visual narrative structure

2013-03-24T17:19:00.003-07:00

I am happy to announce I have a new paper published in the journal Cognitive Science that outlines my theory of sequential image understanding. I argue that approaches like "panel transitions" cannot account for the creation of meaning in sequential images, and I offer a new theory of "Visual Narrative Grammar" that better accounts for how we comprehend visual narratives.

You can download this study along with my other papers, or directly to the pdf here.

This new theory uses narrative categories similar to traditional notions of narrative (though operationalized) that are organized into hierarchic constituents. The basics of this theory are outlined, along with diagnostic methods for testing these categories and constituents. Finally, I outline how this theory can apply beyond the sequential images found in comics to the understanding of film and verbal discourse.

This paper presents the basics of this theory for the first time in a cohesive paper, though this paper actually only consists of part of the broader theory of Visual Narrative Grammar. I hope to discuss the theory in full in subsequent papers and books. I should note also that my experiments on the cognition of understanding comics (also available for download) use this theory as their basis.

Here's the full abstract:

Narratives are an integral part of human expression. In the graphic form, they range from cave paintings to Egyptian hieroglyphics, from the Bayeux Tapestry to modern day comic books (Kunzle, 1973; McCloud, 1993). Yet not much research has addressed the structure and comprehension of narrative images, for example, how do people create meaning out of sequential images? This piece helps fill the gap by presenting a theory of Narrative Grammar. We describe the basic narrative categories and their relationship to a canonical narrative arc, followed by a discussion of complex structures that extend beyond the canonical schema. This demands that the canonical arc be reconsidered as a generative schema whereby any narrative category can be expanded into a node in a tree structure. Narrative “pacing” is interpreted as a reflection of various patterns of this embedding: conjunction, left-branching trees, center-embedded constituencies, and others. Following this, diagnostic methods are proposed for testing narrative categories and constituency. Finally, we outline the applicability of this theory beyond sequential images, such as to film and verbal discourse, and compare this theory with previous approaches to narrative and discourse.

Cohn, Neil. (2013). Visual Narrative Structure. Cognitive Science, 37 (3), 413-452 DOI: 10.1111/cogs.12016

A Caveat: misunderstanding comics and the brain

2013-03-21T11:39:00.000-07:00

Via this article I stumbled onto this dissertation which promotes using comics in educational contexts (a topic I am very interested in).

In one of the chapters of the thesis, it looks at the understanding of comics and includes a "neuroscience" section. Now, even when I disagree with them, I am always one to encourage research and writing on the structure and comprehension of comics (and I am wholly in support of the effort that this thesis is trying to make with regard to graphic textbooks). However, I have also railed on the invocation of neuroscience when used inappropriately.

Given my commitment to educating about comics and the brain, it behooves me to explain exactly what is wrong with a discussion like this, and especially what is misleading/wrong with an image like this:

This graphic—and the accompanying description—reflect the worst of misunderstandings and invocations about the brain. Here are among the problems:

1) It suffers from the "localization" fallacy. Contrary to what the image implies, there is not a "speech" location in the brain. Nor is there a "writing" or "vision" location in the brain. The brain does not localize full behaviors in the brain in modular locations. Rather, the brain is highly interconnected, with portions of the brain devoted to particular functions that interface to produce complex behaviors like speech, language, or vision.

For example, while the occipital cortex (back of the head) is the "primary" visual area, the visual system alone goes up through the top of the cortex (the "dorsal stream"), down into the bottom, temporal lobe (the "ventral stream"), and connections from the eyes go through the middle of the brain on its way to the occipital lobe in the first place. In other words: the visual system engages lots of different parts of the brain, not just back of the head. (...and where is the rest of the brain in this image???)

Reducing the brain into localized areas for each of these complex behaviors displays a lack of understanding about how the brain works (at least, to the extent we understand it so far) and for how these complex behaviors work ("language" alone can be subdivided into at least four different major substructures, which each have more substructures). Beyond this...

2) Discussing the brain is entirely unnecessary in a thesis like this. The overall point that this thesis is trying to make is that comics are very complex and involve numerous interacting parts that we understand almost effortlessly. This point does not rely on discussion of the brain to come across. In fact, I myself have done quite a lot of work describing the complexity of structure found in comics, some of which discusses the brain, and some does not.

This thesis does not need it, and including a poorly-understood discussion of the brain only hurts its overall point. Now, studies have shown that just including an image of the brain somehow convinces people that its point is more believable. However, unless the thesis is actually about the brain, such discussion is unnecessary and borderline dishonest (albeit unintentionally). Which leads us to...

3) If you're going to talk about comics and the brain, at least do the basic research to discuss what work has actually been done on this topic. Granted, there are a limited number of studies that have directly examined this issue. However, they are out there, and reviews of some of these papers can be found on this very blog: Here, here, here, and here.

I myself have now done three studies looking at comics and the brain. I'm currently writing up my latest two studies (which were my dissertation), but my first experiment is online here: Full pdf article, Short "comic" version.

I'll close by reiterating what I said in my previous post on this topic:

My point overall is this: as cool and interesting as it is, not all arguments need to be tied to the brain and cognition. And, in fact, some arguments are made weaker by doing so, since appealing to neuroscience is unnecessary at best and hand-waving at worst.

Figure out what your point is and talk about it. I'm guessing it actually has little to do with neuroscience directly.

Hearts and bulbs

2013-03-14T23:42:00.000-07:00

Here's another great recent Savage Chickens comic by Doug Savage. I've commented on this great strip before, and here's another good "meta"-comic:

As we all know, hearts are symbolic of love (especially over the head), while lightbulbs over the head represent inspiration. I call "over the head" symbols like these "upfixes" since they are like affixes that are "up" (a term coined by my mentor, Ray Jackendoff).

Of course, as in the last panel, a heart and a lightbulb together show a love of lightbulbs, not a love of inspiration or ideas. This combination also does not give you inspiration about love either. What's interesting about Savage's observation here is that it nicely shows that you cannot combine the upfixes together. This is a first interesting trait: these upfixes have meaning on their own, but not in combination.

Related to this, the nature of the lightbulb upfix is to give you "value added" for its meaning. It no longer is just a lightbulb, but out of the relation above the head generates a new conventionalized meaning of inspiration. By adding the heart, it effectively removes this additive meaning, making the lightbulb simply a lightbulb once again.

The heart also changes meaning a bit as well. When the heart is an upfix, it describes the mental state of the person: The chicken is in love—with what, it doesn't matter. However, in the third panel, their combination makes the heart modify the lightbulb now—it's a love of lightbulbs—not merely reflecting a general mental state of the lover (the chicken).

Finally, this combination also changes the thought bubble. In the first two upfixes, the thought bubble mostly gives a depicted link between head and upfix. It doesn't mean thinking about love or about inspiration, but just reinforces these signs as being mental states. However, it is mostly unnecessary. The heart or the lightbulb would retain their meaning without the thought bubble. In the combination though, the bubble now returns to it's usual meaning as encapsulating thoughts. Having a heart and a lightbulb floating above the head wouldn't work as an upfix, nor would it work to convey thoughts.

Altogether, this simple, quirky comic tells us a lot about the structure of these types of signs!

Science and Eddie Campbell's rules of comics comprehension

2013-02-28T18:44:00.001-08:00

In Eddie Campbell's recent article at The Comics Journal, he described several potential "rhetorical rules" that authors of comics can follow in order to make them more understandable to inexperienced readers. In a previous post, I tackled the broader issue of what factors go into limitations to understanding. In this post, I discuss what scientific research tells us about his actual stated rules.

Rule 1: All the information necessary to understand the drama of a sequence must be contained in every panel of the sequence.

This rule actually reflects a very "Western" dialect of visual language that I would claim is even stronger in European comics than American comics (though I don't have data to support that). In two separate studies (first, second) I coded comics from America and Japan for how many characters appeared in each panel (the second breaks down American books into Indy and Mainstream genres).

Overall, I found that American comics used far more panels showing multiple interacting characters than Japanese manga, which used overwhelmingly more panels of single characters or close ups. This would support that American books use more sequences following "Rule #1" than Japanese books.

This difference has an impact on comprehension. Being provided with only parts of a scene (single characters) forces you to infer the larger scene. This requires more machinery in the narrative grammar (what I call "Environmental-Conjunction"), i.e., the rules in people's heads that allows them to comprehend sequential images. Yet, this does not necessarily lead to poor comprehension. Rather, it simply reflects a different grammar along with the need for a different type of fluency. Neither is better or worse. Just different.

So, as described in my last post (and the comments), the problem isn't that sequences like this are "incomprehensible" in some "universal" sense, but rather that those that have difficulty with them either 1) lack fluency in this grammar, or 2) have a different set of patterns in their heads from being fluent in a different visual language (such as European VL vs. American VL vs. Japanese VL).

Nevertheless, for Campbell's purposes of whether this would help an inexperienced (i.e., non-fluent) reader: There is no data at this point suggesting that the "Western" way leads to easier comprehension. I wouldn't doubt that this might be the case though, because it forces less inference.

However, it is worth also considering that were an actual author of comics to change their dialect in this way, it may have an effect on experienced readers. I would bet that doing as Campbell suggests would actually have an adverse effect on the reading experience for a Japanese manga reader, and possibly for a reader of mainstream American comics. So, for an author considering "changing their dialect" to that of Campbell's, they may have to weigh these issues (and for which audience is intended).

Rule 2: Ordering of speech balloons and Rule #3: Speech balloons should follow a system that can be intuited and doesn’t need to be explained.

Campbell claims that "After reading the contents of one balloon, the eye is likely to go to the next nearest balloon, even if that balloon is in another panel and the eye has not yet taken in all the balloons in the current panel. "

Some eye-tracking studies give us insight on this...

First, one study found that balloon position did have an impact on how often people skipped over content. They found that panels were skipped if they followed a panel that had a balloon with a dense amount of text. Breaking apart that balloon into smaller balloons with less text lead to less skipping over panels. This implies that alterations to balloons can have an impact on reading behavior.

Another study compared the eye-movements of "novice" and "expert" readers as they navigated through comic pages. The inexperienced comic reader had erratic eye-movements across a page and focused much more on the text. In contrast, the experienced comic reader had a very deliberate order of reading, and focused on the images much more than the text. This implies that an experienced reader would not jump around to whatever balloons are closest, but an inexperienced reader might.

Thus, in this case Campbell's rules might, as is their aim, help an inexperienced reader.

Rule #4: Timing only exists in comics if the reader agrees to play the game.

Unless I am mistaken, this rule has to do with people who read ahead in a book so that crucial information is known before it's read in the narrative. This has little to do with the structure of the narrative, and has to do with people flipping through a book beforehand. Contrary to Campbell's claim, I don't see how it's any different with skipping ahead in a novel except that images show you content.

I'd be curious for Campbell to expand here on just how authors should prevent readers from skipping ahead. The example he gives is fairly constrained and clever for preventing people from getting too much information by inadvertently reading ahead. How would this work for something like his example of "Magneto [showing] up surprisingly on the last page"?

It's worth mentioning here that how we interact with a comic as a physical object (flipping pages, accidentally looking ahead to the next page, etc) is different from our actual comprehension (the patterns in our heads that allow us to understand/produce sequential images). I don't think that Campbell confuses this issue, but it's a point worth remembering.

Rule #5: In a visual medium, a thing does not exist unless it is seen to exist

This rule applies to characters off-panel and indicated with the tail of a balloon, or to implied aspects of a depiction. Campbell notes in his example that we are to assume a character has his arms tied behind him because we don't see them (and it matches the context of the page), and that this is something that should be depicted.

However, this rule applies also to the legs of the characters: No character on that page is depicted below the waist, yet Campbell doesn't have an issue about that except to say that having a panel where the whole figures shown (feet and all) would help provide good spacing (i.e., Rule #10: have a panel with feet on every page). He doesn't say necessarily that if not shown, we won't believe they have feet.

It is certainly the case that "undepicted" elements are part of the conventional grammar of panel framing. Studies of children show the ability to treat a panel as a "window" on a scene is correlated with experience reading comics. Comic industry lore also tells of interactions with indigenous people who did not read comics who wonder why figures in panels without legs "had no legs at all." Unless you fully lack some type of basic fluency in the visual language, then this shouldn't be an issue.

Nevertheless, there is some validity in providing at least some notice of an element in a panel and not leaving too much to be inferred. People do track elements and concepts across panels. An element even subtly depicted once can then pervade inferentially across a sequence.

In contrast, another constraint on sequences aims at reducing how many elements need to be tracked across panels. Including too much information can lead to overload in working memory and can adversely impact comprehension. Thus, balancing these issues—what should or should not be shown—can be a delicate issue.

---------

So, Campbell's rules cover a wide range of issues. In most cases though, he is correct that inexperienced readers may have trouble with these issues. It is not clear though whether following such rules would help those people further comprehend sequences or if some of these alterations may have adverse ramifications on the reading experience of people who are actually fluent already. However, experiments could easily test these ideas...

Fluency and dialects in understanding comics

2013-02-22T17:08:00.003-08:00

In a recent article at The Comics Journal, Eddie Campbell describes the challenge facing some people who "can't understand comics," and offers "rules of comprehension" to help aid readers along in their ability to read comics. I thought the piece was really interesting, and Campbell intuitively taps into many ideas that are fundamental to sequential image understanding.

Nevertheless, there are several issues here being conflated that it might be nice to tease apart. We can categorize a lack of easy understanding of sequential images to two main things:

1. Lack of visual language fluency

There legitimately are people who cannot understand sequences of images. These people mostly have never had any experience reading comics, and the most compelling I have found are people from non-Westernized countries or have never learned any language (such as deaf individuals who never learned a sign language). These are far from the impoverished conditions that Campbell is talking about, but a lack of exposure to comics would still lead someone to not being able to make sense of sequences of images at all. Not just "be confused," but legitimately have a hard time connecting the meanings between panels.

Even among "fluent" readers of comics, comprehension differs based on experience. These differences can be seen in how people move their eyes across pages, and my studies show differences in brainwave amplitudes correlated with comic reading expertise. While Campbell's rules may aid "non-fluent" people a little, the entire task of reading comics will be difficult for them at a fundamental level.

In contrast, many of the "rules" that Campbell describes are more akin to prescriptive "rules" of "proper English" than the underlying structure of the language. These would be akin to "don't end sentences with prepositions" or "don't split infinitives"—both of which are not actual rules of English grammar. You wouldn't learn these rules in a class on English as a second language.

Rather, real rules of English would be things like "don't put adjectives after nouns" or "don't flip nominative and accusative case when using pronouns" and other rules that significantly impair the structure of a sequence. These rules are rarely produced by fluent English speakers because your mental grammar constrains the language intuitively enough to disallow them.

Similarly, there are significant rules of the grammar of sequential images that can really impair comprehension, and these types of rules are what people who "can't understand comics" really struggle with. Yet, they are rarely violated by authors of comics, who are fluent in their visual language already. Those who are not fluent, on the other hand, lack these core rules of understanding. I plan to discuss Campbell's actual rules in my next blog post.

2. Competition with another visual language grammar

A second type of difficulty in comprehension comes from preferences we have for one type of system over another. Campbell nicely acknowledges this, calling it an "idiom" or "style," and even making the comparison to accents. Accents are the right comparison, but, unlike Campbell's belief about "idioms," accents aren't a choice. They are reflections of the patterns in people's head that they acquire from their language. Furthermore, this may be what leads to distaste in other people's systems.

For example, speakers of one dialect of English (let's say Texan) might grate on the ears of speakers of another dialect (let's say New Yorkers)…and vice versa. This doesn't mean either group lacks fluency in their language, nor are they speaking some degraded form of "pure" English (which does not exist). They simply have patterns in their brains for their languages that differ in certain features, though they are still mutually intelligible. Of course, systems become even more difficult when they are not mutually intelligible—such as English speakers (of any type) and Japanese speakers.

Different "dialects" of visual language work the same. What some may view as incomprehensible storytelling may simply be competition of one visual language grammar (let's say "Indy" comics) with another (let's say "mainstream" comics). To a reader from one camp, it may seem as though the author is "bad" at storytelling or lacks the ability to be a decent visual writer. However, it may just be that the patterns in their head is different than those of the author. They might "speak" different visual languages.

I personally think this accounts for many of the complaints people have made about 1) comics from different countries (ex: people who have trouble with/dislike manga or bande desinée) or 2) younger artists (ex: the old guard's critique in the 90s with many Image Comics storytelling). This latter case simply is an instance of "those kids today are ruining the language," while the former is a cross-cultural reaction to a different narrative grammar.

--------

Nevertheless, the overall idea that there is a fluency to sequential image comprehension—and some people lack it—is just what we would expect from the notion of a visual language. This idea underlies a very different perspective than Campbell's. Rather than believe that sequential images are somehow universal—and thus the problem in understanding is simply certain surface features—the idea of a visual language acknowledges that the production and comprehension of sequential images directly ties to patterns in people's heads. Because of this, despite the mutual intelligibility that iconic drawings offer, these "rules" need to be learned, and they may differ depending on which visual language you read and draw.

The Principle of Equivalence

2013-02-14T00:23:00.003-08:00

An overarching theme across my research is the idea that the structure and cognition of drawing and sequential images is comparable to that of language. I have tried to formalize this notion under the umbrella of the "Principle of Equivalence":

The human mind/brain treats all modalities in an equal way, given modality specific constraints.

There are two parts to this Principle. The first part (in the first clause) is the idea that the mind/brain has general cognitive principles that all factor into different domains in similar ways. We should therefore expect that the cognition of spoken language, sign language, and visual language be similar, just as we might expect facets of actions, music, dance, and others to share parts of that same cognitive orientation.

Among the things that might be similar across domains might be: 1) the ability to store systematic parts into our long term memory (i.e., a "lexicon"), 2) the ability to manipulate those parts to make larger units, 3) the hierarchical organization of sequences that enables those parts to be organized in an infinite number of ways, etc.

The second part of this Principle (the second clause) wards off an overextension of these similarities. In some ways, we should absolutely expect that different human behaviors are processed differently. However, we would expect that the nature of those differences is a direct result of the nature of the behavior itself.

For example, we might expect spoken language to differ from drawings in certain ways, because drawings are analog and spatial, while spoken language is digital and temporally constrained. All of these differences are directly related to the fact that drawings are visual-graphic while spoken language is verbal-auditory. The differences come directly from the nature of the expressions.

Overall, this Principle is affirming to the general processes of the mind. Why should the brain create lots of unique diverse ways to handle different behaviors when it can efficiently make use of various general underlying structures (like those listed above) in a variety of capacities?

Coming back to the overall idea of drawings being structured, processed, and learned comparable to language, the Principle of Equivalence demands a counter to any theory going against it. It's notable that this is the vast majority of theories about drawing from the past century, which do not think drawings are structured in a systematic, conventionalized way, but rather that they represent perception.

To these theories, they must address the key question: Why should drawing and sequential images NOT be processed like language or other human behaviors? The same questions should be asked of language and other behaviors: What makes it alone unique and different, and why is that advantageous to cognition or behavior?

Eye-umlauts

2013-01-23T17:34:00.000-08:00

Savage Chickens never seems to fail to give me a laugh. This one especially had me chuckling...

The reason I liked this one so much is that it plays on a common convention where certain symbols (like hearts) replace the eyes of a character. I've likened this type of substitution to a kind of morphological change in words when a word changes in meaning through an internal change of the sounds. For example, to make the past tense of jump we add -ed to make jumped. But, to make the past tense of run we change the sound of the vowel to make ran. This internal change is called "umlaut."

In a sense, substituting a sign for the eyes or pupils of a character is similar to umlaut, since it makes an internal change in the face of a character to change its meaning. This creates a whole class of "eye-umlauts" that include hearts, stars, dollar signs (or any money signs), spirals, X's, and others. Indeed, we could even make new eye-umlauts too, such as using "+" plus signs in the eyes to insinuate someone is lying (for instance).

In this way, the morphology of images uses a similar strategy as the construction of words in language.

Language and culture development

2013-01-03T11:39:00.002-08:00

I've been a big fan of Jorge Cham's comic PhD Comics for a long time, and I've really been impressed with his recent foray into short educational videos. I found this one to be particularly fun, and since it overlaps with my research interests, I figured I'd share it here too:

Big year behind, big year ahead

2013-01-01T17:27:00.001-08:00

Happy New Year! Between finally finishing my PhD, moving back to California to start at research fellowship at UCSD, and being featured in a Discover Magazine article, 2012 was a big year. However, 2013 looks to be even bigger...

First off, this Fall will see the release of my book, The Visual Language of Comics: Introduction to the Structure and Cognition of Sequential Images, by Bloomsbury. I'm very excited about it, because it finally lays out the full foundation of my theories of visual language.

The other main event is that later this year I plan to fully relaunch my website and blog, including a new web address. If you want to plan ahead, the new url for emaki.net will become www.visuallanguagelab.com (currently a redirect). The blog will maintain it's main url (www.thevisuallinguist.com), and I will retain the RSS feed via feedburner. The new site will better reflect the changes that have occurred in my research over the last few years, and prep the site for when I set up a real lab as a professor, hopefully in a few years.

Spider-Man's naughty adventures within or between panels

2012-12-19T11:37:00.001-08:00

A friend of mine recently pointed me to an unusual debate that's raging about a recent Spider-Man comic where Peter Parker apparently is within the mind of Dr. Octopus and is forced to see his memories. One questionable scene occurs with Aunt May, where the scene is unclear whether she and Dr. Octopus kiss (and thus Peter experiences it), or whether more illicit behavior happens:

As my friend wrote to me, "Amid fan outrage, the writer, Dan Slott, actually started hitting up message boards to claim that his intent was to imply a pre-wedding kiss... most fans have assumed that much more went on. Who is right?"

The short answer is: it is entirely ambiguous about what happens. This is exactly the same as the example McCloud shows with the guy with the axe (right). The unseen actions need to be filled in by your mind given the information you are provided. In both cases, you are given preparatory actions, and then shown something different, with only a word balloon to connect to the preparatory action.

The key is that your mind tries to figure out what happened because of the associated word balloon and because the second panel doesn't show the action. It's worth emphasizing that, contrary to McCloud's claims, the "filling in of the information" does not happen between panels, but within that underspecified second panel. In both cases, the authors choose not to show the action.

In the Spider-Man example, the first panel is also slightly underspecified. It hints at Aunt May about to kiss, but the action isn't defined well enough. The "No Stop!" balloon reads as anticipatory, suggesting that something is about to happen. The most direct answer then would be a kiss in the second panel, because it's set up directly in the previous panel (just like the axe sets up chopping in the second panel).

However, there is nothing to prevent other interpretations, since the first panel's "preparatory kiss" wasn't drawn that clearly and the second panel is entirely ambiguous because all you see is a door and all cues come from the word balloon. The question then becomes how associated is the word balloon "Ahhh!" with just kissing or with something else.

(It is also worth mentioning the implication of duration that an action takes place. Kisses are can be single pointed actions, or can be drawn out (as can...ahem... other actions). Showing a door doesn't give any further information about the duration of time passing in the hidden event—this is also only suggested by the word balloon.)

So... from a structural perspective, there is no real "right answer." It is a (likely intentionally) suggestive and ambiguous depiction, but it nicely plays with properties of the visual language grammar to elicit varying interpretations.

For more information on these types of concerns, see my paper The limits of time and transitions (pdf here).

Basic structures of visual language

2012-12-13T09:00:00.000-08:00

One of the important basic tasks of doing research on the visual language used in comics is to identify the foundational components that go into our comprehension of sequential images. In Understanding Comics, McCloud implicitly broke down the medium into a few parts:

1. Graphic Style
2. Iconography and symbolism
3. Panel-to-panel relationships
4. Text-image relationships

These parts provided a nice initial foray into how the visual language of comics might be segmented. However, the crux of my research outlines that the structure of sequential images actually breaks down similarly to language, and can thereby be researched using similar tools. This gives us several components of the visual language of comics, many of which tie to McCloud's:

1. Graphic Structure is how we understand the visual pieces of an image. Are certain junctions of lines more appropriate for certain parts of an image? How do we understand lines and shapes? This is the equivalent of studying the sound properties of language, only here in the visual-graphic form.

2. The Lexicon is the vocabulary of systematic pieces used to create images and sequential images. These might range from the morphology of visual conventions (like motion lines) to systematic full panels (like those from Wally Wood's 22 Panels) or patterns of storytelling (like the set up-beat-punchline pattern). Basically, anything that is used as a pattern is a part of the lexicon of visual language.

2.2. Morphology is a particular part of the lexicon that deals with small components of meaning (like McCloud's iconography and symbols). However, morphology also includes the principles for how these parts combine together. Why do stars above heads mean one thing but replaced for eyes mean something else? Why can't lightbulbs also replace eyes to mean inspiration, like they do floating above heads? Why do motion lines always trail behind objects, but not in front?

3. Event Structure is how people understand the nature of events, and in sequential images we may have to rely on knowledge about parts of an event to understand the whole. If an image shows a person punching another, we infer that the puncher reached back their arm first. We also need to be able to make sense of the connections in meaning between and across panels.

4. Spatial Structure has to do with the knowledge that panels convey information about a fictitious spatial location. Each panel only frames a glimpse of this location, and our minds build the overall space. If one panels shows the exterior of a house and the next shows someone sitting at a table, how do we know that they are inside that house without overt cues? If panels in a sequence only depict individual characters, how do we know they belong to the same broader environment?

5. Narrative Structure is how we make sense of the meaning of a sequence of images—its grammar. The event or spatial structures convey the meaning of a sequence, but this meaning is guided through its presentation in a narrative structure. Why delay the climax of a sequence until after several lead-in panels? Why show a scene where each panel shows individual characters instead of all characters in just one panel? These have to do with the presentation of meaning, not just the meaning itself.

6. Navigational Structure is the system used to move through a page layout. Why do people read from left-to-right in America instead of vertically down-then-up? What happens when layouts depart from simple grids? These issues go beyond just the meaningful connections between panels and have to do with a reader's preferences for how to move from panel-to-panel on a page.

7. Multimodality is the phenomenon of getting information from different domains. In this case, we receive information from both text and image, and thus need to explore how these multiple signals cohere to form a single conception (or, in reverse for creation: how a single conception results in multiple signals).

------
These are the broad components at work in comprehending sequential images. Many questions have yet to be answered about their parts and their relationships. And, of course, we can also ask how these components might differ across cultures, how people learn these conventions, and how their understanding changes over development.

Importantly, when we look at these components through a linguistic or cognitive perspective, we can't simply think about it terms of the components of the medium. Rather, we must think about these components in terms of what authors or readers must know in order to create/understand this visual language.

In other words, it shifts the focus to what's going on in people's minds and brains. Because of this type of shift, we can then ask how this knowledge may be similar or different from what we know about other cognitive systems, spoken and signed languages in particular.

Blog-iversary and updates

2012-12-04T06:00:00.000-08:00

So, today marks 7 years that my blog has now been online, and 10.5 years for the website. How time flies!

I've received a lot of great feedback from people about my profile in Discover Magazine, so it's worth having a post here to just review what I'm working on. My original study described in the Discover article can be found here (pdf), or a short, "comic" version here (pdf). I do have a few other brainwave studies that examine sequential images (from my dissertation), though they are still being written up for publication.

All of these papers are based on a theoretical model of a "narrative grammar" I've been developing for the past 12 years. The seeds of that theory appear in my book, Early Writings on Visual Language, though the approach in there has been far surpassed by my recent work. A concise version of this narrative grammar is set to be published by Cognitive Science soon.

In addition to that, I will have a new book out next year published by Bloomsbury called, The Visual Language of Comics: Introduction to the Structure and Cognition of Sequential Images. It will lay out the basics of my full theory of visual language, as well as summarize the experimental work done in psychology about how sequential images are comprehended. I'm very excited about the book, since it finally lays out the broader picture of my theories, and should provide a solid foothold for people who are interested in this research. I believe we're looking at a Fall release, so stay tuned for more updates as it gets closer...

Finally, I'm working on new research as part of my postdoctoral fellowship here at the Center for Research in Language at UC San Diego. We're currently designing a new brainwave study that looks at the intersection of how people make predictions and "fill in" missing information in a visual sequence.

These all look to be the foundations of a growing field, so I hope you stick around to see how things look once we really pick up steam.

Discover Magazine article

2012-11-28T15:49:00.002-08:00

I'm proud to say that I'm featured in an article in this month's Discover Magazine! The article was written by the excellent Carl Zimmer, who good-naturedly let me run him through my experiment for the sake of the article. I'd actually read articles by Carl for many years, so it was fun to interact with him for the interview.

I should note a slight correction to the the reported results of my study. While the difference between the Scrambled and Narrative Structure Only sequences did show a "left anterior negativity" (correlated with syntax), the difference between the amplitudes of those sequences and normal ones showed a different waveform, called the N400 (correlated with semantics). So…

Normal vs. Structural Only = N400
Normal vs. Scrambled = N400
Structural Only vs. Scrambled = Left Anterior Negativity

You can read the original article here (pdf), or a short, "comic" version here (pdf).

Overall though, Carl did a great job describing my study and this type of research. I'm very humbled to receive the attention. Go read!

Sequence vs. Singles in "visual language"

2012-11-20T09:00:00.000-08:00

In my last post, I addressed the basic idea for a "visual language" as being a sequence of meaningful images guided by a system of constraints (i.e., a grammar). In the comments, I was asked a very good question:

Why is a sequence necessary for the graphic form to be considered "language"?

There are two main reasons for this, both which relate to the analogy with verbal and sign languages. As I said in that post, my notion of "language" in "visual language" is not metaphorical, but rather based on conceptions from the linguistic sciences.

The first reasons is structural. Languages are made up of three primary components:

1. The conceptual structure of meaning in the mind
2. A sensory modality they can be expressed in (i.e., sound, body motions, graphics)
3. A "grammar" that guides and constrains the sequential expressions of meaning

In the verbal form, the main grammar is syntactic structure, which allows us to sequentially order words and other expressions into coherent sentences. However, technically all of these components (meaning, modality, grammar) are built of rule-systems that constrain them. The phonological system that guides our production of sounds also is constrained by rules. This is why English cannot have words that start with the sound combination "tf" or why the "c" in elastic goes from a "k" sound to an "s" sound in elasticity. These are rules guiding the modality itself.

The analogy for the graphic form holds these same functions. The primary "grammar" guiding images is a "narrative grammar" which guides the presentation of meaning in coherent sequences. Both syntax and narrative function in the same general way: to present meaning in a coherent sequence. They also share methods of doing this, such as chunking units into groupings, making connections between distant units, and embedding groupings inside each other.

However, single images also have a constraining system which is analogous to phonology. You could call it "photology" or "graphology" perhaps. This system similarly constrains the modality itself. This is why certain junctions of lines are awkward, like when you want to show occlusion (one thing in front of another), but instead of using a "T" shaped junction of lines, you use a "Y" or "+" shaped junction.

So, structurally, single images are guided by a rule system, but that system is closer to that of phonology than syntax.

The second reason for sequence being important comes from analogies with development.

By and large, when people are not exposed to a language within the right time period of life, they won't learn language. They seem to be able to still acquire a limited set of vocabulary (i.e., words) but the most problematic component is the syntax.

Even when people are able to learn a spoken language, but never learn sign language, they can still all gesture. The manual modality doesn't disappear as a way to create meaning—it just functions using single expressions without a grammatical sequence.

This same trend is true of drawing and sequential images. Most people cannot draw a coherent narrative sequence. However, they can all use the drawing system ("photology") to create meaningful single images (albeit rudimentary ones if they haven't fluently developed the vocabulary of the drawing system either).

So, the analogy then holds that single images are to visual language what gestures are to sign languages. One type uses a modality for single novel expressions (single images/gestures) while the other uses complete grammars in sequences of expressions (sign language/visual language). The evidence comes because even a rudimentary form of the simpler expressions (single images/gestures) is maintained even if the full grammar is not developed.

Incidentally, quite a lot of my discussion about the structure of single images and the development of the drawing system is available in my recently published paper, "Explaining 'I can't draw'" available as a pdf here.

Revisiting "visual language"

2012-11-13T09:00:00.000-08:00

I've now had this website for over 10 years, and have been blogging for almost 6 years, so it may be worth revisiting the fundamental ideas of my research over the next few posts. Hopefully by this time next year my book, The Visual Language of Comics, will be out and describing these ideas in even more detail. Until then...

Let's start with the obvious: What is "visual language"?

There are several ways that the term "visual language" can be used. Sometimes it is used to talk about general visual information or visual culture. It might be used as a broad term for visual culture, or for any combination of text and images. Some people use it to describe creative ways to use writing in pictorial ways.

None of of these are what I mean by "visual language."

This isn't necessarily a bad thing. These and other applications of the term use "language" in a very metaphorical sense, usually by extension to mean "communication."

My meaning of "language" is actually very literal, based on the scientific definitions of language. By extension, my definition of "visual language" is also very specific.

So, what do I mean by "visual language"?

Human beings as a species can only convey our thoughts in three ways: we can 1) create sounds with our mouths, 2) move our bodies (especially hands and faces), and 3) draw things. That's it.

When any of these channels is put into a sequence, such that some sequences are good and others are bad, then the result is a "language." Thus, sequential sounds (words) become spoken languages, sequential body movements become sign languages (as opposed to gestures) and sequential images literally become visual languages.

Given this, individual images are similar to single expressions (which have their own rich structure), while sequential images form a visual language.

So, what is writing? Writing is the learned importation of the spoken form into the visual form (essentially a learned synesthesia). This is not natural, which is why it's so hard to learn to read and write, and why most of the world's languages use no writing systems.

By contrast, the ability to draw sequential images is a natural ability that is accessible to anyone who receives the proper exposure and practice at it.

Given all that, now what about "comics"? Well, comics are the place that we predominantly find these visual languages used. Just like novels are written in English, comics in America are written in American Visual Language. Or, manga in Japan are written in Japanese Visual Language.

And, of course, comics are not just written in the visual language of sequential images, they also use written language. So, technically, comics use two languages that combine to make a larger whole of communication. This is actually similar to the way we communicate generally. We constantly combine modalities: we gesture when we speak, text and image often come together, etc.

From this basic idea, that sequential images literally create a natural visual modality of language, innumerable other questions emerge about the nature of graphic communication, it's cognition, and how it can be used in society.

New Article: Explaining "I can't draw"

2012-10-23T12:47:00.000-07:00

I'm happy to say that I have a new article published in the journal Human Development that argues that learning how to draw is similar to learning how to speak.

I've always thought that this was among my best ideas, and apparently the journal agreed: they thought it was provocative enough that they invited two additional scholars to comment on my paper. From one of the reviews:

"Cohn’s paper can be viewed not just as an account of the development of drawing but also as representing a paradigmatic shift in the way we conceptualize the role of nature and nurture in development."

Here's the abstract:

Both drawing and language are fundamental and unique to humans as a species. Just as language is a representational system that uses systematic sounds (or manual/bodily signs) to express concepts, drawing is a means of graphically expressing concepts. Yet, unlike language, we consider it normal for people not to learn to draw, and consider those who do to be exceptional. Why do we consider drawing to be so different from language? This paper argues that the structure and development of drawing is indeed analogous to that of language. Because drawings express concepts in the visual-graphic modality using patterned schemas stored in a graphic lexicon that combine using ‘syntactic’ rules, development thus requires acquiring a vocabulary of these schemas from the environment. Without sufficient practice and exposure to an external system, a basic system persists despite arguably impoverished developmental conditions. Such a drawing system is parallel to the resilient systems of language that appear when children are not exposed to a linguistic system within a critical developmental period. Overall, this approach draws equivalence between drawing and the cognitive attributes of other domains of human expression.

The article is available directly here.

The Graphic Canon Vol. 2

2012-10-11T16:06:00.003-07:00

On the non-theory front, I'm happy to announce that I have a piece in the recently released second volume of The Graphic Canon edited by Russ Kick. It's a collection of great literature, in this case from the 1800s, and adapted into graphic form by various authors. The book is beautiful inside and out.

My contribution is the second of my two versions of John Keats' "La Belle Dame Sans Merci." I'd first read the poem in high school and decided to draw a version during my first semester in college. Shortly after, I realized a second interpretation could be done of it that perhaps was closer to the way most interpret the poem, so I drew a second version that drew from the same layout and composition as the first (along with many of the same full pages).

I'm very proud to be able to contribute to this collection, and am glad its gotten so much attention. My piece pales in comparison to many of the others, so it's definitely worth checking out!

Review: How fast can you comprehend comic panels?

2012-10-01T11:09:00.002-07:00

In this study, the authors wanted to know how much time it would take to comprehend each image of a sequence of images, both for how long each panel stayed on the screen, and for how long the time was between each panel ("interstimulus interval" or "ISI"). They compared normal four-panel long strips with sequences where the third panel was reversed with an adjacent panel (1-2-4-3 or 1-3-2-4).

In the first experiment, they varied the length of time that each panel stayed on screen, 83 milliseconds (ms) and 150ms, with a constant ISI of 300ms between each exposure. They found that the responses to whether the sequence was in correct or incorrect order varied per speed. Panels at 83ms were only correctly responded to 24% of the time, while those at 150ms were correctly responded to 71% of the time. They conclude that 150ms is the minimum time necessary for exposure.

Experiment 2 varied ISI—the time between each panel—keeping each panel exposed on screen for 150ms. They found that accuracy increased as ISI increased. At 133ms, accuracy reached around 70%, staying constant through 217ms and 300ms. They thus conclude that an ISI of more than 130ms is necessary.

I actually find these numbers to be blazing fast. In my experiments, we used a consistent ISI of 300ms to avoid the effect of panels seeming like they turned into a flipbook style animation. In self-paced reading, the speed of processing panels depended on both the complexity of the panel and its context in the sequence, but people would often average between 700ms or 1 second for reading each panel. In our measure of brainwaves—which are even more sensitive to the timing of the brain's comprehension—we don't fully get a response for activation of recognizing something is awry in meaningful information until starting around 200ms to 250ms at the very soonest.

Thus, I find it highly surprising that an exposure time of 150ms and an ISI of 130ms would be sufficient to get as accurate responses as they did. I would think that these numbers would be the absolute minimum amount of time necessary, and that these numbers may get larger if the panels were more complex (their stimuli looked even more simple than the Peanuts panels we use in our experiments).

Inui, Toshio, & Miyamoto, Kensaku (1981). The time needed to judge the order of a meaningful string of pictures Journal of Experimental Psychology: Human Learning and Memory, 7 (5), 393-396 DOI: 10.1037//0278-7393.7.5.393

New article: Framing attention in Japanese and American comics

2012-09-24T09:44:00.000-07:00

I'm pleased to say that I now have a new article out about the cross-cultural differences between American comics—both Mainstream and Indy comics—and Japanese manga:

"Framing attention in Japanese and American comics: Cross-cultural differences in attentional structure."

I'm particularly excited about this paper, because my co-authors are two former undergraduates at Tufts University—Amaro Taylor-Weiner and Suzi Grossman—who worked very hard and took this project on.

This paper shows further evidence that the panels in Japanese manga structure space differently than the panels in American comics, regardless of genre. We argue that these patterns connect to deeper differences in cognition that have been found between Americans and Asians (here, Japanese specifically).

Download the new article (pdf)

Download my previous article on cross-cultural differences (pdf)

Full abstract:

Research on visual attention has shown that Americans tend to focus more on focal objects of a scene while Asians attend to the surrounding environment. The panels of comic books— the narrative frames in sequential images—highlight aspects of a scene comparably to how attention becomes focused on parts of a spatial array. Thus, we compared panels from American and Japanese comics to explore cross-cultural cognition beyond behavioral experimentation by looking at the expressive mediums produced by individuals from these cultures. This study compared the panels of two genres of American comics (Independent and Mainstream comics) with mainstream Japanese “manga” to examine how different cultures and genres direct attention through the framing of figures and scenes in comic panels. Both genres of American comics focused on whole scenes as much as individual characters, while Japanese manga individuated characters and parts of scenes. We argue that this framing of space from American and Japanese comic books simulate a viewer’s integration of a visual scene, and is consistent with the research showing cross-cultural differences in the direction of attention.

Cohn, Neil, Taylor-Weiner, Amaro, and Grossman, Suzanne (2012). Framing attention in Japanese and American comics: Cross-cultural differences in attentional structure Frontiers in Psychology - Cultural Psychology, 3, 1-12 DOI: 10.3389/fpsyg.2012.00349