The Visual Linguist

More changes

2012-05-21T18:05:00.001-04:00

As of yesterday I fully received my PhD (and the fancy hood!) from Tufts University. It was a long and great journey, but now it's on to the next venture. As I mentioned in my last post, I will be spending next year working on an introductory book on visual language theory that is due for Fall of 2013. I've been working hard on it lately and am very excited about it.

I will also be joining the Center for Research in Language at UC San Diego as a post-doctoral fellow starting in September. I'll be further investigating the neurocognition of sequential image processing (i.e., what happens in the brain when people are reading comics). I also plan to start learning techniques for measuring eye-movements, so we can begin to examine what people are looking at in comic panels (and how that relates to what goes on in the brain). I'm very excited for this opportunity and the potential for enlightening new collaborations and research.

In the meantime, hopefully I can start posting on the blog more often. Hopefully...

A User's Guide to Thought and Meaning

2012-05-07T01:53:00.001-04:00

Lo and behold, I now have a new book out! My mentor, Ray Jackendoff, has a new book, A User's Guide to Thought and Meaning, and it is chalk full of illustrations by me! (... along with some choice Zippy strips by Bill Griffiths)

He has been working on this book throughout my time as his student, and I think the result is truly excellent. If you're looking for a good book about language, meaning, thought, and their relations, this is a good, non-technical read. I can't recommend it enough, and not just because my name is on the cover page. Check it out!

From the publisher's description:

A User's Guide to Thought and Meaning presents a profound and arresting integration of the faculties of the mind - of how we think, speak, and see the world. Ray Jackendoff starts out by looking at languages and what the meanings of words and sentences actually do. He shows that meanings are more adaptive and complicated than they're commonly given credit for, and he is led to some basic questions: How do we perceive and act in the world? How do we talk about it? And how can the collection of neurons in the brain give rise to conscious experience? As it turns out, the organization of language, thought, and perception does not look much like the way we experience things, and only a small part of what the brain does is conscious. Jackendoff concludes that thought and meaning must be almost completely unconscious. What we experience as rational conscious thought - which we prize as setting us apart from the animals - in fact rides on a foundation of unconscious intuition. Rationality amounts to intuition enhanced by language. Written with an informality that belies both the originality of its insights and the radical nature of its conclusions, A User's Guide to Thought and Meaning is the author's most important book since the groundbreaking Foundations of Language in 2002.

Review: "Language of Comics" in Art of Comics

2012-04-16T20:59:00.000-04:00

The recent compilation, The Art of Comics edited by Aaron Meskin and Royt Cook contains several new articles on a "Philosophical Approach" to comic theory. This book tackles many interesting and pertinent topics about comics, with varying degrees of effectiveness.

Here I'm going to focus on Darren Hudson Hicks' article, "The Language of Comics" since it is especially pertinent to the topics of my research and this blog. Hicks explores the claim that "comics constitute a language" and analyzes that claim in light of Currie's opinions that film cannot be a language. There are many problems with this article, and, I'll focus on some of the largest affronts. In the philosophical tradition then, I don't feel I'd do it any service to pull my punches, so here goes...

In much of this article, Hicks defends the similarities of comics and natural languages, and tries to point out where Currie's arguments fall flat. I should say upfront that I support this aim, since my own work has to respond to the same questions. To be complimentary upfront, I do agree with many of his points, and think that he is overall on the right track with many of his arguments. The biggest problem is his lack of knowing (or just citing) other appropriate literature.

For example, Currie argues that film cannot be considered a language because the signs (shots) are not arbitrary symbols. Hicks then defends comics for having a great deal of conventional signs, while pointing out that language also is not entirely arbitrary. His conclusion is that symbolic and iconic signs actually lie on a continuum, not a discrete categorization.

I agree with Hicks' position overall here and his analysis of it (neither all words nor all images are solely iconic or symbolic). My problem is that Hicks' conclusion misses the mark because of a lack of knowing previous research. He references C.S. Peirce's well known division between iconic, indexical, and symbolic types of reference in discussing the difference (and overlap) between images and words. Yet, he does not acknowledge that within Peirce's system, conventionality is not solely associated with symbols. Peirce recognizes that all three of those types of reference can be conventional, but only symbols derive their meaning solely from conventionality (i.e. Peirce readily would say that smiley faces are Conventional Icons, just like words are most often Conventional Symbols). Had Hicks known this distinction, the idea of a continuum between iconic and symbolic reference would not be needed. This reveals a lack of actually knowing the literature that is being cited.

Another grating section is where Hicks discusses the relationships between panels on a page. Here, he is trying to argue that individual panels on a page interact greatly with other panels. His point is well made, but in his discussion he resorts to reporting where "the eye" moves while reading the example comic page that is reprinted from Xenozoic Tales. As a cognitive psychologist, this grates on my nerves, because if eye-tracking experiments have taught us anything, it's that we often do not consciously know where our eyes are looking. To me, this renders his whole description a bit vacuous, because it is based on a faulty premise that he actually knows where his eyes are looking (which he doesn't: My own first reading of the page completely missed details he claims are "visually prominent" and that my eye should "gravitate towards" showing outright that his analysis might be wrong).

Overall though, the discussion of the relationship between comics and language is framed in the wrong way. Because he mostly adheres to the McCloudian conception that "comics ARE sequential images (± text)", he then must deal with the issue of whether "comics ARE language."

This is the wrong comparison, though it is frequently made. As I have discussed at length for the past 10 years (here, here, here, in talks, and many comixpedia articles and blog posts), "comics" are not a language. Rather, "comics" are a cultural context in which a visual language of sequential images is used, where it often combines with text. Just as novels are written in English, comics are written in a visual language (plus maybe also a written language). Dylan Horrocks hinted towards a similar breakdown in his essay, "Inventing Comics."

If Hicks recognized this disparity, much of his arguments would be simplified, and he would not have to deal with the issue of "defining comics" in relation to language. Also, he would not have to deal with the sticky issue of text-image relationships, since if "comics ARE language, what does it mean for that to enclose another language?" This whole issue is rendered moot if "comics" aren't argued as a language, but "sequential images" are a visual language which combines with written language in a socio-cultural object of "comics". (Another pet peeve here: he unnecessarily appeals to the brain processing text and images differently. Not only is mentioning the brain superfluous, but his citation for this is from over 40 years ago. Again... lit review?).

Not recognizing this argument for separating "visual language of sequential images" and "comics" again shows a lack of reading previous literature. In this case, it's a little personal, because it relates directly to my own work. Despite my work probably being the most vocal advocacy of the relationship between language and sequential images over the past 10 years, nowhere is my work mentioned or cited (though an actual comic of mine is featured and cited in a different essay in the book).

If this omission was on purpose (which I doubt), it raises the issue of "why"? If it was not on purpose (as I suspect), it betrays a lack of basic research on this topic. This is more my issue with the article. It's not so much that my ego is bruised (Horrocks should be mentioned, as should Mario Saraceni's dissertation, and others), but leaving it out seems an oversight in doing the appropriate background research for a paper topic that could greatly benefit from this point of view. (The books editors should also have given feedback on this, especially those I've corresponded with).

Knowing my work would also be useful for his concluding paragraphs. Here, he dismisses the idea of sequential images (re: "comics") being a type of full natural language because he cannot conceive of a "syntax" for sequential images. In fact, my book Early Writings on Visual Language laid out my first model of generative "syntax" for sequential images all the way back in 2003, and my recent research has actually provided empirical evidence for psychological validity of a "grammar" for sequential images. Granted, Hicks does mention two other approaches to "syntax" by Saraceni and Groensteen. But, these approaches receive little attention outside an endnote, and they are not discussed in depth. One would think such an important topic would receive more than passing mention as being a too "difficult concept to wrap one's head around" in the concluding paragraph.

Essentially then, this leaves me with the impression that Hicks is saying, "This idea is beyond me, so I can't address it well enough, and/or it must not actually exist." This does not see like the lasting impression one wants to have about an essay in a book collection purporting to be a solid foundation for a "philosophical" approach to comics.

Looking forward and backward

2012-04-10T11:43:00.003-04:00

So, after 12 years of research and 6 years of grad school, I'm proud to say that I successfully defended my dissertation yesterday and now have a PhD in Psychology studying the how people's minds and brains understand comics. It's been a wild ride!

What's next then? Well, I have several papers that are set to come out soon or will soon be submitted to journals. Also, this seems like a good a place as any to announce that I will have a new book coming out in late 2013 from Continuum Books. It will introduce visual language theory, and will outline the basic structure and cognition of visual narratives. I'm very excited about it, and will post updates periodically as it approaches.

Thanks, dear readers, for your continued support of this work, and I look forward to big things to come.

Public Defense

2012-04-06T10:49:00.002-04:00

For anyone in the Boston area and interested in hearing about comics and the brain, I'm defending my dissertation on Monday, and it's open to the public. I'll be speaking about my research on the "grammar" of sequential images, as found in comics:

"Structure, Meaning, and Constituency in Visual Narrative Comprehension"
Monday, April 9th at 4:30pm
Kreplick Conference room on the first floor of the Tufts Psychology Building (490 Boston Ave, Medford 02155)

New Article: Comics and the brain

2012-03-04T17:03:00.000-05:00

I have a new paper available online (pdf), and I'm proud to say that this is my first brainwave study on comics. In this paper, now published by Cognitive Psychology, we argue that sequential images use a narrative "grammar" to distinguish coherent narrative sequences from random strings of images. We conducted two experiments measuring reaction times and brainwaves to examine the contributions of narrative structure and meaning to processing sequential images. Our findings provide evidence that sequential image comprehension uses a narrative structure that goes beyond "transitions" between panels.

Below is the abstract, though here's a pdf of a "graphic" version of the abstract...

Just as syntax differentiates coherent sentences from scrambled word strings, the comprehension of sequential images must also use a cognitive system to distinguish coherent narrative sequences from random strings of images. We conducted experiments analogous to two classic studies of language processing to examine the contributions of narrative structure and semantic relatedness to processing sequential images. We compared four types of comic strips: (1) Normal sequences with both structure and meaning, (2) Semantic Only sequences (in which the panels were related to a common semantic theme, but had no narrative structure), (3) Structural Only sequences (narrative structure but no semantic relatedness), and (4) Scrambled sequences of randomly-ordered panels. In Experiment 1, participants monitored for target panels in sequences presented panel-by-panel. Reaction times were slowest to panels in Scrambled sequences, intermediate in both Structural Only and Semantic Only sequences, and fastest in Normal sequences. This suggests that both semantic relatedness and narrative structure offer advantages to processing. Experiment 2 measured ERPs to all panels across the whole sequence. The N300/N400 was largest to panels in both the Scrambled and Structural Only sequences, intermediate in Semantic Only sequences and smallest in the Normal sequences. This implies that a combination of narrative structure and semantic relatedness can facilitate semantic processing of upcoming panels (as reflected by the N300/N400). Also, panels in the Scrambled sequences evoked a larger left-lateralized anterior negativity than panels in the Structural Only sequences. This localized effect was distinct from the N300/N400, and appeared despite the fact that these two sequence types were matched on local semantic relatedness between individual panels. These findings suggest that sequential image comprehension uses a narrative structure that may be independent of semantic relatedness. Altogether, we argue that the comprehension of visual narrative is guided by an interaction between structure and meaning.

Cohn, N., Paczynski, M., Jackendoff, R., Holcomb, P., & Kuperberg, G. (2012). (Pea)nuts and bolts of visual narrative: Structure and meaning in sequential image comprehension Cognitive Psychology, 65 (1), 1-38 DOI: 10.1016/j.cogpsych.2012.01.003

Art and books you should check out

2012-02-09T13:16:00.000-05:00

Here's a few links I've been meaning to post...

First, head over to my friend Helena's website. She's an amazing artist, so go check out her paintings!

Second, if you haven't already been seeing it in airport shelves and bookstores everywhere, you should check out the new book Situations Matter, by my friend and colleague Sam Sommers, a professor in the psychology department here at Tufts. The book explores how the contexts and people find themselves in often affect the way they behave...

"Every day and in all walks of life, we overlook the enormous power of situations—of context—in our lives. Just like the museum visitor neglects to notice the frames around paintings, so do most people miss the influence of ordinary situations on the way they think and act. But frames do matter: your experience viewing the paintings wouldn't be the same without them.

The same goes for human nature."

Go check it out! There's a reason it's been getting so much attention! And here's a short video promoting it as well:

Downloadable Un-Defining "Comics" article

2012-02-04T11:00:00.000-05:00

It's not exactly a new article, but I realized that my article "Un-Defining 'Comics'" from the International Journal of Comic Art way back in 2005 was not downloadable from my site. That is now fixed! A downloadable pdf of the article is now available.

This article was among my first published works on visual language (and it kind of shows...gulp), and is the first that argues for a separation between the idea of "comics" and a "visual language" made up of images (i.e. "comics ≠ sequential images"). Enjoy!

Little busy for a bit...

2012-01-23T22:35:00.000-05:00

Unfortunately, I haven't been able to blog as much as I'd like these days, and the coming months are likely to be a little quiet around here. I'm hoping to have a few big announcements to make soon. However, I'm aiming to defend my dissertation this semester (!), so blog posts are likely to stay sparse for awhile.

Eye movement in reading comics

2011-12-07T11:41:00.001-05:00

I've posted a few studies that have looked at how people's eyes move across comic pages (here and here), and I recently found another. This short study looked at when people's eye movements skip panels or go back and re-read them.

They found that people spend more time reading panels with text than with just images, and that panels without text are more likely to be skipped and to be read with peripheral vision. Unusual panel arrangements (i.e. non-horizontal then vertical arrangements) also possibly led to jumping over panels (as was found in another study as well). After skipping these panels, participants then backtrack and re-read them.

These findings are consistent with previous studies that compared the eye-movements of expert and non-expert comic readers. Non-experts tend to focus more on text and read more erratically throughout a page. Experts tend to read more smoothly and focus more on the images.

General studies like this are interesting, though I'd really like to see more studies that specifically target specific issues. Are there particular features of page layouts that motivate skipping panels? Are there features of layouts that impede on the actual comprehension of panels? Once we get beyond these very basic sorts of "what do eyes do generally" studies, we can really start exploring how looking at eye-movements can tell us about the comprehension of comic pages.

Chiba, Shinichi , Takamasa Tanaka, Kenji Shoji, and Fubito Toyama. 2007. "Eye Movement in Reading Comics." In Proceedings of the 14th Annual International Display Workshops, 1255-58

Attention and comic panels

2011-11-23T10:00:00.000-05:00

Craig Fischer has a nice article over at The Comics Journal about how panels focus attention, particularly focusing on the work of Jack Kirby. He nicely keys in on several techniques that authors (like Kirby) use to highlight certain aspects of a panel over others. For example, putting things in the foreground vs. background, thick lines vs. thin lines, or focusing on people vs. objects.

He then goes on to create an interesting taxonomy of ways that content connects with a narrative, and whether the focal and background elements are done in a common style. As a descriptive taxonomy, I think it works pretty well.

At the end of the piece, Fischer wishes there was more empirical work on how people read comics, especially with eye-trackers. Apparently he hasn't been reading this blog much! Amongst the many studies I've reviewed here about comprehending sequential images, there have been some eye-tracking research on comics that I review here and here.

Also, while my empirical work has mostly focused on how sequential images are comprehended, my theoretical work has looked at the capacity of panels to convey attention for many years. For example, I discuss it in this blog post, as well as in my article, A Visual Lexicon (pdf).

My approach to attention has focused less on the individual aspects of a panel's features, and more on how the panel as a whole acts as a "window" onto a scene. The panel then simulates the same type of "window" on the fictitious world that attention does in our visual perception. As I said in that blog post:

"Most of the time though, panels serve to exclude all relevant information except for the elements that need to be focused on, or at least clearly distinguish what is relevant from irrelevant. This lets panels provide a graphic manifestation of this mental "spotlight," allowing the author to control that attention instead of the reader's wandering eyes (which is one of the reasons I formally call panels "Attention Units")."

However, there's much that could be learned by studying the combination of the types of attention that Fischer talks about (those visible in a panel) and those that I talk about (how what is visible connects with what is not visible, or to other parts of a narrative).

Take my online comics experiment!

2011-11-14T15:56:00.001-05:00

At long last, I have another comic experiment ready to go that needs your help, dear reader! This survey will help us prepare our next study looking at how the brain comprehends comics, and your help would be greatly appreciated. It should take roughly 15 minutes and involves reading comics and giving a basic rating for how much they make sense. Participation enters you in a raffle for a $50 gift certificate to Best Buy.

UPDATE: This survey is now closed. Thank you for your participation! If you would like to participate in future experiments, please email me.

A checklist for decent scholarship

2011-10-26T10:00:00.000-04:00

I've read a ridiculous amount of research on the comprehension of sequential images the last few years. Many people have written papers about this topic, often from many different disciplines. While I can respect that not everyone will aim for the linguistic and psychological approach that I use (and nor should they if they have different intents), there are several pet peeves that I found repeated over and over that make me feel like just disregarding what people write.

So, here's a checklist for getting me to take your scholarship on comics seriously:

Get your names right. I once read a paper that was cited as an "authoritative" source by another book only to find it said the author of Calvin and Hobbes was "Bob Watterson" in a trivial throwaway line. I could maybe forgive a spelling error or something accidental, but not knowing that his name is actually Bill made me lose respect for the entire paper and regard the scholar as someone who was a "tourist" in this research.

Get your basic facts right. An otherwise decent article commented that Chris Ware's Acme Novelty Library was "computer-generated." It's actually done by hand. Such a mistake could easily be remedied with a Google search. And, again, this was a trivial descriptor that was unnecessary for any part of the discussion.

Do your background research. Often I read papers where they only major work of scholarship about sequential image comprehension mentioned is McCloud's Understanding Comics. As important as it was for establishing this line of thought, it is not the only work out there. Or, if they want to talk about cognition, they'll cite a textbook written over 30 years ago. This just isn't acceptable. One of these days I'll write a huge paper reviewing all the theories and experiments that I've found that have ever looked at sequential images (and there are more than you'd think... I've been painfully remiss in updating my bibliography and I'll be the first to say that there's probably more out there that I haven't found yet). Until then, do your own research beyond a cursory job. The same goes for any topic of research or else you'll get a response like this.

Say something novel. It's amazing the amount of papers that I read that merely regurgitate McCloud's ideas of panel transitions and closure without adding anything new. At best, they often just reinterpret his same ideas and draw a connection between them and something in another line of thinking. But, they don't add anything to what he said. Almost ten years ago, I just about doubled the amount of transitions McCloud had before abandoning panel transitions altogether. So, even if you're working with transitions, I know there's more to say than he did (and probably what I did too). Truly, if you can't say something new, why bother saying anything? (AND... why should anyone read or cite your paper??)

Seriously though... shouldn't all these things be part of basic research and paper writing? Why then are they so rampantly disregarded when talking about the structure of comics? For a long time, scholars of comics felt the need to justify such research. However, the best way to convince people to take this work seriously is to actually do serious scholarship.

On tap...

2011-10-18T23:34:00.001-04:00

Thank you to everyone who came out for my talk last week at the New York Comic Con! I had a great time giving the presentation, so I hope it was as enjoyable for all you in the audience.

I have a few blog posts planned for the coming weeks if I can find the time to finish them off. As usual, about a dozen projects are being worked on right now. Included among those is one that we will soon be launching an online experiment for. So, if you'd like to help with some comic research, watch this space over the next few weeks!

New York ComicCon 2011

2011-10-06T12:00:00.000-04:00

Next Saturday on October 15th I'll be speaking at the New York ComicCon for the first time. Here's my blurb from the program:

4:00-5:30 Comics Studies Conference 6: Understanding Comics and the Self—Neil Cohn (Tufts University) discusses several psychology experiments measuring reaction time and brainwaves that contribute to our understanding of what goes on in the brain when a person reads a comic and reveals that the understanding of comics involves a complex negotiation between a hierarchic system of narrative and the construction of meaning. [...]

I'm quite excited about this talk because it looks like I'll be presenting brand new data from my latest study of comics and the brain. Hope to see you there!

Development of drawing abilities

2011-09-29T12:12:00.000-04:00

The book Making Sense of Children's Drawings by John Willats puts forth a compelling theory of how kids learn to draw, and the course of that development.

To Willats, drawings link up to our mental conceptions of how things look in the world, thereby rejecting a view that says drawings are entirely based on what we see. A great example of this is when children are given dice and told to draw them. Instead of drawing them as they see them, they draw dice with all six sides, which would be impossible to see.

Willats also provides great detail on the origin of the "don't copy" trend of instruction in drawing (which he, like me, is highly critical of). As he describes, this came originally from the 1800s educator Franz Cižek, based on Romantic ideas that children had a pure "inner creativity" that needed to develop unspoiled by imitation from external influences.

As he nicely points out, this doctrine is largely not reflected by what children actually do. Indeed, closer inspection of Cižek's own students show a consistent group style. They were copying between each other, just not from him.

Also, his general trajectory for learning to draw runs like this:

1-3 yrs: dots, lines, regions. Scribbles denote whole regions of space, not necessarily just random uncontrolled lines.
2/3-8 yrs: Bounded areas depict regions and volumes. Round, long regions denote round long volumes, while long or round regions show flat volumes.
~6-10 yrs: Regions are used as as picture primitives to denote faces rather than volumes. However, lines still denote boundaries of regions, not the contours of shapes.
~6-8 yrs: Regions as volumes, but compensated by more modifiers, resulting in "having a smooth outline" (threading); denote regions n the visual field (starts approaching lines as contours)
~8-10 yrs: Finally, lines are used as picture primitives (instead of using lines for regions). Lines are finally used as contours, as evident by line junctions used for occlusion and foreshortening.

The one drawback to this approach is that, despite his critique of the overall trend against copying, his developmental trajectory does not incorporate the effects of imitation on drawing. This may not be possible for him though: there simply doesn't seem to be enough data, looked at through the right perspective, to offer his model much more (true both of when the book came out, and now).

However, overall, I highly recommend this book to anyone interested in how children learn to draw.

Willats, John. 2005. Making Sense of Children's Drawings. Mahwah, NJ: Lawrence Erlbaum.

Graphical Abstracts

2011-09-20T11:13:00.000-04:00

A friend of mine passed along this interesting link today. The academic publisher Elsevier looks like it's now accepting "graphical abstracts" for scientific papers in journals:

A Graphical Abstract should be a one-image file and should visualize one process or make one point clear. For ease of browsing, the Graphical Abstract should have a clear start and end, preferably "reading" from top to bottom or left to right. Try to reduce distracting and cluttering elements as much as possible.

Also interesting is that they expressly specify how they want the image to be "read." Now I'm curious what they'd think to an abstract using sequential images...

What is the human language faculty?

2011-09-14T12:35:00.000-04:00

My mentor, Ray Jackendoff, has a new article out in the journal Language that mentions my research (as well as has some illustrations by me):

Jackendoff, Ray. 2011. What is the human language faculty?: Two views. Language 87(3):586-624

The piece explores the biological foundation of our capacity for language, and what components of cognition contribute to language understanding. My work comes in because he points out that several cognitive capacities involve the hierarchic organization of structures, including language, music, vision, events, and, yes, the visual narrative in comics. Here's the abstract:

In addition to providing an account of the empirical facts of language, a theory that aspires to account for language as a biologically based human faculty should seek a graceful integration of linguistic phenomena with what is known about other human cognitive capacities and about the character of brain computation. The present discussion note compares the theoretical stance of biolinguistics (Chomsky 2005, Di Sciullo & Boeckx 2011) with a constraint-based PARALLEL ARCHITECTURE approach to the language faculty (Jackendoff 2002, Culicover & Jackendoff 2005). The issues considered include the necessity of redundancy in the lexicon and the rule system, the ubiquity of recursion in cognition, derivational vs. constraint-based formalisms, the relation between lexical items and grammatical rules, the roles of phonology and semantics in the grammar, the combinatorial character of thought in humans and nonhumans, the interfaces between language, thought, and vision, and the possible course of evolution of the language faculty. In each of these areas, the parallel architecture offers a superior account both of the linguistic facts and of the relation of language to the rest of the mind/brain.

Keywords:
narrow faculty of language, recursion, parallel architecture, Merge, Unification, lexicon, consciousness, evolution

Segmentations in visual narrative

2011-09-08T23:32:00.000-04:00

Gernsbacher's 1985 paper "Surface information loss in comprehension" is an important article on the comprehension of sequential images, and one that has informed much of my current research. It is based on her dissertation, and describes several experiments.

Overall, Gernsbacher had participants read the Mercer Mayer book Frog, where are you? to question whether people can accurately recall the exact surface images in the story, or if (like language) they are only able to retain the gist of meaning.

First, she asked participants to read this "picture story" and choose where they would divide it into parts. They simply drew lines between images where they felt that one episode ended and another began. Overall, she found that people greatly agreed on where these boundaries between segments were placed.

She then asked another group of people to read the stories, but the composition of certain images were flipped horizontally. These images either came before or after the boundaries that people agreed upon in the previous experiment. She found that people had a harder time accurately remembering the horizontal composition if the image came after the boundary as opposed to before it. This provided evidence that people were building up context throughout a segment, and that the start of a new segment incurred a cost on memory.

These experiments were important for several reasons. First, it confirmed her hypothesis that people mostly retain the gist of meaning and not the surface information of images. Given that people's comprehension did not appear overly damaged by flipping the composition of images, it could be pertinent to discussions of how much impact is really made by the left-right composition of images, such as in the 180º rule.

However, more importantly, these experiments showed very strong evidence that people group images together into segments. This poses a problem to theories like McCloud's panel transitions, which envision no stopping point for linear transitions: they keep going on and on throughout a visual narrative (either linearly or promiscuously between multiple panel relationships).

Rather, this experiment shows that people have some intuitions for dividing up visual narratives into segments (what I called in my book "visual sentences"), and that moving between those segments incurs a cost to comprehension.

Gernsbacher, Morton Ann (1985). Surface information loss in comprehension. Cognitive Psychology, 17 (3), 324-363 DOI: 10.1016/0010-0285(85)90012-X

Waning days of studenthood

2011-09-03T16:54:00.002-04:00

Tuesday will begin what looks to be my last year of grad school, which means my last year of being a student. Yikes! That means I'm currently balancing finishing my projects, helping teach classes, writing/revising papers, and looking for what's next. Should be a wild semester!

I have quite a few projects underway right now, starting with my second brainwave study looking at the comprehension of sequential images. I got some good data on this experiment over the summer with a "reaction time" experiment, so now its time to stick electrode caps on people! If all looks good with troubleshooting, we could be up and running this week.

Also, coming up in October I'll be making an appearance down at the New York ComicCon, giving a talk on my research on Saturday, October 15th. As the date gets closer, I'll offer more info.

What do you do with a comic?

2011-08-14T19:48:00.005-04:00

Based on some disagreements I've had with people lately, I'm curious what the general populace thinks. Please take my poll (and pass it on!):

Also: If you choose "other" for either one, please feel free to say what you think in the comments. Thanks!

Brainwaves for non-sequitur visual sequences

2011-08-04T13:22:00.000-04:00

Here is another repost of a review I did awhile ago (1/22/09).

This study examines the neurocognitive processes involved with comprehending a series of pictures, like in comics. The experimenters pulled frames from an animated movie to create static picture sequences. There were two possible endings for each sequence: one with a normal ending, and one with a non-sequitur panel that did not make sense.

Comparison of these sequences used a technique called "event-related potentials" (ERP) that examines people's brainwaves with an EEG recording. The electrical field is measured off the top of the scalp through an electrode cap (like in hospitals), and by averaging out the noise at the critical point (the "event" — here the last panel) it can give you a nice smooth waveform that can tell you about the nature of the cognitive process. Unlike fMRI, ERPs don't tell you much about "where" in the brain things happen, but they do tell you a lot about "when" and a little about the nature of the process.

In this case, your brain distinguishes the difference in processing at less than half a second. The result was a "negative" deflection of the waveform roughly 400 milliseconds after the final panel appeared on the screen (panels appeared one-by-one). These waveforms are from the frontal right part of the head:

The BLUE line represents the normal sequence ending, the RED line the non-sequitur ending. Note that the lines separate and there is a bump labeled "N400" that shows the processing difference (negative is up here). Because of the separation, we can tell that the brain is working harder to process the non-sequitur panel. If it was treated the same, the lines would stay together, like at the beginning of the waveforms.

This N400 also appears in language under similar conditions: where the brain is working harder to integrate semantic information into a meaning, though with language it appears in different locations on the scalp (more back of the head than front). In fact, the first paper that found an N400 for language used this same manipulation: comparing normal and incongruous words at the end of a sentence.

Unfortunately, more experiments of this sort have not really been done with sequential images. Fortunately, it's only a matter of months until I do more. Phil Holcomb, one of the authors, is also one of my advisors. My upcoming projects will be doing these types of brainwave studies using more targeted manipulations of the visual grammar.

EDIT (8/4/11): I have now done a study examining visual narrative structure and am soon going to do several more!

West, W., & Holcomb, P. (2002). Event-related potentials during discourse-level semantic integration of complex pictures Cognitive Brain Research, 13 (3), 363-375 DOI: 10.1016/S0926-6410(01)00129-X

I own a metaphor!

2011-07-28T01:20:00.000-04:00

I am now the proud owner of an original piece of Drew Weing art! Drew is selling pieces from his excellent works,and I chose this one:

I like this one so much both because it reflects a feeling common to both comic authors and academics (of which I'm both), as well as it shows a great conceptual metaphor. Here, the head is mapped to the idea of a container that bursts when faced with emotional overload, just like a pot of water bubbles over when the water boils. In fact, I discussed this metaphor for this very comic last year in this post.

So, if you're interested in helping out a great author and buying some great original comic art for ridiculous prices, head on over to his site!

Taking a year off

2011-07-18T00:02:00.000-04:00

Astute followers of this website will have noticed a conspicuous lack of an announcement of my ComicCon talk this year. In fact, I am taking a break from the Con this year and instead presenting at a cognitive science conference this weekend in my current town of Boston instead of going to my hometown of San Diego. This is actually the first time in ten years I won't be at ComicCon, the last time being when I was living in Japan in 2001. So, I hope everyone has a great Con and perhaps I'll see you there next year!

Otherwise, research is still plugging along with vigor. I have several papers being prepped for publication, as well as several studies progressing along nicely. Hopefully within the next month or so we'll have another online study for people to take too.

Looking at Comics in the Brain with Lasers!

2011-07-04T14:44:00.000-04:00

Nagai, Masayoshi, Nobutaka Endo, and Kumada Takatsune. 2007. "Measuring Brain Activities Related to Understanding Using near-Infrared Spectroscopy (Nirs)." In Human Interface and the Management of Information: Methods, Techniques and Tools in Information Design, 884-93. Heidelberg: Springer Berlin

I had originally reviewed this article several years ago back in 2008, but after recent research of my own I think it's worth revisiting...

This study used near-infrared spectroscopy to measure blood flow in the brain while people read comics. This technique emits infrared light into the scalp to measure where blood flows in the brain, which can thus indicate the brain regions involved in various behaviors.

They compared normal comic strips with fully scrambled strips of random panels. They then compared the normal strips with tasks asking people to either pay close attention to the strips and report what they found funny or to just read them passively. They found that there was greater activation in "the left prefrontal lobe region is activated when people actively try to understand the comic stories and to memorize their contents for reporting in the future."

As I reported last time, there are several problems with this study, such as the number of stimuli (only 6 strips) and their population (13 people). Comparatively, my last brain study used 160 stimuli per trial (720 strips total) and 24 participants.

However, the areas of activation that they did find is interesting. The prefrontal lobe in the left hemisphere ("Broca's Area) is associated with the processing of grammar in language, and the authors specifically point out that the areas the found for comprehending comics may overlap with this region.

Previous research has also found that this area plays a role in the comprehension of comics. Indeed, research of my own has been more fine-grained than these previous approaches, and has been finding hints that this area is active as well.

So, while this study may have great limitations, it may have been providing some early precedents for some important understandings about the comprehension of comics and the brain.