Design View | Articles and opinion on design professionalism, technique and culture by Andy Rutledge

Poynter's Complete Waste of Time

September 3, 2006

The Poynter Institute is about to launch the research phase of its run-up to launching version 4.0 of its perennially popular game, The Poynter Institute’s Complete Waste of Time. Many of you may know it by its more common name, the EyeTrack study. It’s a game where knowledge is fun! – knowledge of utterly useless, anecdotal and irresponsibly non–applicable trivia about the viewing habits of website visitors. Let’s play along.

Wait, before we play along let’s first see if these guys know what they’re doing. I’ll save the suspense and skip to the end for you: no, they don’t know what they’re doing. Instead they’re wasting your time and the time of the editors and publishers they claim to want to help.

How are they doing so much harm? By completely ignoring the fact that design exists and so failing to allow for the impact of design in the context of their study. Their work creates and perpetuates vacuous ideas about Web page layout and design effectiveness, keeping online publications in the Stone Age and making it harder for design professionals to do their jobs and appropriately serve their clients (many of whom are online publications).

They do this harm by releasing “conclusions” that are completely anecdotal and which fail to account for varying and vital contexts. Virtually every one of their past conclusions from eye-tracking studies can be refuted or reversed with even the most basic application of design or a changed context. Further, they fail to account for the effect of perhaps the most important element in the study material – the content. The results can be nothing less than irresponsibly flimsy data.

One of the reasons that they so completely miss the mark with their studies is that they don’t understand the usefulness of eye-tracking studies, abuse the purpose and then mischaracterize the applicability of the data.

They’re making these mistakes, I believe, largely because they’re not designers! The Poynter Institute is “a school for journalists, future journalists and teachers of journalists.” That’s great, but it clearly does not stand them in good stead when they so casually wander into the realm of design.

If I sound a bit indignant it’s because I care about my profession and its people (students and pros), so I’m particular when it comes to what others offer up to us as fodder for consumption. If someone is going to presume to teach facets of design to designers, or anyone else, they’d better damn well get it right. Poynter gets it wrong. They’re poisoning the water and they need to be called on it.

Eye-tracking Myopia

Here are a couple of examples of EyeTrack conclusions. Let’s examine some of their claims from the 2004 study about online readers’ habits.

1. Users spend a good deal of time initially looking at the top left of the page and upper portion of the page before moving down and right-ward.

Not if the designer doesn’t want that to happen. Any competent designer can craft a layout and design to elicit any specific entry/focus behavior on the page.

2. Ads perform better in the left hand column over the right column of a page. The right column is treated by users as an "after-thought" area and should be designed with that in mind.

Hogwash. Yes, a designer can make this so, but she can also design the page so that any area of the page allows ads to perform better. The “after thought” area of a page is created by the design, if it is designed to exist at all, not relegated to the right column. Crafting a compelling and consistent experience is the designer’s job.

3. Navigation placed at the top of a homepage performed best.

For the specific designs used in the study, perhaps. But this again is wholly contextual to the design, the content, and the intent of the designer. See again the previous answer.

This is the sort of pap they would have designers, publishers and editors believe and invest in. In their current pre-study promo article, they say, “Because the study adheres to the highest research standards, we'll be able to offer industry leaders scientific accuracy on which to base the editing decisions they make every day.” However, as their conclusions are farcical and myopic, they don’t actually offer much in the way of valuable information. Their definitions of high research standards and scientific accuracy would seem to be rather lax.

Misapplication

One of the reasons that they so completely miss the mark with their studies is that they don’t understand the usefulness of eye-tracking studies, abuse the purpose and then mischaracterize the applicability of the data. Eye-tracking studies as they conduct them have one and only one use: the evaluation of a specific design as applied to a specific layout in a specifically relevant context. In other words, what can be gleaned from the properly conducted eye-tracking evaluation of one design/layout is almost completely worthless to any other page or context, UNLESS you are a designer and practiced at accurately extrapolating variations based on the data.

I’ve said it before and I’ll continue to harp on this theme: we designers must know our craft and understand the psychological and behavioral impact of our design decisions.

Another of the problems inherent in these studies is the fact that they invariably use subject publications that have serious design and layout problems (as they’re going to do again in this new study). Doing so results in flawed data because conclusions drawn from examinations of poor quality designs in an artificially created context are generally inapplicable to good design and natural contexts. This is something they either don’t grasp or simply don’t care to mention.

I wonder, did it ever occur to Poynter to consult designers about the design quality of the subject publications and the context under which the studies would be conducted? Apparently not, because their studies don’t appear to consider even the fundamental sort of end-user roles designers consider as a matter of course in the average design process. Context matters. Content matters. Design matters. Poynter and others, though, can’t be bothered by these trivial facts.

For instance, Poynter (and every other eye-tracking study I’ve ever seen) presumes to scientifically measure how deeply people read into articles. This sort of data might be generally applicable to similar types of online news sites, but it’s wholly inapplicable to specialized publications or commercial sites because of one vital factor: the reader’s interest in the content subject matter. Anyone not interested in the content will read very little of it. This behavior has nothing at all to do with the layout configuration or font size. Yet none of these studies seem to take this vital bit of context into account. And this is only one contextual factor.
There are many others.

Any competent designer can design and lay out a page and its content so that the content is easily and readily consumed in-full (in the right context) – or – can design and lay out a page and its content so that it is merely scanned. Any competent designer can design a page to compel readers to concentrate on one specific area or to systematically cover the entirety of the page’s content. But again, context matters and impacts reader behavior. Designers take context into account. These are design concerns and require design understanding to create and evaluate.

I’m greatly disappointed that Poynter and others like them make such feeble and casual forays into design matters without accounting for relevant context issues. This is irresponsible of them and the fact that their work is held up as consequential and important harms the design profession; not to mention their own credibility.

Designers should first rely on their own practiced understanding of these issues before swallowing what others offer up. I’ve said it before and I’ll continue to harp on this theme: we designers must know our craft and understand the psychological and behavioral impact of our design decisions. Without this knowledge we can easily fall prey to the naïve pseudoscience common to these eye-tracking studies.

If you want to know how a reader will consume a page of content in a specific context, ask a competent designer, not a journalism study coordinator.

Archives

Archive

Practice