InfoVis tests

What makes something “Information Visualization?” At this point there are probably tens of thousands of programs turning numbers into images and many of them purport to help us understand the data. I believe most of them don’t, based on admittedly high standards: is a given tool readable, useful, worth my time? Or is it just visual titillation: fun for a moment’s early visual process engagement, and ultimately unsatisfying?

Here’s my attempt to list what runs through my mind as I see and try to evaluate new work that calls itself InfoVis. I will be expanding each section as time goes on, and welcome questions and additions.

To help facilitate discussion about specific visualization techniques I am trying to develop a terminology similar to Mathematics’ well-defined “properties,” such as closure, commutative, identity. Where I have come up with an expressive and concise term I list it above the test.

(People requested a more easily printed PDF version of this when I presented it at a recent Visual Analytics conference; download it if you like.)



Basic tests: it’s not InfoVis if it doesn’t pass these


DATA-DRIVEN
Is there data in it?

Random number-based images need not apply.


INFORMATION-DRIVEN
Is there information in it?

Following Bateson’s formulation of information as “the difference that makes a difference”: are the visible variations meaningful?

READABLE
Is it readable?

For these purposes I define “readable” very specifically as supporting discourse about the subject.

This can be opposed to supporting discourse about the representation, image, process, creator, or other purely form-related concerns; helping distinguish InfoVis from computer art that uses data as decoration.

The key issue is this: if you can’t get useful information about the subject back out again it’s not information visualization.



Operational tests: it’s not a tool unless it passes these

HYPOTHESIS-GENERATING
Does it support asking questions?

One of the primary uses of good visualization tools is to help people form hypotheses about the subject.

TEST-GENERATING
Does it help formulate ways to answer questions?

If it can help suggest a test for the hypothesis so much the better.



Operational tests 2: it’s a better tool if it passes these

HYPOTHESIS-SUPPORTING
Does it support answering questions?

Slightly dangerous territory: some kinds of questions in scientific exploration, for example, must be answered with more rigorous methods like empirical experimentation or statistical analysis. However, in some areas the image can directly provide the answer, as in TextArc’s ability to show foreshadowing references to the Queen in Alice’s Adventures in Wonderland.

TASK-COMPLETE
Does it display all relevant data needed to answer a question or complete a given task?

[property name needed]
Does it allow easy access to additional needed data?



Expressiveness tests: it’s talking about itself unless it passes these

I am using “expressive” in the way statisticians use it: not in the artistic sense of “self expression.” Statisticians talk about whether a given result expresses the variability in the original data set.

BETWEEN-SUBJECT-VARYING
Do different subjects look different?

WELL-BOUND
Do variations in the image always express variations in the subject?

Or does the image mostly represent the algorithm that generated it more than the data— an extrememly common but critical fault, exhibited in part by the label text layout of my science map, for example.

If small changes in the order of the input cause large changes in the representation (as happens in many popular force directed placement layout simulations) the representation does not have this property.

STRUCTURALLY-EVOCATIVE
Does the visual structure suggest the subject’s structure?

TASK-RANKING-EVOCATIVE
Does the most visually salient feature expose the most important feature of the subject (with respect to a task)? Do less important subject features draw concomittantly less visual attention?

Note that when I use “important” here and below it’s always meant to be measured with respect to a given task—if there’s no defined task I have to wonder why it was built...



Parsimony tests: boy, do a lot of famous visualizations fall down here

GRAPHICALLY-PARSIMONIOUS
Are the visual attributes truly necessary?

Attributes like color, shape, curvature of lines, 3D-ness, animation can serve the goal, or simply sell the representation. Here's the simplest test: is it still as easily readable if the attribute in question is removed? (Also look to see if it’s actually more readable or easier to navigate, like when removing 3D-slanted text.)

MODALITY-PARSIMONIOUS
Can the questions/answers have been elicited with quicker/easier non-visual means?

TECHNOLOGICALLY-PARSIMONIOUS
Can the questions/answers have been elicited with simpler technology? (E.g., a print rather than an interactive computer system)



Reliability tests: I know of no tool that has carefully addressed these issues

RANGE-COMPLETE
Are the smallest or largest meaningful differences in the subject visible?

PERCEPTUALLY-ONE-TO-ONE-MAPPED
Do equal perceptual steps map to equal steps in the importance of the data?



Interpretation tests: is the analysis part of the work?


[property name needed]
Does the image suggest the visual analytical techniques or just allow them?

Bar charts, scatter plots, and TextArc allow people to develop visual recognition tasks that let people tease information out of the display, but the analysis remains distinct from the tool.



Effectiveness tests: it may not be justifiable if it doesn’t pass these

AUTOMATABLE
Can the process be automated?

The academic InfoVis community has been addressing perceptual layering and gestalt “pop-out” issues for several years now, but if the tool only takes advantage of the human visual system to identify these patterns, why not just train a cat or pigeon to do it? And as computational models of the human visual system get more sophisticated, why not just plug the visualization output raster into the input buffer of the vision simulation?

Processes that are not automatable—and therefore in my mind truly justify visual representations—might include higher-level task-based decision-making, for example.

COMMONLY-IN-USE
Do people actually continue to use it in practice?


MARKET-VALUED
Do people buy it? Do they buy the upgrades? Do they ask the creator back to do more like it?



Ecological validity/specificity tests: does the representation help reveal the subject’s essence?


[property name needed]
Does the representation suggest the subject’s nature?


[property name needed]
Does the representation suggest the subject’s uniqueness?



Popularity tests: often confused with tests of value as a tool

Was it in a major museum show?

Was it on the cover (shown as a feature, etc.) in a famous journal or magazine?

Was it an illustration for a scientific article?

Illustrations can help one understand a process by example, but are rarely actual tools.