Computational
Esthetics
Remko
Scha and Rens Bod
Formal theories which compute the "beauty coefficient"
of visual patterns, fail to do justice to the complexity of the
esthetic experience. These "computational esthetic" models
do, however, embody some notions that are needed to build formal
models of human perceptual processes  and these, in their turn,
must be the basis of any empirically adequate esthetic theory.
Though the esthetic
experience remains one of the most enigmatic sideeffects of human
perception, several mathematical models have been proposed which assign
to visual patterns a "beauty coefficient"  a number that
is intended to correlate with the degree of esthetic pleasure the
pattern evokes. Such theories seem a little naive, because they focus
on a quantitative and absolute beauty judgment. They disregard the
qualitative aspects of specific esthetic experiences, and do not account
for the contextdependence and variability of beautyjudgments. It
is interesting, nevertheless, to look at the operation of these excessively
simplistic beauty calculations; if we integrate them with other ideas
from perceptual psychology and computational linguistics, they may
in fact constitute a starting point for the development of more adequate
formal models.
Kant and the beauty experience.
The best analysis
of the esthetic is still Immanuel Kant's. He viewed the experience
of beauty as the consciousness of a psychological process: the pleasing
awareness of the harmony in the free play of our cognitive faculties.
If Kant is right about this, the natural phenomenon or art object
that thrills us is in fact not much more than a trigger. Then we must,
to understand the esthetic, first of all understand the perceptual
processes; apparently these are such that, helped by the properties
of their input, they can bootstrap themselves into esthetic experiences.
Kant's analysis implies that the objectivity of esthetic judgments
is not selfevident. He construed it as an intersubjectivity  as an indirect consequence of the high degree of similarity between
the cognitive machineries of different persons. He nevertheless disputed
the validity of completely arbitrary individual esthetic judgments
by positing the "better developed" taste as the norm. Later
philosophers have often pointed out that this is one of the weaker
spots in Kant's story. A psychological notion of beauty is necessarily
subjective, and certainly not normative.
Against this background, a notion of beauty that only classifies objects
as beautiful, less beautiful, neutral, or ugly, must be viewed as
naive. Nevertheless it is such a notion that underlies all formal
theories of beauty proposed so far. Perhaps we shouldn't be surprised
about this. Many much more commonplace aspects of perception have
not yet been formally analyzed either; it is therefore not realistic
to expect today's mathematical theories to face all complexities of
the esthetic.
That existing formal theories only account for caricatures of the
beautiful, is thus not a sufficient reason to dismiss them altogether.
It would be sufficiently interesting if they analyze certain aspects
of the esthetic in a way that can be extended or refined. From that
perspective, this article looks at some of these theories, and then
reconsiders what would be involved in a more adequate computational
model of esthetic processes.
Birkhoff and harmony.
Twentiethcentury
formal theories of beauty tie in with earlier informal theories which
focussed on the feeling of harmony in the experience of beauty, and
which explained that feeling as arising from our resonance with the
harmonious properties of the object that is being observed  with
selfsimilarities, symmetries, and simple proportions in the appearance
of that object. In this view, beauty is in essence a mathematical
phenomenon. The ancient Pythagoreans were not the only ones who explicitly
held this opinion. G.W. Leibniz, for instance, described the enjoyment
of art as the unconscious calculation of numerical proportions 
between time intervals, in the case of music, or between spatial distances,
in the case of visual art and architecture.
In 1928, the American mathematician George David Birkhoff made the
first attempts to formalize such notions. He introduced the concept
of the Esthetic Measure (M), defined as the ratio between Order
(O) and Complexity (C): M = O/C. The Complexity is roughly
the number elements that the image consists of; the Order is a measure
for the number of regularities found in the image. For different artistic
genres, Birkhoff has indicated specific rules to actually compute
precise values for Order and Complexity.
For polygons he thus defines Complexity as the number of edges, while
the numerical value for Order depends among other things on the presence
of vertical symmetry, point symmetry, and mechanical stability with
respect to an imaginary horizontal plane. Figure 1 shows for some
polygons the value for the Esthetic Measure that is computed in this
way. As one might expect, the highest scores go to patterns with a
minimal number of parts and a maximal number of symmetries. The square
wins.
Figure
1:
The
esthetic measure of some polygons
according to Birkhoff's formula: M = O/C.
(After: G.D. Birkhoff)
Birkhoff's formula
thus turns out to formalize the idea of "orderliness" rather
than the idea of "beauty". And to identify orderliness and
beauty, though not impossible, seems to be a very specific esthetic
choice. Artistic movements such as ZERO, NUL and minimal art actually
made such a choice. In the constructivist tradition this idea also
plays an important role: "If a picture works out without a
remainder, that means that all its elements are logically related
to each other; it means that each color corresponds to every other,
each form to every other, each form to every color and both form and
color to their contents. It means ultimately: that its structure is
homogeneous, from conception to perception." (Gerstner,
1981, p.35.)
Birkhoff, however,
was not making an artistic statement or propounding a normative theory;
he viewed his model as an empirical theory, and was interested in
its validity. He has therefore presented his polygons to students,
and compared their beauty judgments with those of his formula. He
never published the details about these experiments, but he was satisfied
with their results ("the judgments of students seem to indicate
the validity of the formula"). More recent psychological
experiments, however, only yielded a weak correlation between Birkhoff's
measure and the actual beauty judgments of the subjects. This contextdependence
of the esthetic judgments is not surprising: there is no reason to
suppose that people entertain one fixed notion of beauty, which can
be activated with an arbitrary laboratoryexperiment. It is much more
plausible that there are many different classification criteria, all
in some way related to the esthetic dimension of perception, that
people may apply in different situations.
For a different domain, a class of Chinese vases, Birkhoff defined
the numerical value for Order in a rather different way. His point
of departure is the twodimensional projection of the vase. He then
draws tangents, horizontal lines and vertical lines through the points
of maximal, minimal and zero curvature on the outline of the vase;
and he counts how many intersections of such lines coincide with each
other, and how many pairs of intersection points are equidistant.
Figure 2 illustrates how Birkhoff arrives at the vase with the highest
Esthetic Measure.
Figure
2.
Left: the esthetic measure of some
vase shapes according to Birkhoff's formula: M=O/C.
Right:
the 'ideal vase' according to Birkhoff. (After: G.D. Birkhoff)
The formula behaves
in a more interesting way now. The Esthetic Measure now correlates
with a quality of "elegance", rather than a trivial property
of orderedness. The reason is, that the objects which are to be compared
are now defined in a different (more limited) way: all the different
vase shapes are distortions of one basic shape. The shapes can thus
be compared more accurately with each other, in terms of the quantity
of additional internal coherence they display. Once more we find,
in this way, the singularities within a space of possibilities 
but now they are less predictable.
The exemplars classified as "beautiful" now indeed have
something of the "organic unity" that is often viewed as
a characteristic of the succesful artwork: "Every element in a work of art is so involved with other
elements in the making of the virtual object, the work, that when
it is altered (as it may be  artists make many alterations after
the composition is well under way) one almost always has to follow
up the alteration in several directions, or simply sacrifice some
desired effects. [...] This manysided involvement of every element
with the total fabric of the poem is what gives it a semblance of
organic structure; like living substance, a work of art is inviolable;
break its elements apart, and they no longer are what they were 
the whole image is gone." (Langer, 1957, pp. 5557.)
Birkhoff has
also worked out specific versions of his formula for the auditory
dimension of poetry, and for melodies. We will not discuss these in
detail; the above suggests that it is by no means obvious what they
should look like, and what kind of intuition of beauty would be formalized
then. This betrays a weak point in Birkhoff's "theory":
for every genre of input objects, new rules must be formulated, and
the notion of beauty embodied by Birkhoff's formula may therefore
shift a little in each case.
Bense and Information Theory.
It is not surprising,
therefore, that some researchers have tried to use Birkhoff's idea
as a point of departure for developing a more general, encompassing
theory.The most important case in point is a group of literary theorists
in Germany in the fifties, headed by Max Bense. This group has developed
the theory of information esthetics  a Birkhofflike model
of beauty judgments, formulated in terms of Claude Shannon's information
theory.
The starting
point is Birkhoff's original formula: M = O/C. The definition
of the Complexity of an input pattern is then borrowed from Shannon's
notion of Information: if an input pattern specifies n binary
choices from the class of possible patterns, the Complexity equals
n.To be able to compute the Complexity in a direct way, one
introduces the assumption that an input pattern can always be described
as a twodimensional grid of discrete symbols from a predefined repertoire.
If the repertoire contains k symbols which all have an equal
a priori chance of occurring, every symbol has an information content
which correponds to 2log k binary choices. The information
content H1 of an m by n grid is then n * m
* 2log k, and that is the value assigned to the Complexity
C of such a pattern.
Figure 3: Some grid patterns in order of increasing orderliness: increasingly large 'supersymbols'.
(After: Gunzenhäuser, 1975)

To arrive at a similar informationtheoretic articulation of Birkhoff's
notion of Order, we observe that orderliness corresponds to the possibility
of perceiving larger structures. If these larger structures can in
their turn be considered as discrete "supersymbols" within
a welldefined repertoire, we can compute the informationcontent H2 of the pattern as described in terms of these supersymbols.
If not all combinations of elementary symbols are considered as legitimate
distinct supersymbols, the new coding is more parsimonious than the
original one, so H2 is smaller than H1: the description
in terms of supersymbols yields an "Ordnungsgewinn". The degree of orderliness of the pattern
corresponds to the difference between the informationcontent of the
original coding and the informationcontent of the supersymbolcoding:
H1  H2. Birkhoff's Esthetic Measure is thus computed as: M
= (H1  H2)/H1.
Bense's idea thus
stays rather close to Birkhoff's original intuition, but nevertheless
suggests a somewhat different model of the perceptual process. For Birkhoff,
the experience of orderliness is a direct consequence of the perception
of a relatively large number of regularities; in information esthetics,
the experience of orderliness is a result of the transition between
an initial coding of the input (in terms of individual line segments,
words or tones) and its more parsimonious recoding which comes about
after some reflection.
The informationesthetic
formula therefore corresponds to wellknown ideas about the role of
the artwork's perceptual unity in the experience of beauty: "Initially,
the details of the work seem to be just there, and we may seem free
to conjoin them this way or that, whichever way we please. Yet if we
dwell with the art work, and if this work is genuine, it comes to crystallize
into a whole: the parts fit together and we discern a certain necessity
in their cohesion. And since we are now guided by this sense of necessity,
we are forced to discard our "old" freedom. But we do
not experience this necessity as a mere external constraint. Rather
it comes to us as a liberation, a release: we are freed from the fragmentariness
of mere detail and come to be at home in a rich whole. It is not that
we discard or obliterate the details, but in standing beyond their fragmentariness
we ourselves are freed from fragmentation. Such a "standing beyond" which unites and preserves the internal details of a complex whole,
in fact, makes the art work an aesthetic concretion of Hegel's general
principle of Aufhebung". (Desmond, 1986, p. 64.)
Bense's information
esthetics is, however, not more general than Birkhoff's theory.
It is better viewed as an addition to Birkhoff's list of rules for
specific genres. Information esthetics gives rules for computing Complexity
and Order for a very specific kind of image: a grid consisting of
discrete symbols from an explicitly specified finite repertoire. There
is suggestion of generality, because in a technical sense all images
may be viewed that way, at least approximately, if we think of them
as built up out of pixels. But the suggestion is false, because for
most images encountered in practice, a construction out of adjacent
discrete elements is not the perceptually relevant analysis.
Information esthetics also inherits Birkhoff's preference for minimalist
structures. The simpler the image, the more compact its supersymbolcoding
can be, and the larger the resulting "Ordnungsgewinn".
But exactly in the case of grid patterns it is clear that the preference
for "total order" leads to incorrect results. It has often
been remarked that an intuitive measure of beauty should not only
get a null value when a pattern is too complex to to observe any order
in it (random patterns: figure 4, upper left), but also when a pattern is ordered
completely into perfect banality (figure 4, lower right). Complete disorder and
complete order are perceptually approximately identical. The maximal
value of the Esthetic measure should be found somewhere between these
two poles.

Figure
4: Some
grid patterns in order of increasing orderliness.
(After:
Gunzenhäuser, 1975)

There is another
problem with the informationesthetic measure: the computation is
based on a predefined repertoire of supersymbols. But many forms
of orderliness, and not the ugliest ones, employ supersymbols defined
by the artwork itself. A particular combination of elementary symbols
can function as a supersymbol, merely because it (or a pattern derived
from it) occurs more often in the total pattern, and can thus be employed
conveniently for describing the whole pattern. To compute an orderliness
measure on the basis of a recoding of the input pattern in terms of
supersymbols, one must first compute which supersymbols are being
used in the first place. This component of the computation of the
Esthetic Measure is not specified in the informationesthetic literature.
Leeuwenberg
and Prägnanz.
The contextdependence
of the supersymbols was appreciated already by the psychological tradition
of Gestalt perception, initiated in the twenties by Max Wertheimer
and Kurt Koffka. The Gestalt psychologists emphasize that the overall
impression (the "Gestalt") evoked by an input pattern, is
determined by that input pattern in a very complex way. Various possibly
conflicting factors play a role. One of the most important ones, which
settles the outcome in situations which in principle would allow several
possibilities, is the preference for the simplest structure. This
factor is sometimes called the principle of Prägnanz.
The original Gestalt perception theory as developed by Wertheimer
and Koffka was not yet a mathematically formulated model. That step
was made in the late sixties by the psychologist Emmanuel Leeuwenberg
in Nijmegen. Like the informationestheticians, he describes perception
as a recodingprocess. The "raw input" is described as a
simple enumeration of occurrences of elementary constituents. The
perceptual "Gestalt" which this input evokes in the mind
of the observer, is modelled as a more compact coding of the same
image  a coding which explicitly represents the perceived structure
of the pattern.
Information esthetics has given us a first impression of such a recoding.
An informationesthetic recoding of a grid pattern indicates how the
plane is filled by supersymbols; and for each of these supersymbols
it indicates how it is built up out of smaller supersymbols; and so
on, until the level of elementary symbols has been reached. The recursive
constituent structure of the image is thus represented in an explicit
way. The informationesthetic recoding process is limited in several
respects, however: it only deals with grid patterns; it assumes that
supersymbols can only be constructed by putting smaller, independently
defined supersymbols next to each other; and supersymbols cannot be
explicitly represented as variants or transformations of each other.
Though notions such as "repetition", "mirrorimage",
"rotation", etc. play a role in the perceived Gestalt of
an input pattern, they do not occur in the informationesthetic recoding
of such a pattern.
Leeuwenberg therefore proposes a much richer imagecoding language,
with operators which can transform any visual pattern into various
other patterns by rescaling or rotating it, or by repeating it or
alternating it with other patterns. Leeuwenberg's paradigmatic images
are not symbol grids, but drawings built up out of straight linesegments.
The expressions of his coding language thus resemble sequences of
plottercontrol commands, as in the turtle graphics of the LOGO system.
The coding of raw input consists exclusively of commands of this sort:
so many steps ahead; so many degrees to the left; . . . But in recoding
the analysed input, high level operations are also used, which duplicate
, move, or rotate a figure that was defined before.
Leeuwenberg thus broaches a hypthesis about the formalisation of Gestaltperception:
the idea that such a turtlegraphics language can express meaningful
representations of Gestalts. Assuming the correctness of this hypothesis,
he then attempts to describe Gestaltperception phenomena within his
model, by modelling Gestalt perception as a disambiguation process.
The coding of the raw input always allows a large number of alternative
recodings, and the question is: which is the recoding actually generated
by the human brain?
To answer that question, Leeuwenberg identifies the psychological
complexity of a Gestalt with the length of the corresponding turtlegraphics
code, as measured by counting the number of occurrences of basic visual
elements in that code. This formalizes the Prägnanzprinciple:
the preferred recoding of an input pattern is simply the shortes recoding,
and the perceived Gestalt is the gestalt corresponding to that recoding.
In Figure 4, for instance, we see three different structural interpretations
(a, b and c) of two simple patterns. For the first pattern, interpretation
c yields the shortest code. For the second pattern, the shortest code
corresponds to interpretation a.
Figure
5: Two line drawings with three different
analyses each.
For A, the perceptually preferred analysis is c. For B, this is a.
(After H. Buffart)
Leeuwenberg's
theory was tested on different kinds of visual patterns, and on musical
perception. In many cases this yielded satisfying empirical results.
Leeuwenberg's
approach suggests an interesting formulation of the informationesthetic
orderlinessmeasure. A pattern consisting of repetitions of the same
element, is experienced as more orderly than a pattern of elements
which are all different. The informationcontent of a Leeuwenbergcode,
which directly correlates with that distinction, thus results in a
better orderlinessmeasure than the original informationesthetic
proposal, which involved adding up the informationcontent of all
individual imageelements. An additional advantage is that the applicability
of Leeuwenberg's approach is not limited to specific genres such as
grid patterns.
Perception
and experience.
Not all parts
that we distinguish in an image are repetitive patterns or elements in
repetitive patterns, however. The observer of a figurative painting,
for instance, will be struck by resemblances with previously perceived
objects and situations. If we want to take this phenomenon into account
in the computation of the informationcontent of the minimal Leeuwenbergcode
of an inputpattern, the primitive elements of perception theory cannot
be restricted to pixels or simple line segments. We must reintroduce
one of the ideas of informationesthetics: a predetermined repertoire
of "supersigns", to be used in recoding an inputimage.
How is this supersign repertoire to be specified? In the context of
Leeuwenberg's approach this is easier to decide than in the original
informationesthetic framework. Our capacity for recognizing regular
abstract patterns is already accounted for by the structural properties
of the coding language. The supersigns are only needed to put the
role of experience into the picture. To do that, all sign complexes
which occurred as meaningful constituents in previous experiences
should be recognized as supersigns. But not all to the same extent,
because a supersign is recognized more easily the more often it has
occurred. According to Shannon's wellknown formula, the information
content of a supersign is the logarithm of the a priori probability
of its occurrence. This probability can be estimated as the observed
relative occurrence frequency of the supersign. The calculation can
be further refined by working with conditional probabilities,
which reflect the mutual dependencies between the analyses of the
different parts of the image.
For the case of language perception, we have already worked out this
approach in some detail. The preferred analysis of a language utterance
is the analysis which results most often from the process of randomly
combining random subtrees from a corpus with previously experienced
language data. This corresponds to the preference for the shortest
code: the preference for analyses which can be built up from a maximally
small number of maximally probable fragments.
Towards a
process model.
Looking back
at this short history of computational esthetics, we see some progress,
but we also notice obvious limitations. In particular, we see the
gradual development of a conceptual framework which may make it possible
to describe some elementary properties of the Gestalt perception process
in a formal way. But the notion of "beauty" that is being
articulated here, is extremely narrow. We mentioned already that Birkhoff's
"Esthetic Measure" is in fact merely an "orderlinesscoefficient",
and this characterization also applies to the the informationtheoretic
versions of this notion based on Bense or Leeuwenberg. All these models
identify the experience of beauty with the perception of formal regularities
in the object that is observed, and they correlate the intensity of
the experience directly with the number of regularities.
From a Kantian perspective which analyzes the esthetic experience
as the awareness of the free play of the cognitive faculties, these
models are too static; a more adequate model should be concerned with
the nature of the perceptual processes rather than their end
result. Such a process model might also account for the important
role that undefinedness and ambiguity play in the esthetic experience,
both at the level of Gestalt perception and at the level of interpretation.
Though the orderlinessmodels as they stand completely ignore this
aspect of the esthetic, they might nevertheless provide a starting
point for the design of a more adequate processmodel.
The coding
theory that we proposed should not only predict the Gestalt that a
particular input evokes, but also, what inputs are experienced as
ambiguous because they evoke several distinct Gestalts that are roughly
equally plausible. And they should predict in which cases these distinct
Gestalts are mutually related in such a way that they do not compete
with each other, but give rise to associative cycles — superGestalts,
i.e., processes which resemble definite perceptions but which are
much richer since they embrace a large number of different (possibly
incompatible) perceptions in one coherent whole. We conjecture that
the experience of beauty is characterized by processes of this sort,
which allow perception to gain access to itself, because its intermediate
results and alternative interpretive hypotheses are stable enough
to reach consciousness — something which is impossible during
the normal goaldirected perception of clearcut input.
For a specific, narrowly defined class of inputs (such as line drawings
or grids), such a processmodel might be worked out. But it would
be absolutely out of the question to accomplish this in the context
of a complete simulation of all possibilities of human visual perception.
Things get even more difficult when we introduce the semantic dimension — when we acknowledge that the experience of beauty involves
not only the perception of Gestalts, but also the assignment of meanings.
It is not possible to build serious simulations which involve the
semantic realm. But it is possible, of course, to speculate about
the structure that such simulations would have.
It is clear that they would not only involve the literal meanings
of conventional signs and recognizable images, but also the meanings
which are evoked when the structures perceived are mapped onto the
observer's experiential background through metaphorical or metonymical
projection. Again it is crucial that the interpretive processes do
not yield definite interpretations too quickly, but rather give rise
to complexes of mutually related alternatives. As Roland Barthes indicated
in Éléments de Sémiologie, this
machinery is applied recursively: in the context of the other structures
and meanings observed, the first layer of meanings can be reinterpreted
to yield "deeper" meanings, and so on.
For the time being, we cannot work out such a semantic model in any
detail. But it will become more concretely imaginable as soon as a
very limited purely syntactic model would show interesting results.
Thus, the ultimate benefit of the computational approach to the esthetic
will not lie in the models that can be implemented and validated — but in the more speculative and encompassing models which they make
thinkable.
Literature.
Roland Barthes: Éléments de Sémiologie. Paris: Éditions du Seuil, 1964.
Max Bense: Aesthetica. Einführung in die neue Aesthetik. BadenBaden:
AgisVerlag, 1965.
G.D. Birkhoff:
Collected Mathematical Papers. New York: American Mathematical
Society, 1950.
Rens Bod: "Using
an Annotated Corpus as a Virtual Grammar." Proceedings EACL'93,
Utrecht, 1993.
William Desmond: Art and the Absolute. Albany, NY: SUNY Press,
1986.
Karl Gerstner: "The Precision of Sensation" In: H. Stierlin
(ed.): "The Spirit of Colors. The Art of Karl Gerstner".
Cambridge, Mass.: The MIT Press, 1981.
R. Gunzenhäuser: Mass und Information als ästhetische Kategorien. BadenBaden:
Agis Verlag, 1975.
Immanuel Kant: Kritik der Urteilskraft. 1799.
Susanne Langer: Problems of Art. New York: Charles Scribner's
Sons, 1957.
E.L.J. Leeuwenberg: "A Perceptual Coding Language for Visual and Auditory Patterns."
Am. J. Psychology, 84 (1971).
Remko Scha: "Virtual Grammars en Creative Algorithms." Gramma/TTT,
1,1 (1992).
Claude E. Shannon: "A Mathematical Theory of Communication." Bell Syst.
Techn. J., 27 (1948).