FRIDAY MORNING SESSION
What do we want computers to do?
How can HAA inform computer science?
Adriana Kovashka
“Towards Human-like understanding of visual content”
1. Semantic way of finding images
How can we search better through all visual data on the web? Too big to browse.
Keywords for known name. But if you don’t know the name: you can only describe a thing. What do you type? Interaction to describe what you are looking for. Steps in describing details, narrowing it down on physical characteristics.
But computers can’t see.
Interactive search done today: looking for a person. Binary relevance feedback, from the initial search results. But very imprecise. Search via comparisons: whittle search. Relates to what you’re looking for based on particular properties of change and modification.
High-level descriptive properties (adjectives), human-understandable, middle ground between user and system.
20 question game: what questions the computer asks. Computer learns (active learning)
Cf Adobe font selection: with attributes, aesthetic appearance.
Attributes are subjective: adjectives like “feminine” or “fashionable” can be ill-informed. Cannot use a single model.
To build good model, need a lot of data, many various participants. Build a basic model and have a crowd adapt it. Tweak existing model.
Learning attributes using human gaze.
Question of identity: bring a person to pre-defined characteristics.
Temporal adjectives: colors can be more measurable. But “schools of thought”, spatial differences.
2. Vision to analyze aesthetics:
Photographer identification: who took this photograph? 74% accuracy. Human performance is 47%
CNNs: Convolutional Neural Networks
Features: gradiance in image, pixels.
Deep learning: trained to distinguish between 1000 object categories (person, couch, car…). Has multiple layers. Can treat responses of each layer.
Style determined by photograph.
Can generate novel photographs by given creators: crazy idea!
Project looking into trained photographers, who think more about questions of composition and style. But what about artists who have a wide range of subject-matter?
Dataset: wide range temporal.
GIST: captures global shape of the scene.
What captivates a viewer? Where do people look? Semantic notion of what is interesting. Eye-tracking. Difference between original object or screen reproduction.
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3170918/
We see in different ways depending on your training.
Christopher Nygren
“Heads, Shoulders, Knees and Toes” Giovanni Morelli and the Computational History of Art”
What does computational mean?
Algorithmic
Morelli’s process. What did computational art history looked at, at one point?
He might serve as a productive starting point for questions we pose.
Morelli, born 1816, Swiss. Lived Berlin and Paris. Founders of Italian system of museums.
Published in German under pseudonyms. His publications take form of a dialogue with Morelli and an aristocrat.
Contempt for how study of art had developed, in Paris and in Germany. Also for archivists, disdain for both.
Lacked strong understanding of paintings. To distinguish good from the bad, and best from the better. Many bad attributions of paintings in the 19th century. Motivated his approach.
Dresden Venus, not recognized as Giorgione.
Met resistance. What is his method? Focus should be on the detail. Where we spend the least attention. Anatomical features like ears, fingernails, etc. became the locus for his study for authentication.
Illustrates with deceptive sketch from a detail of the hand and thumb that he talks about. Maybe not from Titian’s portrait of the cardinal. Maybe taken form another painting by Titian. And that is precisely his point: reflective of the mind of the artist. These hands are Titian’s hands.
Illustrations that challenge the art historian’s sensibility.
Schematic rendering of the details. Not only recognize but reproduce. Reduction of visual detail to focus on what he defined as the essential: line, not tone.
Renderings lack the tonal aspect of the paintings, that also make a Titian a Titian: flesh.
Through his dialogues, he can give voice to some discomforts to his method.
Renounce the whole in pursuit of the detail.
Morelli trained as a doctor. Maybe influences his technique. Methods going through a revolution (Freud, etc.). Cf Ginsberg article.
Recent articles from medicine on these Renaissance artworks.
Draws to the minutia of a painting in the hope of saying something of consequence of the whole.
Medicine: training tool to learn how to look.
Proto-object: a place that computers see stuff that we don’t.
This way of thinking about the world in 19th century made the computers happen.
Unitary act of creation that Morelli propagates: but really, it’s a workshop system.
Semantic gap: do we have the language that we need? Powerful in its failure. Realizing the limitations.
Are we all Morellians?
Assumptions that it matters WHO painted this painting. MONEY. Image feature that is subjective.
Forgeries.
All different elements that vary the ear: changes in time and space.
Very focused on descriptive texts back then. We are all in a land of interpretation.
We take portraiture as having to be realistically accurate.
Paintings never speak in a single voice.
Morelli wants to find that it is all unconscious. Just like describing objects in an Internet search.
Analysis based on line, reproduced by another hand (sketched and engraved): layers from thing reproduced to object of analysis.
Using images on the screen: the image itself is modified.
Scientific methods today go beyond the surface of the painting to analyze it: deep through layers of paint. Scientific data from pigments and chemical components. Computers raise the data and humans interprate it.
Benjamin Tilghman
“Complexity and Emergence in Early Medieval Art”
Kinds of practices of seeing in Northern Europe in Middle Ages.
From objects and jewels: complex patterns, figurative (animals). Parse out as you are looking at them.
Physical manipulation or mental reconstruction.
Magical assumption or calling of the power of that animal represented.
If you recognize the pattern, then you can trust the person. Creates association.
Strong act of process: a lot of metal work.
Cf Lindisfarne Gospels, f. 94v, Northumbria, early 8th century
Stylized X (for Christ): what kind of viewing experience was this supposed to create? Why illustrate the Bible with this instead of figurative/realistic pictures.
Numerological analysis too.
Challenge to slow down the viewer.
Ornamenting has meaning.
Algorithm to create. What mathematical tools did they use?
Beautiful composition on the basis of the process. Mechanical procedures. Layers of patterns one on top of another.
Sequences of patterns that reinforce the metal (on a sword). Structural power and symbolic. Connection of process and product. Artefacts and digital reconstructions of the patterns (Cf reconstructions of Evans in Crete). But can an artist change the plan when making the object, through the operation?
Artist as author of the work. Creator of design and executor.
But these pages don’t necessarily break down to rational units.
Computational to measure and structure God’s creation (calendar, Easter, etc)
Symbols, like the cross, emerge from the lace of the design, not drawn by artist. Negative space highlighted by black. The artist is not the pride maker, but lets God guide their hands. Think of themselves as transmitters, not creators.
Need for an observer. Depend on pre-existing conditions and people to recognize. Patterns exist in our perception of nature. Calls for interpretation on the part of the viewer. Active. Art as a way of getting to something that is greater than you, and also in the making of it.
Latent and blatant designs. Cf nature of digital technology: we like computers because they make something invisible visible.
Volition: you want to see what you see.
Performative practice of seeing. Elite kind of viewing (Book of Kells: not for the text, but for the beauty of it. Taken out at specific times and for VIPs).
Tom Lombardi
“Interdisciplinary Approaches to Metadata”
Computing projects in different disciplines.
Interdisciplinary metadata analysis: counting a phenomenon.
Used for biology, economics, sociology, computer, ecology. Can it be applied to art history?
Test case: metadata from the Index of Christian art.
Differential expression in iconography: before and after the Black Death. Gets metadata. But cannot explain it. Problem of bias in getting the numbers.
Exploratory analysis. Plus surviving artifacts: destruction through times or during WWII.
How can you use the metadata for something interesting? You get these figures and then we look to explain it.
Different counts of images, not combinations of images. Pairing with groups of people.
How do images survive? As many depictions as possible.
Network structure of the interaction of saints, through art historians through time.
Create directed network: Antony and Francis. Always.
Show popularity of images over time. Synchronized popularity over time. Curbs can show this.
DISCUSSION
Connecting artwork in a system that would not be based on artworks.
What is on display with what, what is in proximity to what else.
Information around the artwork: metadata analysis.
Iconographic analysis: symbols, how often they come up. You can quantify that.
Isms: known sets that beg questions to us.
Can you find a pattern that gives you a method.
Importance of chronology.
Descriptor to the property.
Can you model cubism?
Trying to pick a metric to understand an artist’s hand.
Helpful to say it first, describe it in words.
Art historians will become interested when it can be multi-layered: color, texture, pixel, chronology, all at once. Multi-varied analyses. A computer could compare what a human does not necessarily compare.
What do we mean by visually similar?
Follow the prototype: cf Byzantines. But they did not follow the prototype.
What is likeness?
Need to group things. But computers can deal with things without having to group them.
Topic modeling: Type of algorithm. Corpus of letters sent by someone. Corpus of individual documents.
Topic modeling dumps all the words in one bag so you lose all relative semantics;
Takes all the stop words out: skip them so that most popular words don’t take unnecessary space; harder with poetry
It does math: LDA. Distributes likelihood that words appear.
But computer does not know the topic: just says this is the likelihood that these words appear together.
Run a feature: it can do the same, with the same math. Can group the same way.
Featured intentionally constructed instead of discovered.
Elite viewing practices vs basic viewing practices.
Cf Wolfflin: concepts for Renaissance and Baroque paintings descriptors.
Not applicable to all art.
Matthew Lincoln: Modelling the (Inter)national printmaking networks of Early Modern Europe
Social history of art: metadata studies
Using prints for computer model. Easier than paintings.
Computer model to reconstruct the collector’s portfolio/print album
No Comments