What do we want computers to do?
\nHow can HAA inform computer science?<\/p>\n
Adriana Kovashka 1. Semantic way of finding images Christopher Nygren What does computational mean? Benjamin Tilghman Kinds of practices of seeing in Northern Europe in Middle Ages. Tom Lombardi Computing projects in different disciplines. Test case: metadata from the Index of Christian art. DISCUSSION<\/p>\n Connecting artwork in a system that would not be based on artworks. Featured intentionally constructed instead of discovered.<\/p>\n Elite viewing practices vs basic viewing practices. FRIDAY MORNING SESSION What do we want computers to do? How can HAA inform computer science? Adriana Kovashka \u201cTowards Human-like understanding of visual content\u201d 1. Semantic way of finding images How can we search better through all visual data on the web? Too big to browse. Keywords for known name. But if you don\u2019t know…<\/span><\/p>\n
\n\u201cTowards Human-like understanding of visual content\u201d<\/strong><\/p>\n
\nHow can we search better through all visual data on the web? Too big to browse.
\nKeywords for known name. But if you don\u2019t know the name: you can only describe a thing. What do you type? Interaction to describe what you are looking for. Steps in describing details, narrowing it down on physical characteristics.
\nBut computers can\u2019t see.
\nInteractive search done today: looking for a person. Binary relevance feedback, from the initial search results. But very imprecise. Search via comparisons: whittle search. Relates to what you\u2019re looking for based on particular properties of change and modification.
\nHigh-level descriptive properties (adjectives), human-understandable, middle ground between user and system.
\n20 question game: what questions the computer asks. Computer learns (active learning)
\nCf Adobe font selection: with attributes, aesthetic appearance.
\nAttributes are subjective: adjectives like \u201cfeminine\u201d or \u201cfashionable\u201d can be ill-informed. Cannot use a single model.
\nTo build good model, need a lot of data, many various participants. Build a basic model and have a crowd adapt it. Tweak existing model.
\nLearning attributes using human gaze.
\nQuestion of identity: bring a person to pre-defined characteristics.
\nTemporal adjectives: colors can be more measurable. But \u201cschools of thought\u201d, spatial differences.
\n2. Vision to analyze aesthetics:
\nPhotographer identification: who took this photograph? 74% accuracy. Human performance is 47%
\nCNNs: Convolutional Neural Networks
\nFeatures: gradiance in image, pixels.
\nDeep learning: trained to distinguish between 1000 object categories (person, couch, car\u2026). Has multiple layers. Can treat responses of each layer.
\nStyle determined by photograph.
\nCan generate novel photographs by given creators: crazy idea!
\nProject looking into trained photographers, who think more about questions of composition and style. But what about artists who have a wide range of subject-matter?
\nDataset: wide range temporal.
\nGIST: captures global shape of the scene.
\nWhat captivates a viewer? Where do people look? Semantic notion of what is interesting. Eye-tracking. Difference between original object or screen reproduction.
\nhttp:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3170918\/
\nWe see in different ways depending on your training.<\/p>\n
\n\u201cHeads, Shoulders, Knees and Toes\u201d Giovanni Morelli and the Computational History of Art\u201d<\/strong><\/p>\n
\nAlgorithmic
\nMorelli\u2019s process. What did computational art history looked at, at one point?
\nHe might serve as a productive starting point for questions we pose.
\nMorelli, born 1816, Swiss. Lived Berlin and Paris. Founders of Italian system of museums.
\nPublished in German under pseudonyms. His publications take form of a dialogue with Morelli and an aristocrat.
\nContempt for how study of art had developed, in Paris and in Germany. Also for archivists, disdain for both.
\nLacked strong understanding of paintings. To distinguish good from the bad, and best from the better. Many bad attributions of paintings in the 19th century. Motivated his approach.
\nDresden Venus, not recognized as Giorgione.
\nMet resistance. What is his method? Focus should be on the detail. Where we spend the least attention. Anatomical features like ears, fingernails, etc. became the locus for his study for authentication.
\nIllustrates with deceptive sketch from a detail of the hand and thumb that he talks about. Maybe not from Titian\u2019s portrait of the cardinal. Maybe taken form another painting by Titian. And that is precisely his point: reflective of the mind of the artist. These hands are Titian\u2019s hands.
\nIllustrations that challenge the art historian\u2019s sensibility.
\nSchematic rendering of the details. Not only recognize but reproduce. Reduction of visual detail to focus on what he defined as the essential: line, not tone.
\nRenderings lack the tonal aspect of the paintings, that also make a Titian a Titian: flesh.
\nThrough his dialogues, he can give voice to some discomforts to his method.
\nRenounce the whole in pursuit of the detail.
\nMorelli trained as a doctor. Maybe influences his technique. Methods going through a revolution (Freud, etc.). Cf Ginsberg article.
\nRecent articles from medicine on these Renaissance artworks.
\nDraws to the minutia of a painting in the hope of saying something of consequence of the whole.
\nMedicine: training tool to learn how to look.
\nProto-object: a place that computers see stuff that we don\u2019t.
\nThis way of thinking about the world in 19th century made the computers happen.
\nUnitary act of creation that Morelli propagates: but really, it\u2019s a workshop system.
\nSemantic gap: do we have the language that we need? Powerful in its failure. Realizing the limitations.
\nAre we all Morellians?
\nAssumptions that it matters WHO painted this painting. MONEY. Image feature that is subjective.
\nForgeries.
\nAll different elements that vary the ear: changes in time and space.
\nVery focused on descriptive texts back then. We are all in a land of interpretation.
\nWe take portraiture as having to be realistically accurate.
\nPaintings never speak in a single voice.
\nMorelli wants to find that it is all unconscious. Just like describing objects in an Internet search.
\nAnalysis based on line, reproduced by another hand (sketched and engraved): layers from thing reproduced to object of analysis.
\nUsing images on the screen: the image itself is modified.
\nScientific methods today go beyond the surface of the painting to analyze it: deep through layers of paint. Scientific data from pigments and chemical components. Computers raise the data and humans interprate it.<\/p>\n
\n\u201cComplexity and Emergence in Early Medieval Art\u201d<\/strong><\/p>\n
\nFrom objects and jewels: complex patterns, figurative (animals). Parse out as you are looking at them.
\nPhysical manipulation or mental reconstruction.
\nMagical assumption or calling of the power of that animal represented.
\nIf you recognize the pattern, then you can trust the person. Creates association.
\nStrong act of process: a lot of metal work.
\nCf Lindisfarne Gospels, f. 94v, Northumbria, early 8th century
\nStylized X (for Christ): what kind of viewing experience was this supposed to create? Why illustrate the Bible with this instead of figurative\/realistic pictures.
\nNumerological analysis too.
\nChallenge to slow down the viewer.
\nOrnamenting has meaning.
\nAlgorithm to create. What mathematical tools did they use?
\nBeautiful composition on the basis of the process. Mechanical procedures. Layers of patterns one on top of another.
\nSequences of patterns that reinforce the metal (on a sword). Structural power and symbolic. Connection of process and product. Artefacts and digital reconstructions of the patterns (Cf reconstructions of Evans in Crete). But can an artist change the plan when making the object, through the operation?
\nArtist as author of the work. Creator of design and executor.
\nBut these pages don’t necessarily break down to rational units.
\nComputational to measure and structure God\u2019s creation (calendar, Easter, etc)
\nSymbols, like the cross, emerge from the lace of the design, not drawn by artist. Negative space highlighted by black. The artist is not the pride maker, but lets God guide their hands. Think of themselves as transmitters, not creators.
\nNeed for an observer. Depend on pre-existing conditions and people to recognize. Patterns exist in our perception of nature. Calls for interpretation on the part of the viewer. Active. Art as a way of getting to something that is greater than you, and also in the making of it.
\nLatent and blatant designs. Cf nature of digital technology: we like computers because they make something invisible visible.
\nVolition: you want to see what you see.
\nPerformative practice of seeing. Elite kind of viewing (Book of Kells: not for the text, but for the beauty of it. Taken out at specific times and for VIPs).<\/p>\n
\n\u201cInterdisciplinary Approaches to Metadata\u201d<\/strong><\/p>\n
\nInterdisciplinary metadata analysis: counting a phenomenon.
\nUsed for biology, economics, sociology, computer, ecology. Can it be applied to art history?<\/p>\n
\nDifferential expression in iconography: before and after the Black Death. Gets metadata. But cannot explain it. Problem of bias in getting the numbers.
\nExploratory analysis. Plus surviving artifacts: destruction through times or during WWII.
\nHow can you use the metadata for something interesting? You get these figures and then we look to explain it.
\nDifferent counts of images, not combinations of images. Pairing with groups of people.
\nHow do images survive? As many depictions as possible.
\nNetwork structure of the interaction of saints, through art historians through time.
\nCreate directed network: Antony and Francis. Always.
\nShow popularity of images over time. Synchronized popularity over time. Curbs can show this.<\/p>\n
\nWhat is on display with what, what is in proximity to what else.
\nInformation around the artwork: metadata analysis.
\nIconographic analysis: symbols, how often they come up. You can quantify that.
\nIsms: known sets that beg questions to us.
\nCan you find a pattern that gives you a method.
\nImportance of chronology.
\nDescriptor to the property.
\nCan you model cubism?
\nTrying to pick a metric to understand an artist\u2019s hand.
\nHelpful to say it first, describe it in words.
\nArt historians will become interested when it can be multi-layered: color, texture, pixel, chronology, all at once. Multi-varied analyses. A computer could compare what a human does not necessarily compare.
\nWhat do we mean by visually similar?
\nFollow the prototype: cf Byzantines. But they did not follow the prototype.
\nWhat is likeness?
\nNeed to group things. But computers can deal with things without having to group them.
\nTopic modeling: Type of algorithm. Corpus of letters sent by someone. Corpus of individual documents.
\nTopic modeling dumps all the words in one bag so you lose all relative semantics;
\nTakes all the stop words out: skip them so that most popular words don\u2019t take unnecessary space; harder with poetry
\nIt does math: LDA. Distributes likelihood that words appear.
\nBut computer does not know the topic: just says this is the likelihood that these words appear together.
\nRun a feature: it can do the same, with the same math. Can group the same way.<\/p>\n
\nCf Wolfflin: concepts for Renaissance and Baroque paintings descriptors.
\nNot applicable to all art.
\nMatthew Lincoln: Modelling the (Inter)national printmaking networks of Early Modern Europe
\nSocial history of art: metadata studies
\nUsing prints for computer model. Easier than paintings.
\nComputer model to reconstruct the collector\u2019s portfolio\/print album<\/p>\n","protected":false},"excerpt":{"rendered":"