\nToward Human-like Understanding of Visual Content<\/strong><\/p>\n
We need functional searches for image searching. How do we do it? We type a keyword in a search bar in Google image search\u2014the only reason it works is because images are tagged with text.
\nWhat happens if you have an image of something you want to look for and don\u2019t have a word for it? Keywords don\u2019t help. So you interact by describing and narrowing down what you\u2019re looking for.
\nIn an in-person scenario, a person responds with guesses which get closer and closer with more detailed description.
\nIs there a similar way to interact with a computer? (Well, computers can\u2019t see\u2026)
\nHow is interactive search done today?<\/p>\n
Let\u2019s say you saw a person whom you think committed a crime, and you want to report their description to the authorities:
\nEnter \u201cthin white male\u201d
\nTry binary relevance feedback in the list of results\u2014and exclude some as relevant and some as irrelevant.
\nTraditional binary feedback is imprecise; allows only coarse communication between user and system<\/p>\n
Adriana proposes: Allow people to select one result and say: \u201cSimilar to this\u2026but with curlier hair\u201d: Search via comparisons: a \u201cwhittle search\u201d\u2014lets you \u201cwhittle away\u201d irrelevant images via comparative feedback<\/p>\n
Prior work: Semantic Visual attributes – essentially adjectives
\nHigh level descriptive properties shared by objects
\nHuman-understandable and machine detectable
\nMiddle ground between user and system
\nPrimarily used in object recognition<\/p>\n
Let people make relative comments on properties: yellow, spiky, smooth, natural, perspective open smiling etc.<\/p>\n
Idea: Attribute Pivots for Guiding Feedback
\nHave the computer play a 20-questions game with the user:
\nA person wants to buy shoes.
\nThe computer asks: \u201cAre the shoes you seek more or less feminine than this picture?\u201d
\nWhat questions should the computer ask?
\nSelect series of the most informative visual comparisons that user should make to help deduce target.
\nThis is like active learning\u2014and it\u2019s computationally expensive to evaluate all possible questions and figure out what the next question in a series should be.<\/p>\n
Impact of WhittleSearch: Adobe Font Selection
\nUsers retrieve fonts that match requested attributes and fonts are sorted by relative attribute scores. Example: fonts with these attributes: \u201cnot delicate.\u201d
\nLots of attributes are really subjective. PROBLEM: One Model DOES NOT fit all. But people want to build a single model for all users.
\nValid perceptual differences within an attribute, yet existing methods assume a monolithic model is sufficient.<\/p>\n
Ideas to improve: Learn User-Specific Attributes
\nBuild a generic model that distributes a labelling method among a crowd. Adapt generic attribute model with minimal user-specific labeled examples.
\nTry to discover shades of attributes: Discover \u201cschools of thought\u201d among users based on latent factors behind their use of attribute terms. Allows discovery of the attribute\u2019s \u201cshades of meaning.\u201d
\nUse the human gaze to learn where attributes \u201clive\u201d: Which parts of a face give an \u201cAsian\u201d quality or a \u201cbaby-like\u201d quality?<\/p>\n
Q: Which came first: Your list of adjectives, or people assigning independent adjectives?
\nA: Others have tried to evolve a vocabulary from scratch. She\u2019s refining a generic model with more crowd labelling.<\/p>\n
Q: Could you do any of this model refining by machine learning from where users click most frequently\u2014and NOT based on conscious responses?
\nA: Yes, and that would be interesting, but that\u2019s not what she\u2019s been working on.<\/p>\n
Q: Does the crowd influence people\u2019s sense of attribute meaning?
\nA: Users don\u2019t often deviate entirely from the crowd, but sometimes do.<\/p>\n
Q: Is there \u201ctemporal drift\u201d in a notion like \u201cfashionable\u201d? (How do attributes change over time?) These adjectives aren\u2019t tethered to anything, are they?
\nA: I don\u2019t have an answer to that\u2014but yes there are cultural\/spatial differences in people\u2019s sense of fashionable. There\u2019s a computer vision paper that studied how cars evolve over time.
\nUsing vision to analyze aesthetics<\/p>\n
Photographer identification: Can you tell who took a photograph from the photos themselves? Can you build a computer model that can discriminate among 41 different photographers and some hundreds or thousands of pictures.<\/p>\n
Deep net features achieve 74% accuracy.
\nStudies gradients in a region of the photo\u2014generate a histogram of gradients.
\n\u201cDeep learning\u201d: deep neural networks that are trained to distinguish between 1,000 object categories like person, couch, car, etc. Use whatever the network is giving as a classification. It\u2019s called \u201cdeep\u201d because it has multiple layers. Take the last layer of your network\u2014what\u2019s the probability that each of these 1,000 objects is present in your image? When we do that, the network achieves 74% accuracy. Chance is less than 3%. Human performance is 47%.
\nMethod learns which proto-objects and scenes authors shoot.
\nThis can be used this to develop \u201cfield guides\u201d for human use.
\nThis can generate photographs in the style of a given author: as in, GENERATE an Ansel Adams photo. They\u2019ve had some positive early results.<\/p>\n
Q: Is that done by subject matter, then? Identifying objects in an image?
\nA: Yes\u2014objects and background are cues.<\/p>\n
Q: Are you mostly looking at art photographers?
\nA: Yes, not looking at amateurs. It might matter because professional photographers will care more about image composition, etc.<\/p>\n
Q: What about photographers with a wide range of subject matter?
\nA: Probably the model won\u2019t work as well.
\nMake computer system generate a photo \u201cby\u201d a photographer: object and background comes through.<\/p>\n
Predicting the \u201cmystique\u201d of a photograph:
\nWhat part of the image will viewer stare at? Called saliency: ability to predict WHERE the viewer will stare at a picture?<\/p>\n
Q: How does this saliency model work?
\nA: Computer look for high contrasts, and faces.<\/p>\n
Q: Does it matter whether you\u2019re looking at a digital reproduction or a print photo?
\nA: Work in saliency ignores semantics.<\/p>\n
We can use the conflict between connotations of works to capture \u201cmystique.\u201d
\nAnalyzing body pose and image setting for visual persuasion: which images look more positive, favorable, powerful, comforting, competent for images of politicians?
\nIt\u2019s all human annotated.<\/p>\n
Their test: Show users passages of text and ask questions, and then show them some images (as though it\u2019s a break from answering questions about a text\u2014just to get people away from an unnatural close analysis of pictures\u2014distract the viewer to see how they\u2019d actually look at a photo).<\/p>\n
Christopher Nygren, History of Art and Architecture, University of Pittsburgh<\/strong> Look at portrait of Senator Giovanni Morelli, 1886 by Franz von Lenbach: the social history of art history dismissed the formalism of this picture. Trained as a doctor in Switzerland, Morelli lived off of inheritance with odd jobs. He was associated with art dealers and collectors, served as a senator after the Risorgimento, and became a founder of museums in Italy.<\/p>\n Starting in 1874, Morelli published things in German under pseudonyms (including Johannes Schwarz\u2014\u201cMorelli\u201d means dark or black, as does \u201cSchwarz\u201d). These publications gave art historians a \u201cMorellian method\u201d written in dialogic form.<\/p>\n Morelli\u2019s method animated a deep-seated contempt for academic context of 18th and 19th century Paris and Germany\u2014\u201cspits venom\u201d at these places, and at collectors. He said these people spoke in sweeping generalizations\u2014dismissed them as \u201cart philosophers\u201d wont to look at pictures as though they were mirrors.<\/p>\n Morelli proposes \u201cthe connoisseur\u201d as opposed to these \u201cart philosopher\u201d\u2014The connoisseur has the proper means to judge: \u201cThe only true record for the connoisseur is the work of art itself.\u201d The \u201cgeneral impression\u201d matters to Morelli\u2019s connoisseur.<\/p>\n Studied Giorgone\u2014too many pictures had been ascribed to him\u2014Morelli was trying to purge the corpus of misidentifications. How did he do it, and why does it matter to computational aesthetics? Instead: Anatomical features like fingernails, toes, noses\u2014became the locus of Morelli\u2019s classifications. Features such as Titian\u2019s adherence to a rounded ear and a round ball of the thumb.<\/p>\n But Morelli fudges a bit\u2014images he sketches don\u2019t quite match the pictures he says he\u2019s talking about. It could be that he\u2019s pulling from other pictures as models for the generic hand Morelli sketches as \u201ctypial\u201d of a Titian hand. But that\u2019s the point\u2014it\u2019s an iterative model. Morelli\u2019s work is littered with illustrations which are profoundly alienating: nose, earlobe\u2026 these offer a schematic rendering of those details.<\/p>\n Morelli would probably have used photographic details if technology allowed\u2026 instead he uses line engravings\u2014he\u2019s going for the most salient feature and so eliminates contrast and tone, among other elements of a painting. Morelli was working around the same time as Arthur Conan Doyle and Sigmund Freud. These methods predicated on an emergent form of diagnostics. As Carlo Ginsberg described it, the art historian became a detective.<\/p>\n Medical doctors today like to look at pictures to diagnose issues like breast cancer in them\u2014very dodgy, and not quite what Morelli was doing, but related to the diagnostic approach. Q: Isn\u2019t this what the GIST algorithm does\u2014with identifying proto-objects?<\/p>\n Q: Does Morelli ever show us some sketches that he gets absolutely right? Q: What about Morelli developing a typology\u2014Do we see him draw this in grids? Photographic details would not serve Morelli\u2019s purpose: he needs abstraction. Q: Maybe the abstraction makes it \u201cscalable\u201d? Q: But this is so deeply problematic. Museum community importance, but art historians so deeply rejected Morelli that THIS WAY of thinking about the world helped make the digital computer possible and at the same time alienated art historians Q: Why did art historians reject this and \u201cmove on\u201d? In the worlds of textual scholarship, manuscripts\/medieval paleography, Morelli\u2019s presence is important.\u00a0What we\u2019re rejecting is the ideology of the dilettante white male who kicks around in a museum.<\/p>\n Q: Conservation and chemical based data\u2014that\u2019s connected to Morellian thinking, right? Q: We in computer science do this Morellian abstraction. Computer scientists want the language they need to describe something closely enough to predict. And Morelli\u2019s example is compelling for its failure. Q: The problem with Morelli is that it was all predicated on who was the painter. And that\u2019s all about the question of how much the painting is worth\u2014all about money. On subjective aspect, take Isabella D\u2019Este\u2014the wife of an important man, left in charge when he leaves town. Leonardo portrayed her at the age of 20-25 in 1490 versus one made later by Raphael when she was 60 (and she looks younger, and loved this picture). I suppose the donor matters.<\/p>\n Q: Can we train computers to want to see? The way humans want to see? Benjamin Tilghmann (Lawrence University, WI)<\/strong> Two areas of focus: \n Helmet from Sutton Hoo\u2014looks like it\u2019s got a mustache\u2014but those are tailfeathers attached to a bird:<\/p>\n
\nHeads, Shoulders, Knees, and Toes\u2026Computational Art History<\/strong><\/p>\n
\nMorelli and his benighted method shows us something about \u201csemantic reduction\u201d\u2014how was this done 150 years ago?<\/p>\n
\nDetail matters in places where painters are expending the least attention, rather than the most.
\nBeauty follows very generic patterns based on Petrarchan ideal.<\/p>\n
\nThere are tonal qualities that make a Titian a Titian and \u201cincarnating flesh on canvas\u201d (paint that becomes flesh). It seems that Morelli does \u201caesthetic violence\u201d to this. Morelli himself recognized that his method was disconcerting.<\/p>\n
\nBut the idea of identifying Minimal degrees of variation with maximal consequence is potentially important to computational visual aesthetics.<\/p>\n
\nA: Yes\u2014his ears!<\/p>\n
\nA: When he does that\u2014the ears are again probably the best examples.<\/p>\n
\nIs the abstraction a level of \u201cPlatonic ideal\u201d?<\/p>\n
\nA: Yes: attempt to identify principles that will help people in museums.<\/p>\n
\nA: Art historians call Morelli\u2019s practices a kind of \u201cviolence\u201d to Titian.<\/p>\n
\nA: Because it was too schematic a model: We\u2019ve realized that what we call \u201cTitian\u201d is actually a bunch of people working in his workshop\u2014he has a production model\u2014a number of specialists who help out. Morelli takes a given the unitary act of creation, but art historians have pushed back against us.
\nBut Morelli also taught us how pictures are created. This makes us wonder about the concept of \u201cobsolete.\u201d We\u2019d question whether really Morelli is obsolete\u2014we just don\u2019t want to admit the importance of his presence.<\/p>\n
\nA: Yes\u2014look at canvases and their warp and weft.<\/p>\n
\nA: Yes\u2014it\u2019s compelling for failures combined with major breakthroughs.<\/p>\n
\nQ: How much is art history is still about that?
\nDiscussion: We\u2019re all in a land of subjective interpretation, and the computer can give us something to interpet.<\/p>\n
\nA: We can use the computer to track human eye-path in gazing at picture\u2026 why couldn\u2019t volition be modelled, then?<\/p>\n
\nComplexity and Emergence in Early Medieval Art<\/strong><\/p>\n
\nThe volitional aspects of seeing in medieval northern Europe: What practices of seeing were expected then (not normally associated with practices of seeing in Western art)?
\nCompositional\/geometric method that went into interlace \u201ccarpet\u201d pattern of Lindisfarne gospels.
\nSlide of shoulder clasp from Sutton Hoo site:<\/p>\n![\"shoulder\"](\"https:\/\/sites.haa.pitt.edu\/cva\/wp-content\/uploads\/sites\/11\/2015\/12\/shoulder.jpg\")
<\/a>\n<\/p>\n