Comparing semantics (and relational summaries) of Humans and Deep Networks

One of the primary applications of Incremental Multiresolution Matrix Factorization is to model hierarchical correlations in the data. As motivatved by the construction of the factorization, and the corresponding construction of the MMF graphs (refer to Incremental MMF ), this visualization of correlational stucture if a direct outcome of the MMF graphs.

To make this interesting, we conducted a small survey on 75 participants asking three simple questions -- What is an outlier? Which classes are most related? Which classes are most representative? . The outcome of this experiment on three sets of clases from Imagenet are summarized here. For each of these questions, we visualize its "frequency" in the MMF. An outlier according to humans is expected to be high frequency according to MMF.

---

The Google survey/questionnaire that was used to construct this can be accessed here.
Class_by_Class Hierarchical Relationships


Example: A set of Animal classes



10 classes: cow, insectivore, hound, puppy, garden spider, ptarmigan, phalanger, killer whale, green lizard, kangaroo

This is a versatile set of classes i.e., the contextual information (like background, typical sets of objects that are found etc.) for cow, hound may be drastically different from kangaroo, ptarmigan. Clearly, the most distinct classes -- green lizard, killer whale are resolved at highest levels. The composition at first level involves those classes which share context -- there is most certainly grass or trees ot bark in these images. Given the first level, the graph shows that kangaroo is most expressible by the first level composition, compared to green lizard -- implying a clear sense of hierarchy in learned representations.


Question. What is the outlier?

The class whale is chosen as the outlier by the survey. As shown by the MMF graph, this class in fact corresponds to the highest frequency i.e., the information in whale is accounted for after decorrelating/interacting the remaining 9 classes -- clearly implying that whale is distint to the rest of the animals here.


Question. What are the 5 most related classes?

The survey chose hound, possum, insectivore, puppy and cow as the most related classes. The word related was left to the interpretation of the user (the survey taker). Four out of the five classes are in the lowest level correlation/frequencies of the MMF graph. This is an intersting "match" between the human semantics and deep representational sematics that is decoded by the hierarchical factorization provided by the MMF. Observe that the five animals have contextual relationship i.e., they are animal classes with common background (green grass, trees, hillsides etc.)


Question. What are the 5 most representative classes?

This final question is asking what classes are the most representative. Observe that this is different from the previous question, and as was the case with earlier question, the meaning of representative-ness is left for user interpretation. An interesting pattern emerges with respect to human and deep semantics here. The classes picked by the survey are whale, hound, possum, cow and kangaroo. As can be seen from the MMF graph, these five classes are coming from different frequencies (i.e., different levels of the factorization). In other words, the five classes chosen by the survey correspond to uniform sampling of classes across all hierarchical frequencies. This is a fundamental observation. The choices made by humans to summarize data inherently is multi-scale i.e., they tend to choose entities that span both DC type low-frequency and well as more band-pass type higher frequenc elements. Although the nature of what frequency here means, this aspect of sampling all "kinds" of entities is apparently common among a group of individuals (the survey was on 75 participants and is being extended using Mechanical Turk).