Not only language can sometimes be an obstacle in communication, it also requires a special level of knowledge to understand it. For this blog post, I’ve tried to collect more information about cognitive science and image processing, in order to be able to understand better the process behind our comprehension of words and images.
Since these findings could enlighten some crucial aspects of my thesis, I will share here the facts I found important to mention.
What do we consider as a language?
The definition of a language varies a lot, as you browse through the internet. According to Britannica, a language is a system of conventional spoken, manual (signed), or written symbols by means of which human beings, as members of a social group and participants in its culture, express themselves. The functions of language include communication, the expression of identity, play, imaginative expression, and emotional release.
Any natural language can be described by a system of symbols (lexemes) and the grammars (rules) which manipulate the symbols. What makes one language unique are the symbols, grammars, and their interactions that make a particular language unique from others, although certain languages do have overlapping sets of symbols or grammars. For instance, “cognates” represent words that have similar spelling across languages, usually identified based on some string similarity measure (like longest common subsequence, or Levinstein distance). Such a string similarity will indicate that e.g. “name” (English) and “nome” (Italian) and “nume” (Romanian) refer to the same concept, and were derived from the same root. Despite such evidence, the concept of “cognates” usually holds only between languages from the same family (like Romance languages), and can hardly, if ever, helps the communication process between speakers of languages from different language families.
Cognitive Science
Cognitive science is the interdisciplinary study of mind and intelligence, embracing philosophy, psychology, artificial intelligence, neuroscience, linguistics, and anthropology. Cognitive science explains that we use pictorial representation because they deliver information way much quickly than verbal descriptions.
That would also explain why do people have visual images of situations, but it also claims that people have processes such as scanning and rotation that operate on those images, which produce intelligent behavior.
How do we apprehend sentences?
The meaning of a sentence is encoded in each of its component words. To understand a sentence, all meanings of the individual words must be retrieved and combined. But what if we change the words with pictures? How do we make connections then?
Some studies conducted by M.C. Potter, J.F. Kroll, B. Yachzel, E. Carpenter, and J. Sherman and explained in the Journal of Experimental Psychology 115 “Pictures in sentences: understanding without words” have shown, that situations in which the concrete noun in a sentence is replaced with a pictorial representation which should symbolize the word, and which is then shown to participant in rapid serial visual representation (RSVP) to forty subjects, give no significant delay in the understanding of rebus sentences compared to all words sentences.
What is also worth mentioning, is that the speed of understanding and accuracy of comprehension remains the same, no matter what position the picture has. Also, it did not make the difference whether there it was one picture or two pictured object replacing the nouns.
We could make a conclusion out of that and say, that the lexical representation (dictionary definition) of a noun points to a conceptual system in the human brain (which contains people’s knowledge about the world), where the meaning of the sentence is constructed. This also points to the possibility that humans can communicate with one another through non-linguistic means.
Sources:
https://academicconnections.ucsd.edu/onlinecourses/intro-cogsci.html
https://pubmed.ncbi.nlm.nih.gov/2944988/
https://www.britannica.com/topic/language
https://www.kfki.hu/~csdori/Concepts.pdf
https://digital.library.unt.edu/ark:/67531/metadc5263/m2/1/high_res_d/thesis.pdf
M.C. Potter, J.F. Kroll, B. Yachzel, E. Carpenter, and J. Sherman, Pictures in sentences: understanding
without words, Journal of Experimental Psychology 115 (1986), no. 3, 281–294.
K. Barnard and D.A. Forsyth, Learning the semantics of words and pictures, Proceedings of the IEEE
International Conference on Computer Vision, 2001.
