Quantum GestART: identifying and applying correlations between mathematics, art, and perceptual organization

Abstract

Mathematics can help analyze the arts and inspire new artwork. Mathematics can also help make transformations from one artistic medium to another, considering exceptions and choices, as well as artists' individual and unique contributions. We propose a method based on diagrammatic thinking and quantum formalism. We exploit decompositions of complex forms into a set of simple shapes, discretization of complex images, and Dirac notation, imagining a world of “prototypes” that can be connected to obtain a fine or coarse-graining approximation of a given visual image. Visual prototypes are exchanged with auditory ones, and the information (position, size) characterizing visual prototypes is connected with the information (onset, duration, loudness, pitch range) characterizing auditory prototypes. The topic is contextualized within a philosophical debate (discreteness and comparison of apparently unrelated objects), it develops through mathematical formalism, and it leads to programming, to spark interdisciplinary thinking and ignite creativity within STEAM.

Keywords:

• Gestural similarity
• Gestalt
• diagrams
• Dirac notation
• sonification

2010 Mathematics Subject Classification:

• 81-00
• 18C10
• 42-00

Authors' contributions

M. M. developed the initial idea, the mathematical section, and contributed to discussion and conclusions. F. F. contributed to the conceptual introduction. B. D. D. contributed to the pseudocode section. L. T. contributed to the cognitive application and helped revise the initial manuscript. All the authors exchanged ideas and gave feedback on each other's work.

Acknowledgments

The authors are grateful to the physicist Peter beim Graben and to the mathematicians Giuseppe Metere and Olivia Caramello for comments on the mathematical section. The authors thank the physicist Lucia Rizzuto for reading the manuscript and for her insightful questions, included the role of space separation in image-music translations and the role of beauty. The authors also thank the engineer and computer scientist Davide Rocchesso for his comments and suggestions about crossmodal perception. The authors thank the anonymous reviewers for their comments, as well as the editors in chief Emmanuel Amiot and Jason Yust for their careful work as editors.

Disclosure statement

No potential conflict of interest was reported by the authors.

ORCID

Maria Mannone http://orcid.org/0000-0003-3606-3436

Federico Favali http://orcid.org/0000-0002-8505-6163

Balandino Di Donato http://orcid.org/0000-0001-6993-2445

Luca Turchet http://orcid.org/0000-0003-0711-8098

Notes

1 Deleuze's contribution also involves diagrammatic thinking. According to Deleuze (1986, 43): “Le diagramme ne fonctionne jamais pour représenter un monde préexistant, il produit un nouveau type de réalité, un nouveau modèle de vérité,” that is, “A diagram doesn't work to represent a world as a pre-existing one; it rather produces a new type of reality, a new model of truth.”

2 The term “GestART” has been used by M. Mannone in a European application (H2020) in Summer/Fall 2017, and then used in this framework.

3 Limits and Colimits are concepts from category theory (Mac Lane 1978). As explained by Mannone (2018a, 12): “Intuitively, limits and colimits are generalizations of products and coproducts, respectively. The product is a special case of the limit, with discrete categories. The coproduct (also called sum) is the dual of the product, obtained by reversing the arrows. Given an object P and two maps $p_1:P\to B_1,p_2:P\to B_2$, P is a product of $B_1,B_2$ if for each object X and for each pair of arrows $f_1,f_2$ we have only and only one arrow $f:X\to P$ such that $f_1=p_{1}f,f_2=p_{2}f$ (Lawvere and Schanuel 2009), see diagram (1).”

4 This “unicity” is a rough example of the “universal property” shown by some constructions in category theory, such as colimits and limits, dual of the colimits (Mac Lane 1978).

5 The idea that a shape may contain its story can be also compared with evolution, creating an ideal connection between the evolution of natural forms and the evolution of musical forms. This fact may be contextualized in a wider framework, within the evolution of Western (artistic) thinking.

6 The synesthesia concept is familiar from poetry: let us recall the “black scream” in “[…] All'urlo nero della madre/che andava incontro al figlio/crocifisso sul palo del telegrafo” from the poem Alle fronde dei salici, in Giorno dopo giorno (1947) by Salvatore Quasimodo, awarded the Nobel prize for Literature in 1959.

7 Curiously, Reybrouck (1997) explicitly refers to algebraic methodologies and cites the functor term, but it appears as more in the sense of a function than of a categorical functor. A functor is a morphism between categories (Mac Lane 1978).

8 Gestalt as a colimit: all simple elements are connected into a colimit. We are considering here colimits and not limits because arrows are directed from small entities to the complete, recognizable form, the Gestalt. As explained above in our example with LEGO${}^{TM}$ bricks, let us consider a collection of simple shapes that can be combined together to obtain a recognizable form. Thus, the arrows we are considering are directed from the simple shapes to the ‘complete’ form. A reviewer thought of a colimit of ‘visual things’ as patterns in the brain. This is a very interesting remark that can be addressed in future research; also, there are recent studies between categories, brain, and consciousness, see Northoff, Tsuchiya, and Saigo (2019).

9 In the approach proposed in (Mannone 2017), images are seen as the result of processes, thus the use of Gestalt from images to music in this framework appears as more coherent.

10 This can easily be contextualized within studies about music and gestures in terms of physical movements. If we interpret visuals in terms of gestures, our quest for Gestalt translation from visual domain to auditory domain via mathematics and gestures may appear as well-defined. Also, according to Spence (2015, 11), crossmodal correspondences “can be conceptualized in terms of crossmodal grouping by similarity.”

11 In these works, the $\frac{1}{3}$ law is cited, that is, the connection between timbre variations and velocity profiles. This reminds one of the $\frac{2}{3}$ power law describing the connection between the kinematics of handwriting and the trajectory of a movement, between the curvature of handwriting and the pentip's angular velocity (Lacquaniti, Terzuolo, and Viviani 1984; Plamondon and Guerfali 1998).

12 Curiously, this paper has been submitted on the same day as a paper on categories and composer/conductor/listener, see Mannone (2018a).

13 Diagram (5) from Mannone (2018a), shows a mapping from an abstract collection of points and arrows (Δ) to the topological space $\overrightarrow{X}$ (containing curves in the X, the space of points in physical space and time). In the diagram, $g_{\phi }$ represent the physical gesture ($\varphi$) without any specified dynamics, while $g_{\phi }^F$ is the gesture with the dynamics forte (F). The double arrow F represent the deformation of the unspecified gesture into the “forte” gesture. This is a specific forte operator. We can think of the “forte” gestural generator as a generic “forte” gesture, to be adapted in the specific space of gestures for each musical instrument that deforms anonymous movements into movements that produce forte loudness on musical instruments.

14 A “skeleton” is an abstract structure of points and arrows; see Δ in diagram (5) as an example. Such a generic structure is then specified into a collection of points and arrows in a topological space. In our case, we consider physical space and time.

15 The visual kets of a set can be considered as linearly independent if none of them can be obtained as a linear combination of the other ones; similarly for the sound kets. We can consider two kets $|{\phi }_i^v⟩,|{\phi }_j^v⟩$ as orthogonal if their (gestural) similarity degree is zero (Mannone 2018a), and thus their scalar product is zero: $⟨{\phi }_i^v|{\phi }_j^v⟩=0$. However, if they are identical, $⟨{\phi }_i^v|{\phi }_j^v⟩=1$. Two kets with a similarity degree strictly between 0 and 1 have a scalar product between 0 and 1. Similarly for the sound kets. Thus, the scalar product assumes the meaning of a comparison between simple shapes or between simple musical sequences.

16 Formally, kets would require an Hilbert space, but here the imaginary component is zero or it is not considered. It could be identified with the mind component, that is embodied into the real part, that is, an artwork, during the artistic creation process (Mazzola 2018).

17 In ${\mathbb{R}}^n$ a circle is not a vector but a set of such vectors.

18 Sparavigna and Marazzato (2016) consider kets as living in a subset of the vector space that can be of interest also for our study, both for the space of visuals and the space of sounds. It is required that: there is a zero vector $|0⟩$; given $|a⟩$ and $|b⟩$ their sum $|a⟩+|b⟩$ is in the subset; the product by a scalar is also in the subset; for each $|a⟩$ in the subset there is the opposite element $-|a⟩$. In our study, the sum can be the superposition of shapes or musical sequences, and the zero vector the absence of a visual form (i.e. the empty visual space) and of musical sequences (i.e. silence). The opposite $-|a⟩$ can be the time-reverse of an image $|a⟩$ – that is, imagine starting with the finished simple drawing and going back to the empty space, or playing backward in time and going back to the silence. The operation of time reversing can be seen as some deleting operation, or, more interestingly, as just a time direction change: playing a melody from the latest to the first note, or drawing a simple shape from the last point to the first one.

19 Given two vector spaces V, S over fields $K,K^{\prime }$ respectively, $f:V\to S$ is a semilinear map if there exists a homomorphism $\sigma :K\to K^{\prime }$ such that, for all $u,v\in V$ and $\lambda \in K$, we have $f(u,v)=f\left(u\right)+f\left(v\right)$ and $f\left(\lambda u\right)=\sigma \left(\lambda \right)f\left(u\right)$.

20 We could also talk of ‘dimension’ here, however, specifying that this is about the size of each ket in the space, not about the number of elements in a basis.

21 According to Caramello (2016), “the importance of ‘bridges’ between different areas lies in the fact that they make it possible to transfer knowledge and methods between the areas, so that problems formulated in the language of one field can be tackled (and possibly solved) using techniques from a different field, and results in one area can be appropriately transferred to results in another.” More specifically, Caramello refers to topoi as bridges to unify different areas of mathematics; however, the bridge metaphor can be generalized.