On double cosets of groups GL (n) with respect to subgroups of block strictly triangular matrices

ABSTRACT

We parametrize the space of double cosets of the group $\mathrm{G}\mathrm{L}(n,\mathbb{k})$ with respect to two subgroups T₋, T₊ of block strictly triangular matrices. In the appendix, we consider the quasi-regular representation of $\mathrm{G}\mathrm{L}(n,\mathbb{C})$ in L² on ${\mathrm{T}}_{-}\setminus \mathrm{G}\mathrm{L}(n,\mathbb{C})$, observe that it admits an additional group of symmetries, find the joint spectrum, and observe that it is multiplicity free.

Keywords:

• Double cosets
• linear relation
• flags
• unitary representation
• principal series

2010 Mathematics Subject Classifications:

• 15A21
• 43A85

1. The statement

1.1. Double cosets

Let G be a group, K, L subgroups. A double coset of G with respect to K, L is a set of the type K · g · L, i.e. the set of all elements of G that can be represented in the form kgl, where g is fixed, k ranges in K, l ranges in L. We denote the set of all double cosets by $K\mathrm{\setminus }G/L$.

A description of this set is equivalent to a description of orbits of K on the homogeneous space G/L, and to a description of orbits of L on the homogeneous space $K\mathrm{\setminus }G$. If G is finite, then a description of double cosets is equivalent to a description of intertwining operators between quasi-regular representations of G in ℓ²(G/K) and ℓ²(G/L) (see, e.g. [1, Sect. 13.1]). For Lie groups and locally compact groups the picture is more complicate, in any case understanding double cosets seems necessary for understanding analysis on the corresponding homogeneous spaces.

In any case a problem of the description of double cosets arises quite often, but not quite often it admits a tame solution.

1.2. The problem

Let $\mathbb{k}$ be a field, V be a finite dimensional linear space over $\mathbb{k}$. Denote by GL[V] the group of all invertible linear operators in V. We also use the notation $\mathrm{G}\mathrm{L}\left(n\right)=\mathrm{G}\mathrm{L}(n,\mathbb{k})$ for $\mathrm{G}\mathrm{L}\left[{\mathbb{k}}^n\right]$. Split V into a direct sum $V=V_1\oplus \cdots \oplus V_p,\dim V_j={\alpha }_j.$Denote ${\mathrm{T}}_{+}[V_1,\dots ,V_p]={\mathrm{T}}_{+}({\alpha }_1,\dots ,{\alpha }_p)={\mathrm{T}}_{+}\left(\mathit{\alpha }\right)$the group of all block strictly upper triangular matrices of the size $({\alpha }_1+\cdots +{\alpha }_p)\times ({\alpha }_1+\cdots +{\alpha }_p)$, i.e. matrices of the form (1) $\left(\begin{array}{cccc}{1}_{{\alpha }_1}& \ast & \dots & \ast \\ 0& 1_{{\alpha }_2}& \dots & \ast \\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \dots & 1_{{\alpha }_p}\end{array}\right),$(1) where 1_m denotes the unit matrix of size m. By ${\mathrm{P}}_{+}[V_1,\dots ,V_p]={\mathrm{P}}_{+}({\alpha }_1,\dots ,{\alpha }_p)$we denote the group of all block triangular matrices, i.e. we allow arbitrary invertible matrices on places of units. Clearly, T ₊ is normal in P ₊, ${\mathrm{P}}_{+}({\alpha }_1,\dots ,{\alpha }_p)/{\mathrm{T}}_{+}({\alpha }_1,\dots ,{\alpha }_p)\simeq \prod _{i=1}^p\mathrm{G}\mathrm{L}\left({\alpha }_i\right).$By ${\mathrm{T}}_{-}[\dots ]={\mathrm{T}}_{-}(\dots )$ and ${\mathrm{P}}_{-}[\dots ]={\mathrm{P}}_{-}(\dots )$ we denote the corresponding groups of lower triangular matrices.

In this paper we describe double coset spaces (2) ${\mathrm{T}}_{-}({\text{β}}_1,\dots ,{\text{β}}_q)\mathrm{\setminus }\mathrm{G}\mathrm{L}\left(n\right)/{\mathrm{T}}_{+}({\alpha }_1,\dots ,{\alpha }_p).$(2)

1.3. The statement

Recall some definitions. Let X, Y be linear spaces over $\mathbb{k}$. A linear relation (see, e.g. [2, Sect. 2.5]) L:X ↔ Y is a linear subspace in $X\oplus Y$. A graph of a linear map X → Y is a linear relation but not vice versa. For a linear relation we define:

1. the kernel $\ker L$ is the intersection $L\cap X$;

2. the image $\mathrm{i}\mathrm{m}L$ is the projection of L to Y along X;

3. the domain $\mathrm{d}\mathrm{o}\mathrm{m}L$ is the projection of L to X along Y;

4. the indefiniteness $\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{f}L$ is the intersection $L\cap Y$.

If L is a graph of a linear operator A:X → Y, then kernel and image are the usual kernel and image. In this case also $\mathrm{d}\mathrm{o}\mathrm{m}L=X$, $\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{f}L=0$.

Any linear relation L determines a canonical invertible operator $\mathrm{\Theta }\left(L\right):\mathrm{d}\mathrm{o}\mathrm{m}L/\ker L\to \mathrm{i}\mathrm{m}L/\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{f}L.$Moreover, a linear relation L:X ↔ Y is determined by subspaces $\ker L\subset \mathrm{d}\mathrm{o}\mathrm{m}L$ in X, subspaces $\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{f}L\subset \mathrm{i}\mathrm{m}L$ in Y and the operator Θ (L).

Now consider a linear space $V\simeq {\mathbb{K}}^n$, consider its copy W ≃ V and take two decompositions (3) $\begin{array}{rl}V& ={\mathbb{k}}^{{\alpha }_1}\oplus \cdots \oplus {\mathbb{k}}^{{\alpha }_p}=:V_1\oplus \cdots \oplus V_p;\end{array}$(3) (4) $\begin{array}{rl}W\simeq V& ={\mathbb{k}}^{{\text{β}}_1}\oplus \cdots \oplus {\mathbb{k}}^{{\text{β}}_p}=:W_1\oplus \cdots \oplus W_q.\end{array}$(4) Consider the double cosets (2(2) ${\mathrm{T}}_{-}({\text{β}}_1,\dots ,{\text{β}}_q)\mathrm{\setminus }\mathrm{G}\mathrm{L}\left(n\right)/{\mathrm{T}}_{+}({\alpha }_1,\dots ,{\alpha }_p).$(2) ). For each element A ∈ GL[V] we assign a canonical collection of linear relations ${\chi }_{ij}\left(A\right):V_i\leftrightarrow W_j$defined in the following way. We say that (ξ, η) ∈ χ_ij(A) if there exist x₁ ∈ V₁, …, x_i−1 ∈ V_i−1 and y_j+1 ∈ W_j+1, …, y_q ∈ W_q such that $\left(\begin{array}{c}0\\ ⋮\\ 0\\ \eta \\ y_{j+1}\\ ⋮\\ y_p\end{array}\right)=A\left(\begin{array}{c}{x}_1\\ ⋮\\ x_{i-1}\\ \xi \\ 0\\ ⋮\\ 0\end{array}\right).$

Proposition 1.1

(a)

The relations χij depend only on the double coset containing A;

(b)

The relations χij = χij(A) satisfy the conditions: (5) $\ker {\chi }_{ij}=\mathrm{d}\mathrm{o}\mathrm{m}{\chi }_{i(j+1)},\mathrm{i}\mathrm{m}{\chi }_{ij}=\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{f}{\chi }_{(i+1)j},$(5) and (6) $\begin{array}{rl}& \mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{f}{\chi }_{1j}=0,\mathrm{i}\mathrm{m}{\chi }_{pj}=W_j;\end{array}$(6) (7) $\begin{array}{rl}& \ker {\chi }_{iq}=0,\mathrm{d}\mathrm{o}\mathrm{m}{\chi }_{i1}=V_i.\end{array}$(7)

Proof is contained in Sections 2.1–2.3.

We say that a collection of linear relations ξ_ij:V_i ↔ W_j is a bi-hinge ¹ if it satisfies conditions (5(5) $\ker {\chi }_{ij}=\mathrm{d}\mathrm{o}\mathrm{m}{\chi }_{i(j+1)},\mathrm{i}\mathrm{m}{\chi }_{ij}=\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{f}{\chi }_{(i+1)j},$(5) )–(7(7) $\begin{array}{rl}& \ker {\chi }_{iq}=0,\mathrm{d}\mathrm{o}\mathrm{m}{\chi }_{i1}=V_i.\end{array}$(7) ). We denote the set of all bi-hinges by $\mathrm{H}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{e}=\mathrm{H}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{e}[V_1,\dots ,V_p;W_1,\dots ,W_q].$

Theorem 1.2

The map $A\mapsto {\chi }_{ij}\left(A\right)$ determines a one-to-one correspondence between the double coset space (2(2) ${\mathrm{T}}_{-}({\text{β}}_1,\dots ,{\text{β}}_q)\mathrm{\setminus }\mathrm{G}\mathrm{L}\left(n\right)/{\mathrm{T}}_{+}({\alpha }_1,\dots ,{\alpha }_p).$(2) ) and the set of bi-hinges.

Proof is contained in Sections 2.4–2.6.

1.4. Some known cases of classification of double cosets

We discuss shortly such cases related to classical (also, semisimple, reductive) groups. There are three big important series of solvable problems with tame solutions related to symmetric subgroups. Recall that a subgroup H ⊂ G is symmetric if it is the set of fixed points of some involution σ:G → G (i.e. σ(g₁)σ(g₂) = σ(g₂g₁), σ(σ(g)) = g).

• Let G be a real semisimple group, H, L are symmetric subgroups. In particular, this problem includes the Jordan normal form (more generally, description of conjugacy classes in all semisimple Lie groups ² Q), reduction of pairs of nondegenerate quadratic (or symplectic) forms, canonical forms of pairs of subspaces in a Euclidean space, etc. A formal reference to a ‘general case’ is [3].

• We consider $H\mathrm{\setminus }G/P$, where G is a semisimple group, H is a symmetric subgroup and P is a parabolic subgroup (a block triangular subgroup). A formal reference to a ‘general case’ is [4].

• For p-adic groups, the most important case is related to the Iwahori subgroups, see [5].

There is a big family of minor variations of these series (we can slightly enlarge G or slightly reduce subgroups).

Next, there are different ways to assign spectral data to several matrices (this also can be regarded as a classification of double cosets): a spectral curve with a bundle, see [6–8], or a spectral surface with a sheaf, see [9].

On the other hand, for infinite-dimensional groups quite often a double coset space $K\mathrm{\setminus }G/K$ has a structure of a semigroup. There arise questions about spectral data visualizing such multiplications. This also leads to objects of algebraic-geometric nature as spaces of holomorphic maps of the Riemann sphere to Grassmannians (see [2, Sect.X.3]) or rational maps of Grassmannians to Grassmannians (see [10]).

Our case arose as a by-product of a construction of the latter type in [11, proof of Theorem 1.6], it is quite elementary. However, I could not find it in the literature. Apparently, a natural generality here are spaces $H\mathrm{\setminus }G/T$, where G is a classical (or semisimple) group, H is a symmetric subgroup, and T is the maximal unipotent subgroup in a parabolic subgroup.

A possibility to describe double coset space implies a question about harmonic analysis for L² on ${\mathrm{T}}_{+}({\alpha }_1,\dots ,{\alpha }_p)\mathrm{\setminus }\mathrm{G}\mathrm{L}\left(n\right)$. Such an analysis is possible, see the appendix to this paper (but it is not directly related to the description of double cosets).

2. Proof of Theorem 1.2

2.1. Proof of Proposition 1.1.a.

We must show that χ_ij does not depend on the choice of a representative of a double coset.

Reformulate the definition of χ_ij(A) in the following way. Recall that W is a copy of $V\simeq {\mathbb{k}}^n$, the spaces V and W are decomposed as (3(3) $\begin{array}{rl}V& ={\mathbb{k}}^{{\alpha }_1}\oplus \cdots \oplus {\mathbb{k}}^{{\alpha }_p}=:V_1\oplus \cdots \oplus V_p;\end{array}$(3) )–(4(4) $\begin{array}{rl}W\simeq V& ={\mathbb{k}}^{{\text{β}}_1}\oplus \cdots \oplus {\mathbb{k}}^{{\text{β}}_p}=:W_1\oplus \cdots \oplus W_q.\end{array}$(4) ). We consider the intersection $H=H\left(A\right):=A^{-1}(W_j\oplus \cdots \oplus W_q)\cap (V_1\oplus \cdots \oplus V_i)$and send $H\left(A\right)\to Z:=(V_1\oplus \cdots \oplus V_i)⨁(W_j\oplus \cdots \oplus W_q)$by the formula (8) $h\mapsto (h,Ah).$(8) Next, we send Z to the quotient $Z/((V_1\oplus \cdots \oplus V_{i-1})⨁(W_{j+1}\oplus \cdots \oplus W_q))\simeq V_i\oplus W_j.$Clearly, the relation χ_ij(A) is the image of H(A) in this space.

Now, let C ∈ T ₊, consider AC instead of A. Then H(AC) = C⁻¹ H(A). The set of vectors (h, Ah), see (8(8) $h\mapsto (h,Ah).$(8) ) changes to (C⁻¹h, Ah). But C⁻¹ acts trivially in $V_1\oplus \cdots \oplus V_i/V_1\oplus \cdots \oplus V_{i-1}\simeq V_i.$Therefore χ_ij(AC) = χ_ij(A).

2.2. Proof of (5)

For the sake of concreteness, let us verify that $\ker {\chi }_{ij}=\mathrm{d}\mathrm{o}\mathrm{m}{\chi }_{i(j+1)}$. Write the equation $\left(\begin{array}{c}0\\ ⋮\\ 0\\ y_{j+1}\\ ⋮\\ y_q\end{array}\right)=A\left(\begin{array}{c}{x}_1\\ ⋮\\ x_i\\ 0\\ ⋮\\ 0\end{array}\right).$A vector x_i is contained in $\ker {\chi }_{ij}$ if there are x₁, …, x_i−1, y_j+1, …, y_q such that this equation is satisfied. This implies that (x_i, y_j+1) ∈ χ_i(j+1). In particular, $x_i\in \mathrm{d}\mathrm{o}\mathrm{m}{\chi }_{i(j+1)}$.

Conversely, let $x_i\in \mathrm{d}\mathrm{o}\mathrm{m}{\chi }_{i(j+1)}$. Then there are x_i and x₁, …, x_i−1, y_j+2, …, y_q satisfying the equation. This implies that $x_i\in \mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{f}{\chi }_{ij}$.

2.3. Verification of (6)–(7)

To be concrete, let us prove the statements from the first row (6(6) $\begin{array}{rl}& \mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{f}{\chi }_{1j}=0,\mathrm{i}\mathrm{m}{\chi }_{pj}=W_j;\end{array}$(6) ).

Let us show that $\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{f}{\chi }_{1j}=0$. Let $\eta \in \mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{f}{\chi }_{1j}$. Then there y_j+1, …, y_q such that $\left(\begin{array}{c}⋮\\ 0\\ \eta \\ y_{j+1}\\ ⋮\end{array}\right)=A\left(\begin{array}{c}0\\ 0\\ ⋮\end{array}\right).$But the right-hand side must be zero and η = 0.

The statement $\mathrm{i}\mathrm{m}{\chi }_{qj}=V_j$ follows from the subjectivity of A.

2.4. The action of ∏GL[W_j] × ∏GL[V_i] on the double cosets space

Let G be a group, K, L its subgroups, $\tilde{K}$ and $\tilde{L}$ the normalizers of K and L. Then the group $\tilde{K}/K\times \tilde{L}/L$ acts on $K\mathrm{\setminus }G/L$. Indeed, for $\kappa \in \tilde{K}$, $\lambda \in \tilde{L}$ we have ${\kappa }^{-1}\cdot K\cdot g\cdot L\cdot \lambda =K\cdot {\kappa }^{-1}g\lambda \cdot L.$ So this transformation sends double cosets to double cosets. Clearly, orbits of $\tilde{K}/K\times \tilde{L}/L$ on $K\mathrm{\setminus }G/L$ are in one-to-one correspondence with double cosets $\tilde{K}\mathrm{\setminus }G/\tilde{L}$.

In our case the normalizers of ${\mathrm{T}}_{-}(\dots )$ and ${\mathrm{T}}_{+}(\dots )$ are the groups ${\mathrm{P}}_{-}(\dots )$ and ${\mathrm{P}}_{+}(\dots )$, the quotients are ∏GL[W_j] and ∏GL[V_i].

So let us describe the double coset spaces (9) ${\mathrm{P}}_{-}[W_1,\dots ,W_q]\mathrm{\setminus }\mathrm{G}\mathrm{L}\left(n\right)/{\mathrm{P}}_{+}[V_1,\dots ,V_p].$(9) Our subgroups contain the usual subgroups of lower and upper triangle matrices. Applying the usual Gauss reduction we observe that any double coset contains a permutation matrix (a permutation matrix is a matrix consisting of zeros and units and containing only one unit in each column and each row), i.e. an element of the symmetric group S(n). After this reduction we can permute basis elements in each V_i and in each W_j, so the double coset space (9(9) ${\mathrm{P}}_{-}[W_1,\dots ,W_q]\mathrm{\setminus }\mathrm{G}\mathrm{L}\left(n\right)/{\mathrm{P}}_{+}[V_1,\dots ,V_p].$(9) ) is in one-to-one correspondence with $\prod _{j=1}^{q}S\left({\text{β}}_j\right)\mathrm{\setminus }S\left(n\right)/\prod _{i=1}^{p}S\left({\alpha }_i\right).$Our matrix has a natural decomposition into pq blocks, it is important only the number of units in each block. We formulate our observation in the following complicate form.

Lemma 2.1

For any double coset (9(9) ${\mathrm{P}}_{-}[W_1,\dots ,W_q]\mathrm{\setminus }\mathrm{G}\mathrm{L}\left(n\right)/{\mathrm{P}}_{+}[V_1,\dots ,V_p].$(9) ) there are canonical decomposition of each V_j and each W_i into a direct sum of coordinate subspaces (10) $\begin{array}{rl}{V}_i& ={\mathbb{k}}^{{\alpha }_i}={\oplus }_{j=1}^q{\mathbb{k}}^{{\alpha }_i^j}=:{\oplus }_{j=1}^q{V}_i^j;\end{array}$(10) (11) $\begin{array}{rl}{W}_j& ={\mathbb{k}}^{{\text{β}}_j}={\oplus }_{i=1}^p{\mathbb{k}}^{{\text{β}}_j^i}=:{\oplus }_{i=1}^p{W}_j^i,\end{array}$(11) such that ${\alpha }_i^j={\text{β}}_j^i.$A representative of the double coset is the map sending each $V_i^j$ to the corresponding $W_j^i$ coordinate-wise.

Less formally, we get a matrix of the form (12) $J\left[\right\{V_i^j\},\{W_j^i\left\}\right]=\left(\begin{array}{cccccccccccc}\mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0\\ 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0\\ 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}\end{array}\right).$(12) Here p = 4, q = 3. We present a decomposition of a matrix into blocks corresponding to the decompositions ${\oplus }_i{V}_i$ and ${\oplus }_j{W}_j$, and refined blocks corresponding to decompositions ${\oplus }_i\left({\oplus }_j{V}_i^j\right)$ and ${\oplus }_j\left({\oplus }_i{W}_j^i\right)$. Units are put in bold to make them visible among zeros.

Lemma 2.2

For the matrix (12(12) $J\left[\right\{V_i^j\},\{W_j^i\left\}\right]=\left(\begin{array}{cccccccccccc}\mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0\\ 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0\\ 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}\end{array}\right).$(12) ) the corresponding linear relations χ_ij are the following: (13) $\begin{array}{rl}& \ker {\chi }_{ij}=V_i^{j+1}\oplus \cdots \oplus V_i^p,\mathrm{d}\mathrm{o}\mathrm{m}{\chi }_{ij}=V_i^j\oplus \cdots \oplus V_i^p;\end{array}$(13) (14) $\begin{array}{rl}& \mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{f}{\chi }_{ij}=W_j^1\oplus \cdots \oplus W_j^{i-1},\mathrm{i}\mathrm{m}{\chi }_{ij}=W_j^1\oplus \cdots \oplus W_j^i.\end{array}$(14) The operator $\mathrm{\Theta }\left({\chi }_{ij}\right):\mathrm{d}\mathrm{o}\mathrm{m}{\chi }_{ij}/\ker {\chi }_{ij}\to \mathrm{i}\mathrm{m}{\chi }_{ij}/\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{f}{\chi }_{ij}$is the identical map $V_i^j\to W_j^i$.

We say that such a bi-hinge is standard and denote it by $\mathrm{h}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{e}\left[\right\{V_i^j\},\{W_j^i\left\}\right]$

Proof.

The statement is semi-obvious. First, let us write χ₃₂ for the matrix (12(12) $J\left[\right\{V_i^j\},\{W_j^i\left\}\right]=\left(\begin{array}{cccccccccccc}\mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0\\ 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0\\ 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}\end{array}\right).$(12) ) (the general case differs from considerations below only by longer notation, see below). We apply this matrix to a vector (15) $\left(\begin{array}{cccccccccccc}{x}_1^1& x_1^2& x_1^3& x_2^1& x_2^2& x_2^3& {\xi }^1& {\xi }^2& {\xi }^3& 0& 0& 0\end{array}\right)$(15) and get (16) $\left(\begin{array}{cccccccccccc}{x}_1^1& x_2^1& {\xi }^1& 0& x_1^2& x_2^2& {\xi }^2& 0& x_1^3& x_2^3& {\xi }^3& 0\end{array}\right).$(16) On the other hand, this must be equal to (17) $\left(\begin{array}{cccccccccccc}0& 0& 0& 0& {\eta }^1& {\eta }^2& {\eta }^3& {\eta }^4& y_3^1& y_3^2& y_3^3& y_3^4\end{array}\right).$(17) Recall that the linear relation χ₃₂ consists of vectors $(\left(\begin{array}{ccc}{\xi }^1& {\xi }^2& {\xi }^3\end{array}\right),\left(\begin{array}{cccc}{\eta }^1& {\eta }^2& {\eta }^3& {\eta }^4\end{array}\right))\in V_3\oplus W_2,$for which there are $x_{\dots }^{\dots }$, $y_{\dots }^{\dots }$ such that (16(16) $\left(\begin{array}{cccccccccccc}{x}_1^1& x_2^1& {\xi }^1& 0& x_1^2& x_2^2& {\xi }^2& 0& x_1^3& x_2^3& {\xi }^3& 0\end{array}\right).$(16) ) equals to (17(17) $\left(\begin{array}{cccccccccccc}0& 0& 0& 0& {\eta }^1& {\eta }^2& {\eta }^3& {\eta }^4& y_3^1& y_3^2& y_3^3& y_3^4\end{array}\right).$(17) ). We get (18) $\begin{array}{rl}& {\xi }_1=0,{\xi }_2={\eta }_3,{\xi }_3=y_3^3,\end{array}$(18) (19) $\begin{array}{rl}& x_1^2={\eta }_1,x_2^2={\eta }_2,{\eta }_4=0.\end{array}$(19) The remaining equations contain no information: (20) $\begin{array}{rl}& x_1=0,x_2^1=0,0=0;\end{array}$(20) (21) $\begin{array}{rl}& x_1^3=y_3^1,x_2^3=y_3^2,0=y_3^4.\end{array}$(21) The sets of variables in (18(18) $\begin{array}{rl}& {\xi }_1=0,{\xi }_2={\eta }_3,{\xi }_3=y_3^3,\end{array}$(18) )–(19(19) $\begin{array}{rl}& x_1^2={\eta }_1,x_2^2={\eta }_2,{\eta }_4=0.\end{array}$(19) ) and (20(20) $\begin{array}{rl}& x_1=0,x_2^1=0,0=0;\end{array}$(20) )–(21(21) $\begin{array}{rl}& x_1^3=y_3^1,x_2^3=y_3^2,0=y_3^4.\end{array}$(21) ) do not intersect (since we start with a permutation matrix), On the other hand, the second system (20(20) $\begin{array}{rl}& x_1=0,x_2^1=0,0=0;\end{array}$(20) )–(21(21) $\begin{array}{rl}& x_1^3=y_3^1,x_2^3=y_3^2,0=y_3^4.\end{array}$(21) ) has a solution, since each variable is present in it only one time (again, this is a priori clear, because we start with a permutation matrix).

Finally, we get the linear relation χ₂₃ consisting of vectors $(\left(\begin{array}{ccc}0& {\xi }^2& {\xi }^3\end{array}\right),\left(\begin{array}{cccc}{\eta }^1& {\eta }^2& {\eta }^3& 0\end{array}\right))\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}{\xi }_2={\eta }_3.$For a general case we must write (22) $\begin{array}{rl}& (\dots |\dots ,x_{i-2}^{j-1},x_{i-2}^j,x_{i-2}^{j+1},\dots |\dots ,x_{i-1}^{j-1},x_{i-1}^j,x_{i-1}^{j+1},\dots |\dots ,{\xi }^{j-1},{\xi }^j,{\xi }^{j+1},\dots |\\ & \dots ,0,0,0,\dots |\dots ,0,0,0,\dots |\dots )\end{array}$(22) instead of (15(15) $\left(\begin{array}{cccccccccccc}{x}_1^1& x_1^2& x_1^3& x_2^1& x_2^2& x_2^3& {\xi }^1& {\xi }^2& {\xi }^3& 0& 0& 0\end{array}\right)$(15) ) and $\begin{array}{rl}& (\dots |\dots ,0,0,0,\dots |\dots ,0,0,0,\dots |\dots ,{\eta }_{i-1},{\eta }_i,{\eta }_{i+1},\dots |\\ & \dots ,y_{j+1}^{i-1},y_{j+1}^i,y_{j+1}^{i+1},\dots |\dots ,y_{j+2}^{i-1},y_{j+2}^i,y_{j+2}^{i+1},\dots |\dots )\end{array}$instead of (17(17) $\left(\begin{array}{cccccccccccc}0& 0& 0& 0& {\eta }^1& {\eta }^2& {\eta }^3& {\eta }^4& y_3^1& y_3^2& y_3^3& y_3^4\end{array}\right).$(17) ). We apply the permutation matrix to (22(22) $\begin{array}{rl}& (\dots |\dots ,x_{i-2}^{j-1},x_{i-2}^j,x_{i-2}^{j+1},\dots |\dots ,x_{i-1}^{j-1},x_{i-1}^j,x_{i-1}^{j+1},\dots |\dots ,{\xi }^{j-1},{\xi }^j,{\xi }^{j+1},\dots |\\ & \dots ,0,0,0,\dots |\dots ,0,0,0,\dots |\dots )\end{array}$(22) ), repeat the same steps and come to the linear relation consisting of all vectors of the form $((\dots ,0,0,{\xi }^i,{\xi }^{i+1},\dots ),(\dots ,{\eta }_{j-1},{\eta }_j,0,0,\dots )).$

2.5. The action of ∏GL[V_i] × ∏GL[W_j] on the set of bi-hinges

Clearly, the group ∏GL[V_i] × ∏GL[W_j] acts on the space $V_1\times \cdots \times V_p\times W_1\times \cdots \times W_q,$therefore it acts on the set of bi-hinges.

Lemma 2.3

(a)

Any orbit of the group ∏GL[Vi] × ∏GL[Wj] on the set $\mathrm{H}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{e}[V_1,\dots ,V_p;W_1,\dots ,W_q]$contains a unique standard bi-hinge $\mathrm{h}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{e}\left[\right\{V_i^j\};\{W_j^i\left\}\right]$ (as in Lemma 2.2).

(b)

The stabilizer $G\left[\right\{V_i^j\};\{W_j^i\left\}\right]\subset \prod _{i=1}^p\mathrm{G}\mathrm{L}\left[V_i\right]\times \prod _{j=1}^q\mathrm{G}\mathrm{L}\left[W_j\right]$ of a standard bi-hinge $\mathrm{h}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{e}\left[\right\{V_i^j\};\{W_j^i\left\}\right]$ is the semidirect product of the reductive group (23) $\prod _{i=1}^p\prod _{j=1}^q\mathrm{G}\mathrm{L}\left[V_i^j\right]\simeq \prod _{j=1}^q\prod _{i=1}^p\mathrm{G}\mathrm{L}\left[W_j^i\right]$(23) and the unipotent group (24) $\prod _{i=1}^p{\mathrm{T}}_{-}[V_i^1,\dots ,V_i^q]\times \prod _{j=1}^q{\mathrm{T}}_{+}[W_j^1,\dots ,W_j^p].$(24)

Proof.

(a) In a fixed V_i we have a flag $V_i=\mathrm{d}\mathrm{o}\mathrm{m}{\chi }_{i1}\supset \ker {\chi }_{i1}=\mathrm{d}\mathrm{o}\mathrm{m}{\chi }_{i2}\supset \ker {\chi }_{i2}=\mathrm{d}\mathrm{o}\mathrm{m}{\chi }_{i3}\supset \cdots \supset \ker {\chi }_{iq}=0.$We choose an element of GL[V_i] sending this flag to the flag of decreasing coordinate subspaces of the form (25) $\left(\begin{array}{cccccc}0& \dots & 0& \ast & \dots & \ast \end{array}\right),$(25) this flag is canonically determined by dimensions of $\ker {\chi }_{ij}$.

Similarly, we fix W_j, consider the flag $0=\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{f}{\chi }_{1j}\subset \mathrm{i}\mathrm{m}{\chi }_{1j}=\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{f}{\chi }_{2j}\subset \mathrm{i}\mathrm{m}{\chi }_{2j}=\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{f}{\chi }_{3j}\subset \cdots \subset \mathrm{i}\mathrm{m}{\chi }_{pj}=W_j,$and choose an element of GL[W_j] sending this flag to a flag consisting of an increasing sequence of subspaces of the form $\left(\begin{array}{cccccc}\ast & \dots & \ast & 0& \dots & 0\end{array}\right)$.

So for any element of our bi-hinge we fixed positions of its domain, kernel, image and indefinity. After this fixing, it remains a possibility to choose coordinates in subquotient of the flag (25(25) $\left(\begin{array}{cccccc}0& \dots & 0& \ast & \dots & \ast \end{array}\right),$(25) ).

We get the desired canonical form. The statement (b) also becomes obvious, since the stabilizer must regard the flags in each V_i and W_j and the maps Θ ( · ).

2.6. Coincidence of stabilizers

Thus, we have a map (26) ${\mathrm{T}}_{-}[V_1,\dots ,V_p]\mathrm{\setminus }\mathrm{G}\mathrm{L}\left(n\right)/{\mathrm{T}}_{+}[W_1,\dots ,W_q]\to \mathrm{H}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{e}[V_1,\dots ,V_p;W_1,\dots ,W_q],$(26) which is ∏GL[V_i] × ∏GL[W_j]-equivariant and establishes a bijection of the sets of orbits. For a proof of Theorem 1.2 it is sufficient to show that this map establishes a bijection for each pair of corresponding orbits. So we must check the coincidence of stabilizers of canonical representatives of orbits. So it suffices to prove the following lemma.

Lemma 2.4

Consider a representative of a double coset in the canonical permutation form $J\left[\right\{V_j^j\},\{W_j^i\left\}\right]$, see Lemma 2.2, and the corresponding standard bi-hinge $\mathrm{h}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{e}\left[\right\{V_j^j\},\{W_j^i\left\}\right]$. Then any element of ∏GL[V_i] × ∏GL[W_j] stabilizing the bi-hinge stabilizes J.

Remark 2.1

Notice that the inverse inclusion of stabilizers follows from the ∏GL[V_i] × ∏GL[W_j]-equivariance of the map (26(26) ${\mathrm{T}}_{-}[V_1,\dots ,V_p]\mathrm{\setminus }\mathrm{G}\mathrm{L}\left(n\right)/{\mathrm{T}}_{+}[W_1,\dots ,W_q]\to \mathrm{H}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{e}[V_1,\dots ,V_p;W_1,\dots ,W_q],$(26) ).

Proof.

The stabilizer of a standard bi-hinge is described in Lemma 2.3. It is a product of subgroups (23(23) $\prod _{i=1}^p\prod _{j=1}^q\mathrm{G}\mathrm{L}\left[V_i^j\right]\simeq \prod _{j=1}^q\prod _{i=1}^p\mathrm{G}\mathrm{L}\left[W_j^i\right]$(23) ) and (24(24) $\prod _{i=1}^p{\mathrm{T}}_{-}[V_i^1,\dots ,V_i^q]\times \prod _{j=1}^q{\mathrm{T}}_{+}[W_j^1,\dots ,W_j^p].$(24) ). For the reductive factor (23(23) $\prod _{i=1}^p\prod _{j=1}^q\mathrm{G}\mathrm{L}\left[V_i^j\right]\simeq \prod _{j=1}^q\prod _{i=1}^p\mathrm{G}\mathrm{L}\left[W_j^i\right]$(23) ) the statement is clear. The unipotent factor itself is a product, and it is sufficient to prove the statement for any factor in (24(24) $\prod _{i=1}^p{\mathrm{T}}_{-}[V_i^1,\dots ,V_i^q]\times \prod _{j=1}^q{\mathrm{T}}_{+}[W_j^1,\dots ,W_j^p].$(24) ), say ${\mathrm{T}}_{-}[V_i^1,\dots ,V_j^q]$.

We first study the matrix J given by (12(12) $J\left[\right\{V_i^j\},\{W_j^i\left\}\right]=\left(\begin{array}{cccccccccccc}\mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0\\ 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0\\ 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}\end{array}\right).$(12) ) and a matrix (27) $H:=\left(\begin{array}{ccc}1& 0& 0\\ x& 1& 0\\ y& z& 1\end{array}\right)\in {\mathrm{T}}_{-}[V_2^1,V_2^2,V_2^3]\subset \mathrm{G}\mathrm{L}\left[V_2\right].$(27) Multiplying (12(12) $J\left[\right\{V_i^j\},\{W_j^i\left\}\right]=\left(\begin{array}{cccccccccccc}\mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0\\ 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0\\ 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}\end{array}\right).$(12) ) by this element we get the matrix (28) $\left(\begin{array}{cccccccccccc}\mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& \boxed{\mathbf{1}}& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0\\ 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& \mathbf{x}& \boxed{\mathbf{1}}& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0\\ 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& \mathbf{y}& \mathbf{z}& \mathbf{1}& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}\end{array}\right).$(28) We put nonzero symbols in bold to make them visible on the field of zeros. Clearly, this matrix can be reduced to the initial form (12(12) $J\left[\right\{V_i^j\},\{W_j^i\left\}\right]=\left(\begin{array}{cccccccccccc}\mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0\\ 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0\\ 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}\end{array}\right).$(12) ) by a left multiplication by an element of $T_{-}[W_1,\dots ,W_q]$. The boxed units allow to delete x, y, z.

More precisely, consider the second block column Ξ of our big matrix (28(28) $\left(\begin{array}{cccccccccccc}\mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& \boxed{\mathbf{1}}& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0\\ 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& \mathbf{x}& \boxed{\mathbf{1}}& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0\\ 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& \mathbf{y}& \mathbf{z}& \mathbf{1}& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}\end{array}\right).$(28) ). It contains several zero rows. Removing such rows from Ξ we get precisely the matrix (27(27) $H:=\left(\begin{array}{ccc}1& 0& 0\\ x& 1& 0\\ y& z& 1\end{array}\right)\in {\mathrm{T}}_{-}[V_2^1,V_2^2,V_2^3]\subset \mathrm{G}\mathrm{L}\left[V_2\right].$(27) ). On the other hand, consider a row of (28(28) $\left(\begin{array}{cccccccccccc}\mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& \boxed{\mathbf{1}}& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0\\ 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& \mathbf{x}& \boxed{\mathbf{1}}& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0\\ 0& 0& \mathbf{1}& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& \mathbf{y}& \mathbf{z}& \mathbf{1}& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& \mathbf{1}\end{array}\right).$(28) ) whose part contained in Ξ is non-zero. Then the remaining part of the row is zero. Elements x, y, z of Ξ are located precisely under units, this allows to ‘kill’ them applying a multiplication from the left by a lower triangular matrix.

For a general case, we multiply the corresponding permutation matrix by an element of ${\mathrm{T}}_{-}[V_i^1,\dots ,V_i^q]$ and get the same structure in the ith block column Ξ of the matrix.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by Austrian Science Fund [P31591].

Notes

1 Cf. similar objects in [13].

2 Namely, we set G = Q × Q, H = L = diag Q is the diagonal subgroup.

Appendix 1

The spaces L² on $T_{+}\left(\mathit{\alpha }\right)\mathrm{\setminus }\mathrm{G}\mathrm{L}(n,\mathbb{C})$

A.1. The principal series of unitary representations of the groups $\mathrm{G}\mathrm{L}(n,\mathbb{C})$

Denote by Λ the set of pairs λ|λ′ of complex numbers of the type $\lambda |{\lambda }^{\prime }=\frac{k+is}{2}|\frac{-k+is}{2}\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}k\in \mathbb{Z},s\in \mathbb{R}.$For λ|λ′ ∈ Λ we have a well-defined ‘generalized power’ of any nonzero $z\in \mathbb{C}$: $z^{\lambda |{\lambda }^{\prime }}=z^{\lambda }{\overline{z}}^{{\lambda }^{\prime }}:=(z/|z|{)}^k\cdot |z{|}^{is}.$Consider the subgroup ${\mathrm{B}}_{+}\left(n\right):={\mathrm{P}}_{+}(1,\dots ,1)\subset \mathrm{G}\mathrm{L}(n,\mathbb{C})$ consisting of all upper triangular matrices, (A1) $C:=\left(\begin{array}{cccc}{c}_{11}& c_{12}& \dots & c_{1n}\\ 0& c_{22}& \dots & c_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \dots & c_{nn}\end{array}\right).$(A1) A signature λ is a collection of the form $\mathit{\lambda }:=({\lambda }_1|{\lambda }_1^{\prime },\dots ,{\lambda }_n|{\lambda }_n^{\prime })\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}{\lambda }_j|{\lambda }_j^{\prime }\in \mathrm{\Lambda }.$For such λ denote by χ_λ(A) the character of B₊(n) defined by (A2) ${\chi }_{\mathit{\lambda }}\left(A\right):=\prod c_{jj}^{{\lambda }_j|{\lambda }_j^{\prime }}.$(A2) By ρ_λ we denote the representation of $\mathrm{G}\mathrm{L}(n,\mathbb{C})$ unitary induced in the sense of Mackey (see, e.g. [1, Subsect. 13.2], [12, Sect. 16, Sect.19.1]) from a one-dimensional representation χ_λ of the subgroup B₊(n). Such unitary representations are called representations of the nondegenerate principal series, see, e.g. [12, Sect. 19.3]. Representations ρ_λ and ρ_μ are equivalent if and only if a collection $\{{\mu }_k|{\mu }_k^{\prime }{\}}_{k=1,\dots ,n}$ can be obtained from a collection $\{{\lambda }_j|{\lambda }_j^{\prime }{\}}_{j=1,\dots ,n}$ by a permutation. Denote by Σ_n the set of all signatures defined up to a permutation.

Next, consider the action of $\mathrm{G}\mathrm{L}(n,\mathbb{C})\times \mathrm{G}\mathrm{L}(n,\mathbb{C})$ on $\mathrm{G}\mathrm{L}(n,\mathbb{C})$ by left and right multiplications, $g\mapsto h_1^{-1}g{h}_2$. This determines the left–right regular representation of $\mathrm{G}\mathrm{L}(n,\mathbb{C})\times \mathrm{G}\mathrm{L}(n,\mathbb{C})$ in $L^2\left(\mathrm{G}\mathrm{L}\right(n,\mathbb{C}\left)\right)$. According to Gelfand and Naimark, see, e.g. [12, Sect. 14.4.A], this representation decomposes as a direct integral (A3) ${\int }_{{\mathrm{\Sigma }}_n}{\rho }_{\mathit{\lambda }}\otimes {\rho }_{-\mathit{\lambda }}d\mathit{\lambda }$(A3) of representations of $\mathrm{G}\mathrm{L}(n,\mathbb{C})\times \mathrm{G}\mathrm{L}(n,\mathbb{C})$.

A.2. The space $L^2\left({\mathrm{T}}_{+}\right({\alpha }_1,\dots ,{\alpha }_p\left)\mathrm{\setminus }\mathrm{G}\mathrm{L}\right(n,\mathbb{C}\left)\right)$.

Let us decompose the quasiregular representation of $\mathrm{G}\mathrm{L}(n,\mathbb{C})$ in $L^2\left({\mathrm{T}}_{+}\right({\alpha }_1,\dots ,{\alpha }_p\left)\mathrm{\setminus }\mathrm{G}\mathrm{L}\right(n,\mathbb{C}\left)\right)$. This problem has an additional symmetry. Indeed, the subgroup ${\mathrm{P}}_{+}({\alpha }_1,\dots ,{\alpha }_p)$ normalizes ${\mathrm{T}}_{+}({\alpha }_1,\dots ,{\alpha }_p)$. Therefore, the quotient group ${\mathrm{P}}_{+}({\alpha }_1,\dots ,{\alpha }_p)/{\mathrm{T}}_{+}({\alpha }_1,\dots ,{\alpha }_p)\simeq \prod _{i=1}^p\mathrm{G}\mathrm{L}({\alpha }_i,\mathbb{C})$acts on ${\mathrm{T}}_{+}({\alpha }_1,\dots ,{\alpha }_p)\mathrm{\setminus }\mathrm{G}\mathrm{L}(n,\mathbb{C})$ by left multiplications. So we get a unitary representation of the group (A4) $G:=\prod _{i=1}^p\mathrm{G}\mathrm{L}({\alpha }_i,\mathbb{C})\times \mathrm{G}\mathrm{L}(n,\mathbb{C})$(A4) in our L².

For elements ${\mathit{\lambda }}^j\in {\mathrm{\Sigma }}_{{\alpha }_j}$ denote by ${\mathit{\lambda }}^1\bigsqcup \cdots \bigsqcup {\mathit{\lambda }}^p\in {\mathrm{\Sigma }}_n$the row obtained by concatenation of rows λ^j.

Theorem A.1

The decomposition of $L^2\left({\mathrm{T}}_{+}\right({\alpha }_1,\dots ,{\alpha }_p\left)\mathrm{\setminus }\mathrm{G}\mathrm{L}\right(n,\mathbb{C}\left)\right)$ under the action of the group $G=\prod _{i=1}^p\mathrm{G}\mathrm{L}({\alpha }_i,\mathbb{C})\times \mathrm{G}\mathrm{L}(n,\mathbb{C})$ is multiplicity-free and has the form ${\int }_{{\mathit{\lambda }}^1\in {\mathrm{\Sigma }}_{{\alpha }_1}}\dots {\int }_{{\mathit{\lambda }}^p\in {\mathrm{\Sigma }}_{{\alpha }_p}}({\rho }_{-{\mathit{\lambda }}^1}\otimes \cdots \otimes {\rho }_{-{\mathit{\lambda }}^p})\otimes {\rho }_{{\mathit{\lambda }}^1\bigsqcup \cdots \bigsqcup {\mathit{\lambda }}^p}\mathrm{d}{\mathit{\lambda }}^p\dots \mathrm{d}{\mathit{\lambda }}^1.$

Proof.

We have a space homogeneous with respect to group G. The stabilizer G₀ of the initial point consists of tuples $\left\{b_1\in \mathrm{G}\mathrm{L}\left({\alpha }_1\right),\dots ,b_p\in \mathrm{G}\mathrm{L}\left({\alpha }_p\right),\left(\begin{array}{cccc}{b}_1& a_{12}& \dots & a_{1n}\\ 0& b_2& \dots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \dots & b_p\end{array}\right)\in {\mathrm{P}}_{+}({\alpha }_1,\dots ,{\alpha }_p)\right\}.$By definition our representation is induced from the trivial representation of the stabilizer G₀. Consider a larger group $G_0^{\ast }\supset G$ defined by $G_0^{\ast }=\prod \mathrm{G}\mathrm{L}({\alpha }_i,\mathbb{C})\times {\mathrm{P}}_{+}({\alpha }_1,\dots ,{\alpha }_p).$We apply induction in stages (see, e.g. [1, Subsect.13.1], [12, Sect. 16.2]), first from G₀ to $G_0^{\ast }$, second from $G_0^{\ast }$ to G.

On the first step we have the same normal subgroup ${\mathrm{T}}_{+}({\alpha }_1,\dots ,{\alpha }_p)$ in both G₀ and $G_0^{\ast }$. Since the initial representation of G₀ is trivial, the induced representation is trivial on ${\mathrm{T}}_{+}({\alpha }_1,\dots ,{\alpha }_p)$. In fact we have the induction from $G_0/{\mathrm{T}}_{+}({\alpha }_1,\dots ,{\alpha }_p)\mathrm{t}\mathrm{o}{G}_0^{\ast }/{\mathrm{T}}_{+}({\alpha }_1,\dots ,{\alpha }_p).$The second group is the double $\prod \mathrm{G}\mathrm{L}({\alpha }_j,\mathbb{C})\times \prod \mathrm{G}\mathrm{L}({\alpha }_j,\mathbb{C})$, the first group is $\prod \mathrm{G}\mathrm{L}({\alpha }_j,\mathbb{C})$ embedded to the double as the diagonal. So the induced representation is the left–right representation of the double. Clearly, it is equivalent to the tensor product of the left–right regular representations of the factors $\mathrm{G}\mathrm{L}({\alpha }_j,\mathbb{C})\times \mathrm{G}\mathrm{L}({\alpha }_j,\mathbb{C})$, $L^2(\prod \mathrm{G}\mathrm{L}({\alpha }_j,\mathbb{C}))=\underset{j}{⨂}{L}^2(\mathrm{G}\mathrm{L}({\alpha }_j,\mathbb{C})).$We decompose spaces $L^2\left(\mathrm{G}\mathrm{L}\right({\alpha }_j,\mathbb{C}\left)\right)$ according (A3(A3) ${\int }_{{\mathrm{\Sigma }}_n}{\rho }_{\mathit{\lambda }}\otimes {\rho }_{-\mathit{\lambda }}d\mathit{\lambda }$(A3) ).

In this way, we come to a direct integral of irreducible representations of $G_0^{\ast }$ having the form (A5) $({\rho }_{-{\mathit{\lambda }}^1}\left(b_1\right)\otimes \cdots \otimes {\rho }_{-{\mathit{\lambda }}^p}\left(b_p\right))\otimes {\sigma }_{{\mathit{\lambda }}^1,\dots ,{\mathit{\lambda }}^p},$(A5) where ${\sigma }_{{\mathit{\lambda }}^1,\dots ,{\mathit{\lambda }}^p}$ is the representation of ${\mathrm{P}}_{+}({\alpha }_1,\dots ,{\alpha }_p)$, which is trivial on ${\mathrm{T}}_{+}({\alpha }_1,\dots ,{\alpha }_p)$ and is defined by ${\sigma }_{{\mathit{\lambda }}^1,\dots ,{\mathit{\lambda }}^p}\left(\begin{array}{cccc}{a}_{11}& a_{12}& \dots & a_{1n}\\ 0& a_{22}& \dots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \dots & a_{pp}\end{array}\right)={\rho }_{{\mathit{\lambda }}^1}\left(a_{11}\right)\otimes \cdots \otimes {\rho }_{{\mathit{\lambda }}^p}\left(a_{pp}\right).$Notice that ${\sigma }_{{\mathit{\lambda }}^1,\dots ,{\mathit{\lambda }}^p}$ itself is an induced representation. It is induced from a one-dimensional representation of B₊(n), see (A.1), namely from the character given by the formula (A6) $\begin{array}{rl}\zeta \left(C\right)& =\prod _{j=1}^{{\alpha }_1}{c}_{jj}^{{\lambda }_j^1|({\lambda }_j^1{)}^{\prime }}\cdot \prod _{j=1}^{{\alpha }_2}{c}_{({\alpha }_1+j)({\alpha }_1+j)}^{{\lambda }_j^2|({\lambda }_j^2{)}^{\prime }}\cdot \prod _{j=1}^{{\alpha }_3}{c}_{({\alpha }_1+{\alpha }_2+j)({\alpha }_1+{\alpha }_2+j)}^{{\lambda }_j^3|({\lambda }_j^3{)}^{\prime }}\cdot \dots \\ & ={\chi }_{{\mathit{\lambda }}^1\bigsqcup \cdots \bigsqcup {\mathit{\lambda }}^p}\left(C\right),\end{array}$(A6) where $\chi \dots \left(C\right)$ is determined by (A1(A1) $C:=\left(\begin{array}{cccc}{c}_{11}& c_{12}& \dots & c_{1n}\\ 0& c_{22}& \dots & c_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \dots & c_{nn}\end{array}\right).$(A1) ).

Next, we consider the representation of G induced from an irreducible representation (A5(A5) $({\rho }_{-{\mathit{\lambda }}^1}\left(b_1\right)\otimes \cdots \otimes {\rho }_{-{\mathit{\lambda }}^p}\left(b_p\right))\otimes {\sigma }_{{\mathit{\lambda }}^1,\dots ,{\mathit{\lambda }}^p},$(A5) ) of $G_0^{\ast }$. Since the factor ∏GL(α_j) is present in both groups $G_0^{\ast }$ and G, actually we have the induction from ${\mathrm{P}}_{+}({\alpha }_1,\dots ,{\alpha }_p)$ to GL(n) (formally, we can refer to [12, Sect.16.2.D, Theorem 3]). But the representation ${\sigma }_{{\mathit{\lambda }}^1,\dots ,{\mathit{\lambda }}^p}$ itself is induced from the character (A6(A6) $\begin{array}{rl}\zeta \left(C\right)& =\prod _{j=1}^{{\alpha }_1}{c}_{jj}^{{\lambda }_j^1|({\lambda }_j^1{)}^{\prime }}\cdot \prod _{j=1}^{{\alpha }_2}{c}_{({\alpha }_1+j)({\alpha }_1+j)}^{{\lambda }_j^2|({\lambda }_j^2{)}^{\prime }}\cdot \prod _{j=1}^{{\alpha }_3}{c}_{({\alpha }_1+{\alpha }_2+j)({\alpha }_1+{\alpha }_2+j)}^{{\lambda }_j^3|({\lambda }_j^3{)}^{\prime }}\cdot \dots \\ & ={\chi }_{{\mathit{\lambda }}^1\bigsqcup \cdots \bigsqcup {\mathit{\lambda }}^p}\left(C\right),\end{array}$(A6) ) of B₊(n). Applying induction in stages we get that our representation of GL(n) is induced from the character (A6(A6) $\begin{array}{rl}\zeta \left(C\right)& =\prod _{j=1}^{{\alpha }_1}{c}_{jj}^{{\lambda }_j^1|({\lambda }_j^1{)}^{\prime }}\cdot \prod _{j=1}^{{\alpha }_2}{c}_{({\alpha }_1+j)({\alpha }_1+j)}^{{\lambda }_j^2|({\lambda }_j^2{)}^{\prime }}\cdot \prod _{j=1}^{{\alpha }_3}{c}_{({\alpha }_1+{\alpha }_2+j)({\alpha }_1+{\alpha }_2+j)}^{{\lambda }_j^3|({\lambda }_j^3{)}^{\prime }}\cdot \dots \\ & ={\chi }_{{\mathit{\lambda }}^1\bigsqcup \cdots \bigsqcup {\mathit{\lambda }}^p}\left(C\right),\end{array}$(A6) ) of B₊(n). But this is a representation ${\rho }_{{\mathit{\lambda }}^1\bigsqcup \cdots \bigsqcup {\mathit{\lambda }}^p}$ of the principal series.