CONJUGACY LANGUAGES IN GROUPS

LAURA CIOBANU, SUSAN HERMILLER, DEREK HOLT, AND SARAH REES

Abstract. We study the regularity of several languages derived from conjugacy classes in a finitely generated group G for a variety of examples including word hyperbolic, virtually abelian, Artin, and Garside groups. We also determine the rationality of the growth series of the shortlex conjugacy language in virtually cyclic groups, proving one direction of a conjecture of Rivin.

2010 Mathematics Subject Classification: 20F65, 20E45; 20F67, 20F36.

Key words: Conjugacy growth, regular languages, word hyperbolic groups, Artin groups, Garside groups.

1. Introduction

Many classes of finitely presented groups have been studied via the formal language theoretic properties of their sets of geodesics or shortlex least representatives of the group elements. In this paper we study the regularity of three languages derived from the conjugacy classes, rather than elements, of some of these groups.

A regular language is recognized by a finite state automaton, and one particularly useful application is the existence of an algorithm that uses this automaton to calculate a rational function equal to the growth series (that is, the generating function associated to the growth function) of the language.

Before proceeding further, we need to recall some notation. We use standard notation from formal language theory and refer to [19] for details. Where X is a finite set, we denote by X^{∗} the set of all words over X, and call a subset of X^{∗} a language. We write ε for the empty word, and denote by X^{+} the set of all nonempty words over X (so X^{∗} = X^{+} ∪ {ε}). For each word w ∈ X^{∗}, let l(w) = l_{X}(w) denote its length over X. For subsets A, B of X^{∗}, we define AB to be the set of concatenations wu with w ∈ A, u ∈ B. Similarly we define A^{n} to be the set of concatenations of n words from A, A^{∗} = ∪_{n=0}^{∞}A^{n}, A^{+} = ∪_{n=1}^{∞}A^{n}. A language is regular if it can be built out of finite subsets of X using the operations of union, concatenation, ∗ (and complementation); such an expression for a language is called a regular expression.

All groups we consider in this paper are finitely generated, and all generating sets finite and inverse-closed. Let G = ⟨X⟩ be a group. Let π : X^{∗} → G be the natural projection onto G, and let = denote equality between words and =_{G} equality between group elements (so w =_{G} v means π(w) = π(v)). For g ∈ G, define the length of g, denoted |g| (= |g|_{X}), to be the length of a shortest representative word for g over X. Define a geodesic to be a word w ∈ X^{∗} with l(w) = |π(w)|.

Let ∼ denote the equivalence relation on G given by conjugacy, and G/∼ its set of equivalence classes. Let [g]_{c} denote the conjugacy class of g ∈ G. Define the length up to conjugacy of an element g of G, denoted |g|_{c}, by

|g|_{c} := min{|h| | h ∈ [g]_{c}}.

We say that g has minimal length up to conjugacy if |g| = |g|_{c}.

We call a word w ∈ X^{∗} satisfying l(w) = |π(w)|_{c} a geodesic with respect to conjugacy, or a conjugacy geodesic word. Note that if a word w is a conjugacy geodesic, then so is every cyclic permutation of w. By contrast, a word w ∈ X^{∗} satisfying the property that every cyclic permutation is a geodesic is not necessarily a conjugacy geodesic. Call a word whose cyclic permutations are all geodesics a geodesic with respect to cyclic permutation, or a cyclic geodesic word.

We consider six languages associated to the pair (G, X), the first three being:

Geo = Geo(G, X) := {w ∈ X^{∗} | l(w) = |π(w)|},
ConjGeo = ConjGeo(G, X) := {w ∈ X^{∗} | l(w) = |π(w)|_{c}},
CycGeo = CycGeo(G, X) := {w ∈ X^{∗} | w is a cyclic geodesic},

which we call the geodesic language, conjugacy geodesic language, and cyclic geodesic language, respectively, of G with respect to X. The conjugacy geodesic language was introduced by the first two authors in [7], where it is shown that the property of having both Geo and ConjGeo regular is preserved by taking graph products.

Although Geo and ConjGeo capture much of the geometric information about the elements and conjugacy classes of G, it is desirable also to have languages whose growth functions are exactly the growth functions of the elements, minimal length elements up to conjugacy, and conjugacy classes of G, respectively. To that end, let ≤ be a total ordering of X, and let ≤_{sl} be the induced shortlex ordering of X^{∗} (for which u <_{sl} w if either l(u) < l(w), or l(u) = l(w) but u precedes w lexicographically).

For each g ∈ G, we define the shortlex normal form of g to be the unique word y_{g} ∈ X^{∗} with π(y_{g}) = g such that y_{g} ≤_{sl} w for all w ∈ X^{∗} with π(w) = g. For each conjugacy class c ∈ G/ ∼, we define the shortlex conjugacy normal form of c to be the shortlex least word z_{c} over X representing an element of c; that is, π(z_{c}) ∈ c, and z_{c} ≤_{sl} w for all w ∈ X^{∗} with π(w) ∈ c.

Our remaining three languages are the shortlex language, shortlex minimal conjugacy language, and shortlex conjugacy language for G over X, defined respectively as

SL = SL(G, X) := {y_{g} | g ∈ G},
ConjMinLenSL = ConjMinLenSL(G, X) := {y_{g} | |g| = |g|_{c}}, and
ConjSL = ConjSL(G, X) := {z_{c} | c ∈ G/∼}.

We note that the language ConjMinLenSL of shortlex normal forms for the set ConjMinLen of minimal length elements up to conjugacy in G satisfies ConjMinLenSL = ConjGeo ∩ SL.

Our six languages satisfy the following containments.

ConjGeo  ⊆ CycGeo ⊆ Geo
⊆  ⊆
ConjSL ⊆ ConjMinLenSL  ⊆  SL

Any language L over X gives rise to a strict growth function ϕ_{L} : ℕ ∪ {0} → ℕ ∪ {0}, defined by ϕ_{L}(n) := |{w ∈ L | l(w) = n}|, and an associated generating function, called the strict growth series, given by f_{L}(z) := ∑_{i=0}^{∞} ϕ_{L}(i)z^{i}. It is well known that if L is a regular language, then f_{L} is a rational function. For the three languages above, the coefficient ϕ_{SL}(n) is the number of elements of G of length n, ϕ_{ConjMinLenSL}(n) is the number of minimal length elements of G up to conjugacy of length n, and ϕ_{ConjSL}(n) is the number of conjugacy classes of G whose shortest elements have length n. Note that all three of these numbers depend only on G, X and not on our choice of SL as a language of geodesic normal forms for G; the growth series f_{SL}, f_{ConjMinLenSL}, and f_{ConjSL} would be the same with respect to any geodesic normal form. We consider the last of these series in two examples in this paper. To emphasize this independence from the shortlex ordering, we denote by σ̃ the strict growth series of ConjSL, that is,

σ̃ = σ̃(G, X) := f_{ConjSL(G,X)}(z) = ∑_{i=0}^{∞} ϕ_{ConjSL(G,X)}(i)z^{i},

following the notation of [7], and call that series the spherical conjugacy growth series.

In Section 2, Proposition 2.2 proves that regularity of Geo implies the same for CycGeo. Theorem 2.4 proves that equality of the languages CycGeo and ConjGeo is preserved by taking graph products, giving Corollary 2.5 that ConjGeo and CycGeo are equal (over the standard generating set) for all right-angled Artin and rightangled Coxeter groups.

Section 3 studies regularity of the conjugacy languages for many families known to have regular geodesic languages. These include word hyperbolic groups (for all generating sets) [9, Theorem 3.4.5] and, with appropriate generating sets, virtually abelian groups and geometrically finite hyperbolic groups [24, Theorem 4.3], Coxeter groups [18], right-angled Artin groups [21], Artin groups of large type [16], and Garside groups (and hence Artin groups of finite type and torus knot groups) [6]. For groups in most of these families, we have succeeded in proving the regularity of ConjGeo and ConjMinLenSL. Moreover, for virtually abelian groups, we build on [12, Prop. 6.3] to exhibit a generating set for which ConjGeo has the stronger property of piecewise testability (see Definition 3.2).

We summarize the results known for these families of groups in Table 1, where the symbol * means that the property holds for all generating sets, and if * does not appear, the result is only known to hold for a specific generating set.

In the cases of virtually abelian groups and Garside groups, we found no proofs in the literature of their shortlex automaticity with respect to the generating sets under consideration, and so we have also supplied those in Section 3.

Result Group SL Geo ConjGeo ConjMinLenSL

ConjSL

T hm. 3.1 word hyperbolic reg.* [9] reg.* [9] reg.*  reg.*  –
Thms. 4.2,3.1 virtually cyclic reg.* [9] reg.* [9] reg.*  reg.*  reg.*
Prop. 3.3 virtually abelian reg. PT [12] PT  reg.  –
Thms. 3.10,  extra-large reg. [16] reg. [16] reg.  reg.  not reg.
3.12  type Artin
Prop. 3.13  Garside  reg. reg. [6]  –  –  –
Thm. 3.15 homog. Garside reg. reg. [6] reg.  reg.  –
graph product † reg. [14] reg. [21] reg. [7]  reg.  –

†: assuming that SL, Geo, ConjGeo are all regular for all vertex groups

Table 1. Summary of language properties for groups in Sections 3 & 4

Note that Table 1 includes many families of Artin groups, including those of spherical (i.e. finite) type (which are homogeneous Garside groups), and also rightangled Artin and Coxeter groups, as graph products of appropriate vertex groups. (The graph product results are included in the table for completeness, but are all proved elsewhere.)

For two of the blank entries in Table 1 in the ConjSL column, namely word hyperbolic and graph product groups, we note that there are examples of groups and finite generating sets for which ConjSL is not regular for any total ordering of the generators. In particular, Rivin [27],[26] and Ciobanu and Hermiller [7] have shown that for a free product of two infinite cyclic groups or two finite cyclic groups of order greater than 2, with respect to the cyclic generators (in the infinite case) or the nontrivial elements of the finite cyclic factors, respectively, the spherical conjugacy growth series is not rational.

Some of the results in Section 3 have consequences for groups beyond those treated in this paper. In [1, Section 8] Antolín and the first author have shown that certain relatively hyperbolic groups have regular ConjGeo by using Corollary 3.8.

Section 4 is devoted to a conjecture of Rivin [27], [26] about the rationality of the spherical conjugacy growth series σ̃ for word hyperbolic groups. In Theorem 4.2 we verify one direction of the conjecture, proving that for a virtually cyclic group ConjSL is regular, and hence that σ̃ is rational. This result is included in Table 1.

Section 5 examines two virtually abelian groups to demonstrate how regularity properties can differ between the various languages we consider, and depend on choice of generating set. Cannon [24, p. 268] showed that regularity of Geo depends upon the generating set for G = ℤ^{2} ⋊ ℤ/2ℤ. Using the same two generating sets as Cannon, we show in Propositions 5.1 and 5.2 that regularity of ConjSL can depend upon the generating set. At the same time, we prove for one of these generating sets that, although ConjSL(G, Z) is not regular (with respect to any total ordering of Z), σ̃(G, Z) is rational. Moreover, over the generating set used in Proposition 5.2, we show that ConjGeo(G, X) is regular but Geo(G, X) is not, and similarly ConjMinLenSL and ConjSL are regular but SL is not. Considering a further finite extension K = ℤ^{2} ⋊ D_{8} of G in Propositions 5.3 and 5.4, we show that regularity of the conjugacy geodesic language and shortlex minimal conjugacy language can also depend upon the generating set. These results are summarized in Table 2.

Result Group, SL Geo ConjGeo ConjMinLenSL

ConjSL

generators

Prop. 5.1 ℤ^{2} ⋊ ℤ/2ℤ, Z reg.  reg. [24]  reg.  reg.  not reg.
Prop. 5.2 ℤ^{2} ⋊ ℤ/2ℤ, X not reg. not reg. [24] reg.  reg.  reg.
Prop. 5.3 ℤ^{2} ⋊ D_{8}, Z′ reg.  reg.  reg.  reg.  not reg.
Prop. 5.4 ℤ^{2} ⋊ D_{8}, X′ not reg. not reg. not reg.  not reg. not reg.

Table 2. Summary of language properties for groups in Section 5

The languages studied in this paper merit exploration not just because of the role they play in determining the conjugacy growth of a group, but also for their potential connections to the computational complexity of the conjugacy problem. We observe that a recursively presented group G = ⟨X | R⟩ for which ConjSL(G, X) is recursive has solvable conjugacy problem. (Given u ∈ X^{∗}, we find the shortlex representative of [u]_{c} by simultaneously enumerating conjugates y^{−1}uy of u, words w ∈ ConjSL(G, X) with l(w) ≤ l(u), and products z of conjugates of relators, and computing the free reduction of each y^{−1}uyw^{−1}z. When that is the empty word, w is the shortlex representative of [u]_{c}.) However Chuck Miller has pointed out to us that the example of an amalgamated product T of two free groups of the same rank described in [23, Section IVA] has unsolvable conjugacy problem but has recursive ConjGeo. Another open question of interest is what regularity of ConjSL or ConjGeo implies for the computational complexity of this important decision problem.

2. Conjugacy versus cyclic geodesics

For any language L ⊂ X^{∗}, let Cyc(L) denote the cyclic closure of L; that is, Cyc(L) is the set of all cyclic permutations of words in L.

Lemma 2.1. If a language L is regular, then Cyc(L) is also regular.

Proof. Let M be a finite state automaton accepting L, with state set Q and initial state q_{0}. A word w lies in Cyc(L) if and only if, for some factorisation w_{1}w_{2} of w, and for states q, q′ ∈ Q, with q′ accepting, M contains both a path from from q to q′ labelled by w_{1} and a path in M from q_{0} to q labeled by w_{2}. From this description we construct a set of non-deterministic automata M_{q}, indexed by the states of Q, the union of whose languages is Cyc(L).

The automaton M_{q} is formed from two disjoint copies M_{q,1} and M_{q,2} of the states and transitions of M. The initial state of M_{q} is the state q in M_{q,1} and the single accept state is the state q in M_{q,2}. Additional ϵ-transitions join each state of M_{q,1} that is accepting in M to the state q_{0} in M_{q,2}. □

Proposition 2.2. For G = ⟨X⟩, if Geo(G, X) is regular then so is CycGeo(G, X).

Proof. This follows directly from Lemma 2.1 and the fact that cyclic geodesics are the words that are not cyclic conjugates of nongeodesics; that is,

CycGeo = X^{∗} \ Cyc(X^{∗} \ Geo).
□

The following is an immediate consequence of the above proposition.

Proposition 2.3. For G = ⟨X⟩, if Geo(G, X) is regular and CycGeo(G, X) = ConjGeo(G, X), then ConjGeo(G, X) is also regular.

Given a finite simplicial graph Λ with vertices labeled by groups G_{i}, the associated graph product group G is generated by the groups G_{i}, with the added relations that whenever the vertices labeled G_{i} and G_{j} are adjacent, then the elements of G_{i} and G_{j} commute.

Theorem 2.4. Suppose that G_{i} = ⟨X_{i}⟩, with CycGeo(G_{i}, X_{i}) = ConjGeo(G_{i}, X_{i}), for each i. If G is a graph product of these groups with generating set X = ∪_{i}X_{i}, then CycGeo(G, X) = ConjGeo(G, X).

Proof. The containment CycGeo(G, X) ⊇ ConjGeo(G, X) is immediate from the definitions.

For each index i, let Geo_{i} := Geo(G_{i}, X_{i}), and let $ be a symbol not in X_{i}. Define the map ρ_{i} : X^{∗} → (X_{i} ∪ $)^{∗} by, for a ∈ X_{j}, setting ρ_{i}(a) := a if i = j, ρ_{i}(a) := ε (the empty word) if the vertices i and j are adjacent in the defining graph Λ of the graph product, and finally ρ_{i}(a) := $ if the vertices i and j are neither equal nor adjacent. In [7, Props. 3.3, 3.5], the first two authors show that, for w ∈ X^{∗},

w ∈ ConjGeo(G, X) ⟺ ∀i, ρ_{i}(w) ∈ ConjGeo(G_{i}, X_{i}) ∪ {u_{0}$u_{1} ⋯ $u_{n} | n ≥ 1 and u_{n}u_{0}, u_{1}, ..., u_{n−1} ∈ Geo_{i}};

moreover, w is geodesic if and only if ρ_{i}(w) ∈ Geo_{i}($Geo_{i})^{∗} for all i.

Suppose now that w ∈ CycGeo(G, X). For each index i, we have ρ_{i}(w) = u_{0}$u_{1} ⋯ $u_{n} with n ≥ 0 and each u_{j} ∈ Geo_{i}. Note that whenever w′ is a cyclic permutation of w, then ρ_{i}(w′) is a cyclic permutation of w′. In particular there is a cyclic permutation w′ of w such that ρ_{i}(w′) = u_{n}u_{0}$u_{1} ⋯ u_{n−1}$. Now since w′ ∈ Geo(G, X) as well, we have u_{n}u_{0}, u_{1}, …, u_{n−1} ∈ Geo_{i}. Suppose further that n = 0, that is, ρ_{i}(w) = u_{0} ∈ Geo_{i}. Now for every cyclic permutation u′ of u_{0}, there is a cyclic permutation w′ of w such that ρ_{i}(w′) = u′, and hence we must have u′ ∈ Geo_{i}. Thus in this case u_{0} ∈ CycGeo(G_{i}, X_{i}), and so by hypothesis also u_{0} ∈ ConjGeo(G_{i}, X_{i}). Therefore w ∈ ConjGeo(G, X), as required. □

In [7] the first two authors show that whenever the languages Geo, ConjGeo are both regular for all vertex groups, then both are regular for the graph product group. Theorem 2.4, however, gives a slightly stronger result about the structure of the geodesics involved, which we highlight in the case of right-angled Artin and Coxeter groups.

Corollary 2.5. For every right-angled Artin group (respectively, every right-angled Coxeter group) G with respect to the Artin (respectively, Coxeter) generators X, a word w is a conjugacy geodesic if and only if every cyclic conjugate of w is geodesic; that is, ConjGeo(G, X) = CycGeo(G, X).

3. Results about groups with regular geodesic languages

In this section we study conjugacy languages associated to particular families of groups, including word hyperbolic, virtually abelian, locally testable, (extra) large type Artin, and Garside groups, for which the geodesic language is regular. In each of these cases, we prove that the conjugacy geodesic language ConjGeo is also regular.

3.1. Word hyperbolic groups. For a word hyperbolic group G, regularity of the geodesic and shortlex languages holds for every finite generating set [9, Thms. 3.4.5, 2.5.1]. In the following we show that the same is true for the conjugacy geodesic language.

Theorem 3.1. Let G = ⟨X⟩ be a word hyperbolic group. Then ConjGeo(G, X) and ConjMinLenSL(G, X) are regular.

Proof. Since ConjMinLenSL(G, X) = ConjGeo(G, X) ∩ SL(G, X), and SL is regular, it suffices to prove the result for ConjGeo. We suppose that G has hyperbolicity constant δ ≥ 0 (i.e. geodesic triangles in the Cayley graph are δ-slim). By [4, Lemma III.2.9], two words u, v ∈ CycGeo(G, X) with max{l(u), l(v)} ≥ 8δ +1 represent conjugate elements precisely when there exist u′ ∈ Cyc({u}), v′ ∈ Cyc({v}) representing elements conjugate by a word of length at most 2δ + 1. From this we deduce that a word v ∈ CycGeo of length at least 8δ + 1 is outside ConjGeo precisely when it is in the cyclic closure of the set ⋃_{|α|≤2δ+1} L(α) where, for α ∈ X^{∗},

L(α) := {v′ ∈ CycGeo | ∃u′ ∈ CycGeo such that α^{−1}u′α =_{G} v′, l(v′) > l(u′))}.

Since the set ConjGeo ∩ (∪_{k=0}^{8δ+1}X^{k}) is finite, it is sufficent to prove the regularity of the set

Cyc ( ⋃_{|α|≤2δ+1} L(α)) .

Since the family of regular sets is closed under both finite union and cyclic closure (Proposition 2.1), it is enough to show the regularity of L(α), for a given word α.

Now, since G is hyperbolic it has a biautomatic structure on Geo. By [11, Lemma 8.1], for any word α, the language

L_{1}(α) := {(u, v) | u, v ∈ Geo(G, X), v =_{G} α^{−1}uα}

is regular.

By Proposition 2.2, CycGeo(G, X) is regular and by standard arguments so are the languages

L_{2} := {(u, v) | u, v ∈ CycGeo(G, X), l(v) > l(u)},
L_{3}(α) := L_{1}(α) ∩ L_{2},  and
L_{4}(α) := {v ∈ CycGeo | ∃u ∈ CycGeo, (u, v) ∈ L_{3}(α)}.

We see that L_{4}(α) = L(α). □

In contrast, for the shortlex conjugacy language, Rivin [27], [26] and Ciobanu and Hermiller [7] have given examples of word hyperbolic groups and specific generating sets for which the the spherical conjugacy growth series is not rational, and so ConjSL is not regular for any total ordering of the generators. More precisely, σ̃ was proved non-rational for free products of the form ℤ ∗ ℤ and ℤ/mℤ ∗ ℤ/nℤ with m, n > 2, using as generators in the first case cyclic generators for the two copies of ℤ and in the second case all nontrivial elements of the two finite cyclic groups.

3.2. Virtually abelian groups. In [24, Props. 4.1, 4.4] Neumann and Shapiro show that for every virtually abelian group, there is a generating set for the group such that the corresponding set of geodesic words is regular, and in [12, Prop. 6.3], Hermiller, Holt, and Rees strengthen this result by showing that the geodesic words are a piecewise testable language. In the following we adjust that proof to show that for some choice of generators the language of conjugacy geodesics is also piecewise testable.

Definition 3.2. Let A be a finite alphabet.

(1) A subset L of A^{∗} is called piecewise testable if it is defined by a regular expression that combines terms of the form A^{∗}a_{1}A^{∗}a_{2} … A^{∗}a_{k}A^{∗} using the Boolean operations of union, intersection and complementation, where k ≥ 0 and each a_{i} ∈ A.

(2) A subset L of A^{∗} is called piecewise excluding if there is a finite set of strings W ⊂ A^{∗} with the property that a word w ∈ A^{∗} lies in L if and only if w does not contain any of the strings in W as a not necessarily consecutive substring. In other words,

L = (∪_{i=1}^{n}{A^{∗}a_{i_{1}}A^{∗}a_{i_{2}}A^{∗} … A^{∗}a_{i_{l_{i}}} A^{∗}})^{c},

where W = {a_{1}, …, a_{n}} and a_{i} = a_{i_{1}}a_{i_{2}} … a_{i_{l_{i}}} for 1 ≤ i ≤ n.

According to Definition 3.2, piecewise excluding languages are also piecewise testable.

Proposition 3.3. Let G be a virtually abelian group. There exists a finite generating set Z for G such that ConjGeo(G, Z) is piecewise testable. Furthermore, there is an ordering of Z with respect to which G is shortlex automatic and ConjMinLenSL(G, Z) is regular.

Proof. This argument follows closely the proof of [12, Prop. 6.3], so we omit some of the details.

Let G be a virtually abelian group, and let N ◁ G be abelian of finite index in G with a finite generating set A. Let T ∪ {1} be a transversal of N in G.

We build the generating set Z for G as follows. Let Y := T^{±1}. Let X′ be the set of all x ∈ N such that x =_{G} w ≠_{G} 1 for some w ∈ Y^{∗} with l(w) ≤ 4. Finally, let X be the closure of the set A ∪ X′ in G under inversion and conjugation, and let Z := X ∪ Y. Then

(i) X ⊂ N, Y ⊂ G \ N, (ii) both X and Y are closed under inversion, (iii) X is closed under conjugation by elements of G, (iv) Y contains at least one representative of each nontrivial coset of N in G, and (v) if w =_{G} xy with w ∈ Y^{∗}, l(w) ≤ 3, x ∈ N and y ∈ Y ∪ {ε}, then x ∈ X.

Write the finite set X as X = {x_{1}, ..., x_{m}}. For each x ∈ X and y ∈ Y, let x^{y} denote the generator in X that represents the group element y^{−1}xy. Similarly if v = x_{i_{1}} ⋯ x_{i_{k}} ∈ X^{∗}, the symbol v^{y} denotes the word x_{i_{1}}^{y} ⋯ x_{i_{k}}^{y}. An immediate consequence of properties (i)-(v) above is that the language L := Geo(G, Z) of geodesics of G over Z satisfies the property that L ⊆ X^{⋆} ∪ X^{⋆}Y X^{⋆} ∪ X^{⋆}Y X^{⋆}Y X^{⋆}.

We can now establish the shortlex automaticity of G with respect to a suitable (total) ordering of Z. Choose any such ordering in which the generators in X precede those in Y. Note that every element of G has a geodesic representative in the set X^{⋆} ∪ X^{⋆}Y and since, by properties (iii) and (v), elements of the form yz with y ∈ Y and z ∈ Z have an alternative representative of the form x′y′ with x′ ∈ X ∪ {ε} and y′ ∈ Y ∪ {ε}, we see that all shortlex minimal representatives have this form. Let L′ be the set of words w or wy of this form in which w ∈ X^{∗} is shortlex minimal. By [9, Thm. 4.3.1], a finitely generated abelian group is shortlex automatic with respect to any finite ordered generating set. It follows (again from properties (iii) and (v)) that L′ is the language of an automatic structure for G. The closure of Y under inversion means that elements of G may have two representatives in L′, but by [9, Thm. 2.5.1], we get the shortlex automatic structure for G by intersecting L′ with SL(G, Z).

Turning now to the set L̃ := ConjGeo(G, Z) of conjugacy geodesics of G over Z, we have L̃ := ConjGeo(G, Z) ⊆ L, and so L̃ can be partitioned as the union of the subsets L̃_{0} := L̃ ∩ X^{∗}, L̃_{1} := L̃ ∩ X^{∗}Y X^{∗}, and L̃_{2} := L̃ ∩ X^{∗}Y X^{∗}Y X^{∗}. We show that each L̃_{i} is a piecewise testable language.

We start by showing that L̃_{0} = L ∩ X^{∗}. Suppose that w ∈ L ∩ X^{∗} is not a conjugacy geodesic, and write w = a_{1}a_{2} … a_{n} with a_{i} ∈ X. Then there exist n ∈ N and y ∈ Y such that the element (ny)^{−1}wny of G is represented by a word u that is shorter than w. Since N is abelian we have u =_{G} (ny)^{−1}wny =_{G} y^{−1}wy. Then w =_{G} u^{y^{−1}} and by properties (ii)-(iii) the formal conjugate u^{y^{−1}} is a word in X^{∗} satisfying l(u^{y^{−1}}) = l(u) < l(w), which contradicts the fact that w is geodesic. By the proof of [12, Prop. 6.3], L ∩ X^{∗} is piecewise excluding, and hence also piecewise testable, so L̃_{0} has the same properties.

An operation on words over Z given by replacement ayxb → ax^{y^{−1}}yb with a, b ∈ Z^{∗}, x ∈ X, and y ∈ Y is called a Y -shuffle. An operation on words over X given by a replacement ux_{i}x_{j}v → ux_{j}x_{i}v is a shuffle. Note that whenever a word w can be obtained from a word v by means of finitely many applications of these operations, then v =_{G} w, v and w are words over Z of the same length, and furthermore v ∈ L̃_{i} if and only if w ∈ L̃_{i}.

Next we turn to L̃_{1}. In this case we further partition the set L̃_{1} = ∪_{r∈Y} L̃_{1,r} where L̃_{1,r} := {v_{1}rv_{2} ∈ L̃ | v_{1}, v_{2} ∈ X^{∗}} for each r in Y. Let Λ̃_{1,r} := {w ∈ X^{∗} | wr ∈ L̃}. It is immediate to see that L̃_{1,r} is the set of all words that can be obtained from words of the form wr with w ∈ Λ̃_{1,r} using Y -shuffles.

Next let Ũ_{1,r} := Λ̃_{1,r} ∩ x_{1}^{∗}x_{2}^{∗} ⋯ x_{m}^{∗}, and let

U_{1,r} := {(n_{1}, ..., n_{m}) ∈ ℕ_{0}^{m} | x_{1}^{n_{1}} ⋯ x_{m}^{n_{m}} ∉ Ũ_{1,r}}.

Now U_{1,r} has only finitely many elements that are minimal under the ordering on ℕ_{0}^{m} defined by (n_{1}, ..., n_{m}) ≤ (p_{1}, ..., p_{m}) if and only if n_{i} ≤ p_{i} for all 1 ≤ i ≤ m (for a proof see, for example, [9, Lemma 4.3.2]); let S_{1,r} be this finite set of minimal elements. Let S̃_{1,r} be the set of all words that can be obtained from elements of the set {x_{1}^{n_{1}} ⋯ x_{m}^{n_{m}} | (n_{1}, ..., n_{m}) ∈ S_{1,r}} using shuffles, and let S̃′_{1,r} be the set of all words that can be obtained from elements of the set {wr | w ∈ S̃_{1,r}} via Y -shuffles. Then the set L̃_{1,r} of Y -shuffles of words in Λ̃_{1,r}r is exactly the set of all words in X^{∗} that do not contain a piecewise subword lying in the finite set S̃′_{1,r}. So L̃_{1,r} is piecewise excluding, and hence also piecewise testable, for each r ∈ Y. Therefore L̃_{1} is also piecewise testable.

The proof that L̃_{2} is piecewise testable starts by partitioning L̃_{2} = ∪_{r,s∈Y} L̃_{2,r,s} where L̃_{2,r,s} := {v_{1}rv_{2}sv_{3} ∈ L̃ | v_{1}, v_{2}, v_{3} ∈ X^{∗}} for each r, s in Y. The proof is similar to that for L̃_{1}, but a little more complicated, and we omit the details.

Having established that ConjGeo and SL are regular, it follows that their intersection ConjMinLenSL is also regular. □

An immediate consequence of the proof above is that for abelian groups piecewise testability of the conjugacy geodesic language holds for every generating set.

Corollary 3.4. If G = ⟨X⟩ is abelian, then ConjGeo(G, X) is piecewise testable.

3.3. Groups with locally testable geodesics. This section is devoted to those groups, previously studied in [13], for which Geo(G, X) is locally testable. Free groups, free abelian groups and dihedral Artin groups over the standard generators provide examples, as do direct products of such groups.

Informally, whenever k is a positive integer, a language L is k-locally testable if membership of a word in L depends on the nature of its subwords of length k (where by a subword of a word a_{1}a_{2} ⋯ a_{n}, we mean either the empty word or a contiguous substring a_{i}a_{i+1} ⋯ a_{j} for some 1 ≤ i ≤ j ≤ n); a language L is locally testable if it is k-locally testable for some k.

More precisely, k-local testability is defined as follows. Let k > 0 be a natural number. For u ∈ X^{∗} of length at least k, let pre_{k}(u) be the prefix of u of length k, let suf_{k}(u) be the suffix of u of length k, and let sub_{k}(u) be the set of all subwords of u of length k. If l(u) < k, then we define pre_{k}(u) = u, suf_{k}(u) = u, and sub_{k}(u) = ∅. We define an equivalence relation ∼_{k} on X^{∗} to relate u and v when pre_{k−1}(u) = pre_{k−1}(v), suf_{k−1}(u) = suf_{k−1}(v), and sub_{k}(u) = sub_{k}(v). A subset L ⊆ X^{∗} is defined to be k-locally testable [5, p. 247] if L is a union of equivalence classes of ∼_{k}. We refer the reader to [13] and the references cited there for additional information. Here we show the following.

Theorem 3.5. For a group G = ⟨X⟩, if Geo(G, X) is locally testable, then ConjGeo(G, X) = CycGeo(G, X) \ A, where A is a finite set. Hence ConjGeo(G, X) is regular.

Proof. The key ingredient of this proof is [13, Lemma 5.2], which can be reformulated as follows: if G = ⟨X⟩ and Geo(G, X) is locally testable, then there exists N ∈ ℕ such that, for each word w ∈ X^{∗} with l(w) > N, there is a cyclic permutation w̃ of w with the property that, if w̃ is a geodesic, then w^{j} is a geodesic for all j ≥ 1.

Assume that Geo(G, X) is locally testable. We claim that each w ∈ CycGeo(G, X) with l(w) > N is in fact in ConjGeo(G, X). Notice first that the cyclic geodesic w lies in ConjGeo(G, X) if and only if each of its cyclic permutations is in ConjGeo(G, X). Now let w̃ be the cyclic permutation of w provided by [13, Lemma 5.2]. Suppose that w̃ ∉ ConjGeo(G, X). Then there exist u, v ∈ X^{∗} such that w̃ =_{G} u^{−1}vu with l(v) < l(w̃). Notice that l(w^{j}) = j l(w̃) for all j ≥ 1. Then

|u^{−1}v^{j}u| ≤ 2 l(u) + l(v^{j}) = 2 l(u) + j l(v) ≤ 2 l(u) + (j l(w̃) − j),

which for j > 2 l(u) leads to |w̃^{j}| = |u^{−1}v^{j}u| < j l(w̃) = l(w^{j}), contradicting the fact that w^{j} is a geodesic. □

3.4. Groups with the falsification by fellow traveler property. For a group G = ⟨X⟩ and k ≥ 0, we say that the words w, w′ ∈ X^{∗} k-fellow travel (and write w ≍_{k} w′), if, for each i ≥ 0, |pre_{i}(w)^{−1}pre_{i}(w′)| ≤ k. We say that G = ⟨X⟩ satisfies the falsification by fellow traveler property (FFTP) if, for some fixed constant k, any nongeodesic word w k-fellow travels with a shorter word. We will write k-FFTP if the above constant needs to be mentioned explicitly. In [24, Prop. 4.1] Neumann and Shapiro showed that a group G satisfying the FFTP over a generating set X has Geo(G, X) regular. They also show that the FFTP is dependent upon the generating set. In order to apply this property to conjugacy languages, we begin with a strengthening of the result of [24, Prop. 4.1].

Proposition 3.6. Suppose that G = ⟨X⟩ satisfies the FFTP. Then for every K ≥ 0 the set

NearGeo_{K} := {w ∈ X^{∗} : |w| ≥ l(w) − K}

is regular.

Proof. Suppose that w ∉ NearGeo_{K}. So |w| ≤ l(w)−(K+1). Suppose that G satisfies the k-FFTP. Then there is a sequence of words w_{0} = w, w_{1}, …, w_{N}, all representing the same element of G as w where, for each i, w_{i} ≍_{k} w_{i−1}, l(w_{i}) < l(w_{i−1}), and where l(w_{N}) < l(w) − K. So, assuming that N is minimal with l(w_{N}) < l(w) − K, we have N ≤ K + 1, and hence w ≍_{(K+1)k} w_{N}. So

X^{∗} \ NearGeo_{K} = {w : ∃w′, l(w′) < l(w) − K, w ≍_{(K+1)k} w′, w =_{G} w′}.

This is the projection onto the first coordinate of the intersection of the padded languages L_{1} := {(w, w′) | l(w′) < l(w) − K} and L_{2} := {((w, w′) | w ≍_{(K+1)k} w′, w =_{G} w′}; see [9, Section 1.4] for details on languages of padded pairs. The regularity of X^{∗} \ NearGeo_{K}, and hence also of NearGeo_{K}, now follows from the regularity of L_{1} (which we leave to the reader) and of L_{2}. The set L_{2} is the regular language accepted by the “standard automaton” M_{ϵ} of [9, Definition 2.3.3], associated to the identity element of G and based on (W, N), where N is the ball of radius (K + 1)k centered at the identity in G, and W is a finite state automaton accepting the language X^{∗}. Briefly, given an automaton M with alphabet X ∪{$}, state set S, initial state s_{0} and transition function δ : S × X ∪ {$} → S accepting the regular language L(W)$^{∗} = X^{∗}$^{∗}, then M_{ϵ} has alphabet (X ∪ $)^{2} \ {($, $)}, state set S × S × N, and initial state (s_{0}, s_{0}, ϵ). When M_{ϵ} is in state (s, t, g) and reads a letter (a, b), the automaton goes to the state (δ(s, a), δ(s, b), a^{−1}gb) if a^{−1}gb ∈ N, and fails otherwise; a word is accepted by M_{ϵ} if the automaton finishes in a state of the form (s, t, ϵ) where s and t are accept states of M. □

Proposition 3.7. Suppose that G = ⟨X⟩ satisfies the FFTP. Suppose that, for some fixed k, L is defined to be the set of words w ∈ CycGeo(G, X) for which the following condition holds:

(v ∈ Cyc(w) ∧ α ∈ X^{∗} ∧ l(α) ≤ k) ⇒ |α^{−1}vα| ≥ |v|.

Then L is regular.

Proof. For a word w ∈ CycGeo, we have w ∉ L if and only if w ∈ Cyc(∪_{α∈X^{∗},l(α)≤k}L(α)), where

L(α) = {v ∈ CycGeo : |α^{−1}vα| < |v|}.

Since any word v satisfies

|α^{−1}vα| < |v| ⟺ l(α^{−1}vα) − |α^{−1}vα| > 2l(α),

we have

L(α) = {v ∈ CycGeo : α^{−1}vα ∈ NearGeo_{2l(α)}}.

Since G satisfies the FFTP, the languages NearGeo_{2l(α)} and Geo = NearGeo_{0} are regular by Proposition 3.6, and hence (by Proposition 2.2) so is CycGeo. Standard properties of regular languages now ensure the regularity of L(α).

Since the class of regular languages is closed under finite union, and by cyclic closure (Lemma 2.1), the complement of L in CycGeo given by Cyc(∪_{α∈X^{∗},l(α)≤k}L(α)) is also regular. It follows that L is a regular set. □

The following is immediate.

Corollary 3.8. Suppose that G = ⟨X⟩ satisfies the FFTP. Suppose that, for some fixed k, s, and for all r ≥ s, the set ConjGeo(G, X) ∩ X^{r} is equal to the set of all w ∈ CycGeo(G, X) ∩ X^{r} for which

(v ∈ Cyc(w) ∧ α ∈ X^{∗} ∧ l(α) ≤ k) ⇒ |α^{−1}vα| ≥ |v|.

Then ConjGeo(G, X) is regular.

Applying this corollary with s = 8δ+1 and k = 2δ+1 gives an alternative proof of Theorem 3.1 that uses FFTP rather than biautomaticity in word hyperbolic groups.

In the following section we use Proposition 3.7 (rather than the corollary) to prove the regularity of ConjGeo for an extra-large type Artin group.

3.5. Artin groups of (extra) large type. An Artin group is defined by the presentation

⟨x_{1}, …, x_{n} | _{m_{ij}}(x_{i}, x_{j}) = _{m_{ij}}(x_{j}, x_{i}) for each i ≠ j⟩,

where (m_{ij}) is a Coxeter matrix (a symmetric n × n matrix with entries in ℕ ∪ {∞}, m_{ii} = 1, m_{ij} ≥ 2, ∀i ≠ j), and where for generators a, a′ and m ∈ ℕ we define_{m}(a, a′) to be the word that is the product of m alternating as and a′s that starts with a. The set {x_{1}, ….x_{n}} is usually called the standard generating set of the group, but since we want our generating sets to be inverse-closed, we define X := {x_{1}, ….x_{n}}^{±1} to be the standard generating set. An Artin group has large type if all the integers m_{ij} are at least 3, and extra-large type if they are all at least 4.

Holt and Rees [16, Thms. 3.2, 4.1] have shown that the languages Geo(G, X) and SL(G, X) are regular. Theorems 3.10 and 3.12 below investigate ConjGeo(G, X), ConjMinLenSL(G, X), and ConjSL(G, X) for Artin groups of (extra-)large type.

First we note that it follows from a result of Mairesse and Mathéus [22, Prop. 4.3] that the set of geodesics for a dihedral Artin group (i.e. an Artin group presented above with n = 2) over its standard generating set is locally testable, and so we obtain the following corollary of Theorem 3.5.

Corollary 3.9. For a dihedral Artin group G over its standard generating set X, the set ConjGeo(G, X) is regular.

Theorem 3.10. Let G = ⟨X⟩ be an Artin group of extra-large type with X its standard generating set. Then ConjGeo(G, X) is regular, and for any ordering of the generators ConjMinLenSL(G, X) is also regular.

Proof. Let n := |X|/2. For standard generators x_{i}, x_{j}, we write X(i, j) = {x_{i}, x_{j}}^{±1} and G(i, j) = ⟨X(i, j)⟩. When n = 2, the Artin group is dihedral, and so the result follows from Corollary 3.9.

When n ≥ 3, ConjGeo\{1} can be written as a union L_{1}∪L_{2}∪L_{3} of words involving the generators x_{i}^{±1} for one, two or at least three values of i, respectively. By [17, Prop. 4.1], L_{1} is exactly the set of freely reduced powers of generators, and hence is regular. We aim now to show that ConjGeo(G(i, j), X(i, j)) ⊆ ConjGeo(G, X). So suppose that w is a 2-generator word involving generators x_{i}, x_{j}, and that w ∉ ConjGeo(G, X). If w is non-geodesic in G, then, by results of [16, Section 3], w ∉ ConjGeo(G(i, j), X(i, j)), so from on now we assume that w is geodesic in G. Suppose that some generator g conjugates w to a word with a shorter representative. The results proved in [16] show that none of the reductions used to reduce a words to shortlex normal form could involve g if g ∉ X(i, j), so g ∈ X(i, j), and again w ∉ ConjGeo(G(i, j), X(i, j)). Otherwise, the element π(w) is ‘cyclically reduced’ according to the definition of [17]; that is, for each a ∈ X we have |a^{−1}π(w)a| ≥ |π(w)|. Then we can apply [17, Prop. 5.1], which ensures the existence of words α, u over X(i, j) with π(α^{−1}wα) = π(u) and l(u) < l(w). So in this case too w ∉ ConjGeo(G(i, j), X(i, j)).

Since ConjGeo(G(i, j), X(i, j)) ⊇ (L_{1} ∪ L_{2}) ∩ X(i, j)^{∗}, it now follows that

L_{1} ∪ L_{2} = ∪_{i≠j}ConjGeo(G(i, j), X(i, j)).

Consequently L_{1} ∪ L_{2} is regular.

We shall now apply Proposition 3.7 to show that L_{3} is regular. We note that since ConjGeo(G, X) ⊆ CycGeo(G, X), we have L_{3} ⊆ CycGeo(G, X). By [16, Thm. 4.1], the language Geo(G, X) is regular, and so Proposition 2.2 shows that CycGeo(G, X) also is regular. Standard properties of regular languages now show that the set L̃_{3} of all words in CycGeo involving at least three generators is also regular.

Words in CycGeo are ‘specially cyclically reduced’ according to the criteria of [2, Thm. 4′′], and so that theorem applies to show that two words in L̃_{3} represent conjugate elements if and only if cyclic conjugates of them are conjugate via a power of a generator. Then [17, Prop. 6.2] applies to show that those two cyclic conjugates must have the same length unless one of them represents an element of G that is not ‘cyclically reduced’ according to the definition above; that is, its conjugate by some generator represents a shorter element of G. Also, by [17, Props. 4.2, 5.1], a word in L̃_{3} cannot be conjugate to a shorter element of G that involves fewer than three generators.

Hence, for w ∈ L̃_{3}, w lies in ConjGeo(G, X) precisely when none of its cyclic conjugates is shortened by conjugation by a generator; that is precisely when

(v ∈ Cyc(w) ∧ a ∈ X) ⇒ |a^{−1}va| ≥ |v|.

In [16, Thm. 4.1] it is also shown that the group G satisfies the FFTP with respect to the generating set X. Now Proposition 3.7 applies to show that the set of all words v that lie in CycGeo and satisfy the above condition is regular. But we have just shown that L_{3} is the intersection of this set with the regular set L̃_{3}, so L_{3} is regular, and the proof that ConjGeo(G, X) is regular is complete.

Holt and Rees [16, Thm. 3.2] show that SL(G, X) is regular for any ordering of the generators, and therefore the intersection ConjMinLenSL(G, X) = ConjGeo(G, X) ∩ SL(G, X) is also regular. □

The remainder of this section is devoted to the proof of Theorem 3.12. We need a technical result about a subset of ConjGeo in the case where G is dihedral.

Lemma 3.11. Let G = ⟨a, b | _{m}(a, b) = _{m}(b, a)⟩ be a dihedral Artin group with m ≥ 3, and let Δ = _{m}(a, b). Let w be a positive word in a^{2} and b^{2}. Suppose that w′ is any word in the generators and inverses that represents a conjugate of π(w) in G. Then |w′| ≥ |w| and, if |w′| = |w|, then w′ is a cyclic conjugate of w or of w^{Δ}.

In order to make the proof of the lemma easier to follow, we precede it with a short explanation of the structure of geodesics in dihedral Artin groups; a characterisation is given in [22].

Following [16], we extend the notation for alternating products of generators already introduced above, and define, for any letters (not just generators) a, b, _{m}(b, a) and (b, a)_{m} to be words of length consisting of alternating as and bs that begin and end in a, respectively. So (for example) _{m}(a, b)^{−1} = (b^{−1}, a^{−1})_{m}.

Now, for any word w in the generators (and inverses) of the dihedral Artin group G presented as in the statement of the lemma, we define p(w) to be the minimum of m and the length of the longest subword of w of alternating a’s and b’s. Similarly, we define n(w) to be the minimum of m and the length of the longest subword of w of alternating a^{−1}’s and b^{−1}’s. According to [22, Prop. 4.3], w is geodesic if and only if p(w) + n(w) ≤ m. Furthermore, π(w) has more than one geodesic representative if and only if p(w) + n(w) = m.

We recall also that the element Δ represented by _{m}(a, b) and _{m}(b, a) is central in G when m is even, whereas a^{Δ} = b, b^{Δ} = a and Δ^{2} is central when m is odd. For a word w = a_{1} ⋯ a_{k} with a_{i} ∈ {a, b}^{±}, we define w^{Δ} to be the word a_{1}^{Δ} ⋯ a_{k}^{Δ}.

Proof of Lemma 3.11:

We prove the result using a minimal counterexample argument, and for the induction to work we need to work under the more general hypothesis that w is a cyclic conjugate of a positive word in a^{2} and b^{2}. So either w is itself such a word, or else it has the form xŵx, where ŵ is a positive word in a^{2} and b^{2}, and x ∈ {a, b}. For such a word w, p(w) ≤ 2, and n(w) = 0. So, by [22], as explained above, w is geodesic.

Suppose that g^{−1}wg =_{G} w′, and let v be a shortest word representing g. Choose w, w′ and g to be a counterexample to the theorem in which |v| is minimal.

Then p(v) ≤ m/2, since otherwise there would be a counterexample with shorter v in which a positive alternating subword u of v of length p(v) is replaced by the geodesic representative of Δ^{−1}u, the prefix v_{1} of v before the occurrence of u is replaced by v_{1}^{Δ}, and w′ is replaced by w^{′Δ}. Similarly n(v) ≤ m/2. We assume that v has a positive prefix, that is, that the first letter of v is a or b. (Otherwise the last letter of v^{−1} is a or b, and the argument is similar.) In fact, we may assume without loss that it is a.

If there is any free cancellation in the word v^{−1}wv, then we can replace w by a cyclic conjugate and v by a shorter word so, by the minimality of v, the word v^{−1}wv must be freely reduced, and it cannot be a geodesic word. Since any negative alternating subwords of v^{−1}wv must occur within v or v^{−1}, we have n(v^{−1}wv) ≤ m/2. Since v^{−1}wv is not geodesic, we see (by considering the cases when m is even and odd) that p(v^{−1}wv) > (m + 1)/2. Hence, since p(v) ≤ m/2 and p(w) ≤ 2 ≤ (m + 1)/2, the longest positive alternating subword of v^{−1}wv must overlap the subwords w and v (and hence, since v begins with a, w must end in b).

Suppose first that m is odd. In this case, for any 0 < p < m the equation p(a, b) =_{G} Δ_{m−p}(a^{−1}, b^{−1}) holds, and we use it below. Now, since p(v) ≤ (m − 1)/2 and p(v^{−1}wv) > (m + 1)/2, w must end in a^{2}b and v must have a prefix u := (m−1)/2(a, b). Let w = w_{1}ab, where w_{1} ends in a. Then

u^{−1}wu = (b^{−1}, a^{−1})_{(m−1)/2} w_{1} _{(m+3)/2}(a, b)
=_{G} (b, a)_{(m+1)/2} w_{1}^{Δ} _{(m−3)/2}(a^{−1}, b^{−1})
= (b, a)_{(m−3)/2} (abw_{1})^{Δ} _{(m−3)/2}(a^{−1}, b^{−1}).

So, by replacing w by the cyclic conjugate (abw_{1})^{Δ} of w^{Δ}, and the prefix u of v by the shorter word _{(m−3)/2}(a^{−1}, b^{−1}), we find a counterexample with a shorter v.

So now suppose that m is even. In this case, for any 0 < p < m the equation p(a, b) =_{G} Δ_{m−p}(b^{−1}, a^{−1}) holds, and Δ is central. Suppose first that w does not have ab as suffix. Then v must have a prefix u := _{m/2}(a, b). Let w = w_{2}b. Then

u^{−1}wu = (b^{−1}, a^{−1})_{m/2} w_{2} _{m/2+1}(b, a)
=_{G} (a, b)_{m/2} w_{2}^{Δ} _{m/2−1}(a^{−1}, b^{−1})
=_{G} (a, b)_{m/2} w_{2} _{m/2−1}(a^{−1}, b^{−1})
= (b, a)_{m/2−1} bw_{2} _{m/2−1}(a^{−1}, b^{−1}).

So, by replacing w by its cyclic conjugate bw_{2}, and the prefix u of v by the shorter word _{m/2−1}(a^{−1}, b^{−1}), we find a counterexample with a shorter v.

So now suppose that w does have ab as suffix, and let w = w_{3}ab, where w_{3} ends in a. If v has prefix u := _{m/2}(a, b) then

u^{−1}wu = (b^{−1}, a^{−1})_{m/2} w_{3} _{m/2+2}(a, b)
=_{G} (a, b)_{m/2} w_{3}^{Δ} _{m/2−2}(b^{−1}, a^{−1})
= (a, b)_{m/2} w_{3} _{m/2−2}(b^{−1}, a^{−1})
= (a, b)_{m/2−2} abw_{3} _{m/2−2}(a^{−1}, b^{−1}).

So, by replacing w by its cyclic conjugate abw_{3}, and the prefix u of v by the shorter word _{m/2−2}(a^{−1}, b^{−1}), we find a counterexample with a shorter v.

Finally, if v does not have prefix _{m/2}(a, b), then it must have prefix _{m/2−1}(a, b), and either v or v^{−1} must contain a negative alternating subword of length m/2. In that case, we can replace g^{−1}wg by an equivalent word of length |g^{−1}wg| − 2, in which a negative alternating subword of v or v^{−1} is replaced by a positive alternating subword of the same length, and the subword _{m/2+1}(a, b) of g^{−1}wg that overlaps w and g is replaced by _{m/2−1}(b^{−1}, a^{−1}). The resulting word has no positive or negative alternating subwords of length greater than m/2, and hence it is geodesic. Since |v| > 1, the resulting word is longer than w, and so w, g, and v do not give a counterexample to the theorem, which is a contradiction of the original choice of these words. □

Theorem 3.12. Let G be an Artin group of large type, and X = {x_{1}, …, x_{n}} its standard generating set. Then ConjSL(G, X) is not regular.

Proof. Suppose first that n = 2, so G = ⟨a, b | _{m}(a, b) = _{m}(b, a)⟩ (with a = x_{1}, b = x_{2}) is a dihedral Artin group with m ≥ 3. Assume, without loss of generality, that a < b. We deduce from Lemma 3.11 that, for m, n > 2, the word a^{2m}b^{2}a^{2n}b^{2} is in ConjSL if and only if m ≥ n. Hence ConjSL(G, X)∩(a^{2})^{∗}b^{2}(a^{2})^{∗}b^{2} = {a^{2m}b^{2}a^{2n}b^{2} : m ≥ n}, which is not regular. It follows from the regularity of (a^{2})^{∗}b^{2}(a^{2})^{∗}b^{2} that ConjSL(G, X) is not regular.

When n > 2, we let a = x_{i} and b = x_{j} be any two generators in X. Now, as we remarked in the proof of Theorem 3.10, [17, Prop. 5.1] implies that, if a group element a^{2m}b^{2}a^{2n}b^{2} with m, n > 2 is conjugate in G to a shorter word, then it is conjugate to a shorter word in the subgroup G(i, j) = ⟨a, b⟩ of G. (In fact the proof of [17, Prop. 5.1] is valid in general only for extra-large type Artin groups, but it works for large type provided that the element g′ defined in that proof has at least three syllables, which is true for the 4-syllable word a^{2m}b^{2}a^{2n}b^{2}.) So again, assuming a < b, we have ConjSL(G, X) ∩ (a^{2})^{∗}b^{2}(a^{2})^{∗}b^{2} = {a^{2m}b^{2}a^{2n}b^{2} : m ≥ n}, which is not regular, and hence ConjSL(G, X) is not regular. □

3.6. Garside groups. Garside groups (also known as small Gaussian groups) were introduced in [8] as a generalization of the spherical type Artin groups, which include the braid groups. It is shown there that many of the properties of spherical type Artin groups, including having a geodesic biautomatic structure, generalize to Garside groups. A Garside group G is defined by a finite presentation in which all of the relations are of the form v = u, where v and u are positive words in the group generators. So this presentation also defines the associated Garside monoid G^{+}, and it turns out that this embeds into the Garside group.

There is an element Δ ∈ G^{+} known as the Garside element, and the (typically rather large) set S of divisors of Δ in G^{+} forms the set of Garside generators of G. The geodesic biautomatic structure mentioned above is defined on these generators, and we shall show now that this is also a shortlex automatic structure.

Proposition 3.13. Let G be a Garside group with Garside element Δ and Garside generators S. Then there is an ordering of X := S ∪ S^{−1} with respect to which G is shortlex automatic.

Proof. We first recall some standard notation and results on Garside groups. For a, b, c ∈ G^{+}, with c =_{G^{+}} ab, we say that a is a left divisor and b is a right divisor of c. It can be shown that any two elements a, b ∈ G^{+} have a unique ‘largest’ common left divisor a ∧ b, which is left-divisible by all of their common left divisors. For a ∈ G^{+}, we can write a =_{G^{+}} (a ∧ Δ)a′ for some a′ ∈ G^{+}, and the word a_{1}a_{2}a_{3} ⋯ a_{k} ∈ S^{∗} where the elements â_{1} := a and â_{i+1} := â′_{i} for 1 < i < k satisfy â′_{k} = 1 and a_{i} := â_{i} ∧ Δ for 1 ≤ i ≤ k, is known as the left greedy normal form of a.

Let L be the set of words over X of the form u^{−1}v, where u, v ∈ S^{∗}, u and v are in left greedy normal form, and u ∧ v = 1. It is proved in [8, Thm. 8.1] that L is the language of a geodesic biautomatic structure for G with uniqueness. We shall now define an order of X with respect to which the words in L are the shortlex least representatives of the group elements.

We choose any total ordering of X with the following three properties.

(i) s^{−1} < t for all s, t ∈ S; (ii) if s, t ∈ S and s is a left divisor of t, then t < s. (iii) if s, t ∈ S and s is a right divisor of t, then s^{−1} < t^{−1}. Let w be the shortlex least representative of the group element with normal form u^{−1}v ∈ L. We claim that w = u^{−1}v. Since u^{−1}v is geodesic, we have |w| = |u^{−1}v|. Since the normal forms of elements of G^{+} lie in S^{+}, we see that the least m ≥ 0 with Δ^{m}w ∈ G^{+} is equal to |u| and the least n ≥ 0 such that Δ^{n}w^{−1} ∈ G^{+} is equal to |v|. It follows that w must contain exactly |u| generators from S^{−1} and |v| from S because if, for example, it contained m < |u| negative generators, then we would have Δ^{m}w ∈ G^{+}. So, by property (i) of the ordering, we have w = u^{′−1}v′ with u′, v′ ∈ S^{∗}, |u′| = |u|, |v′| = |v|. So u′ =_{G} u and v′ =_{G} v by [8, Cor. 7.5].

It follows immediately from the definition of the left greedy normal form and from property (ii) of the ordering that v is the shortlex least representative of its group element, and hence v = v′. Let u′ = a′_{1}a′_{2} ⋯ a′_{m} with a′_{i} ∈ S. By [8, Lemma 8.4], if u′ is not in normal form then a′_{i}a′_{i+1} is not in normal form for some i. So, if the normal form word for this element is a_{i}a_{i+1}, then a′_{i} is a left divisor of a_{i}, and hence a_{i+1} is a right divisor of a′_{i+1}. But then, by (iii), a_{i+1}^{−1}a_{i}^{−1} < a_{i+1}^{′−1}a′_{i}, contradicting the shortlex minimality of u^{′−1}. So u′ is in normal form and hence u′ = u, and w = u^{−1}v as claimed. □

Let G be a Garside group with Garside generators S and Δ ∈ S. Conjugation by Δ permutes S, and we define τ : S → S by τ(x) = Δ^{−1}xΔ for x ∈ S. The left greedy normal form of an element g ∈ G (which is a different normal form from the one in the proof of Proposition 3.13, agreeing with that normal form only on elements of G^{+}) has the form Δ^{p}a_{1}a_{2} ⋯ a_{k}, where a_{1}a_{2} ⋯ a_{k} is a positive word in left greedy normal form with a_{i} ≠ Δ (see, for example, [3, 2.6] for details). So p is maximal such that Δ^{p} is a left divisor of g. Define inf(g) := p, and sup(g) := p + k.

The cycling and decycling operations from G to G are defined by

c(g) = Δ^{p}a_{2} ⋯ a_{k}τ^{−p}(a_{1}),
d(g) = Δ^{p}τ^{p}(a_{k})a_{1}a_{2} ⋯ a_{k−1}.

Note that these operations are equivalent to conjugation by τ^{−p}(a_{1}) and a_{k}^{−1}, respectively. The (de)cycled word is not necessarily in normal form, and it has to be put into normal form before the operation can be applied again. Note that cycling and decycling do not decrease inf(g), but they may increase it. Similarly they do not increase sup(g), but they may decrease it.

Let inf([g]) and sup([g]) denote respectively the largest value of inf(h) and the smallest value of sup(h) for an element h in the conjugacy class of g.

In [3, Thm. 1], it is proved that, for the braid group B_{n}, there is a fixed number K (equal to (n^{2} − n)/2 − 1 or n − 2, depending on which set of Garside generators of B_{n} is used) such that

(1) if inf([g]) > inf(g) then inf(c^{k}(g)) > inf(g) for some k ≤ K;

(2) if sup([g]) < sup(g) then sup(d^{k}(g)) < sup(g) for some k ≤ K.

A Garside group is called homogeneous if l(v) = l(u) for each of its defining relations u = v. This implies that the positive words in the group generators that represent an element g ∈ G^{+} all have the same length. The spherical type Artin groups are homogeneous Garside groups, but there are examples of inhomogeneous Garside groups.

It is observed in [10, Section 3.2] that the proof of [3, Thm. 1] works for all homogeneous Garside groups, but it relies heavily on elements of G^{+} having a welldefined length in G^{+}, so it does not appear to extend to general Garside groups. It is proved in [25, Props. 3.7, 3.10] that [3, Thm. 1] holds for general Garside groups, but without the bound on k. For the remainder of this section, we assume that G is homogeneous.

As observed above, G is automatic with a geodesic normal form on X := S ∪S^{−1}. It is pointed out in [3, proof of Corollary 3] that the geodesic length |g| of g over X is equal to max(sup(g), sup(g) − inf(g), − inf(g)); in fact |g| is equal to sup(g) when g is positive, − inf(g) when g is negative, and sup(g) − inf(g) otherwise.

So, if g is conjugate in G to a shorter element h, then at least one of inf(h) > inf(g) and sup(h) < sup(g) is true. In the first case, we are in case (1) above, and so for some k ≤ K, we have inf(c^{k}(g)) > inf(g). We can deduce that | c^{k}(g)| < |g| except possibly when g is positive and |g| = sup(g). But in this case, we must have sup(h) < |g| = sup(g), and so we are also in case (2), and can deduce that | d^{k′}(g)| = sup(d^{k′}(g)) < sup(g) = |g| for some k′ ≤ K. In the second case, we prove the identical result analogously.

Hence, since c^{k} and d^{k} correspond to conjugation by group elements of length k, we have the following.

Proposition 3.14. Let G be a homogeneous Garside group with Garside generators S. If w ∈ (S ∪ S^{−1})^{∗}, and there exists h ∈ G with |h^{−1}wh| < |w|, then there exists such an h of length at most K.

It is proved in [15] that G satisfies the FFTP over the Garside generators, and so Propositions 3.14 and 3.13 together with Corollary 3.8 implies the following.

Theorem 3.15. If G is a homogeneous Garside group with Garside generators S and X := S ∪ S^{−1}, then ConjGeo(G, X) is regular, and there is an ordering of X for which ConjMinLenSL(G, X) is regular. This holds, in particular, for spherical type Artin groups.

4. Shortlex languages and spherical growth series

In [27] and [26], Rivin proved that the spherical conjugacy growth series σ̃(G) is not rational for nonabelian free groups on their free generating sets, and made the following conjecture, one direction of which we shall prove in this section.

Conjecture 4.1. [27, Conjecture 13.1] Let G be a word hyperbolic group. Then σ̃(G) is rational if and only if G is virtually cyclic.

Theorem 4.2. Let G be a virtually cyclic group. Then for all generating sets of G the set of shortlex conjugacy normal forms ConjSL is regular and hence the spherical conjugacy growth series σ̃ is rational.

Proof. We may assume that G is infinite. Then there exists H ⊴ G, H = ⟨x⟩ ≅ ℤ, with G/H is finite. Let C := C_{G}(H) be the centralizer of H in G. Then the conjugation action of G on H defines a map G → Aut(ℤ) with kernel C and so |G : C| ≤ 2. For g ∈ G \ C, we have x^{−1}gx = gx^{2}, and hence the coset Hg is either a single conjugacy class in ⟨H, g⟩ or a union of two such classes, containing g and gx. So G \ C consists of finitely many conjugacy classes of G. On the other hand, for g ∈ C, |G : C_{G}(g)| is finite, so C is a union of infinitely many finite classes.

Since ConjSL ∩ (G \ C) is finite, it is regular, and to prove regularity of ConjSL it is enough to show that ConjSL ∩ C is regular.

Let T be a transversal of H in G. Then for each c ∈ C, the conjugacy class of c is {t^{−1}ct | t ∈ T }, and hence any word w with π(w) = c is in ConjSL if and only if there does not exist t ∈ T for which t^{−1}wt has a representative v with v <_{sl} w.

Now G is word hyperbolic, and we recall from the proof Theorem 3.1 that the set

L_{1}(t) := {(u, v) : u, v ∈ Geo, π(v) = π(t^{−1}ut)}

is regular for any t ∈ T, as is the set Geo. So ConjSL∩C is the intersection of π^{−1}(C) with

Geo \ ∪_{t∈T} ({u ∈ Geo : ∃v ∈ Geo such that (u, v) ∈ L_{1}(t), v <_{sl} u}).

Now standard arguments for regular sets show that this set regular, and |G : C| finite implies that π^{−1}(C) is regular, so ConjSL ∩ C is also regular. □

5. Behavior of conjugacy languages for virtually abelian groups

In this section we present two examples of virtually abelian groups and discuss their conjugacy languages. Our main goal is to demonstrate that regularity of the conjugacy languages can occur even when SL and Geo, or others among the conjugacy languages, are not regular, and to show dependence of regularity of the three conjugacy languages on the generating sets of the groups. However, we recall that we proved in Proposition 3.3 that any virtually abelian group has some generating set with respect to which both Geo and ConjGeo are regular and an ordering on that generating set for which both SL and ConjMinLenSL are regular.

We begin by considering an index 2 extension G of ℤ^{2}, where G is the semidirect product ℤ^{2} ⋊ ℤ/2ℤ, and the ℤ/2ℤ action swaps the generators of ℤ^{2}; that is,

G = ⟨a, b, t | t^{2} = 1, ab = ba, a^{t} = b⟩.

Cannon [24, p. 268] noted that the language of geodesics for G can be either regular or non-regular, depending on the generating set. In the following two propositions we explore Cannon’s generating sets in the case of conjugacy languages.

Proposition 5.1. Let G ≅ ℤ^{2} ⋊ ℤ/2ℤ be as defined above, with the generating set Z = {a^{±1}, b^{±1}, t}. Then ConjSL(G, Z) is not regular with respect to any ordering on Z, but the spherical conjugacy growth series is given by the rational function

σ̃ = \frac{(1 + z)(1 + 2z + 3z^{2} − z^{3} − z^{4})}{(1 − z^{2})^{2}}.

Moreover ConjGeo(G, Z) and ConjMinLenSL(G, Z) are regular.

Proof. Note that, since t = t^{−1}, the generating set Z is inverse-closed. With respect to any ordering on Z with a^{±1} < b^{±1} < t, the shortlex language is

SL(G, Z) = {a^{i}b^{j}t^{ϵ} | i, j ∈ ℤ, ϵ ∈ {0, 1}}.

Using this representation of the group elements, we compute the conjugacy classes as

[a^{i}b^{j}]_{c} = {a^{i}b^{j}, a^{j}b^{i}}, [a^{i}b^{j}t]_{c} = {a^{i+k}b^{j−k}t | k ∈ ℤ}.

Thus, using this ordering, the shortlex conjugacy language is the non-regular set

ConjSL(G, Z) = {a^{i}b^{j} | |i| > |j|, i, j ∈ ℤ} ∪ {a^{i}b^{i} | i ∈ ℤ} ∪ {a^{i}b^{−i} | i ∈ ℕ}
∪{a^{i}t | i ∈ ℤ},

For any ordering of Z, the intersection of ConjSL(G, Z) with the regular set {a, b}^{∗} is either {a^{i}b^{j} | i > j ≥ 0} or {b^{i}a^{j} | i > j ≥ 0}, and so ConjSL(G, Z) is never regular. However, the above expression for ConjSL gives

σ̃ = 1 + 3z + ∑_{k=1}^{∞}(4k + 3)z^{2k} + ∑_{k=1}^{∞}(4k + 4)z^{2k+1},

yielding the required rational function for σ̃.

Finally, applying the structure of the conjugacy classes again together with the geodesic language Geo(G, Z) = ⋃_{ζ,η∈{±1}}{a^{η}, b^{ζ}}^{∗} ∪ {a^{η}, b^{ζ}}^{∗}t{a^{ζ}, b^{η}}^{∗} yields the conjugacy geodesic and shortlex minimal conjugacy languages (where the latter is computed using the ordering at the start of this proof) as

ConjGeo(G, Z) = ⋃_{η,ζ∈{±1}}{a^{η}, b^{ζ}}^{∗} ∪ ⋃_{η∈{±1}}{a^{η}, b^{η}}^{∗}t{a^{η}, b^{η}}^{∗}

ConjMinLenSL(G, Z) = ⋃_{η,ζ∈{±1}}(a^{η})^{∗}(b^{ζ})^{∗} ∪ ⋃_{η∈{±1}}(a^{η})^{∗}(b^{η})^{∗}t.
□

The other generating set for G = ℤ^{2} ⋊ ℤ/2ℤ considered by Cannon in [24] is {a, c, d, t}^{±1} where c = a^{2} and d = ab, with presentation

G = ⟨a, c, d, t | ad = da, c = a^{2}, t^{2} = 1, tat = a^{−1}d⟩.

In the proof below we use these generators, including a few more details for Cannon’s proof that Geo is not regular.

We also require some more notation. Following [20], we define the insertion of w into z, denoted by z ← w, to be the set of all words of the form z′wz′′, where z = z′z′′. Given languages L_{1}, L_{2} ⊆ X^{∗}, we define also the insertion of L_{2} into L_{1},

L_{1} ← L_{2} := {z ← w | z ∈ L_{1}, w ∈ L_{2}}.

Observe that, if L_{1} and L_{2} are regular, then so is L_{1} ← L_{2}.

Proposition 5.2. Let G ≅ ℤ^{2} ⋊ ℤ/2ℤ be as defined above, with generating set X = {a^{±1}, c^{±1}, d^{±1}, t}. Then ConjGeo(G, X) is regular, but Geo(G, X) is not. Moreover, for some orderings of the generating set, ConjSL(G, X) and ConjMinLenSL(G, X) are regular, but SL(G, X) is not.

Proof. The group G = ℤ^{2} ⋊ ℤ/2ℤ can be embedded in Euclidean 3-space ℝ^{3} using the function f : G → ℝ^{3} defined by f(g) := (i_{g}, j_{g}, ϵ_{g}) where a^{i_{g}}b^{j_{g}}t^{ϵ_{g}} (with b = tat) is the shortlex normal form over Z for the element g of G. Multiplication by a^{±1}, c^{±1}, or d^{±1} has the effect of adding a horizontal vector (parallel to the z = 0 plane); in particular, for any g ∈ G and μ ∈ {1, −1}, we have

f(ga^{μ}) = f(g) + μ(1 − ϵ_{g}, ϵ_{g}, 0), f(gc^{μ}) = f(g) + 2μ(1 − ϵ_{g}, ϵ_{g}, 0),
f(gd^{μ}) = f(g) + μ(1, 1, 0).

On the other hand, multiplication by t adds a vertical vector:

f(gt) = f(g) + (1 − 2ϵ_{g})(0, 0, 1).

Using the taxicab metric on ℝ^{3} (defined by the norm ||(x, y, z)|| = |x| + |y| + |z|), we see that the maximum distance between f(g) and f(gx) due to multiplication by a single generator x ∈ X is 2.

For the remainder of the proof we order the generating set X by a < c < A < C < d < D < t, where for ease of notation we use a capital letter A to denote the inverse a^{−1}, etc.

First we show that the languages Geo(G, X) and SL(G, X) are not regular. Consider the element g := a^{2m}b^{2n} in G, where m, n ≥ 1, and let w be any word representing g. Then w must contain an even (possibly zero) number of occurrences of t, and, since the point f(g) = (2m, 2n, 0) has taxicab metric distance 2m + 2n from the identity of the group, and multiplication by t does not affect the the first two coordinates, we see that w must involve at least m + n letters from {a, c, d}^{±1}. If m ≥ n, then the word c^{m−n}d^{2n} of length m + n is a geodesic representative for g, and no geodesic representative can contain t. But if m < n and w does not contain t, then w must contain at least 2n occurrences of d since the y-coordinate of f(g) is 2n; then c^{m−n}d^{2n} is a shortest representative of g over {a, c, d}^{±1}. However, the shorter word c^{m}tc^{n}t also represents g, and is geodesic. Moreover, any word of length m+n+2 representing g must contain two occurrences of t and m+n occurrences of letters among c, d, with at most m occurrences of c before the first t, and so c^{m}tc^{n}t is the shortlex normal form for g. Hence

Geo(G, X) ∩ c^{∗}tc^{∗}t = SL(G, X) ∩ c^{∗}tc^{∗}t = {c^{m}tc^{n}t | m < n}.

Since this last set is not regular, but the set of regular languages is closed under intersection, we conclude that neither Geo(G, X) nor SL(G, X) is regular.

We shall now deduce the regularity of ConjGeo(G, X), ConjMinLenSL(G, X), and ConjSL(G, X) from the structure of the conjugacy classes for this group discussed in the proof of Proposition 5.1. (We note that ConjSL(G, X) and ConjMinLenSL(G, X) are also regular with various other orderings of the generators; we chose the ordering in this proof because it results in the least complicated description.)

We begin by computing the portions of these three languages from conjugacy classes of the form [a^{m}b^{n}]_{c} when 0 ≤ m, n; without loss of generality we assume m ≤ n. Using similar arguments to those above shows that geodesic representatives of the group element a^{n}b^{m} in this conjugacy class have length ⌈(m + n)/2⌉ and have the form (c^{∗}d^{∗})^{∗} for m + n even and (c^{∗}d^{∗})^{∗}a(c^{∗}d^{∗})^{∗} for m + n odd. However, the geodesic representatives of the other element a^{m}b^{n} of this conjugacy class are strictly longer; when n − m ≥ 3 they all involve two occurrences of t, and geodesic representatives of a^{m}b^{m+1} and a^{m}b^{m+2} are given by Ad^{m+1} and Cd^{m+2}, respectively. Thus the only element of [a^{m}b^{n}]_{c} with minimal length up to conjugacy is a^{n}b^{m}. A similar argument shows that the only group element in [A^{m}B^{n}]_{c} for 0 ≤ m ≤ n with minimal length up to conjugacy is A^{n}B^{m}. Thus when m and n are both nonnegative, the portion of the set ConjGeo(G, X) arising from the conjugacy classes [a^{m}b^{n}]_{c} and [A^{m}B^{n}]_{c} is

({c, d}^{∗} ← {ε, a}) ∪ ({C, D}^{∗} ← {ε, A})

and the portion of both ConjMinLenSL(G, X) and ConjSL(G, X) is

{ε, a}c^{∗}d^{∗} ∪ {ε, A}C^{∗}D^{∗}.

It is more complicated to describe the contributions to the three conjugacy languages from [a^{m}B^{n}]_{c} in the case that 0 < m, n, and we shall do this only briefly. Note first that the sets of geodesic representatives of b^{k} for k ≥ 1 are: {Ad, dA} for k = 1; {Cd^{2}, dCd, d^{2}C, tct} for k = 2; {tc^{k/2}t} for k ≥ 4 even; and t(c^{(k−1)/2} ← a)t for k ≥ 3 odd.

In the case that 3 ≤ m, n, geodesics representing both elements a^{m}B^{n} and A^{n}b^{m} of this class require two occurrences of the letter t, and both elements are of minimal length up to conjugacy. The contribution to ConjGeo(G, X) from these conjugacy classes is the regular set L_{1} ∪ L_{2} ∪ L_{3} ∪ L_{1}^{−} ∪ L_{2}^{−} ∪ L_{3}^{−}, where

L_{1} = [c^{+} ← {a, c}] ← [t(C^{+} ← {A, C})t],
L_{2} = (c^{+} ← tC^{2}C^{∗}t) ← d,
L_{3} = (c^{2}c^{∗} ← tC^{+}t) ← D.

and L_{1}^{−}, L_{2}^{−}, L_{3}^{−} are defined similarly to L_{1}, L_{2}, L_{3}, but with all generators replaced by their inverses. The words in ConjMinLenSL(G, X) are the shortlex normal forms of all of the elements in the classes [a^{m}B^{n}]_{c} with 3 ≤ m, n, which we collect in the regular expression

{c^{2}, ac}c^{∗}t{C^{2}, AC}C^{∗}t ∪ {C^{2}, AC}C^{∗}t{c^{2}, ac}c^{∗}t.

Similarly, the intersection of this last set with ConjSL(G, X) is given by the regular expression {c^{2}, ac}c^{∗}t{C^{2}, AC}C^{∗}t.

Suppose, on the other hand, that either m ∈ {1, 2} and n ≥ 3, or m ≥ 3 and n ∈ {1, 2}. Then the conjugacy class [a^{m}B^{n}]_{c} contains a unique shortest group element, namely A^{n}b^{m} when m ≤ 2 or a^{m}B^{n} when n ≤ 2. The words of ConjGeo(G, X) from classes of this type are in the regular set L_{4} ∪ L_{5} ∪ L_{4}^{−} ∪ L_{5}^{−}, where

L_{4} = (c^{2}c^{∗} ← D) ← {ε, a},
L_{5} = ((c^{2}c^{∗} ← D) ← D) ← {a, c}.

and L_{4}^{−}, L_{5}^{−} are similarly defined in terms of A, C, d. The portion of both ConjMinLenSL(G, X) and ConjSL(G, X) from these classes is given by the regular expression

{a, ε}c^{2}c^{∗}D ∪ {a, c}c^{2}c^{∗}D^{2} ∪ {A, ε}C^{2}C^{∗}d ∪ {A, C}C^{2}C^{∗}d^{2}.

The finitely many classes [a^{m}B^{n}]_{c} when both m and n are at most 2 give rise to finite subsets of the three conjugacy languages.

Finally, we consider the conjugacy classes [a^{s}t]_{c} for s ∈ ℤ. Recall that this class contains all group elements of the form a^{m}b^{n}t for m, n ∈ ℤ and m + n = s. Suppose first that s > 0. Geodesic representatives of these elements all contain a single t.

Geodesic representatives of a^{m}b^{n}t have length ⌈s/2⌉ + 1 when m, n ≥ 0, but are longer otherwise. Hence ConjGeo(G, X) consists of the geodesic representatives of a^{m}b^{n}t with m, n ≥ 0. The case in which s < 0 is similar. Thus the contribution to ConjGeo(G, X) from classes [a^{s}t]_{c} is

({c, d}^{∗}t{c, d}^{∗} ← {ε, a}) ∪ ({C, D}^{∗}t{C, D}^{∗} ← {ε, A}).

The subsets of ConjMinLenSL(G, X) and ConjSL(G, X) arising from these conjugacy classes are defined by regular expressions

{ε, a}c^{∗}{ε, d}tc^{∗} ∪ tac^{∗} ∪ {ε, A}C^{∗}{ε, D}tC^{∗} ∪ tAC^{∗} and {ε, a}c^{∗}t ∪ {ε, A}C^{∗}t,

respectively. □

In the next two propositions we change the virtually abelian group under consideration to a semidirect product group K = ℤ^{2} ⋊ D_{8}. One of the generating transpositions of D_{8} acts by swapping the generators of ℤ^{2}, and the other one fixes the generators; that is,

K = ⟨a, b, t, u | t^{2} = u^{2} = (tu)^{4} = 1, ab = ba, a^{t} = b, a^{u} = a, a^{u} = b⟩.

The index 4 subgroup ⟨a, b, t⟩ of K is the group G considered in Propositions 5.1 and 5.2.

Proposition 5.3. Let K ≅ ℤ^{2} ⋊ D_{8} be as defined above, where the generators Z′ = {a^{±1}, b^{±1}, t, u} are ordered with a^{±1} < b^{±1} < t < u. Then Geo(K, Z′), SL(K, Z′), ConjGeo(K, Z′), and ConjMinLenSL(K, Z′) are regular languages, but ConjSL(G, Z′) is not.

Proof. The shortlex language is

SL(K, Z′) = {a^{i}b^{j}v | i, j ∈ ℤ, v ∈ {1, t, u, tu, ut, tut, utu, tutu}}

and the geodesic language Geo(K, Z′) is ⋃_{η,ζ∈{±1}}(L_{1}^{η,ζ} ∪ L_{3}^{η} ∪ L_{2}^{η,ζ}), where

L_{1}^{η,ζ} = {a^{η}, b^{ζ}}^{∗} ← {ε, u},
L_{2}^{η,ζ} = ({a^{η}, b^{ζ}}^{∗} ← {ε, u}) ← t({a^{ζ}, b^{η}}^{∗} ← u)t,
L_{3}^{η,ζ} = {a^{η}, b^{ζ}}^{∗}t{a^{ζ}, b^{η}}^{∗} ← {ε, u}.

By successively conjugating shortlex normal forms by generators, we find that the conjugacy classes are given by

[a^{i}b^{j}]_{c} = {a^{i}b^{j}, a^{j}b^{i}}, [a^{i}b^{j}t]_{c} = [a^{i}b^{j}utu]_{c} = {a^{i+k}b^{j−k}v | k ∈ ℤ, v ∈ {t, utu}},
[a^{i}b^{j}u]_{c} = [a^{j}b^{i}tut]_{c} = {a^{i}b^{j}u, a^{j}b^{i}tut}, [a^{i}b^{j}tutu]_{c} = {a^{i}b^{j}tutu, a^{j}b^{i}tutu},
[a^{i}b^{j}tu]_{c} = [a^{i}b^{j}ut]_{c} = {a^{i+k}b^{j−k}v | k ∈ ℤ, v ∈ {tu, ut}}.

As a consequence,

ConjGeo(K, Z′) = ⋃_{η,ζ∈{±1}}(L_{1}^{η,ζ} ∪ L′_{2}^{η,ζ}) ∪ ( ⋃_{η∈{±1}} L′_{3}^{η}), where

L′_{2}^{η,ζ} = ({a^{η}, b^{ζ}}^{∗} ← u) ← t({a^{ζ}, b^{η}}^{∗} ← u)t,
L′_{3}^{η} = ({a^{η}, b^{η}}^{∗} ← t) ← {1, u},

and

ConjMinLenSL(K, Z′) = ⋃_{η,ζ∈{±1}}(a^{η})^{∗}(b^{ζ})^{∗}{1, u, tutu} ∪ ⋃_{η∈{±1}}{a^{η})^{∗}(b^{η})^{∗}{t, tu, ut}.

However, just as for the finite index subgroup G of K in Proposition 5.1, the intersection ConjSL(K, Z′) ∩ a^{∗}b^{∗} = {a^{i}b^{j} | i > j} is not regular, and so this shortlex conjugacy language is not regular. □

Proposition 5.4. Let

K = ⟨a, c, d, t, u |

t^{2} = u^{2} = (tu)^{4} = 1, ad = da, c = a^{2}, tat = a^{−1}d, uau = a, udu = d⟩

≅ ℤ^{2} ⋊ D_{8},

with generating set X′ = {a^{±1}, c^{±1}, d^{±1}, t, u}. Then Geo(K, X′) and ConjGeo(K, X′) are not regular. Moreover, if the set X′ is ordered by a < c < a^{−1} < c^{−1} < d < d^{−1} < t < u, then none of the languages SL(K, X′), ConjMinLenSL(K, X′), and ConjSL(K, X′) is regular.

Proof. In order to analyze geodesic representatives for elements of K over X′, we use similar arguments to those in the proof of Proposition 5.2, in that we use an embedding f : K → ℝ^{2} × D_{8} with the map f(g) := (i_{g}, j_{g}, h_{g}), where g = a^{i_{g}}b^{j_{g}}h_{g} with b = a^{t} and h_{g} ∈ ⟨t, u⟩ ≅ D_{8}. Geodesic representatives over X′ of the elements of the subgroup ℤ^{2} ⋊ ℤ/2ℤ = ⟨X⟩ of K where X = {a^{±1}, c^{±1}, d^{±1}, t} cannot contain an occurrence of the letter u, and so the proof of Proposition 5.2 also shows that Geo(K, X′) and SL(K, X′) are not regular.

To show further that ConjGeo(K, X′) and ConjMinLenSL(K, X′) are not regular, consider the word w_{mn} := c^{m}tc^{n}tu for m, n ≥ 0.

When n ≤ m, then (as in the proof of Proposition 5.2) w_{mn} =_{K} c^{m−n}d^{2n}u, and so w_{mn} ∉ Geo(K, X′).

When m < n, we claim that w_{mn} ∈ ConjMinLenSL(K, X′); this will imply that the intersections of each of ConjGeo(K, X′) and ConjMinLenSL(K, X′) with the regular language c^{∗}tc^{∗}tu are not regular; hence neither conjugacy language is regular.

The conjugacy class of w_{mn} is [w_{mn}]_{c} = [a^{2m}b^{2n}u]_{c} = {a^{2m}b^{2n}u, a^{2n}b^{2m}tut}, from the proof of Proposition 5.3. Using the embedding above, we have f(a^{2n}b^{2m}tut) = (2n, 2m, tut). Since we need at least m + n letters in {a, c, d}^{±1} to reach this point, and tut is a geodesic word in ⟨t, u⟩ ≅ D_{8}, every geodesic representative of a^{2n}b^{2m}tut has length at least m + n + 3.

On the other hand, f(w_{mn}) = f(a^{2m}b^{2n}u) = (2m, 2n, u). As in the proof of Proposition 5.2, in order to reach a point in ℝ^{2} × D_{8} with first two coordinates (2m, 2n) we need at least m + n occurrences of letters in {a, c, d}^{±1}. Suppose for a contradiction that v is a word with v =_{K} w_{mn} and l(v) < l(w_{mn}). So l(v) ≤ m+n+2 and v involves at least m + n letters in {a, c, d}^{±1}. Since v is not in the normal subgroup ⟨a, c, d, t, (tu)^{2}⟩ of index 2 in K, the number of occurrences of u in v must be odd, and hence must be exactly 1. We see similarly that the number of occurrences of t is even, so it must be 0. So deleting the single occurrence of u from v would give a word over {a, c, d}^{±1} of length at most m+n+1 for a^{2m}b^{2n} which, as we saw in Proposition 5.2, does not exist when m < n. Thus the element a^{2m}b^{2n}u of [w_{mn}]_{c} is of minimal length up to conjugacy, and the word w_{mn} representing this element lies in ConjGeo(K, X′). All geodesic representatives of this element lie in {c, d, t}^{∗} ← u, and among these the word with the longest initial prefix in c^{∗} is w_{mn}. Therefore w_{mn} ∈ ConjMinLenSL(K, X′) as well, finishing the claim.

In order to prove that ConjSL(K, X′) is not regular, we consider the word v_{mn} := c^{m}tc^{n}utu for m, n ≥ 0. The corresponding conjugacy class from the proof of Proposition 5.3 is [v_{mn}]_{c} = {a^{2m}b^{2n}tutu, a^{2n}b^{2m}tutu}. The words c^{m}tc^{n}utu and c^{n}tc^{m}utu are the shortlex least words representing the elements a^{2m}b^{2n}tutu and a^{2n}b^{2m}tutu, respectively. Thus the element of ConjSL(K, X′) corresponding to this conjugacy class is the representative with the longest initial string of the letter c. That is,

ConjSL(K, X′) ∩ c^{∗}tc^{∗}utu = {c^{m}tc^{n}utu | m ≥ n},

which is not regular. □

Acknowledgments

The first two authors were partially supported by the Marie Curie Reintegration Grant 230889, and by scheme 2 grants from the London Mathematical Society. The first named author was also supported by the Swiss National Science Foundation grants Ambizione PZ00P-136897/1 and Professorship FN PP00P2-144681/1. The second author also acknowledges partial support by grants from the Simons Foundation (#245625) and the National Science Foundation (DMS-1313559).

References

[1] Antolín, Y. and Ciobanu, L., On Cayley graphs of relatively hyperbolic groups, Preprint 2014.

[2] Appel, K.I. and Schupp, P.E., Artin groups and infinite Coxeter groups, Invent. Math. 72 (1983), 201–220.

[3] Birman, J.S., Ko K.H. and Lee S.J., The infimum, supremum, and geodesic length of a braid conjugacy class, Adv. Math. 164 (2001), 41–56.

[4] Bridson, M. and Haefliger, A. Metric Spaces of Non-Positive Curvature, Grundlehren der Mathematischen Wissenschaften 319 Springer-Verlag, Berlin, 1999.

[5] Brzozowski, J.A. and Simon, I., Characterizations of locally testable events, Discrete Math. 4 (1973), 243–271.

[6] Charney, R. and Meier, J., The language of geodesics for Garside groups, Math. Zeitschrift 248 (2004), 495–509.

[7] Ciobanu, L. and Hermiller, S., Conjugacy growth series and languages in groups, Trans. Amer. Math. Soc., to appear; arXiv:1205.3857.

[8] Dehornoy, P. and Paris, L., Gaussian groups and Garside groups, two generalizations of Artin groups, Proc. London Math. Soc. 79 (1999), 569–604.

[9] Epstein, D.B.A., Cannon, J., Holt, D., Levy, S., Paterson, M., and Thurston, W., Word Processing in Groups, Jones and Bartlett, Boston, 1992.

[10] Franco, N. and Gonzalez-Meneses, J., Conjugacy problem for braid groups and Garside groups, J. Algebra 266 (2003), 112-132.

[11] Gersten, S.M. and Short, H. Small cancellation theory and automatic groups, II, Invent. Math. 105 (1991), 641–662.

[12] Hermiller, S., Holt, D.F. and Rees, S. Star-free geodesic languages for groups, Internat. J. Algebra Comput. 17 (2007), 329–345.

[13] Hermiller, S., Holt, D.F. and Rees, S. Groups whose geodesics are locally testable, Internat. J. Algebra Comput. 18 (2008), 911–923.

[14] Hermiller, S. and Meier, J., Algorithms and geometry for graph products of groups, J. Algebra 171 (1995), 230–257.

[15] Holt, D.F., Garside groups have the falsification by fellow-traveller property, Groups Geom. Dyn. 4 (2010), 777–784.

[16] Holt, D.F. and Rees, S., Artin groups of large type are shortlex automatic with regular geodesics, Proc. Lond. Math. Soc. 104 (2012), 486–512.

[17] Holt, D.F. and Rees, S., Conjugacy in Artin groups of extra-large type, submitted, arXiv:1309.4341.

[18] Howlett, R.B., Miscellaneous facts about Coxeter groups, notes of lectures given at the ANU Group Actions Workshop, October 1993, available from http://www.maths.usyd.edu.au:8000/res/Algebra/How/anucox.html.

[19] Hopcroft, J. and Ullman, J.D., Introduction to automata theory, languages, and computation, Addison-Wesley Series in Computer Science, Addison-Wesley Publishing Co., Reading, Mass., 1979.

[20] L. Kari, On insertion and deletion of formal languages, Ph.D. thesis, University of Turku, 1991.

[21] Loeffler, J., Meier, J., and Worthington, J., Graph products and Cannon pairs, Internat. J. Algebra Comput. 12 (2002), 747–754.

[22] Mairesse, J. and Mathéus, F., Growth series for Artin groups of dihedral type, Internat. J. Algebra Comput. 16 (2006), 1087–1107.

[23] Miller, C.F., III, On group-theoretic decision problems and their classification, Annals of Mathematics Studies 68, Princeton University Press, Princeton, N.J., 1971.

[24] Neumann, W. and Shapiro, M., Automatic structures, rational growth, and geometrically finite hyperbolic groups, Invent. Math. 120 (1995), 259–287.

[25] Picantin, M., The conjugacy problem in small Gaussian groups, Comm. Algebra 29 (2006), 1021–1039.

[26] Rivin, I., Some properties of the conjugacy class growth function, Contemp. Math. 360, Amer. Math. Soc., Providence, RI, (2004), 113 –117.

[27] Rivin, I., Growth in free groups (and other stories) - twelve years later, Illinois J. Math. 54 (2010), 327–370.

L. Ciobanu, Mathematics Department, University of Neuchâtel, Rue Emile-Argand 11, CH-2000 Neuchâtel, Switzerland

E-mail address: laura.ciobanu@unine.ch

S. Hermiller, Department of Mathematics, University of Nebraska, Lincoln, NE 68588-0130, USA

E-mail address: smh@math.unl.edu

D. Holt, Mathematics Institute, Zeeman Building, University of Warwick, Coventry CV4 7AL, UK

E-mail address: D.F.Holt@warwick.ac.uk

S. Rees, Department of Mathematics, University of Newcastle, Newcastle NE1 7RU, UK

E-mail address: Sarah.Rees@ncl.ac.uk