Title:The automorphism group of Cayley graphs on symmetric groups generated by transposition sets and of the modified bubble-sort graph

Abstract:In this paper, we first determine the full automorphism group of the modified bubble-sort graph MBS(n), for all $n \ge 4$, and we also investigate the normality of these Cayley graphs. We prove that $\Aut(MBS(n))$ is isomorphic to $D_{2n} V_4 R(S_n)$ if $n=4$, and is isomorphic to $D_{2n} R(S_n)$ if $n \ge 5$. Here, $D_{2n}$ is the dihedral group of order $2n$, $V_4$ is the Klein four-group $Z_2 \times Z_2$, and $R(S_n)$ is the right regular representation of $S_n$. We also show that $R(S_4)$ is not a normal subgroup of $\Aut(MBS(4))$, implying that MBS(4) is not normal. Thus, the full automorphism group of MBS(4) is larger than the group $R(S_4) \rtimes \Aut(S_4,S) \cong D_{2n} R(S_n)$ given in the literature as the full automorphism group ($\rtimes$ denotes the semidirect product).
For a set $S$ of transpositions that generates the symmetric group $S_n$, let $T(S)$ be the transposition graph whose vertex set is $\{1,...,n\}$ and with edge set $\{ij: (i,j) \in S\}$. We show that if $t,k \in S$ is such that $tk \ne kt$ and $T(S)$ does not contain triangles or 4-cycles, then the Cayley graph $\Cay(S_n,S)$ contains a unique 6-cycle passing through $e,t, k$ and a vertex at distance 3 from $e$. Thus, we generalize the results in the literature from transposition graphs that are trees to arbitrary connected transposition graphs that do not contain triangles or 4-cycles; furthermore, a converse is seen to also be true.
As a consequence, if the transposition graph $T(S)$ does not contain triangles or 4-cycles, then $\Cay(S_n,S)$ is normal and the automorphism group $\Aut(\Cay(S_n,S))$ is isomorphic to $R(S_n) \rtimes \Aut(S_n,S)$.