Over the past three decades, the low temperature properties of low-dimensional quantum systems have been studied because of the exotic spin states that can arise from quantum fluctuations. The quantum antiferromagnetic Heisenberg (QAFH) model on a triangular lattice is a typical quantum frustrated system. This involves a generalized model with an antiferromagnetic nearest-neighbor (NN) interaction and a next-nearest-neighbor (NNN) interaction , the model Hamiltonian of which given by
where is the component ( ) of the quantum spin at lattice site i, is an exchange anisotropy, and the sums and run over all NN and NNN pairs of sites, respectively. Hereafter we set as the unit of energy scaling. The ground state (GS) of an isotropic QAFH model ( ) with is the central issue of the model. Anderson proposed a resonating-valence-bond state or a spin liquid (SL) state as the GS of the model  . Since then, many authors have used various methods to study the model  -  . The GS of the model is now widely believed to have the classical long-range-order (LRO) in the form of the 3-sublattice structure (120˚ Néel state). However recent experiments on model compounds such as κ-(ET)2Cu2(CN)3  , EtMe3Sb[Pd(dmit)2]2  , and Ba3IrTi2O9  have observed no LRO down to very low temperatures. Motivated by this discrepancy, the present authors  (hereafter referred to as SMFS) reexamined the GS of an anisotropic QAFH model with on finite lattices having used an exact diagonalization technique, and found that the classical LRO is absent for . This includes the GS of the isotropic model ( ) with , i.e., it has no LRO and is the SL state.
In the present paper, we consider the effect of the NNN interaction on the anisotropic QAFH model. The isotropic QAFH model with was studied recently using approximations such as a variational Monte Carlo (VMC) method  , a many-variable VMC (mVMC) method  , a coupled cluster method (CCM)  , and a density matrix renormalization group method   . These approaches showed that the 120˚ Néel state occurs for and that a four-sublattice antiferromagnetic LRO state (stripe Néel state) occurs for , i.e., the SL state appears between them, . The thresholds have been estimated as and     , although the mVMC method suggested a smaller region of the SL state, i.e., and  . However, an exact result for finite lattices by SMFS  suggested . No studies have been carried out on and using the exact diagonalization technique. We therefore apply the same method used by SMFS to the model extended with . We calculate exactly the squared sublattice magnetization of finite lattices, and we estimate and using their Binder ratios over a wide range of and try to draw the phase diagram in theα-J2 plane.
In Section 2, we present our method with the finite lattices. In Section 3, we estimate the threshold between the 120˚ Néel state and the SL state. In Section 4, we consider the stripe Néel state and its threshold with the SL state. In Section 5, we propose a phase diagram of the model.
It is known that the GS of the classical model is a 120˚ Néel state when , and the stripe Néel state when , where for and tends to zero as . The unit cells of these states are shown in Figure 1. Although a similar LRO may be expected to appear in the anisotropic QAFH model, the classical ordered state is not a good quantum state. Therefore a remarkable difference may exist in the phase transition between the classical and quantum models.
We first consider this problem. Hereafter we refer to the spin space of a lattice with sites with three-sublattice symmetry as the three-sublattice space (3SLS), where n is a natural number. Similarly, we refer to the spin space of a lattice with sites with four-sublattice symmetry as the four-sublattice space (4SLS). The minimum energies per site in the 3SLS and 4SLS are labeled as and , respectively. The spin state is in the 3SLS when and in the 4SLS when . In the classical model, the threshold of is one at which the spin space changes from one to the other, and the phase transition at is of the first order. In the quantum model, although the spin space changes at some threshold , no phase transition will take place at , because there would be no LRO in those spin spaces at . We must then consider the thresholds and natures of the phase transitions in the 3SLS and in the 4SLS, separately.
For the 3SLS, we consider the lattices with (and partly ) sites with periodic boundary conditions suitable for the three-sublattice structure (Figure 2(a)).
For the 4SLS, we consider the lattices with N = 24, 28, and 32 sites with periodic boundary conditions suitable for the stripe structure (Figure 2(b)).
Figure 1. (a) Sublattices A, B, and C in the three-sublattice state; (b) Sublattices A, B, C, and D in the four-sublattice state.
Figure 2. (a) The lattices with three-sublattice symmetry (3SLS). The lattices of N = 21, 27, and 36 appear in Ref.  ; (b) The lattices with four-sublattice symmetry (4SLS).
In either case, we obtain the GS eigenfunction of the N sites using the Lanczos method, where s = tri or str for the 3SLS or 4SLS, respectively. The component of the magnetization on the sublattice is defined as
where and , and C for the 3SLS, and and , and D for the 4SLS. The operators of the z, xy, and xyz components of the squared sublattice magnetization are defined as
We calculate the component the squared sublattice magnetization, , as
where , xy, or xyz.
We study the Binder ratios  that are used by SMFS  to estimate the threshold of the model with . At the critical point, the Binder ratio is size invariant. If there is a LRO, the Binder ratio is expected to increase with the system size. In contrast, in the paramagnetic or SL state, the Binder ratio decreases with the system size. This means that the size dependence of the Binder ratio is different from each other with and without a LRO. The z, xy, and xyz components of the Binder ratio can be defined as
Before estimating and , we should examine that no phase transition will take place at . Figure 3 shows and together with and for the case of . As mentioned above, the spin space changes at , whereas no signal of a change in the magnetic state is seen at this point. We consider the 3SLS for and the 4SLS for . A remarkable point is that ( ) changes markedly around , at which ( ) has its maximum value. In the 3SLS, a bending of accompanied by a discontinuous drop of indicates exchange in the GS between the lowest and next-lowest energy eigenstates at . However, we reason that this says nothing about the
Figure 3. The GS energies and the squared sublattice magnetizations of the 3SLS ( ) and the 4SLS ( ). The data of the 4SLS for are averages of those of and . The solid and open symbols are and , respectively. An arrow represents the positions of .
phase transition between the 120˚ Néel state and the SL state because . The phase boundary should be estimated by a different method. In contrast, we expect to be near , because . In Section 4, we consider together with the Binder ratio in order to estimate .
3. Three-Sublattice Néel State
In this section, we estimate the threshold . We consider in the GS of the 3SLS. Special attention should be paid to the subspace in which the GS belongs. For , the GS is in the minimum subspace. For , however, the GS may not be restricted to the minimum subspace depending on . We then consider the cases and separately.
3.1. XY-Like Case (α ≤ 1)
For , the LRO has the 120˚ Néel state symmetry, and and are calculated for various . For , because the spins lie in the xy plane, we consider the component, whereas the component for . In Figures 4(a)-(d), we present for , 0.4, 0.8, and 1.0 as functions of , respectively. As is decreased, increases revealing the development of the 120˚ spin correlation. For small , the finite-size effect (FSE) for is rather weak implying the occurrence of the 120˚ Néel state. As is increased, the FSE becomes stronger. In the isotropic case of , we can see a strong FSE even for
Figure 4. The squared three-sublattice magnetizations (a)-(c) and (d) as functions of . Note that for , .
We consider the Binder ratio   . In Figures 5(a)-(d), we show for different even N as functions of for , 0.4, 0.8, and 1.0, respectively. For a large , decreases with increasing N, which reveals that LRO is absent. As is decreased, increases and the values for different N are close to each other. For , for different N intersect at almost the same (see Figure 5(a) and Figure 5(b)). That is, and . As is increased, the intersection points scatter, as seen in Figure 5(c). In this case, we consider a lower bound and a upper bound of according to the hypotheses described in Sec. II: the LRO is present when the Binder ratio increases with N, whereas it is absent when decreases with increasing N. We evaluate and , and tentatively estimate as their average value. In this way, we get the thresholds as , , and .
In the case of , exhibits a somewhat different behavior from those for . Although increases with decreasing , its increment
Figure 5. Binder ratios (a)-(c) and (d) of the squared 3-sublattice magnetization as functions of .
depends only very weakly on N, especially for . We could see no definite intersection point of for down to , i.e., we could not evaluate . Therefore we believe , because .
3.2. Ising-Like Case (α > 1)
For , we are interested in and because a distorted 120˚ Néel state occurs in the classical model. We obtain the eigenfunction with the minimum energy for each subspaces. Note that we consider only the subspaces of because the GS is in the subspace for . The GS eigenfunction of the system is one which gives the lowest value among ’s. When , the GS eigenfunction is . As is decreased, successively changes to at , respectively. Using , we obtain and for various . A typical result of these is shown in Figure 6 for . For , both and exhibit a strong N dependence which reveals that for . As is decreased, and show different behaviors from each other. For , they remain almost constant down to and drop at . For the whole range of , we see a strong N dependence, that suggests that for . In contrast, gradually increases down to and discontinuously jumps at . It exhibits its own N dependence in different ranges of . For 1) , is almost independent of N. We believe that the classical ferrimagnetic state arises in this range, because and . For 2) , increases with N revealing the occurrence of the LRO of the z component of the spin. However, for 3) , slightly decreases with increasing N. Note that we also obtain the similar results for and 2.5.
Figure 6. Squared three-sublattice magnetizations (solid symbols) and (open symbols) as functions of . Jumps in those quantities occur at from the right.
Figure 7. Binder ratios as a functions of for (a) and (b) .
of for and , respectively. For (a) , we evaluate the lower and upper bounds of as and , i.e., . For (b) , we estimate and , i.e., . Note that we also examined to confirm the speculation given above and found that, in fact, decreases with increasing N for the whole range of .
To close this subsection, we emphasize that the distorted 120˚ Néel state is absent in the QAFH model, in contrast to the classical model. We find that, when , the LRO of the z component of the spin occurs. A question remains as to what the value of for , . If , the LRO of the z component of the spin occurs for . If not, two possibilities exist in the range : either there is still the LRO, or the system is in a critical state that is similar to the spin state of the Ising model with . Further studies are necessary to answer this question.
4. Stripe State
In this section, we consider the stripe state. We obtain the GS as the eigenfunction with energy because the stripe state belongs to the subspace. In Figure 8(a) and Figure 8(b), we show these quantities as functions of for and , respectively. We readily see that the results for the system are quantitatively different from those of the and 32 systems, which lead us to consider mainly data for in order to evaluate . We see similar properties in the cases of and . The magnetization rapidly increases around the peak posion of that implies the occurrence of the phase transition. When , is small and its N dependence is strong which reveals that the stripe state is absent. When , is large, although its N dependence is still considerable especially for . Then we examine the
Figure 8. The GS energies and four-sublattice magnetizations as functions of .
Figure 9. Binder ratios as functions of .
Binder ratio . In Figure 9(a) and Figure 9(b), we plot and as functions of for and , respectively. For , when , increases with N, suggesting the presence of the stripe state, i.e., . In contrast, the lower bound of is evaluated from of the system because increases slightly with N. Therefore we estimate . Similarly, we estimate .
We have also examined for . We may evaluate the lower bound of . However, we could not evaluate because even up to  .
We have studied the anisotropic antiferromagnetic model with nearest-neighbor (J1) and next-nearest-neighbor (J2) interactions on a triangular lattice using the exact diagonalization method. We have obtained the ground-state energy and the sublattice magnetizations for systems of different size N. We have examined Binder ratios to investigate the stability of the long-range order of the system. The N-dependences of Binder ratios suggest the threshold between the three-sublattice Néel state and the disordered state, i.e., the spin liquid (SL) state, and the threshold between the stripe state and the SL state. The results are summarized in the phase diagram shown in Figure 10. For , the classical 120˚ phase or the stripe phase occurs in the xy plane. For , the xy component of the sublattice magnetization vanishes, i.e., the distorted 120˚ state is replaced by the collinear (up up down) antiferromagnetic state because of quantum fluctuations.
We have suggested that the SL state exists over a wide range in theα-J2 plane in contrast with recent approximation studies     which give and . The discrepancy will come from either the finite-size effect or approximations. Further studies are necessary to
Figure 10. The α-J2 phase diagram of the J1-J2 anisotropic Heisenberg model on a triangular lattice. Cross symbols are those estimated in SMFS  .
establish the thresholds for .
Some of the results in this research were obtained using the supercomputing resources at the Cyberscience Center of Tohoku University.