The Selberg trace formula, introduced by A. Selberg in 1956, describes the spectrum of the hyperbolic Laplacian in terms of geometric data involving the lengths of geodesics on a Riemann surface. Motivated by analogy between this trace formula and the explicit formulas of number theory relating the zeroes of the Riemann zeta function to prime numbers, Selberg  introduced a zeta function whose analytic properties are encoded in the Selberg trace formula. By focusing on the Selberg zeta function, H. Huber (  , p. 386;  , p. 464), proved an analogue of the prime number theorem for compact Rie-
mann surfaces with the error term that agrees with Selberg’s one.
Using basically the same method as in  , D. Hejhal (  , p. 475), established also the prime geodesic theorem for non-compact Riemann surfaces with the remainder
. However, in the compact case there exist several different proofs (see,
B. Randol  , p. 245; P. Buser  , p. 257, Th. 9.6.1; M. Avdispahić and L. Smajlović
 , Th. 3.1) that give the remainder. Thanks to new integral repre-
sentations of the logarithmic derivative of the Selberg zeta function (cf.  , p. 185;  , p. 128), M. Avdispahić and L. Smajlović (  , p. 13) were in position to improve
error term in a non-compact, finite volume case up to.
Whereas the authors in  and  approached the prime number theorem in various settings via explicit formulas for the Jorgenson-Lang fundamental class of functions, our main goal is to obtain this improvement for non-compact Riemann surfaces with cusps following a more direct method of B. Randol  .
Let X be a non-compact Riemann surface regarded as a quotient of the upper half-plane by a finitely-generated Fuchsian group of the first kind, containing cusps. Let denote the fundamental region of. We shall assume that the fundamental region of has a finite non-Euclidean area. We put
and denote by v the multiplier system of the weight for. Let be an irreducible unitary representation on and,. For an r dimensional vector space V over we consider an essentially self-adjoint operator
on the space of all twice continuously differentiable functions, such that f and are square integrable on, and satisfy the equality
The operator has the unique self-adjoint extension to the space, a dense subspace of. Let, be the set of parabolic transformations corresponding to cusps of. does not depend on the choice of a representative of the parabolic class and can be considered as a matrix from. By we will denote the multiplicity of 1 as an eigen-value of the matrix
, and will be the degree of singularity of W. We mention that oper-
ator has both the discrete and continuous spectrum in the case, and only the discrete spectrum in the case. The discrete spectrum will be denoted as (). The continuous spectrum is expressed through zeros (or equivalently poles) of the hyperbolic scattering determinant (see,  ).
3. Selberg Zeta Function
Let denotes the set of -conjugacy classes of a primitive hyperbolic element in, and denotes the set of -conjugacy classes of a hyperbolic element P in that satisfy property. Assume that. We define the Selberg zeta function associated to the pair () by
is absolutely convergent for. Analytic considerations given in (  , pp. 499-501) yield that the Selberg zeta function in this setting satisfies the functional equation
with the fudge factor
Here, denotes the hyperbolic scattering determinant. It can be represented in the form
where the coefficients and depend on the group (see,  , p. 437). Here, denotes the degree of singularity of W (see Section 2). An explicit expression for the fudge factor in the Equation (1) is given in (  , p. 501, Equation (5.10)).
The logarithmic derivative of the Selberg zeta function is given by
where denotes the norm of the class P and for a primi-
tive element such that for some. We will omit the indices in in the sequel.
4. Counting Functions
Lemma 1. For,
where for a primitive element such that for some.
We shall spend the rest of this section to derive a representation of in the form (11) bellow. We choose not to write it in a separate statement because of the length of expressions involved. However, it will serve as a base for the proof of the prime geodesic theorem in Section 5.
Let us recall the following theorem given in (  , p. 51, Th. 40).
Theorem 1. If the Dirichlet’s series is summable for and, , then
By Lemma 1,
Therefore, substituting ω = 1, , and hence, in (2), we get
for. Using (  , p. 12, Th. 1.3.5), it is easy to get that
For, let, , be the zeros of in. Let, denote all zeros of the hyperbolic scattering determinant in.
Assume, , , where and for
,. Following (  , p. 468), we may also assume, , where, , are the zeros of the Selberg zeta function for each zero, , , of the hyperbolic scattering determinant f. Let be a large constant such that, ,. We put.
Without loss of generality we may assume that, ,. Let. By the Cauchy residue theorem one has
Arguing as in  (p. 474) and  (pp. 105-108), we easily find that the sum of the
first eight integrals on the right hand side of (5) is. Similarly, taking into account that is bounded for, we obtain that the sum of the first eight integrals on the right hand side of (6) is. Following  (p. 474) and  (p. 85,
Prop. 5.7), we obtain that the ninth resp. the third integral on the right hand side of (5) resp. (6) are. Now, if we take, (5) and (6) will give us
Bearing in mind location of the poles of given in (  , p. 439, Th. 2.16; or  , p. 498, Th. 5.3) and the fact that, we may assume without loss of generality that
Calculating residues and passing to the limit in (7) and (8) we get
The implied constants on the right sides of (9) and (10) depend solely on, m and W. With in (3), in (4), Equations (4), (3), (9) and (10) yield
where the first sum ranges over the finite set of poles s of
with, , the second sum ranges over the set of poles s of the same functions with, and the third sum ranges over the finite set of their poles s with.
5. Prime Geodesic Theorem
In our setting, the prime geodesic counting function is defined by
where the sum on the right is taken over all primitive hyperbolic classes with respect to (see,  , p. 473,  , p. 13).
Theorem 2. For, the formula
holds true, where, for, and the implied constant depends solely on, m and W.
Proof. Following  (p. 245) and  (p. 11), for a positive number, we define the second difference operator by
Here, d is a constant which will be fixed later. By the mean value theorem, we have
for some. It is easy to verify that
Reasoning as in  (p. 475), we may assume without loss of generality that is non-decreasing. Hence, (12) implies
Since (14) holds true, one can easily deduce that, , , , , ,. Thus, (13) and finiteness of the sums contained in on the right hand side (11) yield
In order to estimate, we will first consider
By (14) it is evident that
On the other hand, the mean value theorem (13) gives us
Let be the number of roots of on the critical line in the interval. It is known (  , p. 477, Th. 3.8) that. Taking and following (  , pp. 463-464;  , p. 246), we use (19) resp. (18) in the sums over, , resp. sum over, (below) to get
Observe that (see,  , p. 437, Prop. 2.13). Thus, application of to the third and the fourth sum in gives us
Let us write
where denotes the sum of the first four sums in and denotes the sum of the last four sums in. Now, Equations (11), (16), (17), (20), (21) and (22) give us
Putting, , the Equation (23) becomes
Since the left sides of Equations (20), (21) are for such choice of M and d, we get. Now, it is obvious that. Finally, Equation (24) gives us
Returning to (15), we conclude that inequality
holds true. Following (  , p. 11), we analogously obtain that
Arguing as in  (p. 475) and  (p. 113), one immediately sees that equality (25) proves the theorem.
We thank the Editor and the referee for their comments.