The study of topological string on Calabi-Yau manifolds is interested in mathematical physics for many years. It was found that gauge theories with certain gauge groups can be geometrically engineered from some Calabi-Yau threefolds, and the topological string partition functions on such spaces are related to instanton sums in gauge theories  .
The topological vertex formalism provides a powerful method to calculate the topological string partition function for non-compact toric Calabi-Yau 3-fold. By transfer matrix approach, A. Okounkov, N. Reshetikhin and C. Vafa proposed the topological vertex using Schur and skew Schur functions  :
where are Young diagrams, denotes the transpose of λ, and . The topological vertex has a nice interpretation by statistical mechanics of the melting crystal model   . In this paper two sets of vertex operators constructed specifically by the annihilation and creation generators of Heisenberg algebra play important roles in realizing Schur and skew Schur functions.
On the other hand, gauge theory partition function is a function with two equivariant parameters. In 2007, based on the arguments of geometric engineering, concerning the K-theoretic lift of the Nekrasov partition functions, A. Iqbal, C. Kozçaz and C. Vafa introduced a refined version of topological vertex  . In this refinement, one more parameter t comes in and the theory seems to be deeply related to a Macdonald function with special variables, or what we call a special Macdonald function, :
where . Moreover H. Awata and H. Kanno proposed another formula  which is expressed entirely in terms of the special (skew) Macdonald functions:
where and is the involution on the algebra of symmetric functions defined by , here . Although and have different expressions, they are supposed to give the same result.
Therefore it seems that the key problem is to change Schur function for the unrefined case to Macdonald function for the refined one. Hence to find a vertex operator formalism for the refined topological vertex will be interesting. The essential step is to realize the special Macdonald function . However a vertex operator formalism for does not exist so far.
In this paper, we get the operator product formula for the special Macdonald function . We also extend this formula to the case when n goes to infinity.
• : the set of rational numbers;
• : the field of rational functions of q, t over ;
• The q infinite product: .
A partition is any (finite or infinite) sequence of non-negative in decreasing order: and containing only finitely many non-zero terms. We denote by the size of the partition, i.e. and by the number of non-zero . The set of all partitions is denoted by .
A pictorial representation of a partition λ is called 2D Young diagram , it can be obtained by placing boxes at the i-th row. For example, Figure 1 represents a partition .
The transpose of λ is denoted by , , here . For example, the transpose of is .
We denote by for each square of a partitionλ, here. For each square, let
The numbers and may be called respectively the arm-length and the arm-colength of s, and, the leg-length and the leg-colength.
2.3. Macdonald FunctionWe
define a scalar product
here, where is the number of parts of λ equal to i; for each partition and.
Macdonald function depends rationally on two parameters, i.e., here and means tensor product over. They are characterized by the following two properties :
1) is of the form:, where;
When, reduce to the schur function.
Figure 1. The Young diagram for.
3. Operator Product Formula for.
We introduce an algebra generated by bosons and, they satisfy the following relations:
Let be the vacuum state which satisfies the conditions and. For a partition, we use a short notation.
The bosonic Fock space is generated from the vacuum state:
The dual vacuum state is defined by the conditions and. The dual boson Fock space is generated by the dual vacuum state:
There is a paring denoted by between two spaces, defined by the following properties:
3.2. The Vertex Operators
To construct the vertex realization for, we propose two sets of vertex operators depending on q and t.
we can obtain
We define another set of vertex operators
likewise we obtain
3.3. Operator Product Formula for
With the help of the vertex operator, we define vertex operators and as follows:
We propose operator product formula
After some careful computation via the commutative relation (6) and (9), the Formula (12) is equal to
Using the identity (we will prove it in the appendix)
we get the vacuum expectation value of this operator product formula
In other words,
Therefore we get the operator product formula for.
Similarly, by using the identity
Hence we get the operator product formula for.
The operator product formula for a special Macdonald function when n is finite as well as when n goes to infinity are given in this paper. A further investigation is to find a possible relation with the refined topological vertex.
The Proof of Identity (13) and (16)
Firstly, we will proof the identity (13).
Suppose, the infinite product of the left side (13) can be separated into three parts, and.
For the first part
For the second part
For the third part, the numerator and denominator cancel out each other.
Next, we will simplify the left hand side of the (13).
Since, we can get
Before combing them all, we can check
To show the identity (13), we need to use some properties of Young diagram λ, namely we need to interpret those powers of q in terms of arm lengths, leg lengths, arm co-lengths and leg co-lengths of those squares of Young diagram λ.
Now let us take i-th row as an example. We can classify all the arm lengths denoted as (where s means a specific square) of all squares of this row according to their leg lengths (denoted as). For example, for all squares s whose leg length, there must be squares counting from the end of row i. Likewise for leg length, there must be squares (See Figure 2). For, there must be squares etc. The leg length for i-th row must satisfy where r is the number of rows of λ. For there must be squares. For there must be squares.
For those squares which have leg length, their arm lengths are ranged from 0 to. For, those squares have arm lengths ranged from to. Similarly for, those squares have arm lengths ranged from to. Therefore the set on the i-th row with leg length becomes (where).
Similarly for leg co-length, these squares have arm co-lengths ranged from 0 to. For, these squares have arm co-lengths ranged from 0 to. At most.
Now from previous computation of and the analysis about the properties of Young diagram, we can deduce the identity (13).
Next we will prove the identity (16).
we notice that if n goes to infinity
Figure 2. Some information of a Young diagram.
So when n goes to infinity,
From previous analysis about the properties of Young diagram, we can deduce the identity (16).
This work is partially supported by National Natural Science Foundation of China (No. 11475116, 11401400, 11626084, 11647123).