The development of wireless communication provides great convenience to people’s life. With the commercial use of 4 G mobile communication system, the 5 G mobile communication receives more and more attention in academia and industry. The research shows that more than 50% of the voice services and 70% of the data services will be mainly from the indoor terminal in the future. Therefore, UDN (Ultra Dense Network) plays an important role in the future 5 G network, which will become one of the key technologies of 5 G and also become one of the global hot topics  .
The basic idea of UDN is to enhance the spectral efficiency, the energy efficiency and spectral reuse of the system through reducing the distance between the transmitter and receiver. In UDN, a large number of FBS (Femto Base Station) are generally located in hot spots, such as airports, supermarkets, railway stations and etc, which can greatly increase user data rate  .
Because of the high density of FBS, the communication coverage between FBSs is often overlapping  . The past network-centric clustering algorithm is not practical, while the user-centric clustering algorithm is more applicable, where each FUE (Femto User) can be connected to multiple FBSs. In order to improve the user transmission rate and channel utilization, collaboration between FBS and FUE becomes more critical. But the dense deployment of FBSs will bring severe cross-layer interference and co-layer interference. The interference is expected to be canceled through designing pleasant downlink precoding matrix on transmitter with Massive MIMO, which can enhance the security and the growth of data rate without increasing the transmission power.
The downlink precoding studied in this paper solves the problem of cross-layer interference and co-layer interference of MUEs (macro users) and FUEs in heterogeneous and ultra dense networks, and lays a theoretical foundation for the deployment of heterogeneous and ultra dense networks. The rest of this paper is organized as follows. The second part introduces the heterogeneous UDN system model and user-centric multi connection architecture; the third part introduces the downlink precoding to achieve interference cancellation for MUE; the fourth part introduces the downlink precoding to achieve interference cancellation for FUE.
2. Heterogeneous UDN System Model and Multi Connectivity in User-Centric Architecture
A heterogeneous UDN is described in this paper, which includes a MBS (Macro Base Station) and many FBSs arbitrarily located. All FBSs are within the coverage of the MBS. MBS is responsible for information connection of all FBSs. The transmitting power of MBS is higher and it has larger coverage than FBS; The FBS with low transmitting power can improve indoor coverage and increase network capacity. If Massive MIMO are deployed at MBS and FBS, the system capacity will be increased exponentially. However, due to severe co-layer interference and cross-layer interference, the efficiency of heterogeneous UDN networks will be greatly reduced. Figure 1 is the network model of heterogeneous UDN.
In Figure 1, all FBSs within the MBS coverage are connected to the MBS. The MBS can obtain channel status information of all FBSs and can interact among FBSs. For each FUE, a user-centric clustering strategy with multi connectivity is deployed  . The multiple FBSs staying in the FUE coverage are recognized as multiple access points. Because of the limited capability of FBS, each FBS cannot be joined to more FUEs. The FBS should be banned when the number of FUEs joined to the FBS reaches its upper bounder.
Figure 1. Heterogeneous UDN system model with user-centric architecture.
Due to the high density of FBSs, the interference is severe. Those FBSs not within the user’s coverage may still cause interference to the user. Therefore, the ability to cancel interference should be enhanced when user has multi connectivity with the FBSs. Otherwise it is difficult to obtain a higher SINR.
In Figure 1, it is assumed that the number of transmitting antenna of MBS is , the number of transmitting antenna of FBS is , the number of receiving antanna of MUE and FUE are both . The receiving signal of the kth MUE can be expressed as Equation (1)
where is the channel metrix from MBS to the kth MUE; is downlink precoding metrix; is normalized transmitting signal vector; is the channel metrix from FBS to the kth MUE; M is the total number of MUE; is the total number of FBS; is the precoding metrix of the ith FBS; is the FUE number connecting to the ith FBS; is the noise vector at the kth user, which is an additive Gaussian noise following the distribution CN(0, ).
Obviously, the interference on kth MUE can be divided into two parts. First part is the multi-user interference within MBS coverage, also called co-layer interference, which is expressed as the second part of the Equation (1); Second part is cross-layer interference on the kth MUE derived from the FBSs, which is expressed as the third part of the Equation (1).
In heterogeneous UDN, the receiving signal of the mth FUE is expressed as Equation (2)
The interference on the mth FUE obviously consists of two parts seen from Equation (2). First part is the interference from the non-joining FBSs, also called co-layer interference, which is expressed in the second part of Equation (2); Second is the interference on the MUE from the MBS, also called cross-layer interference, which is expressed as the third part of Equation (2). is the noise vector at the mth FUE, which is an additive Gaussian noise following the distribution CN (0, ).
From the analysis above, both the MUE and FUE are suffered from the co-layer and cross-layer interference. Thus, designing the pleasant downlink precoding to canceling the two kinds of interference is crucial.
3. Interference Cancellation of the MUE
The cross-layer interference on MUE is considered primarily. The different frequency is allocated to macro cell and femto cell. Generally, The communication frequency between the MBS and the MUE is allocated from 2 G to 6 G. This frequency band has a large communication coverage, and can achieve non-line-of-sight communication. The communication frequency between FBS and FUE is allocated with higher frequency, called millimeter wave communication  . The path loss of the millimeter wave communication is serious, the FBS coverage is thus limited. Only the ultra dense deployment of FBS can achieve full coverage. This Frequency division scheme can greatly reduce the cross-layer interference between macro cell and femto cell  .
Due to the orthogonal resource assignment strategy, cross-layer interference is canceled. And the co-layer interference can be canceled from Equation (1). The receiving signal of the kth MUE can be rewritten as Equation (3)
The parameters in Equation (3) are referred to Equation (1). For any MUE, only the co-layer interference needs to be canceled.
Still taking the kth MUE for example, the interference channel metrix of the kth MUE can be expressed as below
Generally, the dimension of the MUE’s interference channel is relatively small. The block diagonalization method can be used to calculate the downlink precoding of the MBS. Define SVD of as 
where and are both unitary matrix; , is the rank of  , , if the non-zero eigenvalue of  .
where are the eigenvectors which corresponds to non-zero eigenvalues; are the eigenvectors which corresponds to zero eigenvalues, and satisfy
The downlink precoding of the MBS is
4. Interference Cancellation of the FUE
Due to the cross-layer interference cancellation through orthogonal resource assignment between the macro and femto cell, any FUE is also suffered from the co-layer interference. And due to the ultra dense deployment of the FBS, the dimension of the channel is relatively large. The receiving signal of the mth FUE is as follows
Comparing to the Equation (2), the cross-layer interference is canceled in Equation (9). The pleasant downlink precoding needs to be designed to cancel the co-layer interference. The combining channel matrix is denoted as . Let define the pseudo-inverse channel matrix as
Assumed is the partial vectors of , and satisfies that is null space of the interference channel matrix of the mth FUE. Null space can be achieved through . The non-zero solution of is null space which can be assigned to . It satisfy
If is invertible, can be made QR decomposition, which is expressed as below 
Where is an orthogonal matrix and is an upper triangular matrix, such that
is the invertible upper triangular matrix, such that
And it satisfies , such that the receiving FUE needs to add a receiving matrix .The downlink precoding matrix of mth FUE is expressed as Equation (16)
The upper calculation is based on the invertibility of  . Actually, is not necessarily invertible. Firstly, is not necessarily a square matrix; Secondly, even though is square matrix, but it is not necessarily of full rank. Therefore, before QR decomposition, the invertibility of has to be ensured. selected from null space of must be invertible, such that the vectors of are irrelevant from each other.
Ultra dense network (UDN) can increase the frequency reuse by deploying a large number of low power femto cells, which can greatly improve the spectrum efficiency and energy efficiency. At the same time, there are also many problems and challenges. The cross-layer interference and the co-layer interference in heterogeneous UDN make it difficult to improve the signal to interference noise ratio of the users, which leads to the limited capacity of the system.
Interference can be precanceled at the transmitter by applying massive MIMO downlink precoding technology. This paper studies a heterogeneous network with a user-centric clustering, where FUE have multi connectivity with the FBSs. In this heterogeneous UDN, downlink precoding to cancel co-layer interference is mainly studied. The co-layer interference problem with small dimension is resolved by block diagonalization. The co-layer interference problem with large dimension is resolved based on QR decomposition. The scheme provides a theoretical basis for practical network deployment.
This work was supported by the Special scientific research project of Shaanxi Education Department (No. 16JK1688).
 Andrews, J.G., Buzzi, S., Choi, W., Hanly, S.V., Lozano, A., Soong, A.C.K., et al. What Will 5G Be? IEEE Journal on Selected Areas in Communications, 32, 1065-1082.
 Chen, S.Z. and Zhao, J. (2014) The Requirements, Challenges, and Technologies for 5G of Terrestrial Mobile Telecommunication. IEEE Communications Magazine, 52, 36-43.
 Mehrpouyan, H., Matthaiou, M., Wang, R., Karagiannidis, G. and Hua, Y. (2015) Hybrid Millimeter-Wave Systems: A Novel Paradigm for Hetnets. IEEE Communications Magazine, 53, 216-221.
 Pan, C.H., Zhu, H.L., Gomes, N.J. and Wang, J.Z. (2017) Joint Precoding and RRH Selection for User-centric Green MIMO C-RAN. IEEE Transactions on Wireless Communications, 16, 2891-2906.
 Li, C., Zhang, J., Haenggi, M. and Letaief, K.B. (2015) User-Centric Intercell Interference Nulling for Downlink Small Cell Networks. IEEE Transactions on Communications, 63, 1419-1431.
 Baldemair, R., Irnich, T., Balachandran, K., Dahlman, E., Mildh, G., Selén, Y., et al. (2015) Ultra-Dense Networks in Millimeter-Wave Frequencies. IEEE Communications Magazine, 53, 202-208.
 Han, S.F., Chih-Lin, I., Xu, Z.K. and Rowell, C. (2015) Large-Scale Antenna Systems with Hybrid Analog and Digital Beamforming for Millimeter Wave 5G. IEEE Communications Magazine, 53, 186-194.
 Alkhateeb, A., El Ayach, O., Leus, G. and Heath, R.W. (2014) Channel Estimation and Hybrid Precoding for Millimeter Wave Cellular Systems. IEEE Journal of Selected Topics in Signal Processing, 8, 831-846.
 Alkhateeb, A., Mo, J., Gonzalez-Prelcic, N. and Heath, R.W. (2014) MIMO Precoding and Combinin Solutions for Millimter-Wave Systems. IEEE Communications Magazine, 52, 122-131.