Heterogeneous networks (HetNet) are a promising solution to improve network performance in terms of spectrum efficiency and energy efficiency. Nevertheless, HetNets suffer from two main sources of interference: mutual interference between macrocells and small cells (called cross-layer interference), as well as inter-cell interference among small cells themselves (called co-layer interference). In this thesis, we study the resource allocation and interference issues of HetNets. First, in HetNet systems with a moderate number of small cells, we integrate two popular approaches: spectrum
avoidance and spectrum sharing, using optimization and game theory. In this solution, small base stations (SBSs) opportunistically avoid the parts of the spectrum that are used by macro BS (MBS), thereby controlling cross-layer interference, while the co-layer interference is controlled using a spectrum sharing technique. We then exploit recent advances in the mean-field game theory in order to control the co-layer interference between a large number of small cells. Meanwhile, a spectrum avoidance technique is applied to control the cross-layer interference. Next, we design a full spectrum
sharing technique based on the mean-field game theory for interference management in hyper dense HetNet systems. The joint cross-layer and co-layer interference management issue is formulated as two nested problems, which are solved via distributed algorithms. Furthermore, tools from the optimization theory are employed to enhance the performance of cell edge users in open access HetNets. Simulation results are presented to show the effectiveness of the proposed schemes.