In wireless channels, the signal quality degrades mainly due to the additive noise and the random variation of attenuation of the signal, known as fading. The additive noise can be compensated to some extent using forward error correction (FEC) coding and automatic repeat request (ARQ). The fading can be compensated not only with FEC codes and ARQ schemes but also using spatial diversity and multiplexing achieved by employing multiple antenna systems, known as multiple-input multiple-output (MIMO) systems. MIMO schemes fall into different categories based on system requirements, e.g., space-time block codes (STBCs) and limited feedback schemes.

FEC codes, as a method of substantial performance improvement, are employed in most modern communication systems. However, the optimal design of the concatenation of FEC codes and modulation schemes for different applications is an open problem. Polar codes are a new class of FEC codes that benefit from simple rate matching and a variety of low-complexity decoders which facilitate the design of efficient systems. Multilevel coding with multistage decoding (MLC/MSD) is a low-complexity capacity achieving coded-modulation technique that can be designed efficiently for polar codes due to the conceptual similarity.

In this thesis, in order to achieve low-arithmetic-complexity/high-performance coded-modulation schemes for wireless channels, multilevel polar coded-modulation (MLPCM) schemes are designed and analyzed. To this end, new methods are proposed to decrease the MSD log-likelihood ratio estimation complexity and to improve the set-partitioning based bit-to-symbol mapping (SPM) design. Furthermore, some novel methods are proposed for the low-memory-space/low-arithmetic-complexity design of polar codes in additive white Gaussian noise and slow fading channels. In addition, a set of algorithms are proposed to design MLPCM based on the throughput for hybrid ARQ and channel-adaptive schemes. Moreover, to design the signal for MLC/MSD, a variety of multidimensional constellations are optimized using union bounds on the error rate of uncoded MIMO schemes, and their performance in the presence of polar codes is evaluated. Finally, MLPCM is designed in slow fading channels by optimizing STBCs based on the outage probability, generating SPM based on a bound on the outage, and the joint optimization of polar codes and STBCs.