This dissertation will be presented to the Faculty of the Graduate School of The University of Texas at Austin in partial fulfillment of the requirements for the degree of

Ph.D. in Electrical Engineering

FD systems enable bidirectional transmission over the same resource block, with the potential to significantly reduce latency and double spectral efficiency. However, self-interference (SI) remains a key challenge, as the power of the SI signal can be several orders of magnitude higher than the desired signal, potentially saturating the analog-to-digital converters (ADCs) and degrading system performance. Massive antenna arrays offer the necessary degrees of freedom for spatial multiplexing and effective SI suppression. Current FD solutions, primarily implemented for sub-6 GHz radios, provide a valuable foundation for extending FD capabilities to the millimeter wave (mmWave) spectrum. Nevertheless, this extension requires more than just a frequency shift—it necessitates a reevaluation of the underlying assumptions in sig- nal processing developed for the sub-6 GHz band. Adapting FD solutions to the mmWave band thus demands the development of advanced signal processing tech- niques that account for the specific challenges of mmWave communication. Recently, the combination of FD and mmWave communication has been proposed in 3GPP Releases 17 and 18, where FD is being considered for integrated access and backhaul (IAB) systems. Throughout the literature, various SI cancellation techniques have been explored, typically classified into passive and active methods. Passive techniques include antenna array directionality, strategic antenna placement, and isolation, while active cancellation involves suppression through analog and digital circuitry. In this research, I leverage the available degrees of freedom provided by massive antenna arrays to design robust hybrid analog and digital beamformers capable of suppress- ing SI and enabling FD operation in mmWave systems. The design of these hybrid beamformers is subject to several constraints, including the unit modulus constraint imposed by mmWave hardware, which necessitates phased-array implementation for beam direction tuning. Additionally, significant SI power must be mitigated in the analog domain to prevent ADC saturation, while residual interference is eliminated in the digital domain.

First, I focused on a point-to-point system where two FD nodes communicate over bidirectional links. I formulated the problem of maximizing the sum spectral efficiency under the constraints imposed by hybrid analog/digital beamformers. Di- rectly maximizing the sum spectral efficiency or signal-to-interference-plus-noise ratio (SINR) involves jointly addressing both the signal power and the self-interference (SI) components of the SINR. Consequently, the problem was reformulated as minimiz- ing the SI power, a key step toward enhancing the overall sum spectral efficiency. Specifically, I formulated the SI power minimization problem under the constraints of the analog beamformers, including the unit modulus constraint and maintaining the rank of the effective channel, i.e., optimizing the beamformed received channel. To address this, I relaxed the unit modulus constraint to preserve the convexity of the minimization problem and derived a closed-form solution for the analog beamformers at both FD nodes. Following this, I proposed a digital beamforming design aimed at maximizing the sum spectral efficiency. Simulations confirmed the efficacy of the pro- posed algorithm, demonstrating significant improvements in spectral efficiency and SI power reduction. Furthermore, I developed a practical design framework, ana- lyzing trade-offs in terms of the number of antennas, RF chains, system complexity, and power consumption necessary to enable full-duplex operation. In this context, I established that a minimum of 50 dB of self-interference must be mitigated through analog beamforming to achieve effective FD performance.

Second, I examined a single-cell, single-user scenario where both uplink and downlink communications are handled by a FD base station (BS). In this setup, the uplink user is affected by self-interference, while the downlink user experiences inter- user interference (IUI) caused by the uplink transmission. I formulated the problem of maximizing the sum of uplink and downlink spectral efficiency, subject to minimizing SI, inter-user interference, and adhering to unit modulus and semi-unitary constraints. The objective was to design low-complexity hybrid analog/digital beamformers that require few iterations to converge. I further investigated factors influencing the algo- rithm’s convergence, such as the regularization of the interference covariance matrix and the normalization of beamforming both before and after convergence. The perfor- mance of the algorithm was assessed based on key metrics, including self-interference reduction, computational cost, rate of convergence, and convergence stability. Simu- lation results demonstrate the robustness of the proposed approach, achieving significant improvements in interference suppression and spectral efficiency compared to existing methods.

Third, I investigated an IAB system, where the IAB node receives data from the gNB donor via the backhaul link and transmits to the user through the access channel. Operating in full-duplex mode, the IAB node experiences self-interference (SI), which degrades the performance of the backhaul channel. To address this, I formulated an SI power minimization problem and applied a similar approach as in my previous work to design the analog beamformers at the IAB node. For the gNB donor and user, I derived regularized zero-forcing (RZF) and minimum mean square error (MMSE) beamforming designs, respectively. The computational complexity of designing the hybrid beamformers was found to be relatively low compared to exist- ing solutions. By using SI reduction as the objective in both the analog and digital domains, the proposed algorithm demonstrated rapid convergence, typically requir- ing only 3 − 5 iterations to efficiently suppress SI. Additionally, simulation results confirmed that the proposed hybrid analog/digital beamforming design outperforms half-duplex systems and other state-of-the-art approaches in terms of spectral effi- ciency.

Fourth, I extended the second contribution to a wideband multiuser uplink and downlink communication system. In this scenario, the users employ a fully digi- tal architecture, while the BS utilizes a hybrid analog/digital architecture. To reduce power consumption, the phase shifters at the BS are quantized, and the transmis- sion/reception arrays at the BS implement a partially connected structure. I formu- lated the problem of maximizing the per-subcarrier sum spectral efficiency for both uplink and downlink users, subject to minimizing self-interference, inter-user interfer- ence (uplink users interfering with downlink users), and multiuser interference (among uplink or downlink users), while also satisfying unit modulus and semi-unitary con- straints. To address these challenges, I designed hybrid analog/digital beamformers that meet the imposed requirements. Simulation results demonstrate the optimization of beampatterns at both the BS and user devices. Furthermore, I tracked interference reduction across different stages of the receiver chain and over the entire bandwidth. The findings also highlight the advantages of using a partially connected architec- ture, showing improvements in both energy efficiency and spectral efficiency when compared to existing methods.

In the fifth contribution, I extended the previous contribution to a massive multiple-input-multiple-output (MIMO) multicell, multi-user scenario, where BSs op- erate in FD mode and are equipped with a large number of antennas. I developed a unified analytical framework for both uplink (reverse) and downlink (forward) links, applicable to both LTE and mmWave bands, while incorporating the use of low- resolution data converters. The analysis accounted for various system imperfections, including low-resolution quantization noise, pilot contamination, network irregular- ities, imperfect channel state information (CSI), user mobility, cellular interference, and self-interference. I derived the signal-to-quantization-to-interference-to-noise ra- tio (SQINR) as a key performance metric, and based on this, I investigated the effective spectral and energy efficiency of the system. In addition, I derived the spe- cial cases, asymptotic results and power scaling laws to unpack several engineering insights of the proposed model. This comprehensive analysis provides insights into the challenges and trade-offs involved in implementing FD massive MIMO systems in real-world scenarios.

These five contributions offer significant advancements in the feasibility and development of power-efficient FD systems by addressing the communication require- ments of modern wireless networks. They effectively mitigate interference below the noise floor, optimize spectral and energy efficiency, reduce latency, and maximize the utilization of available resources and degrees of freedom. These enhancements are critical for improving the performance and scalability of next-generation wireless systems, particularly in the context of FD communication.