Fiber nonlinearity limits the use of coherent optical orthogonal frequency division multiplexing （CO- OFDM） to upgrade wavelength-division multiplexing （WDM） systems using legacy non-return-to-zero-on- off-keying （NRZ-OOK） channels. This letter proposes to compensate for the fiber nonlinearity of COOFDM with NRZ-OOK neighbors by combining digital signal processing （DSP）-based self-phase modulation （SPM） post-compensation with pilot-tone-based cross-phase modulation （XPM） compensation. The simulation results demonstrate that the optimum low-pass filter bandwidth for pilot-tone-based XPM compensation depends on the pilot-to-signal ratio value and launch optical power. Our method allows a 4-dB increase in the launch power for a 40-Gb/s single polarization CO-OFDM channel placed in the middle of six 10.7-Gb/s NRZ channels in a 50-GHz space and 1 200-km WDM system.
