A more effective and accurate improved Sobel algorithm has been developed to detect surface defects on heavy rails. The proposed method can make up for the mere sensitivity to X and Y directions of the Sobel algorithm by adding six templates at different directions. Meanwhile, an experimental platform for detecting surface defects consisting of the bed-jig, image-forming system with CCD cameras and light sources, parallel computer system and cable system has been constructed. The detection results of the backfin defects show that the improved Sobel algorithm can achieve an accurate and efficient positioning with decreasing interference noises to the defect edge. It can also extract more precise features and characteristic parameters of the backfin defect. Furthermore, the BP neural network adopted for defects classification with the inputting characteristic parameters of improved Sobel algorithm can obtain the optimal training precision of 0.0095827 with 106 iterative steps and time of 3 s less than Sobel algorithm with 146 steps and 5 s. Finally, an enhanced identification rate of 10% for the defects is also confirmed after the Sobel algorithm is improved.