Effect of Nb contents on microstructure and magnetic properties of FeCoNbB films has been studied. X-ray diffraction (XRD) analysis reveals that Nb plays a role in refining grain and in facilitating formation of amorphous structure. When Nb content is more than 8 at%, the films transform from crystalline to amorphous structure, accompanied by variations in magnetic properties. An unusual out-of-plane anisotropy component is consequently observed. It can be suggested that the anisotropy induced by increasing Nb contents be attributed to stress-related magnetoelastic anisotropy. The undesirable anisotropy is greatly reduced by thermal annealing, reducing film thicknesses or applying an external magnetic field together with Si addition.