By default gimp feathers out to your background colour if you do not have an alpha channel on your photo. (and you can make your background colour whatever you like beforehand)
or you can add an alpha channel to your photo before you delete and it will feather out to transparent.