After installing vim from MacPorts I noticed that neither the arrow keys or backspace work as expected.
The solution is simple, create a ~/.vimrc file with the following contents:
set nocompatible set bs=2 Reload vim and voila, insert mode behaves as I’d expect it to.