After installing vim from MacPorts I noticed that neither the arrow keys or backspace work as expected.
The solution is simple, create a ~/.vimrc file with the following contents:
set nocompatible set bs=2
Reload vim and voila, insert mode behaves as I’d expect it to.
Note that you don’t really need to set nocompatible, simply having a .vimrc file in your home directory does this automatically. I’ve left it in to remember the solution in future.