The story of Goat Island is an unsettled one. Some people believe that it was named because it was shaped like a goat. This one is easy enough to dispel because an old map of the harbor definitely shows Goat Island, and it wasn't shaped like a goat. Not in the slightest.
The most credible cause for the name was the presence of a "goat" on the island, along with the Officer's Club and so on. Navy personnel that I've spoken to from that time period confirm the presence of a goat. Great, but where did that goat come from?