Rich in human history, the story of this area began with the arrival of the ancestors of today's Māori people. Hunting the southern right whale, Europeans arrived over 600 years later. Today this house, along with the stories it tells of a developing town and the three families that lived here, are all that remains of the "Waiopuka Fishery", Robert Fyfe's whaling station that was established in 1842.
The earliest part of the House is the single storied wing which sits on whale vertebrae foundations. It was originally built by Robert Fyfe for his cooper, who made barrels in which the whale oil was transported and stored. Robert Fyfe drowned in 1854 and his cousin George Fyffe (spelled with a double f, and after whom the House was named) managed the whaling station and associated farming activities. George built the rest of the House in two stages, the southern end in 1857 and it was completed in the early 1860s when he married his wife Catherine.
George died in 1867, having fallen off the jetty near his house. Catherine died the following year and, since there were no children, all of their property was sold. Fyffe House was bought by Joseph Wilkinson Goodall (1835-1911), who transferred to Kaikoura as the district constable in 1865.The Goodalls made few changes to Fyffe House itself, apart from enclosing part of the verandah to form an office. It seems likely that it was the Goodalls who first established a flower garden in the space between the two wings of the building, and who wallpapered the interior for the first time.
The land surrounding the house was subdivided into five sections after Goodall's death in 1911. The section on which Fyffe House stands passed through several hands until George Low bequeathed the house to the New Zealand Historic Places Trust. After his death in 1980, the NZHPT repaired the house and some years later repainted the exterior in its original pink. Fyffe House remains Kaikoura's oldest building, and has strong connections to Māori and non-Māori.