Race Rock, located at the west end of Fishers Island and the eastern entrance to Long Island Sound, was considered "one of the most dangerous obstructions to navigation on the coast". Rising from a depth of seventy or more feet of water, several small spurs of rock broke the water's surface, while a large rock formation was covered with only three feet of water at low tide. During the early 1800's, there was hardly a summer month that a vessel did not strike the rock reef with some times disastrous results.

The Gothic Revival styled Race Rock Lighthouse marks a most dangerous location with perhaps hundreds of shipwrecks to its dubious credit, including the steamer “Atlantic” in which 45 people perished in November 1846. Its' completion in 1878 marked the end of masonry lighthouses on wave swept or water-bound sites. Most of all, it is a fitting monument to its courageous engineers, Francis Hopkinson Smith and Captain Thomas Albertson Scott. The construction on the “Boulder” (really a ledge that is 3 to 13 feet below water) required 7 years, thousands of tons of riprap, numerous acts of courage and amazing persistence. Smith also built the government seawall at Governors Island, NY and the foundation for the Statue of Liberty in New York Harbor.