Isaac Vennell died in 1895 and was buried at the nearby Bethel Cemetery. The Pennsylvania Railroad was busy erecting a bridge across the Delaware River just up the road in Delair to provide better access to the agricultural center of South Jersey.

With the bridge came additional tracks and the railroad determined it would need the land upon which the Vennell Tavern sat. So, in 1896, Isaac’s heirs discontinued the tavern and sold the house and lot to the railroad. But the railroad agreed to relocate the house farther up Cove Road and the former tavern remains in this new location today.

As the twentieth century dawned, recreational activities at Fish House were waning. The beach was still used for bathing and cove for boating, but its extensive use by Philadelphians was slowly dying out. The Tammany Pea Shore Fishing Company more or less disbanded and the clubhouse was taken over by a group of Alsatian immigrants. In 1915, the Pennsylvania Railroad closed its Fish House station, forcing the Post Office to relocate to the lean-to shed behind the westernmost house of the seven-unit row across Cove Road from the Vennell Tavern. In 1924, the postal facility closed for good.