Two of Little Bay Park's boundaries are important thoroughfares in the borough of Queens: Cross Island Parkway and Utopia Parkway. The third boundary is beachfront. Parks acquired the property via condemnation in September 1950. The agency planned to develop the shoreline area but the construction of the Throgs Neck Bridge and the fiscal crisis of the 1970s put an end to the plan. Previously known as Clearview Park, Parks assigned the property's present name in 1973.

The Mantinecock tribe originally inhabited the shores of Little Bay. But in 1644, King Charles I of England gave 16,000 acres of land to his countryman William Lawrence. The grant, which included today's Bayside, stimulated European settlement in the area. In 1683, when the Crown organized the New York colony into ten separate counties, the land became part of Queens County, which encompassed both present-day Queens and Nassau Counties.