Castle Garden started out as a military fort, which was built between 1807 and 1811 to defend the city of New York during the Napoleonic Wars. The 28-gun West Battery, as the fort was called, was manned by soldiers but never fired a shot in battle. Its mere presence was enough to spare New York from attack. By 1822 it was decided it had no more military use and was ceded to New York City. Renamed Castle Garden when its central parade ground was decorated with flowers and shrubs, it became the setting for band concerts, fireworks, balloon ascensions, and other entertainments.

After a few years of rather cheap shows, (tightrope walkers, Indian war dances, and public wrestling matches) two entrepreneurs leased the structure, put in a new floor and fitted it with a stage, in front of which 6000 seats curved in a giant semicircle. With the new renovations came grander entertainments, the triumph of which was the American debut of singer Jenny Lind in 1850.

It was masterminded by the greatest showman of them all, Phineas Taylor Barnum. Everything, from the Swedish Nightingale's arrival to 30,000 cheering fans at the dock to the advance ticket sales of $17,000 was planned with practiced skill. Miss Lind would donate her share (over $10,000) of the opening night's gross receipts to city charities.

Jenny Lind, the greatest coloratura soprano of her time, had been singing for twelve years and was just 30 years old when her two-year tour began on this triumphant note.

At every one of her six performances in Castle Garden, the 6000 seats and all the standing space were filled, with crowds outside still clamoring for tickets. The management took in over $87,000 for her performances, a vast sum of money in those days.

Jenny's was a hard act to follow. Italian Operas and Equestrian exhibitions didn't draw the crowds, and Castle Garden's days as a theater were numbered. Because of its size and waterfront location, it was just the place the city need for its long needed immigration depot. So when the lease expired in 1845, the Commissioners of Emigration selected Castle Garden to be the new place where all immigrants could land.

After approval by the New York State Legislature, extensive repairs were made. The once elegant refreshment rooms near the main entrance were converted into functional bathrooms, with rows of basins and towels, and huge 20 foot tubs with continuous running water. The seats on the floor were replaced by plain wooden benches, leaving most of the area empty for the crowds of immigrants. Outside, billboards that had once featured Jenny Lind now advertised "Entrance for Emigrants Only."

On August 1, 1855, Castle Garden was officially opened as America's first receiving station for immigrants. Eight million immigrants would pass through Castle Garden in the next 35 years.