Archaeological evidence suggests that Native Americans, know as the
St. Johns people, were living in the Casselberry area between 800 and 1565 CE. A blockhouse known as Fort Concord was constructed on the shores of Lake Concord in 1849 to protect local settlers during conflicts between the Seminole Wars. After the Homestead Act was passed in 1862, settlers began homesteading the land around Lake Concord and the Triplet Chain of Lakes. Stephen J. L. Hooker, the nephew of "Florida Cattle King" William B. Hooker, came to the area in the 1850s and established one of the first families in what would become Casselberry. Real estate broker John W. Griffin purchased land in the early 1880s looking to homestead with his family. During this process he purchased the homestead of Silas B. Carter who allegedly assassinated him in Orlando. Griffin's wife, Annie E. Griffin, platted the area's first subdivision in 1889 after inheriting his land.
In 1925, Gordon J. Barnett of New York came to Altamonte Springs and 
acquainted himself with all aspects of the local fern industry and soon opened his own fernery. Barnett then began a housing development west of Lake Concord called Fern Park Estates comprised of small, individual lots that each included a fernery. The ferneries produced Asparagus Plumosus ferns for floral arrangements throughout the United States and Canada and the area's ferneries became one of the world's largest producers. These ferneries were staffed mainly with African-Americans who lived in shotgun houses in the local Evergreen cemetery, as well as in the surrounding areas.
In 1926, Hibbard Casselberry of Winnetka, Illinois was visiting his wife's aunt in nearby Winter Park, Florida when he met Gordon Barnett and signed on as his exclusive sales agent for Fern Park Estates. Casselberry and Barnett split soon after but Casselberry purchased land surrounding Fern Park Estates and began platting his own subdivision: W
inter Park Ferneries. The area became known as Fern Park and a post office was established under that name in 1928. Casselberry continued to develop and expand his fernery and real estate business throughout the 1930's while Barnett was elected to the Florida House of Representatives in 1937. When Barnett introduced a bill to incorporate the Town of Fern Park it was passed and moved to the senate before Fern Park residents could petition the bill, and Barnett was forced to withdraw it.
Hibbard Casselberry convinced the residents of Fern Park that
the best way to avoid property taxes was to incorporate their own tax-free town. A town meeting was held on October 10th, 1940 where the tax-free Town of Casselberry was officially incorporated, including parts of Barnett's Fern Park Estates. As more property was brought into the city limits of Casselberry, the area of Fern Park was pushed further south towards SR 436. With the beginning of World War II the ferneries were declared non-essential and Hibbard Casselberry secured government contracts for the manufacturing of bandoliers, bomb parachutes, and hospital tent liners. During this time Casselberry also purchased a local horse racing track and hotel, which became one of Seminole County's major attractions.
Despite the fact that Casselberry was an incorporated town and Fern Park remained unincorporated the United States Postal Service refused to grant Casselberry its own post office due to a policy prohibiting the naming of post offices after living persons. Casselberry established a contract post office in 1957 at its own expense and the town was 
finally granted a post office in 1959, although Hibbard Casselberry lived another ten years. The City of Casselberry was incorporated on July 25th, 1965 and Casselberry's citizens eventually voted to have property taxes in 1976.