The square was designed with a series of paths emanating from a central circle where a fountain was located, a pattern retained in its current design. A multi-tiered fountain, located in the center of the park, was later moved to a location near Sussex Street.
At the center of Orange Square today is a monument dedicated on July 5, 1886, in the presence of some 10,000 people to the veterans of the Civil War. The monument was constructed of Quincy granite. Monies toward the construction of the monument were given by Diana Farnum, who set aside funds for it in her will.
In recent years, other veteran memorials have been added to honor those who served during the following conflicts and wars: War on Terrorism, Vietnam, Korean, WW I & WWII - with one for the Revolutionary War being the most recent one added.
The First Presbyterian Church, the oldest church structure in Port Jervis, faces the Square - as does the Drew United Methodist Church - where the father of novelist, Stephen Crane, who wrote The Red Badge of Courage - the Rev. Jonathan Townley Crane served as minister from 1878 til his death in 1880.