Borden Gymnasium was begun in May 1900 and officially opened in March 1902. It was designed by Peabody & Stearns of Boston. Fundraising began with Principal Cecil P. F. Bancroft's campaign to raise $60,000. Phillips alumnus Mathew C. D. Borden of Fall River, MA Class of 1860 was the principal donor and the building was name in his honor.
The original building held basketball courts, a mezzanine level track, and exercise areas, 360 lockers and 15 showers. All the athletic apparatus equipment was selected by Dr. W.G. Anderson of Yale from A. G. Spaulding & Co.

In 1909 fundraising began with the students for the addition of a swimming pool wing. Theodore Woodbury York, Class of 1911 was chairman of the Swimming Pool Committee. They raised $15,000 and construction began in May 1911 by Holt-Fairchild of Boston and completed on December 2, 1911. Every effort was made to match the brick and granite of the Borden Gym. The original pool was 39 ft. wide and 75 ft. long, 8' 9" deep to 4' 9" at the shallow end.

In 1943 the pool was reconstructed and named York Pool. In 1952 a new and enlarge gymnasium wing was added to the west side of Borden Gym and named Memorial Gymnasium in honor of the 144 Phillips Academy Veterans killed in action in WWII and Korea. A large tablet was incorporated into the front west corner of the façade which included all Alumni names lost to war to 1945. The pool also was enlarged, to six lanes, made deeper and a diving pool added.

With the merger with Abbot Academy in 1973 extensive renovations were made to accommodate the female students. A $2.9 million addition was begun in 1979 named the Abbot Wing. This necessitated the permanent closure of Highland Road from Main St. to Salem Street.

A new multi-million dollar athletic facility was begun in 2020 on the sites of the former Case Memorial Gage and the Sumner Smith Hockey Rink.