The empty lot west of Cochran Chapel on the northeast corner of Main Street & Chapel Ave. was once the site of the fourth academic building of Phillips Academy. The building was designed by Squire Samuel Farrar, the Academy Treasurer and Trustee. He also supervised the construction of the building which cost $10,352.90. The funds were raised with the selling of property in Maine and Canada owned by the Academy as well as accrued income from their endowment.

Claude Fuess wrote the following description in his book:
"In the spring of 1829 workmen were excavating a cellar on the northeast corner of Main Street and Chapel Avenue and soon arose an oblong, two-storied, massive edifice with thick walls of rough gray stone and a slanting roof surmounted by a high wooden cupola or bell tower on which was perched an equally tall weather vane. The architect was Squire Farrar who had created a style that was all his own, not Grecian or Gothic or Colonial by "Farraresque". Bare, somber, and unrelieved by ornamentation, the building resembled a jail or tomb and seemed at once the strangest and ugliest structure ever produced by the hand of man."

This critique of the building was harsh and the granite block exterior was a departure from traditional red brick facades of the Seminary buildings near by. But there was a nod to the Federal style hip roofs, coining detail on the exterior corners and the roof cupola to make a statement.

In Robert A. Domingue's book "Phillips Academy Andover Massachusetts, an illustrated history of the property, published in 1990 he writes the following about the Stone Academy:
The original "purpose of the building was to house the "English Classical School" or in 1835 referred to as the "Teacher's Seminary" which was established independent of Phillips Academy. The Seminary had a chemical laboratory in the basement, There was also a room for "Philosophical apparatus" for experiments in what is now known as physics. Cabinets housed and extensive collection of minerals for the study of geology, a complete field set of instruments for practical surveying and civil engineering. A library held 850 volumes of books. "The Teacher's Seminary rapidly became the English High School, and on August 12, 1842, merged with the Academy to become the English Department of Philips Academy."

"On the first floor held a large room where students met for morning and evening prayers. On the left as one entered at the door near the southwest corner, was a low platform on which Dr. Taylor's chair was placed. Incoming students had to face those who were already seated. In the northeast alcove stood a wheezy organ, around which were stools for the choir. Along the middle aisle were desks for day scholars and on either side were rows of had wooden benches, certainly not designed for physical ease. The desks and benches were specimens of ancient carpentry, cut through and through with jackknives and worn away by the boots of many generations of youths. The classrooms were upstairs and were poorly lighted and wretchedly ventilated; they also possessed cruel seats."

On December 21, 1863, the Stone Academy was destroyed by fire. It was generally believed, but never proven, that this conflagration was the result of arson perpetrated by a student who had just been expelled.

A plaque to commemorate the former "Stone Academy" now rests on the site where it once stood.