It is the remains of an old "bank barn". We designed a trellis fence to complete the enclosure to prevent deer from entering the garden.

More to the question, the walls are made of local Ellicott City, MD granite.