The smear is just the mortar that the stone was laid up in, and was not completely cleaned off. We used a buff mortar mix which is regular brick sand with white mortar mixed together.

So buff mortar mix is just mixing regular sand to white mortar? I wanted to use a desert buff mortar but they said it would look better with white sand which is pricey.

This particular project used White Mortar Mix and regular sand. White sand is more expensive and will definitely impact your costs. Have your mason construct a mock up using both regular sand and white sand. This will help you get the color you are looking for.