You do not have to worry about the moss and scab. It is alright to brush it off as this cleans the bark. But the moss and scab itself are not harmful as they only sit on the outer bark. In general, moss and scab are indicators for high humidity in your garden microclimate. This could be caused by a dense tree top but also by the grown surroundings in your garden. If you lighten out the tree top, the moss and scab should become less. You should also lighten out the adjacent garden plants to guarantee a better ventilation.