(4)When a number has been affixed under sub-section (1), the owner of the building shall be bound to maintain such number and to replace it if removed or defaced; and if he fails to do so, the authority referred to in sub-section (1) may, by notice, require him to replace it.