Works to demolish a sports ground structure which had to be shut down due to vandalism and asbestos have finally begun.