Victoria University will knock down the building which lost its protected heritage status this year.