The Tamil Nadu government on Friday informed the Madras High Court that the Governor cannot take a unilateral decision to remove V Senthil Balaji from the Council of Ministers.