The Federal Reserve Act states that governors can only be removed by the president “for cause”, meaning they would have to commit serious misconduct in office. Do not let these facts put you at ease.