The top champs outside of lane to counter Graves include Aphelios, Xayah, and Jinx. These are all great at countering Graves with decent win rates over 53%. Even if these Graves counter picks are not chosen in the same position as you, it's not a good idea to take him into one of these matchups. They will usually destroy Graves.
Conversely, if the other team brings either Zyra or Malphite, you should strongly think about picking him, as he easily crushes these champions with win rates over 54% against both ofthem.