The Iraqi military has said the grave, in the grounds of an agricultural college in the town of Hamam al-Alil, contains about 100 decapitated bodies.
Most have been reduced to skeletons, so it is unclear who the victims were.
IS militants have carried out many mass killings and have been accused of fresh atrocities in the area in recent weeks.
They are being forced to retreat to Mosul, the last major IS urban stronghold in Iraq, in an offensive by pro-government forces that began on 17 October.
