ORANGE, Va. (AP) — The board that manages James Madison’s Montpelier estate in Virginia has voted to strip power-sharing status from a group representing descendants of people once enslaved there.

Friday’s vote comes after the board announced last year that it would share authority equally with descendants.

The Washington Post reports that the vote means the descendants committee can’t name future members. Critics of the foundation say the board is trying to tell a “whitewashed” version of history.

Foundation chairman Gene Hickok says the board isn’t backing away from its commitment to fully represent descendants on the board, but says working with the descendants committee has been difficult and the board wants to be able to chose descendant members from a wider pool.