Gravesen played a key role in Everton's rise to the upper reaches of the 2004-05 Premiership table, from his position as an attacking midfielder. As his contract was due to expire in the summer of 2005, which would enable him to leave on a free transfer, Everton sold Gravesen to Real Madrid for a fee of £2.5 million, on January 14, 2005.

At Real Madrid, he looked to fill a hole in the defensive midfield, a position somewhat different from his time at Everton. He found immediate playing success at Real Madrid, as he scored in one of his first games, a 4-0 win against RCD Espanyol.

However, Gravesen would eventually find himself benched. Following public outcry over his tough style of play and the sacking of Real coach Vanderlei Luxemburgo and sporting director Arrigo Sacchi, who were in charge of acquiring Gravesen, he looked ready to leave Real Madrid in the winter of 2005, especially since Manchester United from England showed interest.

However, new coach López Caro would revive Gravesen's career at Real, picking him regularly, using him in the holding role in the new 4-1-4-1 formation Real adopted. Towards the end of the season, Gravesen once again found himself out of the team, and in May 2006 a number of clubs were reported to be interested in him.

] In August 2006, a scuffle erupted at a Real Madrid training session, following a hard tackle by Gravesen on teammate Robinho, who retaliated by hitting Gravesen.

Fabio Capello, the recently appointed Real Madrid manager, said about Gravesen: "The way he is, we won't have problems with him. He's just a little bit particular. I don't mess with him, he works well tactically. His behaviour is like this, and I don't like it, everything has to be done like he wants it to be done".

Gravesen looked likely to leave Real Madrid before the start of the 2006-07 season, with his agent John Sivebæk confirming that a number of clubs - including Newcastle United in England and Scottish club Celtic - were interested in signing the midfielder.