It definitely suggests that someone had his inaccessible memories screaming at him about who the Dark Magician was. Yugi/Atem played with cards he liked in his deck, not with the most well built deck. Probably why people were always so shocked he kept winning.

Also, he probably just had one of both and Summoned Skull kept ending up at the bottom of the deck.