Because it talks about other people in other places. Also, in context it could be referring to the British forces themselves. It had already been used as a rhetorical device for that after the revolutionary war.

Really though the idea that he would take a break in a poem about the war of 1812 and specifically the bombardment of Fort McHenry to dunk on slaves is just weird too. It doesn’t fit.

Here’s the complete extra stanzas.

On the shore dimly seen through the mists of the deep Where the foe’s haughty host in dread silence reposes, What is that which the breeze, o’er the towering steep, As it fitfully blows, half conceals, half discloses? Now it catches the gleam of the morning’s first beam, In full glory reflected now shines in the stream, 'Tis the star-spangled banner - O long may it wave O’er the land of the free and the home of the brave!

And where is that band who so vauntingly swore, That the havoc of war and the battle’s confusion A home and a Country should leave us no more? Their blood has wash’d out their foul footstep’s pollution. No refuge could save the hireling and slave From the terror of flight or the gloom of the grave, And the star-spangled banner in triumph doth wave O’er the land of the free and the home of the brave.

O thus be it ever when freemen shall stand Between their lov’d home and the war’s desolation! Blest with vict’ry and peace may the heav’n rescued land Praise the power that hath made and preserv’d us a nation! Then conquer we must, when our cause it is just, And this be our motto - “In God is our trust,” And the star-spangled banner in triumph shall wave O’er the land of the free and the home of the brave.