When Sam Gamgee believed Frodo to be dead outside Shelob's Lair, he left the shirt with Frodo. Frodo was taken by the orcs, who fought over the shirt. Frodo was saved, but one of the orcs escaped with the shirt. The shirt was, along with Frodo's other possessions, shown to Frodo's allies at the Black Gate to falsely imply that he was captured. Gandalf took the shirt and other tokens, but refused any offer of parley.
Ah yes, showing a material object having belonged or rightfully belonging to someone does not imply owning that person.