There are three factors that should be considered
1) Time
2) How it was taken
3) Location Context
Time
Many items have been around for a very long time and the civilisations that had ownership have come and gone. At some point in time the links con be broken with the past. A trinket from mesopotamia 6000 years ago has a very weak link to the current concept of the nation state
How it was taken
Items get sold, stolen, plundered, lost. If you buy something on the open market that may of been sold or stolen in the murky past it is different from plunder taken as prize. History is littered with wars where people do very desperate things in very desperate situations.
Location context
Some items lose context if not in their original location.