The keepers of the pigs are evangelists of a sort, who rush off from the scene of extraordinary happenings to witness to what they have seen (5:14). And the people, understandably come flocking. Significantly though, when they see the man sitting quietly and no longer deeply disturbed they are frightened. I guess this sort of restoration can be a frightening thing.
The story is unfolded in more detail by the eye-witnesses (5:16), and it seems that at this point the full import unfolds for the citizenry and they plead for Jesus to leave the region (5:17). Sometimes the security of what you know, even if it is oppressive, is less scary that the alternative. Remember that the Children of Israel were barely days out of Egypt when they thought wistfully of the vats of food they had including the pomegranates (Num 20:5). In fact, barely 6 weeks out of Egypt their memories are about how they lounged around pots of meat and had all the food they wanted (Ex 16:3).
Sometimes food in slavery is preferable to scarcity in freedom.
Likewise for these people. They read the signs; they must have seen the calculated sign to the Roman oppressors; they must have remembered how, in living memory, following failed rebellions, tens of thousands of people died horrible deaths to act as a deterrent to future rebellions. Cities reduced to rubble. Roadsides lined with the crucified. In this way the powerful ensure that the powerless are kept in check because dread and fear are terrible things.
So don’t disturb the Romans. Don’t invite violence. Please leave.