In the previous chapter the Refugees talking about this made a statement like "Since we left our cities they've been with us" Meaning the moment Ousen sent them from the cities hidden soldiers were in their numbers. And also the hidden Refugees in this chapter said they we're Ousen's men.
So the entire reason why Ousen sent the Refugees away from their cites and made sure they came to this current city wasn't just to over crowd the city to force them to use up some of their food, it was also to infiltrate Ousen's men into the city to burn the rest of the food.
Meaning this was Ousen's plan, but there was the chapter when Ousen gave the reigns to Kanki to take this city, Kanki realized what Ousen had done, and btw Kyoukai figured it out to.