Yes. Absolutely yes. Just as an American citizen can be arrested and put in jail in any foreign country they visit.
As for whether they would be deported or not, it depends largely on the crime they have committed and whether or not their country of origin would continue the punishment sentenced by the US courts. Generally speaking, the foreign citizen would serve their full term in prison (with time off for good behavior), then undergo deportation proceedings.