1) The impacted same sex couples paid real money to have have to their marriages supposedly officially sanctioned by Federal government. By denying the validity of these marriages, the government is essentially saying that it perpetrated a fraud against these people.
2) Saying that Canada can shore up its residency requirement by denying that x number of SSM are not in fact marriages is, obviously, not going to work. After all, same sex couples, from places without SSM, are not the only foreign couples getting married in Canada.
3) The government position is that because "marriages" conflicted with the domicile laws in the country of residence these marriages never happened. As such, the decision could also impact other groups. For example, any inter racial couples from States with Anti-miscegenation laws, who married in Canada, were never, in fact, ever married under Canadian law. Remember the Anti-miscegenation laws in some US states existed up until 1967. South Africa's law existed until 1985.
4) The "legal option" is manifestly unconstitutional. SSM is legal in Canada and their is no residential requirement. There is not a hope in hell that the courts are going to let domicile comity trump equality under the law.