Don't worry, the Supreme Court has interpreted that to mean it only applies in criminal cases, and even then there are only 4 kinds of situations in which it applies [1].

1st. Every law that makes an action done before the passing of the law, and which was innocent when done, criminal; and punishes such action.

2nd. Every law that aggravates a crime, makes it greater than it was, when committed.

3rd. Every law that changes the punishment, and inflicts a greater punishment, than the law annexed to the crime, when committed.

4th. Every law that alters the legal rules of evidence, and receives less, or different, testimony, than the law required at the time of the commission of the offence, in order to convict the offender.

[1] https://en.wikipedia.org/wiki/Calder_v._Bull