Maybe you’ve joked about not paying your taxes because, say, your candidate didn’t win or the Feds don’t seem to be using your payments to fix potholes on your interstates. Should you ever seriously think you may not be paying your taxes on time, though, it’s no joking matter. Penalties and restrictions are set in motion over a period of time, so it’s not exactly a “my life is over at 12:01 a.m. April 16” thing, but that’s just at the beginning. You probably already know the IRS doesn’t play, but here are all the steps that happen before you get in serious trouble for not paying taxes.

The second you didn’t pay your taxes in full by April 15

If you didn’t file your taxes by the federal due date, which is April 15 unless that day fell on a weekend, you immediately set yourself up for three negatives. First, the clock starts ticking on interest on the tax you owe. This interest also accrues if you filed the automatic extension that grants you an extra six months to get your tax return in or got Form 1127, Application for Extension of Time for Payment of Tax Due to Undue Hardship, approved.  In other words, if you more than 10 percent of your taxes after April 15 there’s no way to avoid accruing interest without paying the tax you owe.

MarketWatch

For those who didn’t pay the taxes or file for either extension, the IRS also heaps on two types of penalty. One is for filing late, the second is for paying late. And this might sound incredible, but the IRS would rather have you file without paying what you owe than fail to file. So it exacts a heavier fine for late filing, equal to five percent of the unpaid amount on that tax year for each month you’re late. What makes it even harsher: If you go over 60 days late before filing, they charge a minimum of $205 and up to 100 percent of tax owed just as a penalty.

If you go ahead and file, your late payment penalties will be comparatively small. The IRS charges 0.5 percent of unpaid tax each month you don’t ante up, up to a total of 25 percent. The IRS will also give you a chance to prove you had reasonable cause either for failing to file or failing to pay. If that’s genuinely your situation it may pay to look into it. But if you just didn’t do the tax thing this year, it’s better to use your time and money filing and trying to arrange a payment plan.

Let this go another 1-3 months and…

While five percent extra for one month of failure-to-file doesn’t feel great, you can expect to feel worse sometime between one and three months after you didn’t file or pay taxes. That’s when you’ll start getting friendly reminders from the IRS. At first, they just note you’ve got a balance due, kind of “Hey, perhaps you overlooked this please get on it,” only in government language. But after the first letter of that sort, the tone starts getting more strict. Could it get any worse? Yes, it could, if you keep ignoring the matter for a month or more longer.

At the 2-6 month mark of not paying taxes

A couple of months in, the IRS starts playing hardball. They’re out to get that tax balance (and the ever-increasing interest) paid, and they may employ a tax lien to do it. This is a court-ordered claim on property or financial assets you have that dictates the proceeds come to the IRS should you sell. (This is different from the IRS out and out seizing the property itself.) A lien also applies to assets you have coming to you, like an inheritance or back pay. Liens could hurt your borrowing power or have a negative effect when employers check your credit history for employment or a security clearance.

Around this same time, the IRS may opt to sic a private collection agency on you. If you owe tens of thousands, this may also be the point where the IRS involves a revenue officer who comes to the house to collect taxes in person.

After three months, the heat is on

While the timetable may vary by a few months, after three months of not paying taxes you are in levy territory. That means the IRS legally seize your assets. It can auction property like cars or boats to pay down the tax, and also seize anything in your bank account and the money you get from Social Security. Unlike ordinary creditors, the IRS can also call in monies you have in an IRA or 401K.

In this time frame, the IRS can also refuse to issue a passport or renew an old one. It may even revoke an active passport for failure to file or pay back taxes.

Protect your passport and truck with a payment plan

Getting hooked up with a formal payment plan lets the IRS know you’re serious about getting them their money and can help you avoid the drastic measures like levies or even liens. The plans available range from short-term for people who can pay up within four months, to long-term for those who need more than 120 days but owe less than $50,000 for the tax, interest, and penalties combined. There are also installment agreements for people who owe over $50,000.

Both installment and long-term payment plans come with set-up fees, and you may have to go more in-depth (and offline) to get the long-term deal. Sure, it’s a pain. But it’s not prison, and you’ll eventually have the IRS off your case if you’re proactive.