How to Legally Stiff-Arm the IRS

To earn the highest net return on your investments, you’re going to need to take one important step… You need to tax-manage your investment portfolio.

One of the biggest burdens to accumulating generational wealth and something that can do serious damage to the long-term value of your portfolio is taxes. As Arthur Godfrey said, ’’I’m proud to pay taxes in the United States; the only thing is, I could be just as proud for half the money.”

If you’re like most investors, there’s a good chance you’re paying more in taxes on your portfolio each year than you need to. So let’s look at how you can keep your tax bite to an absolute minimum and legally stiff-arm the IRS.

(I use mutual funds in the examples below but many of the same principles apply when deciding to purchase specific stocks or bonds.)

Pre-Tax vs. Post-Tax Returns

Mutual fund advertisements and the financial media love to tout funds’ total returns. These returns take operating costs (expenses) into account, but not taxes. However, two funds with identical gross returns can deliver drastically different returns after taxes.

Mutual funds are required by law to distribute at least 90% of their realized gains each year. You can get hit with a big tax bill even if you haven’t sold a share. How? Inside the fund, the manager may be buying and selling like mad, turning over the entire portfolio in less than a year. Although this doesn’t necessarily hurt the fund manager’s annual bonus, it can have a dramatic effect on your real-world returns. After all, you may owe taxes on all those short ­and long-term capital gains, even if you haven’t sold a single share.

Back in 2007, Lipper, a global leader in fund information and analytical tools, published a study, Taxes in the Mutual Fund lndustry-2007: Assessing the Impact of Taxes on Shareholder Returns. It found that taxable mutual fund investors surrendered at least $23.8 billion to Uncle Sam in 2006, just for buying and holding their funds! Taxes gobbled up 15% of the gross return of the average U.S. diversified equity fund. And the tax hit was even worse for the average U.S. taxable bond fund. Here, 38% of the gross return was lost to taxes, nearly double the cost of operating expenses and loads combined.

Believe me, nothing has changed since then…

If anything, this study may have actually understated the tax costs. Why? Because it included the 2000 to 2002 bear market in stocks, so tax-loss carry forwards (and favorable changes in the tax code) actually mitigated the tax burden.

If you are voluntarily surrendering thousands of dollars to the IRS each year, it makes it much tougher to meet your long-term financial goals. For this reason, you need to tax-manage your portfolio to increase your real-world returns. Here’s how.

The Taxman Cometh

Your annual tax liabilities will depend, in part, on both your tax bracket and how much of your portfolio is held outside of qualified retirement plans. I’m going to run through a few different scenarios, allowing you to easily adopt the strategy that is closest to your own personal situation.

Let’s start with the easiest scenario. If all your long-term money is in a tax-advantaged account like an IRA, Keogh, 401(k), 403(b), private pension plan, or annuity, you can stop sweating. You don’t have to be concerned with the tax ramifications of your asset allocation and rebalancing strategy because you don’t owe any annual taxes on the investments in these accounts. You’re safe from the tax­man until you begin making withdrawals. So if all your long-term money is in a qualified retirement plan, you can skip the rest of this section. You’re already home free.

But, if you’re like most investors, your personal situation is probably a bit more complex. You likely have liquid assets both inside and outside of retirement accounts. In that case, you’ll need to do a bit of tax planning.

The first order of business is to place the appropriate funds in the right accounts for maximum after-tax returns. You’ll need to put the most tax-inefficient funds into your tax-deferred accounts and the remaining funds in your taxable accounts.

For example, REITs are highly tax-inefficient. Most of your return will come in the form of dividends and these are taxable at your income tax rate, not the 15% or 20% capital gains tax rate. For this reason, a REIT Index Fund should be one of the things you place in your tax-deferred account.

Another tax-inefficient asset is high-yield bonds. Here the majority of the return comes from interest income and all of it is taxable. A junk bond fund will typically make capital gains distributions from time to time, as well. So a high yield fund should also be placed in your tax-deferred account, if possible.

Also highly tax-inefficient are inflation-protected securities (TIPS). The semiannual interest payments on TIPS are taxable, the same as other Treasury securities. However, investors are also taxed on inflation adjustments to the principal, a situation that is commonly described as taxing phantom income. For these reasons, you should also hold your inflation-adjusted Treasuries in your tax-deferred account.

High-grade corporate bonds and ordinary Treasuries pay taxable income, too. They, too, should be held in your tax-deferred account, if possible. However, if you’re running out of room in your retirement account at this point – and especially if you reside in an upper tax bracket – you should make a substitution here.

Instead of buying a short-term corporate bond fund, invest in an intermediate-term municipal bond fund. You’ll get a slightly lower yield, but your dividends will be exempt from federal taxes. (As an added tax benefit you can buy a fund which only buys municipal bonds issued by your home state. Thus shielding you from state taxes as well).

A similar tax-reducing strategy should be considered for your stock holdings. If you still have room in your tax-deferred account, own a small-cap index fund there. If you don’t, consider owning a tax-managed small-cap fund in your taxable account. Again, this is especially important for investors who reside in the top tax brackets. That’s because long-term capital gains rates can be as high as 20% depending on your income-tax bracket.

In tax-managing your assets, it is important to put your high­-yield bonds and REITs in your retirement accounts first. Why? Because these are the highest-yielding components of your portfolio and there is no tax-advantaged substitution you can make. There is no tax-free substitution for inflation-adjusted Treasuries, either. So plunk all those type of investments in your retirement accounts.

Everything In Its Place

Tax managing your portfolio is essentially your asset “location” strategy. Ideally, you want to own your least tax-efficient assets inside your retirement account and the most tax-efficient outside them. Effective tax-management of your portfolio is critical and can dramatically increase your long-term, real-world returns.

Please don’t think that this step isn’t worth the trouble… It is. For example, Vanguard founder John Bogle gave a lecture at Washington State University where he pointed out that the average mutual fund takes 2.5% in annual costs each year. Taxes take another 2%, on average. No wonder the average mutual fund investor feels like he’s on a slow boat to China. You can’t reach financial independence as quickly if you’re surrendering so much of your annual returns to the taxman and the mutual fund industry.

A brief illustration shows you why. Let’s say one investor owns a $100,000 tax-managed portfolio of mutual funds with an average expense ratio of 0.5%. Another invests the same amount, but is surrendering a total of 4.5% each year in taxes and expenses. Even if both portfolios have 10% gross annual returns, the results over time become dramatically different. (See table.)

The investor who keeps his taxes and expenses to a minimum ends up with a portfolio worth more than three times as much… and that’s without generating gross returns that are any better! Clearly, if you’re not doing everything possible to minimize your investment costs and taxes, you’re at a serious disadvantage.

As a financial writer, I’ve written and spoken about this topic many times. Occasionally, this strategy provokes anxiety from some investors who see tax-management strategies as an abdication of their civic responsibilities.

Nothing could be further from the truth. As a law-abiding U.S. citizen, you need to pay all the taxes you are obligated to pay and not one penny more. As Judge Learned Hand, who served for years as Chief Judge of the U.S. Court of Appeals for the Second Circuit, famously wrote:

Amen, Judge.

Let me remind you, too, that our strategy is to wait at least a year and a day before rebalancing your portfolio. That means you will never be subject to short-term capital gains taxes, which can run as high as 37%. If you choose to rebalance your portfolio every 18 months, you can reduce your annual tax liabilities further, as you will only be creating a taxable event approximately every other year.

As Vanguard founder John Bogle has said, “Fads come and go and styles of investing come and go. The only things that go on forever are costs and taxes.”

In short, taxes matter… a lot. Take the basic steps I’ve outlined here to tax manage your portfolio and you’re assured of higher real world, after-tax returns.

In Conclusion…

1. Voluntarily surrendering a significant percentage of your annual returns to the IRS each year makes it much tougher to meet your long-term financial goals.
2. Taxes can potentially be larger than your other annual investment costs combined. Studies show that typical investors surrender 2% of his annual return to the taxman each year.
3. Maximizing your real-world returns means tax-managing your portfolio to minimize the annual tax liabilities.
4. Implement an asset “location” strategy. Keep your tax-inefficient investments – such as bonds, REITs, and small-cap funds – in your retirement accounts. Keep your tax-efficient investments – such as large-cap stock index funds – in your non-retirement accounts.
5. If needed, substitute a tax-managed stock fund for a stock fund that is not tax-managed. The same goes for bonds; substitute a tax-exempt fund for a traditional bond fund.
6. Wait at least a year and a day before rebalancing. If you want to be even more tax-conscious, you can choose to rebalance every 18 months, further reducing your annual tax liabilities