Declining businesses are companies whose revenues are shrinking on a permanent basis, usually because their products have become obsolete. In my experience, these are almost always bad investments.
Most declining businesses were once growing and highly profitable-- e.g., newspapers or yellow pages-- and as a legacy of that, they carry large debt loads. For a stable or growing business, debt is a non-issue because the market is happy to refinance it absent a crisis and carrying the debt doesn't preclude paying dividends or buying back stock.
The situation is very different for declining businesses, for which debt becomes a kind of supra-equity: not only does the company have to use declining profits to make fixed interest payments, it has to repay the debt's principal before shareholders can receive any money.
Even without the problem of debt, investing in a declining business is an uphill battle: long-term investment returns depend on the power of compound growth, and declining businesses have the opposite of that.
Another issue is that it's hard to know how quickly a declining business will fade away since there are no precedents for its decline, at least not in the same industry. That uncertainty makes it tough to value these companies with much confidence.
A good case study is US Mobility (USMO), which provides paging services. At the beginning of 2005, USMO traded at $35/share. Today, after paying $16 of dividends, the stock trades at $19/share. USMO had a lot of advantages that other declining businesses don't: it had large tax-loss carryforwards, it was debt-free, its management was committed to distributing profits to shareholders, and its business wasn't going away completely: pagers have no alternatives in certain places, like hospitals, and paging networks are much more robust than cellular networks, so pagers will always have a use as a backup after natural disasters. Despite all those strengths, USMO has produced a return of exactly zero over the past nine years.
A common argument for investing in declining businesses is that they tend to be undervalued because they have no natural constituency: institutional investors don't want them, growth investors don't want them, et cetera. That might have been the case historically, but I don't think it's true today. Many hedge funds look specifically for out-of-the-way stocks like micro-caps, spin-offs, and declining businesses. Many investments that were traditionally overlooked are now picked over.