There's a legend (reported by one of our casebook authors, no less) that while on the Court, Douglas had been audited by the IRS over questions about the taxability of expense reimbursements for taking his wife along on business trips, and he held a grudge. Certainly those who have reviewed his tax opinions from a scholarly perspective can't explain his change of heart. The late Penn and Harvard tax professor Bernard Wolfman, along with co-authors, spent many years studying Douglas's behavior in tax cases, which culminated in a law review article and later a book. They were pretty much left scratching their heads.
Douglas was an enigma in every respect. Hailed by many for his courageous and unwavering stances in favor of personal privacy, civil liberties, and concern for the environment, he's reviled by many for his unabashed judicial activism and the lack of discipline in his opinion writing.
For his personal life, "Wild Bill" is universally condemned. He was divorced three times while he sat on the High Court. He abused his staff and law clerks. He drank. He was slow to pay his debts. Judge Richard Posner of the Seventh Circuit has written:
Apart from being a flagrant liar, Douglas was a compulsive womanizer, a heavy drinker, a terrible husband to each of his four wives, a terrible father to his two children, and a bored, distracted, uncollegial, irresponsible, and at times unethical Supreme Court justice who regularly left the Court for his summer vacation weeks before the term ended. Rude, ice-cold, hot-tempered, ungrateful, foul-mouthed, self-absorbed, and devoured by ambition, he was also financially reckless—at once a big spender, a tightwad, and a sponge—who, while he was serving as a justice, received a substantial salary from a foundation established and controlled by a shady Las Vegas businessman.Other than that, he was a nice guy!
Douglas used to spend some of his summer vacations in Glenwood, Washington, at the foot of Mount Adams. During those months, he used to show up in Portland to party. Our city was a hotbed of vice for many decades, and Douglas took advantage of some of the illegal diversions that were freely available to wealthy and powerful people in those days.
Ace reporter Phil Stanford, who wrote a splendid book on the subject called Portland Confidential, includes several pages on Douglas, especially his connection with a local prostitute named "Little Rusty" Kronberg. The Oregonian picked up on the story a while later, here.