A Florida Billionaire's Investment Playbook - Forbes

Tom James built Raymond James Financial into a regional brokerage and banking powerhouse, now encompassing $1.25 trillion in assets. After sixty years of successfully navigating bull and bear markets, the octogenarian shares his investment wisdom.

^By ^{John Dobosz}

Born in Sandusky, Ohio but a Floridian from the age of three, Tom James excelled as a student at St. Petersburg High School and matriculated at Harvard. He graduated magna cum laude in 1964 and from Harvard Business School in 1966, where he was a Baker Scholar and graduated with high distinction. In Cambridge, he was in a rock band, played tennis and squash and ran several investment clubs before returning to the Sunshine State to the Raymond James investment firm founded by his father, Bob James in 1962.

After earning a Juris Doctor from Stetson College of Law in St. Petersburg in 1969, James assumed the role of chief executive officer in 1970, a position he held for the next 40 years until 2010. Over his tenure, Raymond James expanded from financial management into equity research and investment banking, and it pioneered the model of buying up independent wealth management firms that it effectively ran as franchisor. Raymond James Financial (RJF) became a public company in 1983. If you had invested $10,000 into Raymond James stock at its IPO on July 1, 1983, your investment would have mushroomed into $415,310 by the time James retired as CEO on April 30, 2010. By comparison, a $10,000 investment into an S&P 500 Index fund would have grown into a much more modest sum of $70,260 over the same stretch. With a current market capitalization of $22 billion and annual revenue topping $11 billion, Raymond James Financial has 8,700 financial advisors, $1.25 trillion in assets under administration and FDIC-insured banking operations with loans totaling $44 billion.

Long term investors in NYSE-listed RJ Financial have been handsomely rewarded.

YCharts

Tom James, who’s net worth is estimated to be $2.2 billion including a 9% stake in the company’s stock, is now chairman emeritus and member of Raymond James’ board of directors. Along with his wife, Mary, Tom James is a major philanthropic force in the Tampa Bay area, serving as president of the Salvador Dali Museum from 1994 until 2017 and still serving as a trustee. He’s also chairman of the Chi Chi Rodriguez Youth Foundation and the James Museum of Western and Wildlife Art in St. Petersburg.

FORBES: Tell me about your experience as a Florida kid going to Harvard as an undergraduate in the early 1960s.

THOMAS JAMES: My father had been a Harvard graduate, but he didn't even try to sell me. I went to public school. I didn't go to prep school. I chose Harvard because when I looked at Yale, I didn't like New Haven as much as I liked Boston. I played varsity tennis and squash, and I played house football and basketball. I didn't even know what squash was when I got there. I belonged to a social club that had all athletes in it, and at Harvard Business School I was on a rugby team. I quit after about four games. I said, “I'm gonna get hurt doing this.”

I was a very unusual major. I switched my majors from physics and math my sophomore year. I said I don't want to teach, and what am I doing spending all this time in organic chemistry labs, and all these courses. I don't want to be a doctor. I don't want to be any of this: I want to be a businessperson, so I switched to economics and philosophy. I stayed the summer and caught up on economics courses, although I'd already taken one in freshman year.

Because of my mathematical background I took game theory and I started getting interested in more games and more stuff. It just so happened that Tom Schelling, a very famous game theorist who worked with Henry Kissinger advising the defense department, was teaching at Harvard. He’s also the inventor of the red hotline that sat on the president’s desk. I decided that I'd focus on game theory type stuff and got very interested in war strategy. I wrote my thesis on missile hardness and Russian psychology, in which I said deterrent strategy only works if the people on the other side value life. Nobody did anything like that. I went to the war colleges when I was there. I went to RAND Corp. and I got into things thanks to Tom Schelling. He was one of the four guys who started the Kennedy School, and he was a very Influential guy for me. When I got out of Harvard College in 1964, if I was going to be drafted and didn't have a deferment, I would have been recruited by the defense department. If you remember those days, the head of the Defense Department, Robert McNamara, was a Harvard Business School guy, and another Harvard Business School guy ran the accounting department at the defense department. (James was never drafted.)

You had several options it seems. If you didn’t go to the Pentagon or into finance, you might have become a rock’n’roll star. Is that right?

I saw Elvis at a theater in downtown St. Petersburg in the 1950s, and I walked out of there convinced I was going to be a rock and roll singer and I said I’ve got to get a guitar. At Harvard, I went to a mixer when I was a freshman and there was a rockabilly band run by a Canadian named Gordie Main. I went up to him, and he needed a rhythm guitarist, so he invited me to a rehearsal, and I joined the band, appropriately called the Maniacs. I played guitar and was lead singer. I actually did a record called “I’m Losing Irv To The Ready Reserve,” and another song was “John Foster Dulles Rock.”

You’re best known as the guy who made Raymond James a major national player in financial services, but you actually have a background in portfolio management, too.

I built a company. I was a portfolio manager, but it was a part time job when I started our asset management division. I had a good record through the 1974 downturn and coming out of it. A lot of my reps were bringing me money to manage. I said I can't do this anymore. I don't have enough time to do an adequate job of this and I hired Herb Ehlers, who became one of the most famous money managers in the business. Thank God. He has a great track record, and I was good through the down market, so we had a record that was marketable. We started out just with retail clients, but pretty soon we started getting phone calls from institutions. I learned early in business that when you see opportunities, don't fall down when you trip over them.

How did you get your start in investing?

My father. I went to the mutual fund conventions when I was 12 or 13 years old. I’d go down on Saturday mornings to my father's office and watching him interview retirees who were coming to Florida and doing financial plans for them. Mainly they were smaller and medium sized companies that I got interested in. I read the Wall Street Journal when I was in college and business school and so I followed all these smaller companies and actually formed a stock investment club while I was at Harvard and then started a new one at the business school that turned into a corporate form of ownership when I graduated with 13 other business school graduates. We put about $7,000 each into the fund. I was the managing partner, so to speak. I was the guy that did all the stock execution since I had a firm even though a lot of these guys were famous in the investment business. We also did some venture capital investing and we'd let the group in for a very small investment to have a share.

Is there a single investment among the many you've made over the years that you regard as your best ever?

It's hard for me to pick individual names, but I owned a lot of Coca-Cola (KO) and Pepsi (PEP) over the years. I thought they were very attractive low-risk stocks to own, and they performed well. I owned a lot of the healthcare stocks, too, like Baxter Labs (BAX) and made a lot of money in Medtronic (MDT). I owned General Electric (GE) because I really respected Jack Welch as a manager. I had a defense manufacturer named ABA Industries, which made parts for GE here in St. Petersburg. I made a lot of money in oil stocks. I was an oil general partner starting in 1969 and kept doing that until about 1985 when I stopped taking new money. We created investment bankers who specialized in limited partnerships, and we did oil and gas real estate, cable television and leasing. We made a ton of money for our investors. At the bottom of the market in 2009, I bought the big banks. I bought more JPMorgan (JPM) than anything else, partially because I’ve known Jamie Dimon for more than 35 years. I also bought a lot of Bank of America (BAC). It was the cheapest. Overall, my best investments were in myself, my team and in these new businesses that I started. I've loved doing business in Florida and building a company.

What is something you would tell your 20-year-old self about investing that you wish you knew back then?

I’ve made some mistakes, but none of them were fatal. The one thing that I would say is to avoid that guy who comes to you with a stock idea because it’s acting right, yet you can't figure out what's really going on at the company and it sounds like so much bullshit. Back in the bear market of 1974, I went to the best business school in history called the school of hard knocks. If you want to have your arrogance knocked out of your head, just go through a period like that. You have to take risk, but there are times when you do it and there are times when you don't do it. Every retail investor doesn't want to invest at the bottom of the market, but I want to invest at the bottom of the market. That's what I would say my mentality always was.

Are there some fundamental criteria that you use to select companies worthy of investment?

I don't really like to look at stocks that don't have real GAAP earnings. I want to know they have great management and that they are in an industry that looks like it has a bright future and isn’t overpopulated so you can’t determine who’s going to be a winner. This is what’s happening in many tech companies, especially with artificial intelligence (AI) right now, with everybody getting on board. I guarantee that over half those companies are going to fail and disappear. Trying to pick the one out of the group you want to buy is hard right at the beginning, but I did that back in the early days because I could ask a lot of questions. I could find people who knew. More recently, I was involved in the vaccines with Moderna (MRNA). I had a good ride. Why did I get involved? Because I heard about the stock from Harvard Business School people who were talking about Moderna being able to do things faster than anybody did in the industry.

What kinds of investments catch your eye and your money lately?

My biggest investments now are those I think I really understand, like the major banks. Everybody got all upset about this banking crisis, but this was not a crisis. It was terrible management by a few companies that didn't understand duration risk. I mean, you’ve got to be out of your mind. I own a bank here at the firm, and there is no way we would do this stuff. If a company could go down the tubes for one bad thing happening, I’m not going to buy it. There are too many good stocks where you don't have that risk, so you need to be careful. That's why I talk about good management first, and then go look at the business and look at their results and projections. I want growth. My friend Herb Ehlers always said, I want to be a ticket taker at the toll bridge and get more money every year as more and more people go across the bridge. That's really the kind of company I want to own. Right now I'm buying Verizon (VZ). It has a good yield [7%] and it’s priced at the low end of its range over the last five years [ P/E ratio of 7]. I'm not worried about it going broke.

Along with your father, who has informed your approach to investing?

I've always followed Mario Gabelli, and our friend at Fidelity, Peter Lynch. Bill Miller from Legg Mason. He had a great run and then he started cheating on his own formula. Next thing, he was in some trouble. There have been small company managers that I knew. I really respected Jeremy Grantham, a classmate of mine at Harvard. My own guy, Herb Ehlers. He was solid. He would go in his office and lock the door during market hours, and at lunchtime he'd come out, go across the street with a stack of about ten annual reports to the country club and have lunch by himself and read the annual reports, go back to the office and lock the door again, then come out and talk to everybody on the market close. I was really lucky when I found Herb, who had worked at [Pittsburgh’s] Parker/Hunter as a housing analyst and then went to U.S. Steel as vice president of financial reporting. Then he retired in his late 40s, and traveled to Europe. He’d been managing money on the side for his family, himself, and all kinds of friends. [James recruited Ehlers, who took over asset management at Raymond James in 1984, renaming it Eagle Asset Management. He left in 1994 to form his own asset management company, Liberty Investment Management, which had $5.4 billion under management when it sold to Goldman Sachs in 1996.]

What is the biggest risk that an investor faces either from a broad strategy standpoint or from the current investment environment?

The first thing to know, if you look at almost any ten-year period, stocks go up. People are lousy at determining bottoms and tops. I like to use dollar-cost averaging, and putting money into the market over a number of years. If you walk into my office with a lot of money, I say if you want to invest it over the next two years, we'll do it quarterly eight times. You can't beat the market long term.

Thank you.