Can You Beat the S&P 500 by Picking Individual Stocks?

Here’s a question that has fueled dinner-party debates, Reddit threads, and financial advisor consultations for decades: can you, an individual investor sitting at your kitchen table with a brokerage app and a cup of coffee, actually beat the S&P 500 by picking your own stocks?

The short answer is: almost certainly not. The longer, more nuanced answer is: probably not, but there are specific circumstances where you might have a fighting chance — and even if you try, you should do it with eyes wide open about the odds you’re facing.

In 2023, Warren Buffett — arguably the greatest stock picker in history — made a telling admission in his annual letter to Berkshire Hathaway shareholders. He noted that among the hundreds of stocks he and Charlie Munger had bought over their careers, only a dozen or so decisions accounted for Berkshire’s extraordinary performance. The rest? Merely adequate. And this is coming from someone widely considered the best to ever do it.

If the greatest investor of all time credits his success to roughly twelve decisions across a sixty-year career, what does that tell the rest of us?

This article is going to give you the honest, data-driven answer. We’ll look at the professional track record (it’s ugly), understand why the market is such a merciless opponent, explore the few genuine advantages retail investors possess, and figure out whether there’s a sensible way to scratch the stock-picking itch without torching your retirement savings.

The Sobering Reality: What SPIVA Data Actually Shows

Before we even talk about individual investors, let’s look at the professionals — the people who went to the best business schools, who have Bloomberg terminals on their desks, who spend every waking hour analyzing companies. How do they fare?

S&P Global publishes the SPIVA (S&P Indices Versus Active) Scorecard twice a year, and the results are devastating for active management. They’ve been publishing this data for over two decades now, and the conclusion has been remarkably consistent.

Read that table again. Over a 15-year period, more than nine out of ten professional fund managers — people who do this for a living, with teams of analysts, proprietary data, and institutional resources — fail to beat a simple S&P 500 index fund.

And this isn’t a uniquely American phenomenon. The SPIVA data covers markets globally, and the results are similarly bleak almost everywhere. European fund managers, Japanese fund managers, emerging market fund managers — the vast majority underperform their benchmarks over longer time horizons.

But wait, it gets worse. The SPIVA data actually flatters active managers because it only measures funds that still exist at the end of the measurement period. This leads us to one of the most insidious statistical problems in investing.

The Survivorship Bias Problem

When you look at a list of active funds available today and check their track records, you’re only seeing the survivors. Funds that performed terribly have been quietly shut down or merged into better-performing funds, erasing their poor track records from the dataset.

S&P Global tracks this too. Over a 15-year period, roughly 40-50% of all active U.S. equity funds are either merged or liquidated. They simply disappear. The funds that remain look better than the full cohort that started — because the worst performers have been culled.

This is the same statistical illusion that makes successful stock pickers seem more common than they are. You hear about the person who bought Amazon at $18 or Tesla at $30. You don’t hear about the thousands of people who bought Enron, WorldCom, Lucent Technologies, or any of the hundreds of dot-com darlings that went to zero. The winners tell their stories. The losers quietly move on.

The Persistence Problem

Even among the minority of managers who do beat the market over a given period, there’s an uncomfortable question: can they do it again? The research here is equally discouraging. S&P Global publishes a “Persistence Scorecard” that tracks whether top-performing managers maintain their rankings over subsequent periods.

The findings are brutal. Among funds that ranked in the top quartile over a five-year period, less than 20% remain in the top quartile over the next five years. That’s roughly what you’d expect from random chance. In other words, a fund manager’s past outperformance is about as predictive of future outperformance as a coin flip.

This is critical because it means that even if you could identify a skilled manager (or develop skill yourself), the market’s competitive dynamics tend to erode any edge over time.

Why Beating the Market Is So Brutally Hard

Understanding why so many smart people fail at this game is essential before you decide whether to play it. There are several powerful forces working against you.

You’re Competing Against the Best

When you buy or sell a stock, someone is on the other side of that trade. And that “someone” is increasingly not a human at all — it’s an algorithm executing trades in microseconds based on patterns no human could detect.

The modern stock market is dominated by institutional players with staggering resources:

• Quantitative hedge funds like Renaissance Technologies, Two Sigma, and DE Shaw employ hundreds of PhDs in mathematics, physics, and computer science. Renaissance’s Medallion Fund — exclusively for employees — has reportedly averaged 66% annual returns before fees since 1988. These are the people you’re competing against.
• High-frequency trading firms like Citadel Securities and Virtu Financial process millions of trades per day, profiting from tiny inefficiencies that exist for milliseconds. They spend hundreds of millions on infrastructure to shave microseconds off execution time.
• Institutional analysts at firms like Goldman Sachs, Morgan Stanley, and JP Morgan have direct access to company management, attend private conferences, build proprietary models, and have teams of analysts covering every major company.
• Artificial intelligence systems are now scanning satellite imagery of parking lots to estimate retail sales, analyzing natural language in earnings calls for sentiment shifts, and processing alternative data sets that individual investors can’t access or afford.

This isn’t to say these players always win — they clearly don’t, as the SPIVA data shows. But it does mean the market is extraordinarily efficient at pricing in available information. The easy money has been picked up. What remains requires either genuine insight, genuine patience, or genuine luck.

The Relentless Drag of Costs

Even small costs compound into enormous drags on performance over time. When you pick individual stocks, you incur several costs that index fund investors largely avoid:

Dalbar’s annual Quantitative Analysis of Investor Behavior consistently finds that the average equity investor underperforms the S&P 500 by 3-4 percentage points per year over long periods, largely due to poor timing decisions. They buy after prices have risen and sell after prices have fallen — driven by emotions, not logic.

The S&P 500 returned roughly 10% per year historically. If you’re giving up 3-4 points to behavioral errors and costs, you’re earning 6-7%. Over a 30-year career, the difference between 10% and 6.5% on a $500,000 portfolio is approximately $5.7 million. That’s not a rounding error. That’s a retirement.

The Information Asymmetry Illusion

Many stock pickers believe they have an information edge. They read a company’s 10-K filing, listen to the earnings call, analyze the balance sheet, and feel confident they understand something the market doesn’t. But consider this: by the time you’ve read that 10-K, thousands of professional analysts have already read it, modeled it, and traded on it. The information is in the price.

True information edges are rare, often fleeting, and sometimes illegal (insider trading). The efficient market hypothesis, in its semi-strong form, suggests that all publicly available information is already reflected in stock prices. You don’t have to believe markets are perfectly efficient to recognize that they’re efficient enough to make consistent outperformance extremely difficult.

The Math That Works Against You: Bessembinder and the Skewness of Returns

Perhaps the most important — and least understood — reason stock picking is so hard comes from the underlying mathematics of stock returns. A landmark 2018 study by Hendrik Bessembinder at Arizona State University revealed something profound about how wealth is created in the stock market.

The Shocking Findings

Bessembinder studied all U.S. stocks traded on major exchanges from 1926 to 2016 — roughly 26,000 stocks over 90 years. His findings should give every aspiring stock picker serious pause:

• The majority of individual stocks underperform Treasury bills over their lifetimes. Yes, you read that correctly. Most stocks — the majority — do worse than risk-free government bonds.
• Just 4% of stocks accounted for the entire net wealth creation of the U.S. stock market over those 90 years. The other 96% collectively just matched Treasury bills.
• The top 86 stocks (out of ~26,000) accounted for half of all wealth creation. That’s 0.33% of all stocks.
• The single most wealth-creating stock (ExxonMobil) accounted for more wealth creation than the bottom 20% of all stocks combined.

Why This Skewness Matters for Stock Pickers

This distribution of returns creates an asymmetric problem for stock pickers. When you buy an index fund, you automatically own every single one of those 4% of superstars. You own Apple, Microsoft, Amazon, Nvidia, and whatever the next transformative company turns out to be. You don’t have to predict which companies will be the winners — you own them all.

When you pick individual stocks, you’re playing a game where the base rate is against you. Most stocks you pick will underperform. You need to find the needles in the haystack — the handful of companies that will generate outsized returns — while avoiding the vast majority that won’t.

Think of it this way: if you build a portfolio of 20 randomly selected stocks, the probability that your portfolio includes enough of those rare wealth-creating stocks to match the market is quite low. You’re more likely to end up with a portfolio of stocks that collectively underperform.

This is the fundamental mathematical challenge that makes stock picking so difficult. It’s not just about being smart or doing your homework. The distribution of returns itself works against concentrated portfolios.

The Lottery Ticket Analogy

Individual stock returns follow a pattern similar to lottery tickets: most of them lose money (or earn modest returns), and a tiny fraction generate enormous payoffs. The expected value of the entire pool might be positive, but the median outcome for any individual ticket is negative.

An index fund is like buying every lottery ticket. You’re guaranteed to hold the winners. A concentrated stock portfolio is like buying 15-30 tickets and hoping you picked the right ones. Sometimes you will. Usually you won’t.

The Advantages Retail Investors Actually Have

After three sections of doom and gloom, you might be wondering why anyone would even consider picking stocks. And that’s a fair question — for most people, the answer is “they probably shouldn’t.” But intellectual honesty requires acknowledging that retail investors do possess a few genuine structural advantages over institutional investors. The question is whether those advantages are enough.

The Patience Advantage

This is the big one. Professional fund managers operate under enormous pressure to deliver short-term results. If a fund underperforms for two or three years, investors start pulling their money. If it continues for four or five years, the fund may be shut down. This quarterly earnings pressure forces managers to make decisions that may not be optimal for long-term returns.

You, the retail investor, don’t have this problem. Nobody is going to fire you for underperforming for three years. You can buy a stock and hold it for a decade without anyone looking over your shoulder. This is a genuine, meaningful advantage.

Warren Buffett himself has said: “The stock market is a device for transferring money from the impatient to the patient.” If you can genuinely maintain a multi-year time horizon and resist the urge to trade, you have an edge that most professional managers structurally cannot replicate.

The Size Advantage

When you’re investing $10,000 or $100,000, you can buy stocks that large funds simply cannot invest in meaningfully. A fund managing $10 billion can’t take a meaningful position in a $200 million market cap company — the position would be too small to move the needle, and they’d face liquidity issues getting in and out.

This means the small-cap and micro-cap space is one area where retail investors can theoretically find inefficiencies that institutional capital hasn’t competed away. Academic research has consistently found that smaller companies are less efficiently priced than large ones, precisely because large institutional investors can’t operate there.

No Benchmark Pressure

Professional fund managers are judged against a benchmark, usually quarterly. This creates perverse incentives. A manager who trails the S&P 500 by 5% halfway through the year faces career risk, which may lead them to “closet index” — building a portfolio that looks very much like the benchmark — or to take excessive risk to catch up.

You don’t have these constraints. You can hold 100% cash if you see no opportunities. You can hold five stocks instead of fifty. You can invest in unfashionable sectors. You have complete freedom to follow your convictions without career risk.

No Forced Selling

When a mutual fund experiences heavy redemptions — investors pulling their money — the manager is forced to sell stocks to raise cash, often at the worst possible time (since redemptions spike during market downturns). This forced selling locks in losses and prevents funds from benefiting from subsequent recoveries.

As a retail investor, no one can force you to sell. During the March 2020 COVID crash, when markets dropped 34% in weeks, you had the option to do nothing — or even buy more. Fund managers facing heavy redemptions didn’t have that luxury.

An Honest Assessment of These Advantages

These advantages are real, but let’s not overstate them. Having the ability to be patient is different from actually being patient. Research on retail investor behavior shows that individual investors are actually worse at timing than professionals — they trade too frequently, hold losers too long (hoping to break even), sell winners too early (locking in gains), and chase performance.

The structural advantage only matters if you have the psychological discipline to exploit it. Most people don’t.

Strategies That Have Historically Worked

If you’ve read this far and still want to try stock picking — and there’s nothing wrong with that, as long as you do it responsibly — it’s worth understanding which approaches have the strongest historical track records. No strategy works all the time, but some have been more robust than others.

Concentrated Quality

This approach, associated with investors like Terry Smith, Chuck Akre, and to some degree Buffett himself, involves owning a relatively small number (15-25) of the highest-quality businesses you can find and holding them for very long periods.

The criteria typically include:

• High and sustainable returns on invested capital (ROIC above 15-20%)
• Durable competitive advantages (brands, network effects, switching costs, regulatory moats)
• Strong free cash flow generation
• Competent, shareholder-friendly management
• Long reinvestment runways for growth

The logic is compelling: if a company earns 20% on its capital and can reinvest that capital at similar rates, the stock price should compound at a similar rate over time — regardless of short-term market fluctuations.

Terry Smith’s Fundsmith Equity Fund, which follows this philosophy religiously, has delivered roughly 15% annualized returns since its 2010 inception. That’s exceptional — but note the survivorship bias at play. We’re talking about this fund precisely because it succeeded. Plenty of quality-focused funds have not.

Deep Value Investing

This is the original Benjamin Graham approach: buying stocks trading below their intrinsic value, often measured by metrics like low price-to-book ratio, low price-to-earnings ratio, or low enterprise value relative to operating cash flow.

The academic evidence for value investing over very long time periods (decades) is strong. The Fama-French three-factor model identified value as a persistent return factor. But there are important caveats:

• Value has dramatically underperformed growth for most of the past 15 years. Anyone who rigidly followed a deep value strategy from 2010 to 2024 significantly underperformed the S&P 500.
• Many “cheap” stocks are cheap for good reasons — they’re facing secular decline, disruption, or structural problems. This is the “value trap.”
• Deep value investing requires exceptional patience and emotional fortitude. You’re buying things nobody else wants, and you may be wrong for years before the market comes around — if it ever does.

Following Insider Activity

Company insiders — CEOs, CFOs, directors, and other officers — are required to publicly report their stock purchases and sales through SEC filings (Form 4). Research has consistently shown that stocks with significant insider buying tend to outperform the market over subsequent months.

This makes intuitive sense. Who knows a company better than its own management? When a CEO is spending their own money to buy shares, it’s a meaningful signal — especially if it’s a significant purchase relative to their compensation.

Key nuances to understand:

• Insider buying is more meaningful than insider selling. Insiders sell for many reasons (diversification, taxes, buying a house), but they buy for only one reason: they think the stock will go up.
• Cluster buying — multiple insiders buying around the same time — is a stronger signal than a single insider’s purchase.
• The size of the purchase matters. A CEO buying $5 million worth of stock in the open market is more meaningful than buying $50,000.
• This isn’t insider trading. You’re acting on publicly available information (SEC filings), which is completely legal.

Several academic studies have found that portfolios mimicking insider purchases outperform the market by 2-5 percentage points annually. However, transaction costs and the practical difficulties of implementing such a strategy in real-time can erode much of this edge.

Other Approaches Worth Mentioning

Dividend Growth Investing: Focusing on companies with long histories of increasing dividends. The appeal is a growing income stream and the discipline that dividend-paying companies tend to maintain. Companies in the “Dividend Aristocrats” (25+ years of consecutive dividend increases) have historically outperformed — though as this strategy has become crowded, the edge has narrowed.

Special Situations: Spinoffs, merger arbitrage, post-bankruptcy equities, and other corporate events create situations where stocks may be mispriced because institutional investors are forced sellers or the situation is too complex for casual analysis. Joel Greenblatt wrote extensively about this approach, and it has a solid historical track record for investors willing to do the work.

Sector Expertise: If you work in an industry, you may have genuine insight into which companies are winning and which are failing — insights that Wall Street analysts, covering 20 companies across multiple sectors, might miss. A software engineer might genuinely understand why one cloud infrastructure company is technically superior to another, in ways that a generalist analyst wouldn’t.

The “Core + Explore” Compromise

For most investors who’ve read the data and still feel drawn to stock picking, the most sensible approach is what financial advisors call “Core + Explore” (sometimes called “Core + Satellite”).

The concept is simple:

This approach has several important benefits:

It limits your downside. If you allocate 85% of your portfolio to an S&P 500 index fund and your individual stock picks go to zero (which won’t happen with a diversified set of picks, but hypothetically), you’ve lost only 15% of your portfolio. The core holding ensures that your overall portfolio still captures the majority of market returns.

It satisfies the psychological need. Let’s be honest — investing in index funds is boring. Human beings are wired to find patterns, to feel like they’re in control, to compete. Allocating a small portion to stock picking scratches that itch without putting your financial future at risk.

It provides a genuine learning experience. There’s no better way to learn about investing, business analysis, and your own psychological tendencies than to have real money at stake. The lessons you learn from your “explore” portfolio will make you a better investor and a more informed financial consumer.

It gives you an honest benchmark. With the “core” sitting right there in the same brokerage account, you can directly compare your stock picking performance against the index. No mental gymnastics, no cherry-picking time periods — just a straightforward comparison.

How to Size Your “Explore” Allocation

The right percentage for your explore allocation depends on several factors:

• Your financial situation: If you’re 25, have a stable income, and a long time horizon, you can afford a larger explore allocation (maybe 20%). If you’re 55 and approaching retirement, keep it under 10% — or skip it entirely.
• Your knowledge and experience: If you’re new to investing, start with 5% and increase gradually as you learn. Don’t allocate 20% to stock picking if you’ve never read a 10-K filing.
• Your emotional temperament: If you check your portfolio daily and feel anxious about losses, keep the allocation small. The goal is to learn and explore without disrupting your sleep or your relationships.
• The strength of your conviction: Only allocate more if you’ve developed a genuine, repeatable investment process — not because you have a “feeling” about a stock.

Rules for the Explore Portfolio

If you’re going to pick stocks, do it with discipline:

1. Write down your thesis before you buy. Why are you buying this stock? What would make you sell? What’s the specific insight you have that you believe the market is missing? If you can’t articulate it clearly, don’t buy.
2. Set position limits. No single stock should be more than 5% of your total portfolio (including the core). Concentration can work for your best ideas, but a single bad pick shouldn’t devastate your wealth.
3. Minimize trading. Every trade is a decision, and research shows that more decisions means more mistakes. Aim to hold stocks for at least 2-3 years.
4. Ignore the noise. Turn off financial news. Don’t check your portfolio daily. Quarterly reviews are sufficient for a long-term investor.
5. Keep learning. Read annual reports, study investment history, and learn from your mistakes. The explore portfolio is an education as much as an investment strategy.

Tracking Your Performance Honestly

This might be the most important section in this entire article. If you’re going to pick stocks, you must track your performance honestly — and most people don’t.

Common Tracking Mistakes

Here’s how most stock pickers deceive themselves about their performance:

Cherry-picking time periods. “I’m up 40% since March 2020!” Sure — but so is the S&P 500. What’s your return since you started investing? That’s the number that matters.

Ignoring closed positions. You remember your winners and forget your losers. That Netflix pick that tripled lives rent-free in your memory. The Peloton pick that lost 80% has been conveniently forgotten. Track every single position — winners and losers.

Not accounting for timing. Money-weighted returns (which account for when you added and withdrew funds) often look different from time-weighted returns. If you added a lot of money right before a downturn, your actual returns are worse than your portfolio’s returns.

Comparing to the wrong benchmark. If you’re picking small-cap value stocks, comparing to the S&P 500 isn’t entirely fair. But for most investors, the S&P 500 (or a total market fund) is the right benchmark because it’s the alternative — it’s what you’d earn with zero effort.

Ignoring risk. If you beat the S&P 500 by 2% but your portfolio was twice as volatile, you didn’t actually do better on a risk-adjusted basis. The Sharpe ratio (return per unit of risk) is a better measure than raw returns.

How to Track Properly

Here’s a practical system for honest performance tracking:

1. Use a spreadsheet or portfolio tracking tool that calculates your internal rate of return (IRR), also known as money-weighted return. This accounts for the timing of your deposits and withdrawals. Many brokerages provide this automatically.
2. Track from day one. Start tracking from the very first dollar you deploy. Don’t start your “official” track record after your first few picks work out.
3. Include all costs. Commissions, taxes, subscription fees for research tools — everything. These are real costs that reduce your actual returns.
4. Compare to the right benchmark on the same time frame. How much would the same dollars have earned in a total-market index fund, invested at the same times? Most brokerage accounts make this comparison easy.
5. Review annually. Calculate your trailing 1-year, 3-year, and 5-year returns (once you have the history) and compare each to the benchmark. One good year doesn’t mean you have skill. Three or more good years start to be meaningful, though even that could be luck.

When to Stop Picking Stocks

This is perhaps the hardest question, because our egos resist the answer. But here are clear signals that you should move your explore allocation back into the core index fund:

• You’ve underperformed the S&P 500 for 3+ consecutive years after accounting for all costs and taxes.
• You find yourself checking your portfolio multiple times a day and feeling stressed.
• You’ve had a significant loss (more than 50% on a position) that you didn’t have a plan for.
• You’re spending more time on stock research than is reasonable given the dollar amount at stake.
• You’re taking bigger and bigger risks to try to “catch up” to the benchmark.

There’s no shame in concluding that index investing is the right approach for you. In fact, it’s the empirically optimal strategy for the vast majority of investors. Recognizing that takes more intellectual honesty than most people possess.

Conclusion: The Honest Answer

So, can you beat the S&P 500 by picking individual stocks?

The data says: probably not. Over 90% of professionals can’t do it over 15 years, and they have every conceivable advantage over you — resources, training, information, teams, and technology. The mathematical structure of stock returns (with a tiny minority of stocks generating all the gains) makes the odds even worse. And the behavioral challenges of investing — emotions, biases, overconfidence — tend to erode whatever analytical edge you might possess.

But “probably not” isn’t the same as “definitely not.” A small number of investors do beat the market, and some of them do it consistently. The common threads among them are instructive: they have a clearly defined and repeatable investment process, they operate in areas of the market where they have genuine advantages (often small-cap stocks or special situations), they have extraordinary patience and emotional discipline, and they track their performance honestly.

For most readers of this article, the sensible path is clear:

1. Make index investing your default. An S&P 500 or total-market index fund should be the foundation of your portfolio. This gives you market returns at minimal cost — an outcome that beats 90%+ of professionals over the long term.
2. If you want to pick stocks, use the Core + Explore framework. Keep 80-90% in the index and use 10-20% for individual stock picks. This satisfies your curiosity and provides learning without risking your financial future.
3. Track your performance honestly. Compare your stock picks to the index, include all costs, and resist the urge to cherry-pick favorable time periods. Let the data, not your ego, guide your decisions.
4. Be willing to stop. If the data shows you’re not adding value through stock picking after several years, have the intellectual honesty to redirect that capital into the index fund. Your future self will thank you.

The greatest irony of investing is that accepting average market returns — through index funds — actually puts you ahead of the vast majority of investors, including most professionals. In a world where everyone is trying to be above average, the simple act of buying the market and staying the course is itself an extraordinary strategy.

The stock market doesn’t care about your intelligence, your credentials, or how many hours you spent on research. It humbly reminds all of us, every single day, that being smart and being right are very different things.

References

• S&P Global — SPIVA U.S. Scorecard (published semi-annually, available at spglobal.com)
• S&P Global — Persistence Scorecard (published semi-annually, available at spglobal.com)
• Bessembinder, Hendrik — “Do Stocks Outperform Treasury Bills?” Journal of Financial Economics, 2018
• Dalbar Inc. — Quantitative Analysis of Investor Behavior (QAIB), published annually
• Fama, Eugene and French, Kenneth — “The Cross-Section of Expected Stock Returns,” Journal of Finance, 1992
• Buffett, Warren — Berkshire Hathaway Annual Letters to Shareholders (berkshirehathaway.com)
• Greenblatt, Joel — You Can Be a Stock Market Genius, Simon & Schuster, 1997
• Lakonishok, Josef and Lee, Inmoo — “Are Insider Trades Informative?” Review of Financial Studies, 2001
• Bogle, John C. — The Little Book of Common Sense Investing, Wiley, 2007