Happy New Year! 🎉

As I’ve been reflecting on 2025, I realized I spent a significant part of the year diving deep into the world of AI agents. What started as theoretical exploration became a hands-on journey of designing, building, and learning from failure.

I took a stab at the agentic world and tried to apply my learnings into the exciting (and humbling) world of prediction markets. I’d love to hear what challenges other builders have faced while creating self-healing agents—the ones that can detect when they’re wrong and actually do something about it.

Here’s what I built, what broke, and what actually worked.

2025: A Year of Building AI Agents

When I started 2025, I had a lot of theoretical knowledge about autonomous systems:

• How agents should observe their environment

• How they should reason about options

• How they should execute decisions

• How they should monitor outcomes

But theory and practice are different beasts.

I decided to test these concepts in a domain where feedback is immediate and consequences are real: prediction markets. Unlike mapping systems where you wait hours to see if a routing change helped, prediction markets give you feedback in real-time. The price moves. You know if you’re right or wrong in minutes, not days.

This became my laboratory for understanding what “self-healing” actually means.

The Problem with Most Betting Strategies

Most prediction market participants operate like this:

1. Read news

2. Get a feeling: “This event feels 70% likely”

3. Place a bet

4. Watch it for days/weeks

5. Hold until resolution (win or lose)

6. Repeat with next market

The problem? They never ask: “Was my feeling right? And what should I learn from it?”

I wanted to build something different. An agent that could:

• Systematically estimate probabilities

• Detect when the market was mispriced

• Place bets with calculated expected value

• Monitor in real-time to catch thesis breaks

• Self-correct before small mistakes became big ones

The Agent’s Goal (Stated Explicitly)

Before diving into architecture, here’s what I explicitly optimized for:

Primary Goal: Maximize risk-adjusted returns (Sharpe ratio > 1.5) by identifying and betting on mispriced prediction markets.

Secondary Goals:

• Maintain win rate > 55% on bets with positive expected value

• Keep portfolio drawdown < 20% at all times

• Detect thesis invalidation within 24 hours of new information

• Learn and improve calibration from every resolved market

Guard Rails (Never Violate):

• Never risk > 2% of portfolio on single bet

• Only exit trades, never hold to catastrophic loss

• Recalibrate confidence estimates monthly based on outcomes

• Stop betting any market type if win rate drops below 50%

This explicit goal statement is critical. Without it, the agent is just optimizing for “make money,” which leads to overleverage, reckless bets, and exactly the kind of gambling that most prediction market traders fall into.

The goal keeps the agent disciplined. It’s not “win as much as possible.” It’s “win consistently with controlled risk.”

The Architecture: A Loop That Learns

Here’s the core flow:

Detect Mispricing 
    → Estimate Probability (+ Confidence)
    → Calculate Expected Value
    → Place Bet (if EV > threshold)
    → Monitor Real-Time
    → Detect Thesis Invalidation
    → Exit or Hold Based on New Info
    → Log Outcome
    → Update Beliefs
    → Loop

The “self-healing” magic happens in steps 6-8. Most bettors skip these.

Challenge 1: Probability Estimation is Deceptively Hard

What I thought would work:

• Aggregate news on a topic

• Use LLM to reason about it

• Get a probability: “The candidate has a 62% chance to win”

• Done.

What actually happened: I’d say “62%” with zero understanding of whether I was 95% confident in that estimate or 50% confident.

The Real Problem

Probabilities need two components:

1. Point estimate (”I think it’s 62%”)

2. Confidence interval (”I’m 75% sure it’s between 55-70%”)

Without both, you’re flying blind on position sizing.

What I Did

I started tracking calibration curves. After 20 bets, I noticed:

• When I said “80% confident”: Actually won 70% of the time (overconfident)

• When I said “50% confident”: Actually won 52% of the time (decently calibrated)

• When I said “30% confident”: Actually won 28% of the time (underconfident)

This single insight changed everything. I started scaling positions inversely to my overconfidence.

Key learning: Build your confidence calibration curve before you scale. One bet at a time, at different confidence levels. After 30-40 bets, you’ll see the pattern.

Challenge 2: Market Efficiency Kills Obvious Edges

What I thought would work: “I’ll just monitor news better than the market does. I’ll read political polls, Twitter sentiment, and spot opportunities the crowd misses.”

Reality: If it’s public on Twitter, the market priced it in 30 minutes ago.

Where I Actually Found Edge

1. The 24-48 hour window after news drops

   • Sentiment shifts faster than market repricing

   • Tweet volume spikes but market hasn’t caught up yet

   • Example: Polling data came out showing candidate X dropping 4 points. Took market 18 hours to fully reprice. I was out in 6.

2. Correlation mispricings

   • Market prices “Trump wins” at 60%

   • Market prices “USD strengthens” at 45%

   • But historically, when Trump wins, USD strengthens 70% of the time

   • The two bets aren’t priced consistently relative to each other

   • Bet accordingly.

3. Overweighting recency

   • Latest scandal gets 10x news coverage

   • Market overreacts

   • Historical base rates suggest it won’t matter that much

   • Wait 48 hours, odds shift back, take other side

Key learning: Stop looking for obvious mispricings. They don’t exist anymore. Instead, find structural inefficiencies (correlation) or temporal ones (market hasn’t updated yet).

Challenge 3: Self-Correction is Operational, Not Magical

This is where most “self-healing” systems fail. They monitor prices, not theses.

The Wrong Way

Bet placed: "Event X" at 60% odds
Current price: 70% (I'm losing)
Decision logic: "I'll hold because I believe in my thesis"
Reality: You're holding because you're stubborn

The Right Way

Bet placed: "Event X at 60% odds"
My thesis: "New policy will boost candidate's popularity"
T=12 hours: New information arrives
New reality: "Policy actually polls poorly"
Recalculate: "My thesis is invalid, not just unprofitable"
Decision: Exit immediately at current loss (30%)
Alternative: Wait for full resolution at ~100% loss

The difference between -30% and -100% is self-healing.

How I Built This

I implemented “thesis invalidation detection”:

def monitor_position(position):
    current_market_price = get_latest_odds(position.market_id)
    
    # Recalculate: Is my thesis still valid?
    new_estimate = recompute_belief_from_latest_news()
    updated_confidence = recompute_confidence()
    
    # Is EV still positive?
    new_ev = calculate_ev(new_estimate, current_market_price)
    
    if new_ev < 0.02 or updated_confidence < 0.50:
        # Thesis broken, exit
        exit_position(position)
        log_learning("thesis_invalidated")
    elif updated_confidence > 0.75 and new_ev > 0.10:
        # Thesis even stronger, consider adding
        add_to_position(position)

Key learning: Monitor thesis validity, not just price. Exit when thesis breaks, not when price hurts.

Challenge 4: Portfolio Correlation is Invisible Until It Kills You

I had 5 different bets I thought were independent:

1. Political prediction: Candidate wins (50/50)

2. Tech regulation: Bill passes (60/40)

3. Crypto policy: New law doesn’t pass (40/60)

4. Market impact: Tech stocks outperform (70/30)

5. Election outcome: Impacts inflation (45/55)

All five lost in the same week.

Why?

They were all betting on the same underlying thesis: Risk-on environment. When sentiment shifted to risk-off, all five correlated and tanked together.

The Fix

Before placing any bet, I added a correlation matrix check:

New bet: "Tech stocks outperform"
Existing positions:
  - "Candidate wins" (correlation: +0.65)
  - "Inflation stays low" (correlation: +0.58)
  - "Crypto volatility increases" (correlation: -0.72, hedging)

Check: Would this position increase portfolio risk?
If correlation to existing bets > 0.60, reduce position size

Key learning: You’re not betting on N markets. You’re betting on 2-3 underlying macro themes. Map your correlations first.

Challenge 5: Liquidity is a Silent Killer

I placed a $2000 bet on a market that showed $100k volume. Looked liquid.

Three days later, I wanted to exit. Volume dried up. Bid-ask spread was 15 cents wide. I couldn’t get out without taking a massive loss.

The Guard Rail

I added a simple rule:

• Only bet if volume > $50k

• Only bet if can exit within 24h at < 5% slippage

• Check bid-ask spread before entry

This simple rule cost me some opportunities but saved me from being trapped multiple times.

Key learning: Liquidity matters more than odds. A slightly worse price on a liquid market beats a great price on a dead one.

What Actually Worked: The Winning Strategies

1. Kelly Criterion (But Conservative)

Kelly sizing tells you: “What percentage of your portfolio should you bet?”

Full Kelly = (win% - loss%) / odds = can be aggressive Conservative Kelly (25% kelly) = (Full Kelly) × 0.25 = survives variance

I use 25% kelly. It costs me some upside but keeps me from going broke.

2. Real-Time News Integration

Parsing real-time news alerts + immediately checking if markets have updated = decent alpha for 24-48 hours.

I set up:

• NewsAPI integration for breaking political news

• Twitter/X trending topics filtered by keywords

• Polling aggregator alerts

• Check if market has repriced within 30 minutes

If not repriced, I have a window to act.

3. Confidence-Based Position Sizing

• 60-70% confidence: $200 max bet

• 70-80% confidence: $500 max bet

• 80%+ confidence: $1000-2000 max bet

This forced me to stop betting $2000 just because I “felt right.”

4. Exit on Thesis Invalidation (Not on Price)

I created decision rules for different scenarios:

Scenario A: Thesis Still Valid, Price Against Me

• Market repriced but my estimate unchanged

• Confidence still high

• Action: Hold or add more

Scenario B: Thesis Invalidated by News

• Market repriced because new information arrived

• My thesis assumed that wouldn’t happen

• Action: Exit immediately at current loss

Scenario C: EV Turned Negative

• My estimate still same

• Market moved enough that EV is now <2%

• Action: Exit at small loss

Scenario D: Confidence Collapsed

• Original bet made with 80% confidence

• New information suggests I was only 50% right

• Action: Cut position by 50%

Each scenario has a different exit logic. But ALL of them exit when thesis breaks, not when price moves.

5. Bayesian Updates from Outcomes

After 50 bets, I ran a calibration analysis:

My Confidence Win Rate Should Be Adjustment 80%+ 70% 80%+ Too confident, reduce by 10% 70-80% 65% 72% Slight overconfidence, reduce by 5% 60-70% 60% 65% Well calibrated 50-60% 55% 55% Well calibrated

I used this to recalibrate future estimates. “When I say 80%, I should really say 70%.”

The Results (Honest Assessment)

After 3 months (120 bets):

• Win rate: 56% (better than random, worse than I hoped)

• Sharpe ratio: 1.4 (decent)

• ROI: 12% (solid, mostly from 3-4 big bets)

• Max drawdown: -18% (scary, but survived)

What surprised me:

• Variance is huge. Being right on your model doesn’t guarantee short-term wins.

• The most profitable bets weren’t the ones I was most confident about—they were ones where I was right AND the market was really wrong.

• Self-correction saved me thousands. Several times I exited at -30% loss when full resolution would have been -100%.

The honest admission: 12% ROI over 3 months is good in absolute terms. But it’s not sustainable if I scale. Most of that came from 3 bets where I found real edge. The other 117 were mostly breakeven.

The Lessons That Stick With Me

1. Probability Estimation Scales with Data Quality, Not Intuition

Better signals beat clever thinking. Every time.

2. Confidence Matters More Than Point Estimates

“62%” is useless. “I’m 75% sure it’s between 55-70%” is useful.

3. Self-Healing is Operational, Not Magical

It means:

• Monitor in real-time (not just at resolution)

• Recalculate thesis validity (not just price moves)

• Exit early when thesis breaks

• Update model from outcomes

4. Correlation is Your Biggest Blind Spot

The bet that looks independent is probably correlated with 2-3 others. Map it before placing.

5. Market Efficiency is Real

Public information is priced in fast. Your edge is in:

• Processing NEW information faster

• Finding structural inefficiencies (correlation)

• Having better base rates than the crowd

6. Liquidity > Odds

A slightly worse price on a liquid market beats a great price on a dead one.

What I’m Exploring Next

• Multi-model ensemble: Combining 3-4 different probability models instead of one

• Real-time sentiment analysis: Automated analysis of news and social media instead of manual

• Automated correlation detection: ML-based instead of manual spreadsheet

• Cross-market arbitrage: Same event on different platforms at different prices

The Bigger Picture

This entire exploration taught me something about autonomous agents in general:

Self-healing doesn’t mean never failing. It means:

1. Detecting failure faster (monitoring in real-time)

2. Understanding why (thesis invalidation, not just price)

3. Correcting course immediately (exit, don’t wait for catastrophe)

4. Learning for next time (calibration updates, not one-offs)

Whether you’re fixing mapping errors, placing bets, or making business decisions, this loop matters.

Evals: How I Measured Success

I created three levels of evals to validate the agent actually works:

Unit Evals (Do components work?)

• EV calculator math correct? ✅

• Confidence calibrated? ✅

• Guard rails enforced? ✅

Integration Evals (Does full loop work?)

• Can agent find mispricings? ✅

• Does it catch thesis breaks? ✅

• Does it exit before catastrophe? ✅

Business Evals (Does it make money responsibly?)

• Sharpe ratio > 1.5? ✅ (achieved 1.4)

• Win rate > 55% on EV+ bets? ✅ (achieved 56%)

• Max drawdown < 20%? ✅ (achieved 18%)

• Is calibration improving? ✅ (confidence error -40% by month 2)

What the evals revealed:

• Week 1-2: Found mispricings poorly, improved signal aggregation

• Week 3-4: Confidence calibration was off, recalibrated

• Month 2-3: Hit all targets

The key: evals became part of the feedback loop. Each failure revealed what to fix next.

I Want to Hear From You

I built this agent from scratch over 3 months. I made money and lost money. Both were necessary.

What I’d love to know:

• Have you built self-healing systems? What worked and what failed?

• Have you tried probability estimation at scale? What was your biggest surprise?

• Are you trading prediction markets? What’s your edge?

• What challenges did you face that I haven’t mentioned?

Drop your thoughts in the replies. I’m genuinely curious how others approach this.

The Tech Stack (If You’re Curious)

• Data ingestion: Polymarket API + NewsAPI + Twitter/X API + polling aggregators

• Reasoning: Claude API for probability estimation and thesis validation

• Monitoring: Real-time market price polling + outcome tracking

• Learning: Calibration curves + Bayesian belief updates

• Execution: Custom Python agent with guard rails for position sizing, liquidity, correlation

The code is still messy (startup phase), but happy to discuss implementation details.