Tests in flight

A/B tests we're running on real bids. Update with results as data lands.

Active tests

Test 1: TLDR + 700+ word combo

• Hypothesis: Combining a 50-word TLDR with a 700+ word detailed proposal outperforms either alone.
• Why testing: Data shows sub-50 wins (9.4%) AND 700+ wins (18.5%); the middle (100-149) is the worst (6.7%). We're guessing the combo works but no data exists.
• Sample size needed: 10 bids of each format minimum.
• Tracking: reply rate, interview rate, contract win rate.
• Status: Not started.

Test 2: Pre-built spec landing pages

• Hypothesis: Pre-building a watermarked landing page for $2K+ bids beats text proposals on win rate by enough margin to justify the 1-2 hour build cost.
• Sample size needed: 10 attempts.
• Tracking: time spent per pre-build, Lovable credit cost, view rate, interview rate, win rate.
• Kill criteria: win rate <30% after 10 attempts.
• Status: Not started.

Test 3: Slack alerts on 7+ vs 9+

• Hypothesis: Faster Slack notification on 7+ jobs improves bid timing enough to lift reply rate from ~3% (4h+ lag) toward 9-24% (within-hour bids).
• Tracking: time from job-post to bid (currently averaging ?), reply rate before/after.
• Status: Not yet implemented (system change pending).

Closed tests

(none yet)