The short answer
Dark social is sharing your analytics cannot see: a link pasted into a private Slack, a name dropped on a call, a recommendation in a Discord. A GTM engineer owns the measurement. You add a free-text "How did you hear about us?" field, code the answers with an LLM, join them to pipeline (the deals in progress in your CRM, the system that stores customers and deals), and say honestly what the numbers do and do not prove. You also help run the community where much of the sharing happens.
Why this matters
The roadmap behind this course says: "B2B buyers buy from peers in private Slack groups, not from vendor whitepapers." Treat that as a claim to test.
Where the idea comes from. Alexis Madrigal coined "dark social" in The Atlantic in October 2012, working with the analytics firm Chartbeat. The Atlantic blocked my automated fetch, so I could not re-read the article today. As widely quoted, he found that most sharing happens by email and instant message, with dark social at 69% of social referrals on the sites he studied and Facebook at 20%. Treat those two numbers as unverified until you read the article yourself. Today the private channels are Slack, Discord, WhatsApp and group chats.
Analytics really do lose these clicks. In a SparkToro and Really Good Data test, a panel of about 100 people clicked links on 11 networks, giving 1,113 visits to 16 URLs over 10 days, measured in Google Analytics 4 and Universal Analytics. Every visit from TikTok, Slack, Discord, Mastodon and WhatsApp was recorded as "direct", with no referrer (the note a browser sends to say where a click came from).
Buyers do lean on peers. Both studies below come from companies that sell research or software, so weigh the incentive.
- TrustRadius 2026 B2B Buying Disconnect, as reported by HG Insights, which owns TrustRadius: 1,862 buyers and 444 vendors. 53% spoke to a peer, and all found it at least somewhat helpful. 74% consulted customer reviews. Vendor marketing collateral ranked last.
- 6sense 2025 Buyer Experience Report: nearly 4,000 responses. Buyers purchased from their Day One shortlist 95% of the time. 85% of buyers had prior experience with the vendor they chose, and 68% already knew a seller from that winning vendor.
Self-reported attribution says "dark" too, with caveats. Refine Labs added a required free-text "How did you hear about us?" field to its "Book a Strategy Call" form and compared it with software attribution over 12 months and 620 conversions. Software credited web search (direct and organic) with 78% of conversions, while customers named web search 12% of the time. For closed-won revenue (deals that were signed), customers named dark social (social media, podcasts, word of mouth, community) for 98% of it and named web search only 3% of the time, where software credited search with 79%. The article reports no confidence intervals, and Refine Labs is a B2B marketing consultancy, so weigh the incentive.
Verdict. The roadmap line is half right. Peers matter, and your tools undercount them. But no study above measures "private Slack groups" specifically, and buyers still read vendor material and reviews. What 6sense and TrustRadius show is that buyers favor vendors they already know. That is reputation. Community can build it, but no study here shows it builds it best. Measure it on your own data.
Core concepts
Dark social and the dark funnel
Dark social is sharing in private channels that carries no referrer (the note a browser sends to say which page or app a click came from). The dark funnel is the wider idea: the part of the buying journey your software cannot see, including private chats, podcasts, events and AI assistants.
| What happened | What analytics shows | What is true |
|---|---|---|
| Buyer clicks a link in a private Slack | Direct | Peer recommendation |
| Buyer hears the founder on a podcast, then searches the brand | Organic search | Podcast |
| Buyer asks an AI assistant, then types your URL | Direct | AI recommendation |
Last-click attribution gives all the credit to the final click before the signup. So it does not only miss dark social. It gives the credit to whatever came last.
Self-reported attribution
Self-reported attribution means you ask the buyer. You put one open question ("How did you hear about us?", shortened to HDYHAU) on a high-intent form (signup, demo request) and store the answer word for word.
Use free text, not a dropdown. A dropdown lets people pick only channels you already thought of, so someone who heard about you from a named person in a Slack group picks the nearest option, often "Other". Free text captures the podcast title, the person and the group. The cost is that you must code the answers.
| Answer | Channel | Named source |
|---|---|---|
| "Priya on our data team sent me the link" | peer_word_of_mouth | Priya |
| "someone in the dbt slack" | community | dbt Slack |
| "chatgpt suggested it" | ai_assistant | ChatGPT |
| "not sure" | unknown | none |
Report named sources too: "The dbt Slack" is a place you can go and show up.
Survey design and response bias
- Ask at the moment of intent, on the form where the buyer commits, with one open, non-leading question. Do not write "How did our amazing podcast reach you?".
- Required or optional is a trade. Required gets more answers but invites "n/a". A Recast write-up says nearly 30% of respondents leave optional fields blank. Pick one and keep it fixed so trends compare.
- Know the biases. Recall bias: people forget. Recency bias: they name the last touch. Salience bias: a podcast or a friend beats a boring search. Non-response bias: people who answer differ from people who skip.
- Keep the click data. Recast warns that stopping SEO because the survey says "social" would lose the search traffic that closes the deal. Software shows the last click, the survey shows what started the interest. Use both.
Coding free text with an LLM
Coding means assigning each answer to one category from a fixed list.
- Fix the taxonomy. About 10 channels, plus
unknown, plusai_assistant, which is now a real answer. - Force structured output. Claude's structured outputs constrain the reply to a JSON schema. You pass
output_configwithformat.typeset tojson_schema, and every object needsadditionalProperties: false. - Use a small model. Claude Haiku 4.5 costs $1 per million input tokens and $5 per million output tokens on Anthropic's pricing page today, half that through the Batch API. At about 350 input and 40 output tokens per answer (my estimate), 1,000 answers cost about $0.35 plus $0.20, so $0.55 at list price.
- Audit it. Pull 50 answers at random and label them yourself, blind, which means you do not see the model's label first. Compute Cohen's kappa, the agreement between two coders after removing chance. Example: 90% agreement when luck alone gives 40% means kappa = (0.90 - 0.40) / (1 - 0.40) = 0.83. As a rule of thumb, aim above about 0.7 before you trust the totals. If it is low, fix the prompt, bump
PROMPT_VERSIONand recode with--recode. The project includesaudit.pyfor this.
Answers can contain names and emails. Send only the answer text to the model, never the account record, and check your vendor terms.
Correlation is not causation
Say accounts that name word of mouth (WOM) open deals more often. Does community cause it? Two traps:
- Confounding. Large companies are more likely to have someone in your community, more likely to name a peer, and more likely to have budget. Size drives all three.
- Reverse causation. An account already in a buying process may join your community to ask questions. Pipeline caused the membership.
Worked example with made-up numbers. Lift is the share of word-of-mouth accounts that open a deal minus the share of other accounts that do. Here an "opportunity" is a deal record in the CRM, and a size band is a group of companies of similar size.
| Size band | WOM accounts with a deal | Other accounts with a deal | Lift inside the band |
|---|---|---|---|
| Small (200 accounts) | 2 of 20 (10%) | 9 of 180 (5%) | +5 points |
| Large (100 accounts) | 40 of 80 (50%) | 8 of 20 (40%) | +10 points |
| All 300 accounts, raw | 42 of 100 (42%) | 17 of 200 (8.5%) | +33.5 points |
Most word-of-mouth accounts are large, and large accounts open deals more often anyway, so the raw 33.5 points is mostly size. Average the in-band lifts weighted by band size: (5 x 200 + 10 x 100) / 300 = +6.7 points. That is the size-adjusted lift.
The project generates fake data with the first trap built in: size drives both community use and pipeline. It does not simulate reverse causation, so keep that trap in mind for real data. A seeded run with the offline coder prints:
accounts 1500
raw WOM lift in P(opp) = +0.072
size-adjusted WOM lift = +0.023
95% bootstrap CI (adjusted) = [-0.024, +0.070]Read it line by line. opp means the account opened a deal within 90 days of its first signup. The raw lift is 7.2 points. Inside size bands it is 2.3 points, so about two thirds of the raw lift was company size. The bootstrap redraws the 1,500 accounts with replacement 2,000 times, recomputes the adjusted lift each time, and keeps the middle 95% as an interval: -2.4 to +7.0 points. The interval includes zero, so this data cannot tell you whether word of mouth matters at all. Report the adjusted number with its interval. The strongest evidence is an experiment, such as holding out one cohort from the community invite. Without one, say "associated with", never "caused".
Community platforms
- Discord: free-form chat, strong with developers, with roles and a bot API.
- Slack: familiar from work. The free plan keeps 90 days of history, so old answers vanish. Apps created on or after May 29, 2025 and sold outside the Slack Marketplace get 1 call per minute and 15 messages per call on
conversations.history. Internal apps are not affected. - Discourse: an open source forum. Answers stay searchable on the public web.
- Circle: community plus courses and paid memberships.
Chat is for speed. A forum is for memory.
Community ops: moderation, onboarding, rituals
These are working practices, not research findings. Test them.
- Moderation. Write a short code of conduct on day one. Use an escalation ladder: reply, warn, mute, remove. Name two moderators so one person is never the bottleneck.
- Onboarding and rituals. Ask each new member to post an intro on day one. Keep two repeating rituals, such as a weekly "show what you shipped" thread.
- Seeding and giving. Invite your first 20 to 30 people by hand. Answer questions for weeks before you promote anything.
Measuring community: Orbit, Common Room, DevRel metrics
The Orbit Model ranks members in four orbit levels (Advocates, Contributors, Participants, Explorers). It scores love (involvement) and reach (influence) and defines gravity = love x reach. Status in October 2026: Postman acquired Orbit on April 11, 2024 and said the product would sunset over 90 days. The model lives on in an MIT-licensed repo whose README says it is no longer under active development. Use it as vocabulary.
Common Room is a paid signal platform that joins community, product and web activity into account views. Its pricing page lists Essential at $2,500 per month billed annually, other plans custom, no free tier. Build the small version first.
DevRel metrics. Developer relations (DevRel) builds relationships with developers. Two metrics matter:
- DevRel Qualified Leads (DRQL). Mary Thengvall presented it at DevRelCon London in December 2019, in her talk slides. It widens "lead" to anyone who can add value: a case-study candidate, a beta tester, a partner contact, a hire.
- Time to first hello world (TTFHW). A July 2022 Moesif post describes it, and says Moesif computes it by default as the time from when a user was added to their first API call. You choose what "hello world" means for your product. Example: sign up Monday, first successful API call Wednesday, TTFHW of 2 days. Report the median, and also the share of signups who arrive within 7 days, so users who never arrive still count.
Avoid vanity metrics such as member count. Prefer weekly active members and members who answer someone else.
Key terms
The tools
| Tool | What it does | Pricing model | Notes |
|---|---|---|---|
| Discord | Chat server with roles, forums and a bot API | Free tier not stated on the pages I fetched (unverified); Nitro is an optional paid subscription | Put /api/v10 in the URL, because the API reference lists deprecated v6 as the default. |
| Slack | Workplace chat used as a community | Free plan, then Pro at $8.75 per user per month, monthly billing | Free plan: 90 days of history, up to 10 apps. |
| Discourse | Searchable community forum | Free plan to explore, Pro $100 per month, Business $500 per month, Enterprise custom | Also free to self-host, open source. |
| Circle | Community plus courses and events | Professional $89, Business $199, Scale $419 per month, all on annual billing | 14-day trial, no card. API starts at Business. |
| Common Room | Joins community and product signals into account views | Essential $2,500 per month billed annually, others custom | No free tier listed. Calls itself a "Revenue OS". |
| PostHog Surveys | In-product and web surveys | Usage-based. Free allowance not verified today | Good for asking the HDYHAU question on a web app. |
| Tally | Form builder | Free forms and submissions within fair use, Pro $24 per month, Business $74 per month | Free option for the no-budget path. |
| Claude API | LLM for coding free text | Haiku 4.5: $1 input, $5 output per million tokens | Batch API is half price. |
Build the project
You will launch a small niche technical community and build a survey-based model that tests whether word of mouth lines up with pipeline. A real community takes weeks to produce data, so there are two tracks. Track A is the real community and form. Track B is a synthetic dataset, so you can build and test everything today with no company and no budget. Build B first, then point the same code at A.
| Layer | What it does | Where it lives |
|---|---|---|
| Capture | Free-text HDYHAU field, one Discord invite per source | Your form, Discord |
| Store | Accounts, signups, coded answers, opportunities | SQLite file dark_social.db |
| Code | LLM turns each answer into channel, named source, confidence | code_answers.py |
| Model | Join, correlate, adjust for size, bootstrap | analyze.py |
| Audit | Sample 50 answers, label them blind, score agreement | audit.py |
- Pick a niche narrow enough that you can name 30 people in it, such as people who self-host Postgres backups.
- Create a Discord server with a code of conduct, an intro channel, a help channel and one weekly ritual thread.
- Create one invite link per source (podcast, newsletter, partner Slack, your site), set each to never expire, and note each code. Discord reports
usesper invite. - Add one free-text HDYHAU input to your signup form, such as
<input name="hdyhau" type="text">, and store the text word for word. - Run
pip install anthropic requests, saveschema.sqlandgen_data.pyin one folder, and runpython gen_data.pyto create fake accounts. - Save
code_answers.pyand runpython code_answers.py --offlineto code every answer with keyword rules. - Save
analyze.pyandaudit.py, then runpython audit.py exportto write 50 random answers toaudit_sample.csv. - Fill the
my_channelcolumn ofaudit_sample.csvby hand, then runpython audit.py scoreto see Cohen's kappa and every disagreement. - Set
ANTHROPIC_API_KEY, then runpython code_answers.py --recodeto code every answer with Claude. Runpython audit.py scoreagain, and fix the prompt until kappa passes about 0.7. - Run
python analyze.pyand read the raw lift, the size-adjusted lift and the bootstrap interval. - Save
discord_invites.pyand run it daily, then compare joins per invite code with the survey's named sources. - Write one page: what you measured, the adjusted number, three reasons it may not be causal, and one experiment that would settle it.
schema.sql:
CREATE TABLE IF NOT EXISTS accounts (account_id INTEGER PRIMARY KEY, domain TEXT NOT NULL,
employees INTEGER NOT NULL);
CREATE TABLE IF NOT EXISTS signups (signup_id INTEGER PRIMARY KEY, account_id INTEGER NOT NULL
REFERENCES accounts(account_id), created_at TEXT NOT NULL, hdyhau_text TEXT);
CREATE TABLE IF NOT EXISTS coded_answers (signup_id INTEGER PRIMARY KEY REFERENCES
signups(signup_id), channel TEXT NOT NULL, named_source TEXT NOT NULL,
confidence TEXT NOT NULL, prompt_version TEXT NOT NULL);
CREATE TABLE IF NOT EXISTS opportunities (opp_id INTEGER PRIMARY KEY, account_id INTEGER NOT NULL
REFERENCES accounts(account_id), created_at TEXT NOT NULL, amount_usd REAL NOT NULL);gen_data.py plants a small true community effect and a large size confounder:
"""Fake data. Planted truth: community adds a small lift; company size drives
BOTH community use and pipeline (a confounder). Run: python gen_data.py"""
import math
import random
import sqlite3
from datetime import date, timedelta
random.seed(42)
db = sqlite3.connect("dark_social.db")
db.executescript(open("schema.sql").read())
for t in ("opportunities", "coded_answers", "signups", "accounts"):
db.execute(f"DELETE FROM {t}")
DARK = ["a friend sent me the link", "my old coworker told me", "saw it in a slack group",
"someone in the discord", "word of mouth"]
OTHER = ["google", "searched for dictation api", "a podcast, forgot which",
"chatgpt suggested it", "G2 reviews", "not sure", "n/a"]
for acc in range(1, 1501):
employees = max(5, int(math.exp(random.gauss(5.5, 1.4))))
size = math.log(employees) - 3.5
in_comm = random.random() < min(0.95, max(0.02, 0.12 * size))
db.execute("INSERT INTO accounts VALUES (?,?,?)", (acc, f"co{acc}.example", employees))
first = date(2026, 1, 1) + timedelta(days=random.randint(0, 120))
answer = random.choice(DARK if random.random() < (0.55 if in_comm else 0.12) else OTHER)
db.execute("INSERT INTO signups (account_id, created_at, hdyhau_text) VALUES (?,?,?)",
(acc, first.isoformat(), answer))
if random.random() < min(0.9, max(0.01, 0.10 * size + (0.06 if in_comm else 0))):
db.execute("INSERT INTO opportunities (account_id, created_at, amount_usd) VALUES (?,?,?)",
(acc, (first + timedelta(days=random.randint(5, 80))).isoformat(),
round(employees * random.uniform(8, 20), 2)))
db.commit()code_answers.py. The Anthropic SDK reads ANTHROPIC_API_KEY from your environment, so no key lives in code:
"""Code 'How did you hear about us?' answers into channels.
python code_answers.py (Claude; needs ANTHROPIC_API_KEY in the environment)
python code_answers.py --offline (keyword rules, free)
Add --recode to delete old codes and code every answer again."""
import json
import sqlite3
import sys
MODEL = "claude-haiku-4-5-20251001"
PROMPT_VERSION = "v1" # change this when you edit SYSTEM, then run with --recode
CHANNELS = ["peer_word_of_mouth", "community", "podcast", "social_post", "search",
"ai_assistant", "review_site", "event", "ad", "other", "unknown"]
SYSTEM = ("You code survey answers to 'How did you hear about us?'. Pick one channel. "
"peer_word_of_mouth is a person (friend, coworker). community is a Slack, Discord "
"or forum group. Use unknown when no source is stated. Copy any named person, "
"podcast, group or site into named_source, else an empty string. Never guess.")
SCHEMA = {
"type": "object",
"properties": {"channel": {"type": "string", "enum": CHANNELS},
"named_source": {"type": "string"},
"confidence": {"type": "string", "enum": ["low", "medium", "high"]}},
"required": ["channel", "named_source", "confidence"],
"additionalProperties": False,
}
RULES = [("peer_word_of_mouth", ("friend", "colleague", "coworker", "word of mouth")),
("community", ("discord", "slack")), ("podcast", ("podcast",)),
("ai_assistant", ("chatgpt",)), ("review_site", ("g2", "review")),
("search", ("google", "search"))]
def code_with_claude(client, text):
resp = client.messages.create(
model=MODEL, max_tokens=300, system=SYSTEM,
messages=[{"role": "user", "content": f"<answer>{text}</answer>"}],
output_config={"format": {"type": "json_schema", "schema": SCHEMA}},
)
if resp.stop_reason != "end_turn":
raise RuntimeError(f"unexpected stop_reason {resp.stop_reason}")
return json.loads(next(b.text for b in resp.content if b.type == "text"))
def code_offline(text):
low = text.lower()
for channel, words in RULES:
if any(w in low for w in words):
return {"channel": channel, "named_source": "", "confidence": "low"}
return {"channel": "unknown", "named_source": "", "confidence": "low"}
if __name__ == "__main__":
offline = "--offline" in sys.argv
client = None
if not offline:
import anthropic
client = anthropic.Anthropic()
version = f"offline-{PROMPT_VERSION}" if offline else f"claude-{PROMPT_VERSION}"
db = sqlite3.connect("dark_social.db")
if "--recode" in sys.argv:
db.execute("DELETE FROM coded_answers")
rows = db.execute("""SELECT s.signup_id, s.hdyhau_text FROM signups s
LEFT JOIN coded_answers c USING (signup_id) WHERE c.signup_id IS NULL""").fetchall()
for signup_id, text in rows:
text = text or ""
r = code_offline(text) if offline else code_with_claude(client, text)
db.execute("INSERT INTO coded_answers VALUES (?,?,?,?,?)",
(signup_id, r["channel"], r["named_source"], r["confidence"], version))
db.commit() # keep paid-for work if a later call fails
print(f"coded {len(rows)} answers with {version}")analyze.py holds the model, and audit.py imports its cohen_kappa function:
"""Does word of mouth line up with pipeline? Run: python analyze.py"""
import random
import sqlite3
from collections import Counter
SQL = """
WITH f AS (
SELECT s.account_id, MIN(s.created_at) AS first_day,
MAX(c.channel IN ('peer_word_of_mouth', 'community')) AS wom
FROM signups s JOIN coded_answers c USING (signup_id) GROUP BY s.account_id)
SELECT a.employees, f.wom,
EXISTS (SELECT 1 FROM opportunities o WHERE o.account_id = a.account_id
AND o.created_at <= date(f.first_day, '+90 days'))
FROM accounts a JOIN f USING (account_id)"""
def band(n):
return "small" if n < 200 else "mid" if n < 2000 else "large"
def cohen_kappa(a, b):
"""Agreement of two coders beyond chance. 1 is perfect, 0 is chance."""
po = sum(x == y for x, y in zip(a, b)) / len(a)
ca, cb = Counter(a), Counter(b)
pe = sum(ca[k] * cb[k] for k in ca) / len(a) ** 2
if pe == 1: # both coders used one single label, kappa is undefined
return 1.0 if po == 1 else 0.0
return (po - pe) / (1 - pe)
def lift(rows):
yes = [r["opp"] for r in rows if r["wom"]]
no = [r["opp"] for r in rows if not r["wom"]]
return sum(yes) / len(yes) - sum(no) / len(no) if yes and no else None
def adjusted_lift(rows):
"""Compare inside each size band, then weight bands by size."""
parts = [[r for r in rows if r["band"] == b] for b in ("small", "mid", "large")]
pairs = [(lift(p), len(p)) for p in parts if lift(p) is not None]
return sum(d * n for d, n in pairs) / sum(n for _, n in pairs)
if __name__ == "__main__":
rows = [{"band": band(e), "wom": w, "opp": o}
for e, w, o in sqlite3.connect("dark_social.db").execute(SQL)]
print(f"accounts {len(rows)}")
print(f"raw WOM lift in P(opp) = {lift(rows):+.3f}")
print(f"size-adjusted WOM lift = {adjusted_lift(rows):+.3f}")
rng = random.Random(7)
boots = sorted(adjusted_lift(rng.choices(rows, k=len(rows))) for _ in range(2000))
print(f"95% bootstrap CI (adjusted) = [{boots[50]:+.3f}, {boots[1949]:+.3f}]")audit.py exports a sample without the coder's labels, so your labeling stays blind:
"""python audit.py export (writes audit_sample.csv; fill my_channel by hand)
python audit.py score (prints Cohen's kappa and each disagreement)"""
import csv
import random
import sqlite3
import sys
from analyze import cohen_kappa
from code_answers import CHANNELS
db = sqlite3.connect("dark_social.db")
if sys.argv[1:] == ["export"]:
rows = db.execute("SELECT signup_id, hdyhau_text FROM signups").fetchall()
with open("audit_sample.csv", "w", newline="", encoding="utf-8") as f:
w = csv.writer(f)
w.writerow(["signup_id", "answer", "my_channel"])
w.writerows([i, text, ""] for i, text in random.Random(1).sample(rows, 50))
print("channels:", ", ".join(CHANNELS))
elif sys.argv[1:] == ["score"]:
with open("audit_sample.csv", newline="", encoding="utf-8") as f:
mine = list(csv.DictReader(f))
if any(r["my_channel"].strip() not in CHANNELS for r in mine):
sys.exit("every my_channel cell must be a channel name")
coder = dict(db.execute("SELECT signup_id, channel FROM coded_answers"))
human = [r["my_channel"].strip() for r in mine]
model = [coder[int(r["signup_id"])] for r in mine]
for r, h, m in zip(mine, human, model):
if h != m:
print(f"DISAGREE {r['answer']!r}: you={h} model={m}")
print(f"kappa = {cohen_kappa(human, model):.2f} on {len(human)} answers")
else:
sys.exit(__doc__)discord_invites.py. Create a bot in the Discord developer portal, invite it with the Manage Server permission, and export DISCORD_BOT_TOKEN and DISCORD_GUILD_ID. The endpoint GET /guilds/{guild.id}/invites needs MANAGE_GUILD or VIEW_AUDIT_LOG. The guild docs I read give no field naming the invite a new member used, so the script diffs uses between snapshots. That blurs when several people join between snapshots through different invites.
"""Snapshot Discord invite use counts and print joins per invite since the last run.
Needs a bot with MANAGE_GUILD in your server. Run: python discord_invites.py"""
import json
import os
import time
import requests
API = "https://discord.com/api/v10"
GUILD_ID = os.environ["DISCORD_GUILD_ID"]
HEADERS = {"Authorization": f"Bot {os.environ['DISCORD_BOT_TOKEN']}"}
def get(path, **params):
r = requests.get(f"{API}{path}", headers=HEADERS, params=params, timeout=30)
r.raise_for_status() # a daily run stays far below Discord's rate limits
return r.json()
def snapshot():
invites = get(f"/guilds/{GUILD_ID}/invites") # list of invite objects with metadata
members = get(f"/guilds/{GUILD_ID}", with_counts="true")["approximate_member_count"]
return {"ts": int(time.time()), "members": members,
"uses": {i["code"]: i["uses"] for i in invites}}
if __name__ == "__main__":
try:
with open("invite_snapshots.jsonl", encoding="utf-8") as f:
before = json.loads(f.read().splitlines()[-1])
except (FileNotFoundError, IndexError):
before = None
snap = snapshot()
with open("invite_snapshots.jsonl", "a", encoding="utf-8") as f:
f.write(json.dumps(snap) + "\n")
print(f"{snap['members']} members, {len(snap['uses'])} invites")
if before:
for code, uses in sorted(snap["uses"].items()):
gained = uses - before["uses"].get(code, 0)
if gained:
print(f"invite {code}: +{gained} joins since the last snapshot")No-budget path. Use a Discord server, a free Tally form, SQLite and the synthetic generator, and skip Claude with --offline. Nothing here needs a company, a CRM or paid seats.
What done looks like. A running Discord with one ritual. Every signup has a stored free-text answer. A kappa above about 0.7 against your hand labels. A one-page note with the adjusted lift, its interval and two confounders. The synthetic run prints the numbers in the correlation section. I got them on Python 3.14, and other versions can differ.
Stretch goal. Replace the size bands with a logistic regression that adds region and signup month. Then design a holdout: invite half a new cohort to the community and compare pipeline after 90 days.
Common mistakes
- Using a dropdown for the source question. It hides the podcast, the person and the group. Use free text and code it.
- Reading the raw number. The raw lift mixes in company size and other confounders. Compare like with like and show an interval.
- Chasing member count. A big quiet server is a bad sign. Count members who answer someone else.
- Trusting the coder without an audit. Hand-label 50 answers with
audit.py, check kappa and version your prompt. - Treating one vendor study as fact. The 98% figure comes from one firm's own forms. Report the source, the sample and the incentive.
Check yourself
1. Why does a free-text "How did you hear about us?" field usually beat a dropdown?
Why: A dropdown caps answers at the channels you thought of, while free text captures named sources.
2. Your raw word-of-mouth lift is +7.2 points, but +2.3 after comparing inside company-size bands. What does this most likely mean?
Why: Size drove both community use and pipeline, so most of the raw gap was size, not community.
3. In the SparkToro and Really Good Data test, what happened to clicks from Slack and Discord?
Why: Every visit from Slack and Discord appeared as direct, which is how community traffic hides.
4. You code 1,000 answers with an LLM. What is the best check before you trust the totals?
Why: Valid JSON says nothing about correct labels. Chance-corrected agreement with a human is the audit.
5. Which statement do the studies in this stage support?
Why: TrustRadius and 6sense show the weight of peers and familiarity, and the 98% figure is one firm's own data.
6. What is the status of the Orbit product and the Orbit Model in October 2026?
Why: Postman said the current product would sunset over 90 days, and the public repo's README says the model is no longer under active development.
How to learn this faster
Learn first: the survey field and the coding loop. They take an afternoon and work anywhere. Then learn confounding with the synthetic data, then run a real community. Skip for now: paid platforms such as Common Room.
Drill: write 20 messy answers yourself, such as "my friend Sam", "G2 maybe?" and "lol idk". Code them by hand, then with the model. For each disagreement, decide who was right and fix the prompt.
- Day 1: Read the Madrigal article and the SparkToro test. Build the form field.
- Day 2: Run the generator and the offline coder. Read the
analyze.pyoutput line by line. - Day 3: Code with Claude using
--recode, label 50 answers by hand withaudit.pyand compute kappa. Fix the prompt twice. - Day 4: Launch the Discord, seed 20 people, post a ritual thread, create invites.
- Day 5: Snapshot invite counts and write the one-page note.
Resources
- The Atlantic: Dark Social, We Have the Whole History of the Web WrongArticleFree
The 2012 piece that named the idea. My fetch was blocked, so check the figures yourself.
- SparkToro: Dark social falsely attributes traffic as directResearchFree
Which networks lose their referrer.
- Refine Labs: The Attribution MirageArticleFree
The 620-conversion study. Read the caveats.
- Recast: HDYHAU surveys and their limitsArticleFree
Recall bias and what surveys miss.
- 6sense 2025 B2B Buyer Experience ReportResearchFree
Day One shortlist data. Vendor-run.
- TrustRadius 2026 Buying Disconnect, summarized by HG InsightsResearchFree
Peer and review usage in 2026.
- Orbit Model repositoryRepoFree
MIT license, and the not-under-active-development note.
- Postman: Postman has acquired OrbitArticleFree
Primary source for the Orbit product sunset.
- Mary Thengvall: Don't Create; Re-use! (DRQL talk)ArticleFree
December 2019 DevRelCon London slides with five DRQL examples.
- Discord developer docs: Guild resourceDocsFree
Invites, members and counts endpoints.
- Slack changelog: rate limits for non-Marketplace appsDocsFree
Read before you plan a Slack bot.
- Claude docs: Structured outputsDocsFree
The JSON schema feature the coder uses.
Next module
Stage 11 turns the same data discipline toward existing customers: the automated QBR (Quarterly Business Review), a script that builds a branded ROI deck for a customer without anyone assembling it by hand.
Built the project and passed the quiz? Mark the stage done. Your progress stays in this browser only.
Talk to Claude instead of typing to it.
Ducktate turns what you say into clean text, on your Mac, and pastes it anywhere.
Get early access