Home / Learn / GTM Engineering / Stage 10: Dark social and community

Stage 10 of 1224 min readUpdated 2026-10-07

Stage 10: Dark social and community

Learn to measure word of mouth you cannot track: self-reported attribution, LLM-coded survey answers, community ops, and honest correlation with pipeline.

  • Build: Launch a niche technical community and build a survey-based model that tests how word of mouth lines up with enterprise pipeline.
  • Free, no signup

The short answer

Dark social is sharing your analytics cannot see: a link pasted into a private Slack, a name dropped on a call, a recommendation in a Discord. A GTM engineer owns the measurement. You add a free-text "How did you hear about us?" field, code the answers with an LLM, join them to pipeline (the deals in progress in your CRM, the system that stores customers and deals), and say honestly what the numbers do and do not prove. You also help run the community where much of the sharing happens.

Why this matters

The roadmap behind this course says: "B2B buyers buy from peers in private Slack groups, not from vendor whitepapers." Treat that as a claim to test.

Where the idea comes from. Alexis Madrigal coined "dark social" in The Atlantic in October 2012, working with the analytics firm Chartbeat. The Atlantic blocked my automated fetch, so I could not re-read the article today. As widely quoted, he found that most sharing happens by email and instant message, with dark social at 69% of social referrals on the sites he studied and Facebook at 20%. Treat those two numbers as unverified until you read the article yourself. Today the private channels are Slack, Discord, WhatsApp and group chats.

Analytics really do lose these clicks. In a SparkToro and Really Good Data test, a panel of about 100 people clicked links on 11 networks, giving 1,113 visits to 16 URLs over 10 days, measured in Google Analytics 4 and Universal Analytics. Every visit from TikTok, Slack, Discord, Mastodon and WhatsApp was recorded as "direct", with no referrer (the note a browser sends to say where a click came from).

Buyers do lean on peers. Both studies below come from companies that sell research or software, so weigh the incentive.

  • TrustRadius 2026 B2B Buying Disconnect, as reported by HG Insights, which owns TrustRadius: 1,862 buyers and 444 vendors. 53% spoke to a peer, and all found it at least somewhat helpful. 74% consulted customer reviews. Vendor marketing collateral ranked last.
  • 6sense 2025 Buyer Experience Report: nearly 4,000 responses. Buyers purchased from their Day One shortlist 95% of the time. 85% of buyers had prior experience with the vendor they chose, and 68% already knew a seller from that winning vendor.

Self-reported attribution says "dark" too, with caveats. Refine Labs added a required free-text "How did you hear about us?" field to its "Book a Strategy Call" form and compared it with software attribution over 12 months and 620 conversions. Software credited web search (direct and organic) with 78% of conversions, while customers named web search 12% of the time. For closed-won revenue (deals that were signed), customers named dark social (social media, podcasts, word of mouth, community) for 98% of it and named web search only 3% of the time, where software credited search with 79%. The article reports no confidence intervals, and Refine Labs is a B2B marketing consultancy, so weigh the incentive.

Verdict. The roadmap line is half right. Peers matter, and your tools undercount them. But no study above measures "private Slack groups" specifically, and buyers still read vendor material and reviews. What 6sense and TrustRadius show is that buyers favor vendors they already know. That is reputation. Community can build it, but no study here shows it builds it best. Measure it on your own data.

Core concepts

Dark social and the dark funnel

Dark social is sharing in private channels that carries no referrer (the note a browser sends to say which page or app a click came from). The dark funnel is the wider idea: the part of the buying journey your software cannot see, including private chats, podcasts, events and AI assistants.

What happenedWhat analytics showsWhat is true
Buyer clicks a link in a private SlackDirectPeer recommendation
Buyer hears the founder on a podcast, then searches the brandOrganic searchPodcast
Buyer asks an AI assistant, then types your URLDirectAI recommendation

Last-click attribution gives all the credit to the final click before the signup. So it does not only miss dark social. It gives the credit to whatever came last.

Self-reported attribution

Self-reported attribution means you ask the buyer. You put one open question ("How did you hear about us?", shortened to HDYHAU) on a high-intent form (signup, demo request) and store the answer word for word.

Use free text, not a dropdown. A dropdown lets people pick only channels you already thought of, so someone who heard about you from a named person in a Slack group picks the nearest option, often "Other". Free text captures the podcast title, the person and the group. The cost is that you must code the answers.

AnswerChannelNamed source
"Priya on our data team sent me the link"peer_word_of_mouthPriya
"someone in the dbt slack"communitydbt Slack
"chatgpt suggested it"ai_assistantChatGPT
"not sure"unknownnone

Report named sources too: "The dbt Slack" is a place you can go and show up.

Survey design and response bias

  • Ask at the moment of intent, on the form where the buyer commits, with one open, non-leading question. Do not write "How did our amazing podcast reach you?".
  • Required or optional is a trade. Required gets more answers but invites "n/a". A Recast write-up says nearly 30% of respondents leave optional fields blank. Pick one and keep it fixed so trends compare.
  • Know the biases. Recall bias: people forget. Recency bias: they name the last touch. Salience bias: a podcast or a friend beats a boring search. Non-response bias: people who answer differ from people who skip.
  • Keep the click data. Recast warns that stopping SEO because the survey says "social" would lose the search traffic that closes the deal. Software shows the last click, the survey shows what started the interest. Use both.

Coding free text with an LLM

Coding means assigning each answer to one category from a fixed list.

  1. Fix the taxonomy. About 10 channels, plus unknown, plus ai_assistant, which is now a real answer.
  2. Force structured output. Claude's structured outputs constrain the reply to a JSON schema. You pass output_config with format.type set to json_schema, and every object needs additionalProperties: false.
  3. Use a small model. Claude Haiku 4.5 costs $1 per million input tokens and $5 per million output tokens on Anthropic's pricing page today, half that through the Batch API. At about 350 input and 40 output tokens per answer (my estimate), 1,000 answers cost about $0.35 plus $0.20, so $0.55 at list price.
  4. Audit it. Pull 50 answers at random and label them yourself, blind, which means you do not see the model's label first. Compute Cohen's kappa, the agreement between two coders after removing chance. Example: 90% agreement when luck alone gives 40% means kappa = (0.90 - 0.40) / (1 - 0.40) = 0.83. As a rule of thumb, aim above about 0.7 before you trust the totals. If it is low, fix the prompt, bump PROMPT_VERSION and recode with --recode. The project includes audit.py for this.

Answers can contain names and emails. Send only the answer text to the model, never the account record, and check your vendor terms.

Correlation is not causation

Say accounts that name word of mouth (WOM) open deals more often. Does community cause it? Two traps:

  • Confounding. Large companies are more likely to have someone in your community, more likely to name a peer, and more likely to have budget. Size drives all three.
  • Reverse causation. An account already in a buying process may join your community to ask questions. Pipeline caused the membership.

Worked example with made-up numbers. Lift is the share of word-of-mouth accounts that open a deal minus the share of other accounts that do. Here an "opportunity" is a deal record in the CRM, and a size band is a group of companies of similar size.

Size bandWOM accounts with a dealOther accounts with a dealLift inside the band
Small (200 accounts)2 of 20 (10%)9 of 180 (5%)+5 points
Large (100 accounts)40 of 80 (50%)8 of 20 (40%)+10 points
All 300 accounts, raw42 of 100 (42%)17 of 200 (8.5%)+33.5 points

Most word-of-mouth accounts are large, and large accounts open deals more often anyway, so the raw 33.5 points is mostly size. Average the in-band lifts weighted by band size: (5 x 200 + 10 x 100) / 300 = +6.7 points. That is the size-adjusted lift.

The project generates fake data with the first trap built in: size drives both community use and pipeline. It does not simulate reverse causation, so keep that trap in mind for real data. A seeded run with the offline coder prints:

accounts 1500
raw WOM lift in P(opp)      = +0.072
size-adjusted WOM lift      = +0.023
95% bootstrap CI (adjusted) = [-0.024, +0.070]

Read it line by line. opp means the account opened a deal within 90 days of its first signup. The raw lift is 7.2 points. Inside size bands it is 2.3 points, so about two thirds of the raw lift was company size. The bootstrap redraws the 1,500 accounts with replacement 2,000 times, recomputes the adjusted lift each time, and keeps the middle 95% as an interval: -2.4 to +7.0 points. The interval includes zero, so this data cannot tell you whether word of mouth matters at all. Report the adjusted number with its interval. The strongest evidence is an experiment, such as holding out one cohort from the community invite. Without one, say "associated with", never "caused".

Community platforms

  • Discord: free-form chat, strong with developers, with roles and a bot API.
  • Slack: familiar from work. The free plan keeps 90 days of history, so old answers vanish. Apps created on or after May 29, 2025 and sold outside the Slack Marketplace get 1 call per minute and 15 messages per call on conversations.history. Internal apps are not affected.
  • Discourse: an open source forum. Answers stay searchable on the public web.
  • Circle: community plus courses and paid memberships.

Chat is for speed. A forum is for memory.

Community ops: moderation, onboarding, rituals

These are working practices, not research findings. Test them.

  • Moderation. Write a short code of conduct on day one. Use an escalation ladder: reply, warn, mute, remove. Name two moderators so one person is never the bottleneck.
  • Onboarding and rituals. Ask each new member to post an intro on day one. Keep two repeating rituals, such as a weekly "show what you shipped" thread.
  • Seeding and giving. Invite your first 20 to 30 people by hand. Answer questions for weeks before you promote anything.

Measuring community: Orbit, Common Room, DevRel metrics

The Orbit Model ranks members in four orbit levels (Advocates, Contributors, Participants, Explorers). It scores love (involvement) and reach (influence) and defines gravity = love x reach. Status in October 2026: Postman acquired Orbit on April 11, 2024 and said the product would sunset over 90 days. The model lives on in an MIT-licensed repo whose README says it is no longer under active development. Use it as vocabulary.

Common Room is a paid signal platform that joins community, product and web activity into account views. Its pricing page lists Essential at $2,500 per month billed annually, other plans custom, no free tier. Build the small version first.

DevRel metrics. Developer relations (DevRel) builds relationships with developers. Two metrics matter:

  • DevRel Qualified Leads (DRQL). Mary Thengvall presented it at DevRelCon London in December 2019, in her talk slides. It widens "lead" to anyone who can add value: a case-study candidate, a beta tester, a partner contact, a hire.
  • Time to first hello world (TTFHW). A July 2022 Moesif post describes it, and says Moesif computes it by default as the time from when a user was added to their first API call. You choose what "hello world" means for your product. Example: sign up Monday, first successful API call Wednesday, TTFHW of 2 days. Report the median, and also the share of signups who arrive within 7 days, so users who never arrive still count.

Avoid vanity metrics such as member count. Prefer weekly active members and members who answer someone else.

Key terms

The tools

ToolWhat it doesPricing modelNotes
DiscordChat server with roles, forums and a bot APIFree tier not stated on the pages I fetched (unverified); Nitro is an optional paid subscriptionPut /api/v10 in the URL, because the API reference lists deprecated v6 as the default.
SlackWorkplace chat used as a communityFree plan, then Pro at $8.75 per user per month, monthly billingFree plan: 90 days of history, up to 10 apps.
DiscourseSearchable community forumFree plan to explore, Pro $100 per month, Business $500 per month, Enterprise customAlso free to self-host, open source.
CircleCommunity plus courses and eventsProfessional $89, Business $199, Scale $419 per month, all on annual billing14-day trial, no card. API starts at Business.
Common RoomJoins community and product signals into account viewsEssential $2,500 per month billed annually, others customNo free tier listed. Calls itself a "Revenue OS".
PostHog SurveysIn-product and web surveysUsage-based. Free allowance not verified todayGood for asking the HDYHAU question on a web app.
TallyForm builderFree forms and submissions within fair use, Pro $24 per month, Business $74 per monthFree option for the no-budget path.
Claude APILLM for coding free textHaiku 4.5: $1 input, $5 output per million tokensBatch API is half price.

Build the project

You will launch a small niche technical community and build a survey-based model that tests whether word of mouth lines up with pipeline. A real community takes weeks to produce data, so there are two tracks. Track A is the real community and form. Track B is a synthetic dataset, so you can build and test everything today with no company and no budget. Build B first, then point the same code at A.

LayerWhat it doesWhere it lives
CaptureFree-text HDYHAU field, one Discord invite per sourceYour form, Discord
StoreAccounts, signups, coded answers, opportunitiesSQLite file dark_social.db
CodeLLM turns each answer into channel, named source, confidencecode_answers.py
ModelJoin, correlate, adjust for size, bootstrapanalyze.py
AuditSample 50 answers, label them blind, score agreementaudit.py
  1. Pick a niche narrow enough that you can name 30 people in it, such as people who self-host Postgres backups.
  2. Create a Discord server with a code of conduct, an intro channel, a help channel and one weekly ritual thread.
  3. Create one invite link per source (podcast, newsletter, partner Slack, your site), set each to never expire, and note each code. Discord reports uses per invite.
  4. Add one free-text HDYHAU input to your signup form, such as <input name="hdyhau" type="text">, and store the text word for word.
  5. Run pip install anthropic requests, save schema.sql and gen_data.py in one folder, and run python gen_data.py to create fake accounts.
  6. Save code_answers.py and run python code_answers.py --offline to code every answer with keyword rules.
  7. Save analyze.py and audit.py, then run python audit.py export to write 50 random answers to audit_sample.csv.
  8. Fill the my_channel column of audit_sample.csv by hand, then run python audit.py score to see Cohen's kappa and every disagreement.
  9. Set ANTHROPIC_API_KEY, then run python code_answers.py --recode to code every answer with Claude. Run python audit.py score again, and fix the prompt until kappa passes about 0.7.
  10. Run python analyze.py and read the raw lift, the size-adjusted lift and the bootstrap interval.
  11. Save discord_invites.py and run it daily, then compare joins per invite code with the survey's named sources.
  12. Write one page: what you measured, the adjusted number, three reasons it may not be causal, and one experiment that would settle it.

schema.sql:

CREATE TABLE IF NOT EXISTS accounts (account_id INTEGER PRIMARY KEY, domain TEXT NOT NULL,
  employees INTEGER NOT NULL);
CREATE TABLE IF NOT EXISTS signups (signup_id INTEGER PRIMARY KEY, account_id INTEGER NOT NULL
  REFERENCES accounts(account_id), created_at TEXT NOT NULL, hdyhau_text TEXT);
CREATE TABLE IF NOT EXISTS coded_answers (signup_id INTEGER PRIMARY KEY REFERENCES
  signups(signup_id), channel TEXT NOT NULL, named_source TEXT NOT NULL,
  confidence TEXT NOT NULL, prompt_version TEXT NOT NULL);
CREATE TABLE IF NOT EXISTS opportunities (opp_id INTEGER PRIMARY KEY, account_id INTEGER NOT NULL
  REFERENCES accounts(account_id), created_at TEXT NOT NULL, amount_usd REAL NOT NULL);

gen_data.py plants a small true community effect and a large size confounder:

"""Fake data. Planted truth: community adds a small lift; company size drives
BOTH community use and pipeline (a confounder). Run: python gen_data.py"""
import math
import random
import sqlite3
from datetime import date, timedelta

random.seed(42)
db = sqlite3.connect("dark_social.db")
db.executescript(open("schema.sql").read())
for t in ("opportunities", "coded_answers", "signups", "accounts"):
    db.execute(f"DELETE FROM {t}")

DARK = ["a friend sent me the link", "my old coworker told me", "saw it in a slack group",
        "someone in the discord", "word of mouth"]
OTHER = ["google", "searched for dictation api", "a podcast, forgot which",
         "chatgpt suggested it", "G2 reviews", "not sure", "n/a"]
for acc in range(1, 1501):
    employees = max(5, int(math.exp(random.gauss(5.5, 1.4))))
    size = math.log(employees) - 3.5
    in_comm = random.random() < min(0.95, max(0.02, 0.12 * size))
    db.execute("INSERT INTO accounts VALUES (?,?,?)", (acc, f"co{acc}.example", employees))
    first = date(2026, 1, 1) + timedelta(days=random.randint(0, 120))
    answer = random.choice(DARK if random.random() < (0.55 if in_comm else 0.12) else OTHER)
    db.execute("INSERT INTO signups (account_id, created_at, hdyhau_text) VALUES (?,?,?)",
               (acc, first.isoformat(), answer))
    if random.random() < min(0.9, max(0.01, 0.10 * size + (0.06 if in_comm else 0))):
        db.execute("INSERT INTO opportunities (account_id, created_at, amount_usd) VALUES (?,?,?)",
                   (acc, (first + timedelta(days=random.randint(5, 80))).isoformat(),
                    round(employees * random.uniform(8, 20), 2)))
db.commit()

code_answers.py. The Anthropic SDK reads ANTHROPIC_API_KEY from your environment, so no key lives in code:

"""Code 'How did you hear about us?' answers into channels.
python code_answers.py            (Claude; needs ANTHROPIC_API_KEY in the environment)
python code_answers.py --offline  (keyword rules, free)
Add --recode to delete old codes and code every answer again."""
import json
import sqlite3
import sys

MODEL = "claude-haiku-4-5-20251001"
PROMPT_VERSION = "v1"  # change this when you edit SYSTEM, then run with --recode
CHANNELS = ["peer_word_of_mouth", "community", "podcast", "social_post", "search",
            "ai_assistant", "review_site", "event", "ad", "other", "unknown"]
SYSTEM = ("You code survey answers to 'How did you hear about us?'. Pick one channel. "
          "peer_word_of_mouth is a person (friend, coworker). community is a Slack, Discord "
          "or forum group. Use unknown when no source is stated. Copy any named person, "
          "podcast, group or site into named_source, else an empty string. Never guess.")
SCHEMA = {
    "type": "object",
    "properties": {"channel": {"type": "string", "enum": CHANNELS},
                   "named_source": {"type": "string"},
                   "confidence": {"type": "string", "enum": ["low", "medium", "high"]}},
    "required": ["channel", "named_source", "confidence"],
    "additionalProperties": False,
}
RULES = [("peer_word_of_mouth", ("friend", "colleague", "coworker", "word of mouth")),
         ("community", ("discord", "slack")), ("podcast", ("podcast",)),
         ("ai_assistant", ("chatgpt",)), ("review_site", ("g2", "review")),
         ("search", ("google", "search"))]

def code_with_claude(client, text):
    resp = client.messages.create(
        model=MODEL, max_tokens=300, system=SYSTEM,
        messages=[{"role": "user", "content": f"<answer>{text}</answer>"}],
        output_config={"format": {"type": "json_schema", "schema": SCHEMA}},
    )
    if resp.stop_reason != "end_turn":
        raise RuntimeError(f"unexpected stop_reason {resp.stop_reason}")
    return json.loads(next(b.text for b in resp.content if b.type == "text"))

def code_offline(text):
    low = text.lower()
    for channel, words in RULES:
        if any(w in low for w in words):
            return {"channel": channel, "named_source": "", "confidence": "low"}
    return {"channel": "unknown", "named_source": "", "confidence": "low"}

if __name__ == "__main__":
    offline = "--offline" in sys.argv
    client = None
    if not offline:
        import anthropic
        client = anthropic.Anthropic()
    version = f"offline-{PROMPT_VERSION}" if offline else f"claude-{PROMPT_VERSION}"
    db = sqlite3.connect("dark_social.db")
    if "--recode" in sys.argv:
        db.execute("DELETE FROM coded_answers")
    rows = db.execute("""SELECT s.signup_id, s.hdyhau_text FROM signups s
        LEFT JOIN coded_answers c USING (signup_id) WHERE c.signup_id IS NULL""").fetchall()
    for signup_id, text in rows:
        text = text or ""
        r = code_offline(text) if offline else code_with_claude(client, text)
        db.execute("INSERT INTO coded_answers VALUES (?,?,?,?,?)",
                   (signup_id, r["channel"], r["named_source"], r["confidence"], version))
        db.commit()  # keep paid-for work if a later call fails
    print(f"coded {len(rows)} answers with {version}")

analyze.py holds the model, and audit.py imports its cohen_kappa function:

"""Does word of mouth line up with pipeline? Run: python analyze.py"""
import random
import sqlite3
from collections import Counter

SQL = """
WITH f AS (
  SELECT s.account_id, MIN(s.created_at) AS first_day,
         MAX(c.channel IN ('peer_word_of_mouth', 'community')) AS wom
  FROM signups s JOIN coded_answers c USING (signup_id) GROUP BY s.account_id)
SELECT a.employees, f.wom,
       EXISTS (SELECT 1 FROM opportunities o WHERE o.account_id = a.account_id
               AND o.created_at <= date(f.first_day, '+90 days'))
FROM accounts a JOIN f USING (account_id)"""

def band(n):
    return "small" if n < 200 else "mid" if n < 2000 else "large"

def cohen_kappa(a, b):
    """Agreement of two coders beyond chance. 1 is perfect, 0 is chance."""
    po = sum(x == y for x, y in zip(a, b)) / len(a)
    ca, cb = Counter(a), Counter(b)
    pe = sum(ca[k] * cb[k] for k in ca) / len(a) ** 2
    if pe == 1:  # both coders used one single label, kappa is undefined
        return 1.0 if po == 1 else 0.0
    return (po - pe) / (1 - pe)

def lift(rows):
    yes = [r["opp"] for r in rows if r["wom"]]
    no = [r["opp"] for r in rows if not r["wom"]]
    return sum(yes) / len(yes) - sum(no) / len(no) if yes and no else None

def adjusted_lift(rows):
    """Compare inside each size band, then weight bands by size."""
    parts = [[r for r in rows if r["band"] == b] for b in ("small", "mid", "large")]
    pairs = [(lift(p), len(p)) for p in parts if lift(p) is not None]
    return sum(d * n for d, n in pairs) / sum(n for _, n in pairs)

if __name__ == "__main__":
    rows = [{"band": band(e), "wom": w, "opp": o}
            for e, w, o in sqlite3.connect("dark_social.db").execute(SQL)]
    print(f"accounts {len(rows)}")
    print(f"raw WOM lift in P(opp)      = {lift(rows):+.3f}")
    print(f"size-adjusted WOM lift      = {adjusted_lift(rows):+.3f}")
    rng = random.Random(7)
    boots = sorted(adjusted_lift(rng.choices(rows, k=len(rows))) for _ in range(2000))
    print(f"95% bootstrap CI (adjusted) = [{boots[50]:+.3f}, {boots[1949]:+.3f}]")

audit.py exports a sample without the coder's labels, so your labeling stays blind:

"""python audit.py export  (writes audit_sample.csv; fill my_channel by hand)
python audit.py score   (prints Cohen's kappa and each disagreement)"""
import csv
import random
import sqlite3
import sys

from analyze import cohen_kappa
from code_answers import CHANNELS

db = sqlite3.connect("dark_social.db")
if sys.argv[1:] == ["export"]:
    rows = db.execute("SELECT signup_id, hdyhau_text FROM signups").fetchall()
    with open("audit_sample.csv", "w", newline="", encoding="utf-8") as f:
        w = csv.writer(f)
        w.writerow(["signup_id", "answer", "my_channel"])
        w.writerows([i, text, ""] for i, text in random.Random(1).sample(rows, 50))
    print("channels:", ", ".join(CHANNELS))
elif sys.argv[1:] == ["score"]:
    with open("audit_sample.csv", newline="", encoding="utf-8") as f:
        mine = list(csv.DictReader(f))
    if any(r["my_channel"].strip() not in CHANNELS for r in mine):
        sys.exit("every my_channel cell must be a channel name")
    coder = dict(db.execute("SELECT signup_id, channel FROM coded_answers"))
    human = [r["my_channel"].strip() for r in mine]
    model = [coder[int(r["signup_id"])] for r in mine]
    for r, h, m in zip(mine, human, model):
        if h != m:
            print(f"DISAGREE {r['answer']!r}: you={h} model={m}")
    print(f"kappa = {cohen_kappa(human, model):.2f} on {len(human)} answers")
else:
    sys.exit(__doc__)

discord_invites.py. Create a bot in the Discord developer portal, invite it with the Manage Server permission, and export DISCORD_BOT_TOKEN and DISCORD_GUILD_ID. The endpoint GET /guilds/{guild.id}/invites needs MANAGE_GUILD or VIEW_AUDIT_LOG. The guild docs I read give no field naming the invite a new member used, so the script diffs uses between snapshots. That blurs when several people join between snapshots through different invites.

"""Snapshot Discord invite use counts and print joins per invite since the last run.
Needs a bot with MANAGE_GUILD in your server. Run: python discord_invites.py"""
import json
import os
import time

import requests

API = "https://discord.com/api/v10"
GUILD_ID = os.environ["DISCORD_GUILD_ID"]
HEADERS = {"Authorization": f"Bot {os.environ['DISCORD_BOT_TOKEN']}"}

def get(path, **params):
    r = requests.get(f"{API}{path}", headers=HEADERS, params=params, timeout=30)
    r.raise_for_status()  # a daily run stays far below Discord's rate limits
    return r.json()

def snapshot():
    invites = get(f"/guilds/{GUILD_ID}/invites")  # list of invite objects with metadata
    members = get(f"/guilds/{GUILD_ID}", with_counts="true")["approximate_member_count"]
    return {"ts": int(time.time()), "members": members,
            "uses": {i["code"]: i["uses"] for i in invites}}

if __name__ == "__main__":
    try:
        with open("invite_snapshots.jsonl", encoding="utf-8") as f:
            before = json.loads(f.read().splitlines()[-1])
    except (FileNotFoundError, IndexError):
        before = None
    snap = snapshot()
    with open("invite_snapshots.jsonl", "a", encoding="utf-8") as f:
        f.write(json.dumps(snap) + "\n")
    print(f"{snap['members']} members, {len(snap['uses'])} invites")
    if before:
        for code, uses in sorted(snap["uses"].items()):
            gained = uses - before["uses"].get(code, 0)
            if gained:
                print(f"invite {code}: +{gained} joins since the last snapshot")

No-budget path. Use a Discord server, a free Tally form, SQLite and the synthetic generator, and skip Claude with --offline. Nothing here needs a company, a CRM or paid seats.

What done looks like. A running Discord with one ritual. Every signup has a stored free-text answer. A kappa above about 0.7 against your hand labels. A one-page note with the adjusted lift, its interval and two confounders. The synthetic run prints the numbers in the correlation section. I got them on Python 3.14, and other versions can differ.

Stretch goal. Replace the size bands with a logistic regression that adds region and signup month. Then design a holdout: invite half a new cohort to the community and compare pipeline after 90 days.

Common mistakes

  • Using a dropdown for the source question. It hides the podcast, the person and the group. Use free text and code it.
  • Reading the raw number. The raw lift mixes in company size and other confounders. Compare like with like and show an interval.
  • Chasing member count. A big quiet server is a bad sign. Count members who answer someone else.
  • Trusting the coder without an audit. Hand-label 50 answers with audit.py, check kappa and version your prompt.
  • Treating one vendor study as fact. The 98% figure comes from one firm's own forms. Report the source, the sample and the incentive.

Check yourself

1. Why does a free-text "How did you hear about us?" field usually beat a dropdown?

2. Your raw word-of-mouth lift is +7.2 points, but +2.3 after comparing inside company-size bands. What does this most likely mean?

3. In the SparkToro and Really Good Data test, what happened to clicks from Slack and Discord?

4. You code 1,000 answers with an LLM. What is the best check before you trust the totals?

5. Which statement do the studies in this stage support?

6. What is the status of the Orbit product and the Orbit Model in October 2026?

How to learn this faster

Learn first: the survey field and the coding loop. They take an afternoon and work anywhere. Then learn confounding with the synthetic data, then run a real community. Skip for now: paid platforms such as Common Room.

Drill: write 20 messy answers yourself, such as "my friend Sam", "G2 maybe?" and "lol idk". Code them by hand, then with the model. For each disagreement, decide who was right and fix the prompt.

  • Day 1: Read the Madrigal article and the SparkToro test. Build the form field.
  • Day 2: Run the generator and the offline coder. Read the analyze.py output line by line.
  • Day 3: Code with Claude using --recode, label 50 answers by hand with audit.py and compute kappa. Fix the prompt twice.
  • Day 4: Launch the Discord, seed 20 people, post a ritual thread, create invites.
  • Day 5: Snapshot invite counts and write the one-page note.

Resources

Next module

Stage 11 turns the same data discipline toward existing customers: the automated QBR (Quarterly Business Review), a script that builds a branded ROI deck for a customer without anyone assembling it by hand.

Built the project and passed the quiz? Mark the stage done. Your progress stays in this browser only.

Talk to Claude instead of typing to it.

Ducktate turns what you say into clean text, on your Mac, and pastes it anywhere.

Get early access