Opportunity to Insights and Beyond

• Funnel drop‑offs (e.g. sign‑up → activation, activation → invite, free → paid).
• Under‑performing segments (plan tiers, geos, new vs existing customers).
• Usage gaps: important features with low discovery or repeat use.

• Instrument key events in the product (e.g. sign up, create, edit, share, invite, admin actions).

• Define clear funnels (onboarding, first-value, upgrade), cohorts (plan, tenure, feature usage), and retention views.

• Review how similar products solve comparable problems.

• Identify best practices or gaps where Atlassian can differentiate.

• Spot new patterns in the market (e.g. new onboarding paradigms).

• Top‑down priorities (e.g. activation, retention, admin success, expansion).

• Areas where even a small uplift has large business impact.

• Bets that support longer‑term product or platform strategy.

• Expert reviews of key flows against UX and content heuristics.
• Spot inconsistencies, complexity, or unclear value propositions that can be iterated on and A/B tested.

• Crazy 8s – 8 rapid sketches in 8 minutes to generate many variations.

• How Might We (HMW) – turn problems into “How might we…” prompts for ideas.

• Silent brainstorm + dot voting – individual idea generation, then group voting on favourites.

• Impact / Effort mapping – plot ideas on a 2×2 to surface high‑impact, low‑effort bets.

Reusing a pattern that has already “won” in one part of the product and testing it in a new but similar area (e.g. a successful nudge, layout, checklist, or empty state).

When to use it:

• You have clear past learnings from a previous experiment (what worked, for whom, and why).

• You see a similar problem or funnel step in another surface (e.g. activation vs migration vs feature adoption).

How it works:

• Start from the insight, not just the UI (e.g. “time‑boxed checklists help new admins feel guided”).

• Design a variant that adapts that pattern to the new context.

• Run an experiment to confirm whether the pattern generalises or needs tuning for this new audience/flow.

Why it’s useful:

• Compounds value from previous experiments.

• Faster to design and build than net‑new concepts.

• Builds a library of reusable, proven patterns rather than one‑off wins.

• Start from the problem and hypothesis – who actually experiences the problem (e.g. new org admins, evaluators, power users)?

• Define inclusion criteria – product, surface, platform, plan, geo, language, tenure (e.g. “new Jira Cloud orgs in EN, on Standard+Premium, web only, first 30 days”).

• Define exclusion criteria – edge cases to exclude (e.g. very large orgs, internal sites, certain regulated regions, existing betas).

• Align with metrics – ensure the people you target are those who can move your primary metric.

• MDE is the smallest change in a metric your experiment is designed to reliably detect.

• Example: if MDE is +5% activation, you’re saying: “With this sample and duration, we can confidently tell if activation improves by at least 5%. Smaller changes may be too small to see clearly.”

It’s the “resolution” of your experiment:

• Lower MDE → need more users/time but can detect smaller improvements.

• Higher MDE → can run faster/with fewer users but only see bigger effects.

Choose a realistic MDE

• Decide the smallest improvement that would be worth the effort and realistically detectable (e.g. +3–5% relative uplift).

• Use this as the target effect size for:

  • Experiment design (sample size / duration).

  • Rough business impact sizing.

Learning experiments

Primary goal: reduce uncertainty and deepen understanding (the “why”).

Focus:

• Customer problems, behaviours, preferences.
• Testing big assumptions early.

Success criteria:

• Clear insights and direction, even if metric impact is neutral or negative.

Traits:

• Smaller samples, shorter duration.

• More tolerance for risk and rough UX.

• Often paired with qualitative methods.

Here are some common experimentation strategies used in product teams:

A/B tests (controlled experiments)

• Randomly split users into control vs variant.

• Measure impact on key metrics (activation, conversion, retention, etc.).

Feature flags & gradual rollouts

• Use flags to toggle features without redeploying.

• Start with internal/staff/beta cohorts.

• Ramp from small to full rollout while monitoring guardrails.

Experiment cohorts by segment

• Target specific segments (new vs existing, plan tier, geo, admin vs end-user).

• Compare heterogeneous treatment effects across segments.

Dogfooding / internal betas

• Ship early to employees and close partners.

• Collect qualitative feedback plus usage data before external rollout.

Holdback groups / long‑term control

• Keep a small % of orgs permanently without the feature.

• Validate long‑term impact and detect metric drift.

Multi-variant & factorial tests

• Test multiple versions of copy/layouts/flows at once.

• Or test combinations of factors (e.g. pricing layout × CTA copy).

Switchback / time-based experiments

• For non-user-level randomisation (e.g. infra changes).

• Alternate between control and treatment over time windows.

Experimentation in release channels

• Ship via internal → sandbox → early access/beta → production.

Qual + quant paired experiments

• Run an A/B test and parallel interviews or usability tests.

• Use qual insights to understand why variants win or lose.

Combined experiments

• Combine multiple smaller experiences that individually don’t reach significance to achieve significance together.

• Use when the concept is new/novel, directionally unclear, or you need more evidence behind a decision.

• Get real customer feedback via usability tests, interviews, or other methods.

Start from the problem & hypothesis

Ask: “If this works, what changes in user behaviour or business outcome?”

• Pick one primary success metric that directly reflects that (e.g. org activation, setup completion, invite-sent, upgrade rate).

Make the primary metric specific and time‑bound

e.g. “Increase activated orgs within 14 days” or “Increase admins who complete setup checklist in first 7 days.”

• Ensure the experiment audience can actually move this metric.

Add guardrail metrics to protect user experience

Choose 2–4 metrics you don’t want to hurt, e.g.:

• Performance: latency, error rates.

• Engagement elsewhere: completion of adjacent key flows.

• Support/risk: support contacts, abuse, cancellations.

Define clear red lines (e.g. “do not increase error rate by >X% vs control”).

Write them down in the brief

For each:

• Primary success metric: what it is, how it’s calculated, target uplift.

• Guardrails: what they are and what counts as breach → roll back/pause.

This is about estimating likely impact before investing heavily.

Start from the funnel and baselines

• Pick the step to improve (activation, invite-sent, upgrade).

• Document:

  • Volume hitting that step per period.

  • Current conversion rate.

Translate uplift into impact

Combine:

• Volume.

• Baseline rate.

• Target uplift (MDE).

Estimate:

• Extra activations/invites/upgrades per period.

• Rough dollar impact if you have ARPU/LTV or a proxy.

Compare impact vs effort

Plot ideas on impact vs effort:

• High impact / low–medium effort → strong experiment candidates.

• Low impact / high effort → deprioritise or convert to small learning experiments.

Use sizing to shape experiments

If opportunity is small:

• Accept a higher MDE or lighter-weight/learning-focused experiment.

If opportunity is large:

• Invest in better instrumentation.

• Design a stronger, longer-running experiment with lower MDE.

Are you solving the right problem, with clear success measures?

A good problem statement should…
Aim for 1–2 sentences that are:

• User‑anchored – “For which user, when, doing what?”

• Evidence‑based – reference how you know (funnels, interviews, etc.).

• Impact‑connected – tie to the metric that matters.

Template:

“For [who], when [situation], [problem] happens, which leads to [negative outcome / metric impact], as seen in [data / research].”

A good hypothesis looks like…
Make it specific, testable, and metric‑tied:

• If we do X… then Y will happen… measured by Z… for W group.

  • Example: “If we surface a guided setup checklist on the admin home, then more new admins will complete critical setup tasks, measured by +5% relative uplift in ‘activated orgs’ within 14 days, for new Jira Cloud orgs in EN on Standard+Premium.”

Checklist:

• Clear change: one primary idea.

• Clear target audience: matches targeting.

• Clear primary metric and direction: “increase/decrease X by ~Y%”.

• Optionally mention important guardrails.

Simple hypothesis template:

“If we [change] for [who], then [metric] will [increase/decrease] by ~[X%], because [reason / insight from qual/quant].”

• Understand the details of the current experience.

• Ensure you’re designing for what actually exists.

• Avoid out-of-date design files.

• See what others are doing and what “good” looks like.

• Understand customer mental models and expectations.

• Investigate and uncover technical constraints.
• Explore Good, Better, Best options.
• Advocate for the “Best” direction where feasible.

• Iterate quickly in tools like Replit or Figma Make.

• Learn how the experience feels in code.

• Create an artefact to centre discussion.

• Share across functions and squads to build alignment.

• Check for blockers or overlapping experiments in the same space.

• Use 30/60/90 check-ins.

• Take work to Design Crit for feedback and ideas.

• Take it to Product Design Jam with senior leadership for visibility.

• Use when the concept is new/novel, directionally unclear, or you need more evidence behind a decision.

• Get real customer feedback via usability tests, interviews, or other methods.

• Complete Proud To Make Scorecard and get stakeholder sign-off.

• Ensure components and tokens adhere to brand guidelines.

• Factor in accessibility requirements.

• Visual and functional QA of implemented designs.

• Work closely with the Feature Lead.

• Pivot as needed if new technical challenges appear.

• Provide feedback to get implementation as close as possible to the intended design.

• Add feature flags for control and variants and wire into code.

• Ensure the control experience is explicitly defined.

• Emit “experiment exposure” events with name, variant, and user/org ID.

• Add/verify events needed for primary and guardrail metrics.

• Encode inclusion/exclusion rules from the brief.

• Double-check random, consistent bucketing (same user/org → same variant).

Test locally and in staging/dark environments with flags on/off.

Verify:

• Correct users see the variant.

• Events fire with correct properties and metadata.

• No obvious performance or error regressions.

• Agree initial rollout percentage and ramp schedule with PM/analytics.

• Document rollback plan and ownership for guardrail breaches.

• Enable the flag for the agreed cohort and percentage.

• Monitor guardrail dashboards (errors, latency, major flows) especially early.

• Coordinate with PM/analytics on reaching required sample size and duration.

Confirm sample size and run time match the plan (no early stopping).

Verify targeting and exposure were correct.

Ensure primary and guardrail metrics are populated for control and variants.

After analysis, either:

• Roll out winning variant to 100%, or

• Roll back to control and clean up dead code/flags.

Link experiment, rollout decision, and Jira issues for traceability.

• Inconclusive results – effect in the right direction but not statistically significant, or under‑powered sample.
• Mixed results – some segments win while others are flat/negative.
• Clear UX or tech issues – evidence of problems that likely suppressed impact.
• Learning but not earning – strong insights but limited metric movement; clear next idea to test.
• Small win worth improving – modest uplift with room to compound gains.