Product Analytics¶

Metrics¶

• 3 ways to look at metrics
  1. Hits
  2. Sessions
  3. Users

• Why are sessions usually preferable to hits?

  • Hits may inflate/deflate the metrics, and may mislead the analysis

• Define KPIs = key metrics

  • If a KPI going up/down does not instigate any action, then it is useless
• Have a Northstar KPI
  • Best growth KPI: Retained MAU YOY
  • Best performance KPI: Total GMV
• Decompose KPI into drivers to form a metric tree
  • For eg:
    • Total GMV
      • = # Orders x AOV
      • # Orders
        • = # Customers x Order Frequency

Frameworks¶

• Customer Funnel
• Customer Lifecycle
• Cohort Analysis

Things to define

Customer Funnel/Loop¶

• Exclude easy-to-convert aspects from the analysis, such as platform-associated vendors' funnel
• Consider Item = interested thing
  • Ecommerce item
  • Restaurant/Vendor

Loop = Funnel + Cycle

Stages¶

flowchart LR

pa[Platform<br/>Awareness]
ia[Item<br/>Awareness]
mm
conv[Conversion]
r[Referral]

subgraph mm[Messy Middle]
eng[Engagement]
c[Consideration]
c -->|Doubt| eng
end

aq(( )) -->
|Acquisition| pa -->
ia -->
eng -->
c -->
conv -->
r -.->
|Retention<br/>+<br/>Expansion| ia

Note: make sure to backfill - For eg, if there is a click, there should be a view

Customer Lifecycle¶

Each of the Customer Funnel stages have the following states in the lifecycle - Primarily, for reporting, we look at user lifecycle states in terms of conversion

States¶

flowchart LR

a[Acquisition]
r[Retention]
c[Churn]
w[Winback]

a --> r
c ---> w

a & r & w -.-> c
w & r --> r

Better Understanding¶

Think of it as - a leaky pipeline, - with a funnel at each time point, - determining the customer lifecycle for each funnel stage

Growth Accounting¶

Users_t
= Acq_t + Ret_t + Winback_t

Users_t-1
= Ret_t + Churn_t

Growth_t
= Users_t - Users_t-1
= (Acq_t + Winback_t) - Churn_t

Cohort Analysis¶

Curve	Common	Visualization		Visual	Desired
Retention Curve	Usually, conversion retention % from conversion acquisition	Line chart with cohorts as color and X axis as horizon	Metric action trend of cohort of different cohorts from first cohort action		- Flatter curve - Higher curves - Across later cohorts above both
		Line chart with horizon as color and X axis as rolling cohorts	Cohort trend for different metric horizons eg: 1 month retention		- Improving trend
Layer cake	Usually, user base/GMV	Stacked chart	Composition of metric at different time points from different cohorts		Thick layers coming from old cohorts
		100% Stacked chart	Proportion of metric at different time points from different cohorts

Top of Funnel vs Bottom of Funnel¶

When the cost of top-of-funnel reach is zero, the conversion rate matters more than the volume of engagement - long-term success is predicated on intentionally filtering for the highest-probability, customers to maximize conversion - Spending time on skeptical leads is a waste of resources. An intentional "top of the funnel" acts as a qualifier, not just a reach mechanism

The Nigerian prince scam - Scammers send mass emails offering a share of a massive fortune in exchange for a small upfront fee. - The goal of the initial email is not just to get a reply, but to identify victims with high gullibility and low tech literacy. - The infamous grammatical errors and bizarre narratives are not accidental - By filtering out skeptical, rational, or busy individuals early, scammers ensure they only spend time engaging with highly gullible, high-intent leads

Churn Attribution¶

When did user churn?

• jan 15^th: user makes order
• ⁠feb: no order

Idea - If comes back on app in Feb, then they are churned in Feb, as they came on top of funnel but did not convert which could be poor app experience - If they do not come back on app in Feb, then they are churned in Jan as they did not even come to top of funnel, either due to poor order experience or no longer interested

Comments