After publishing an interview about our Scrum-based process at Upstatement with one of our awesome clients (Harvard Gazette), I thought it would be helpful to write up a more detailed appendix for the Agile devotees out there. We’re often asked by these folks to get specific: When applying Scrum methodology in an agency setting, how does it actually work?
One personal disclaimer is that I am not an Agile/Scrum diehard believer. I arrived at Upstatement when the company was coalescing on switching from waterfall to agile framework, so I dove in to learn and helped make adjustments accordingly . But as time has gone forward, I’ve wished for longer periods for focused work (2 week sprints can be stressful) and more of a long-term plan for our projects. We’ve adjusted accordingly and continue to make smart exceptions. Since Scrum was designed for continuous software development within internal teams, we’d be destined to fail if we went by the book.
But otherwise, here it is — our scrummy secrets exposed! After a few years, we have settled on some trends in the process:
Why track product usage?
How do you drive usage or get more people to use your app/ service over a period? By ensuring your app delivers value to them. In other words, helps them solve a problem or perform a task or make life simpler, smarter, or better in a specific way. While this value proposition is the true north for your business and development teams might have built stellar features to fulfill this promise, users’ perception of this value could differ drastically from yours. Therefore, the need to track and understand how they engage with your product through meaningful product analytics.
By measuring product engagement, you’ll have data and numbers to guide:
- Product development. Understand which features attract the maximum users, which ones are less used, whether usage increased or reduced after your last feature release, to inform product roadmaps and guide development teams.
- Customer success. Know which customer accounts are thriving and who needs support, where. Take proactive steps to deliver customer success, reduce churn, and keep the renewals coming.
- Overall business. Get a clear picture of user behavior by tracking key factors like churn, retention, and loyalty, which impact product usage and therefore, the overall business.
When done well, product analytics gets you deeper insights into user behavior which helps you build sharper engagement strategies and drive product success.
How to measure engagement?
Step 1: Define engagement
The idea of engagement varies based on the product – a grocery delivery app gets used differently from an airport app. Therefore, before discussing how to measure engagement, it’s essential to define what engagement looks like for your specific product.
In essence, engagement suggests users taking meaningful action with your app over a period. Begin by listing the activities users can perform on your app. If you run a grocery shopping app, your list could look like this:
- Log in
- Create a profile
- Make first purchase
- Sign-up to get notified about a product
- Refer a friend
- Claim referral bonus
- Write a review on social media
Remember, this list is likely to change and evolve in sync with your product.
Step 2: Track actions that translate into engagement
Now that you know which user actions are meaningful and relevant, set up a mechanism to track them. We suggest using an app tracking and analytics tool, which gives you a holistic picture of user behavior across touchpoints and all through the user journey. Some tools allow you to track both web and mobile apps, even providing video captures of how users interact with your app.
Step 3: Assign a weightage for different activities
For SaaS analytics, each key activity you had listed in step 1, assign a weightage (say values ranging from 1-10, where 10 indicates the highest weightage). Why do this? Some activities are more valuable and meaningful than others. For instance, inviting a friend or writing a review on social media is more meaningful than asking to be notified about a product. The former should get a higher weightage in your scoring system, say 7 as against a 4 for choosing product notification.
Step 4: Calculate an overall score
For each activity, you wish to measure, multiply the weightage by the number of times the activity took place for a single user account, in a defined reporting period. For instance, if tracking logins over two weeks, your score could look like
Event (login): 1 (weightage) x 7 (number of occurrences in the last 2 weeks) = 7
Tabulate this score for all events to create an overall engagement score for each user account for that defined reporting period.
Your table could look like this:
|Event||Weightage||Occurrences in the last 2 weeks||Total event score|
|Total engagement score= 25|
Run a similar exercise for each account, to get the engagement score for each account. Divide by the number of accounts to get an average product engagement score.
Step 5: Interpret and make meaning
We have a product engagement score of 25. But what does this suggest – is 25 an indicator of high engagement, or is that an average score? Moreover, each of your user accounts come with their own unique attributes. Account A could have 25 users, with 12 of them logging in daily. Account B with 10 users could see 8 user logins daily. How would you account for these nuances and still have a reliable way of interpreting these scores? We suggest using an approach like percentile calculation to compare these raw scores and make meaning from them.
Why do it yourself when you can automate it?
Now that you know how a product engagement score can guide your business, you’ll want to track it periodically. But tracking events and calculating the scores can be time and effort consuming. SmartKarrot’s Product Engagement Score calculates product engagement automatically, removing the need for rows and columns of data on your spreadsheet.
What is SmartKarrot’s Product Engagement Score?
It’s an aggregate of four scores that track different aspects of product engagement to provide analytics.
1> Duration Index tracks app usage. It shows the median time (in hours) spent by a user on the platform per day, and also the usage trend, by comparing the length of sessions or time spent, every 15 days. As a product company, one needs to carefully watch app usage and trends, and this index lets one do exactly that.
2> Retention Index measures regularly active users through cohort analysis. It measures the percentage of users who continue to revisit the app and use the platform over time. Each of these pre-defined intervals gets a weighted score, which is then used to calculate the composite score for retention.
3> Performance Index is a composite of two key sub-parameters – the average time for a page to load, and the percentage of successful items loading for that page. Both affect user retention in today’s digital economy thriving on accessibility and immediacy.
The latter is particularly relevant for pages that see high request volumes, giving the business a clearer view of how different pages/ sections of their app perform. Performance Index is replaced by Rating Index for mobile apps, where we pick up ratings from the Mobile App Store (if it is available in the public domain).
4> Loyalty Index measures the overall continuity of usage of the platform’s features. It is calculated based on the number of users who are dropping out (churn), measured by a lack of logins for longer than a month. The index also accounts for the type of app (Mobile or Web) to create an appropriate benchmark.
Through product analytics track all key events and indicators that matter for product engagement, in a single place. See changes in product usage in real-time and take the right steps to improve engagement. Reliable product analytics and actionable insights, without the need to build a calculation mechanism or manage the process manually. That’s what SmartKarrot offers.