Over the past couple months, I have been thinking a lot about adoption metrics around SharePoint and Office 365, and wanted to share some thoughts about the Office 365 Adoption Content Pack for PowerBI that was announced at Microsoft Ignite 2016 in Atlanta. Specifically, what it delivers and what it still lacks, and why every enterprise should be thinking about adoption and engagement metrics, in general.
What is the Adoption Content Pack?
In an early October blog post by Anne Michels (@Anne_Michels) from the Office 365 marketing team, Microsoft opened a limited preview program (through October 16th) of the Adoption Content Pack, with broader availability by the end of December. Its purpose is to enable customers to visualize and analyze Office 365 usage data, create custom reports, share the insights within their organization, and pivot by various attributes, such as location and department.
Of course, you may have noticed a few other updates recently around the usage reports in the new Office 365 admin center, which are a great starting point to understand usage data and better monitor and measure what is happening within your environment. What is new about this Adoption Content Pack is that they allow you to customize and personalize your usage insights, and build your reporting around the metrics that are unique to your own business. To make this possible, Microsoft is combining the intelligence of their out-of-the-box usage reports with the interactive reporting capabilities of Power BI.
As highlighted in Anne's blog post, the dashboard is split up into four areas: Adoption, Communication, Collaboration and Activation. Authorized admins can access detailed dashboards for each area by clicking on any of the metrics. According to the post, here is what is included:
- Adoption report—Helps you understand how your users have adopted Office 365 as well as how usage of the individual services has changed month-over-month. Admins can easily see how many users they have assigned a license to, how many users actively use the products and how many are first time users or returning users that use the product each month. This helps admins identify the products for which additional user training might be needed to increase adoption.
- Communication report—Shows admins how users use Office 365 to communicate. The dashboard includes a communication activities report that provides details about how the usage of different communication methods—such as email or Yammer message posts—has changed over time allowing admins to understand how their users adopt new ways of communication. Additional metrics include average number of emails sent, average number of Yammer posts read and average amount of time spent using Skype. The dashboard also shows which client apps are used to read email or to use Skype.
- Collaboration report—Gives you the ability to see how people in your organization use OneDrive and SharePoint to store documents and collaborate with each other and how this is changing. Admins can also see how many users share documents internally versus externally.
- Activation report—Helps you understand Office 365 ProPlus, Project and Visio activations. Admins can see total activations across users, number of users that have activated the products, number of devices they have activated them on and the type of device.
Her post goes on to provide details on several additional reports that admins can access at the bottom of the admin center:
- Yammer Usage report—Useful for organizations that are in the process of rolling out Yammer or are focused on increasing usage. The report provides helpful information about how various parts of your organization adopt Yammer as a form or communication including how many people post messages, how many consume content by liking or reading a message and how new user activation has changed over time.
- Skype for Business Usage report—Provides a consolidated view of Skype activity as well as with details about how many users leverage Skype to connect with others through peer-to-peer messages and how many communicate their ideas by participating or organizing video conferences.
- OneDrive for Business Usage report—Shows admins how users leverage OneDrive to collaborate with others in new ways. They can easily see how many users use OneDrive to share files and how many utilize it mostly for file storage. The report also includes information about how many OneDrive accounts are actively being used, and how many files are stored on average.
- SharePoint Usage report—Shows how SharePoint team sites and groups sites are being used to store files and for collaboration. The report also includes information about how many SharePoint sites are actively being used, and how many files are stored on average.
- Office 365 Top User report—Enables admins to identify Office 365 power users and the individual products they are using. Power users can often help to drive product usage by sharing their experience about how they use the products to get their work done faster and more efficiently.
But wait—there's more! At the end of October, Microsoft announced several new usage reports that provide some additional depth to your Office 365 reporting:
- Active Users report—Lets you see which of your users actively use one or more of the different Office 365 services. This report is especially helpful for admins to identify users for whom they might want to plan some additional training and communication. Often, after being assigned an Office 365 license, users need a helping hand to get started with the different services. They might not know how to activate the product or how the product can help them to be more productive.
- Skype for Business clients used report—Shows the devices on which users have installed the Skype for Business app and whether they are using those apps for instant messaging or meetings. The report includes both Windows devices and mobile devices and makes it easy for you to see which device types are most commonly used by your users to connect to Skype for Business.
- Email clients used report—Provides information about which apps are used to connect to Exchange, including information about the specific Outlook version.
- Office 365 Groups activity report—Office 365 Groupsis a service that provides a single identity for teams in Office 365 and equips them with a set of group collaboration assets, such as a shared calendar, notebook, project planning tool, as well as conversations powered by Outlook or Yammer. The Office 365 Groups activity report helps you understand how groups facilitate collaboration across your organization by surfacing information about which groups are active, how many members and guests they have, and how they are being used. The report is currently based on email activity and will be enhanced by other group activities in the future, including SharePoint Group Sites and Yammer Groups.
Admins also have the ability to share these usage reports with anyone else within your organization -- as long as they have access to the Office 365 admin center. And because the Adoption Content Pack combines usage data with the Active Directory (AD) information of your users, admins can also pivot the reporting data by AD attributes such as location, department, and organization. If you have been concerned with the level of adoption and engagement of Office 365 within your organization AND you're a fan of PowerBI, then this should all hit your sweet spot!
You can find out more about the Adoption Content Pack in Ignite session BRK2008 -- understand Your users: What's New in Office 365 Usage Reporting, available through the Microsoft Tech Community.
Are there gaps?
Well, that's a great question! To help answer that, I reached out to someone who I knew was eager to play with the new capabilities -- and who also knows a thing or two about PowerBI and adoption metrics: MVP John White (@diverdown1964), CTO of UnlimitedViz and TyGraph.
John's biggest concern about the Adoption Content Pack is that it works as part of a new reporting API, which is new to the community, and not something partners have been able to play with yet. Getting access to this API will be key, as there are a number of metrics available in the admin portal that cannot be derived from the Microsoft Graph alone, or even from the Management API -- which is our primary source of data. According to John, the content pack will provide insights into some, but not all of the data contained in the reporting API.
John also points out that almost all of the metrics are around adoption of the product, rather than providing metrics on influencers and collaboration between users. For example, they inform you on how many licenses are active, how much each product is being used -- from a time basis in the case of Skype, number of messages for Exchange, and the volume of storage for OneDrive. However, there are few collaboration metrics (such as top influencers), and it’s unclear as to how they’re calculated or comprised. For organizations that are more advanced in the monitoring and measurement of engagement and collaboration, the out-of-the-box metrics may not provide what you need.
Another gap that John identifies is the limited trending data, noting that the maximum timeframe for data detail is 180 days. The service will not retain data any further than that. There are some aggregated metrics that stretch out to a year, but they’re just that – pre-aggregated and fixed datasets. The focus is on month-over-month comparisons, and year-over-year is impossible. While set up and provisioning of the service is fairly easy (Microsoft needs to set it up, which takes time, just like Delve Analytics and you are up and running with the canned dashboards almost immediately, according to John, you get what you get. There is currently no facility to extend these reports in any meaningful way (with Power BI Desktop). You could use Analyze in Excel to work directly with the data model, but that’s about it. Having said that, John does expect that to change within the next six months. As we've seen with most features in Office 365, the pace of change has increased dramatically, and the more feedback Microsoft receives from the community, the more quickly we will see changes and improvements.
One last point that John makes is that the design is also locked to the data in the service. Even with some report extensibility, the data in the model is all that will be available, and there’s no real effective way to mash it up with organizational data. While the content pack will work with AD data, John states that data is almost always incorrect and incomplete, and that real value comes when you can amplify these metrics by mashing them up with organizational data.
These gaps may close sooner rather than later. Microsoft has more recently stated that they will be adding more reports focused on Yammer Groups and clients used to access SharePoint, OneDrive for Business, and Yammer. Additionally, they have shared that they are planning to provide new public APIs that will enable admins to programmatically access the usage data and integrate it into custom applications, like a company reporting portal.
Adoption Data in the Real-World
It is exciting to see things moving so quickly, and Microsoft focusing on ways in which to help customers better focus on adoption -- but organizations will need to go beyond these reports to truly understand who their influencers are and how to replicate that success across the entire organization. As John mentions, adoption data in the real world is more than simply tracking the volume of storage in OneDrive or monitoring the number of Outlook Groups created. The out-of-the-box tools may not yet be in place to accurately measure the success of Office 365, which means that organizations must get creative in what they measure -- and how they measure. They need to clearly define what successful collaboration looks like, and then put into place the metrics and reporting to help them monitor and measure that success.
The Office 365 Adoption Content Pack may not provide 100% of what organizations need, but it's another huge step in the right direction by Microsoft.