Before our IT team rolls it out to the whole organization to everyone in our organization, some of our internal users have been beta-testing it which prompted this article (see what I did there? 😉).
This article covers:
To pilot Microsoft 365 Copilot, start by creating a clear plan and building a cross-functional team that’s willing to embrace the future of AI-assisted work. The team we built includes a mix of employees from various departments, including R&D, Sales, Marketing, HR and others. It’s also important to include users with various levels of technical acumen, ranging from foundational proficiency in Microsoft 365 to more advanced users who can script or even build their own apps.
Here’s a step-by-step rundown of how to approach a pilot program for Copilot. These are the steps we took, which can serve as a blueprint for other organizations looking to explore Copilot’s capabilities.
Before embracing this new technology, all team members must have access to the tool. We activated a number of 365 Copilot licenses and assigned them to the appropriate accounts.
In our case, there was an initial hiccup with licenses assigned to the wrong accounts, but it was quickly resolved. Keeping a keen eye on these details from the get-go can prevent delays in deployment.
When piloting 365 Copilot, everyone needs to get the most out of their subscription. We used our internal M365 management solution to keep tabs on usage, but other options are available for monitoring usage and adoption. (E.g. a different Microsoft management tool or even PowerShell) But right now, there's no ready-made tool that can do this for you.
Ultimately, this approach helps us continuously identify who is actively using the tool and who may need a nudge or additional support.
Moreover, I recommend following the principle of use-it-or-lose-it—if there’s no usage within a week, the license can be given to someone else who is keener to try out Copilot's possibilities.
To make sure all feedback gets back to IT, teams should establish a system for gathering feedback from your beta testers.
We created a Teams chat where all Copilot users could share how they were using Copilot. Users were then required to prefix all feedback messages with “Feedback:”
That way, the team can easily sort through the conversations (or have Copilot do it for you!) and extract feedback.
Encouraging the beta testers to use Copilot for a variety of tasks—from scheduling meetings to summarizing discussions—allows you to collect a wide range of data on its performance.
Document both positive and negative feedback in real-time,to help the other users and the IT team get a nuanced understanding of Copilot’s capabilities and limitations.
One of the best ways to evaluate the effectiveness of a new technology is to look at the real-world scenarios and advantages it creates.
Through our 365 Copilot trial, these are the best use cases that surfaced:
One of the standout features of Copilot is its ability to digest lengthy conversations and distill them into concise summaries. This is particularly useful for meetings, where key points and decisions can be quickly reviewed.
During a meeting, one of our beta testers used the prompt"Recap the meeting so far" with Copilot. The tool provided a quick summary to the users who joined the meeting late, helping them quickly catch upon the discussion.
Moreover, the "Get action items" command,used at the end of a call, proved to be beneficial, albeit with varying degrees of accuracy.
It provides a solid starting point for defining tasks and responsibilities post-meeting. However, you should expect to manually adjust action items after a meeting since Copilot isn’t always super accurate.
Example of “Recap the meeting” functionality:
Another highly praised use case is Copilot's ability to sift through an overflowing inbox and summarize the content, extracting a list of to-dos.
Microsoft MVP nominee and in-house M365 expert, Kas Nowicka, has shared her personal experience using Copilot:
Team members found this particularly valuable, as it helped them catch up on missed correspondence and prioritize tasks without having to read every single email.
Extending beyond the confines of internal operations, Copilot also demonstrated its utility in customer-facing situations. For instance,when asked to generate a bilingual presentation description for an upcoming event, Copilot delivered a succinct and professional summary that me tour expectations, saving valuable time in the process.
Here’s the Copilot prompt that Vincent Convert, one of the Regional Sales Managers responsible for French-speaking territories, used to create his presentation description in both French and English:
Copilot provided a suitable description, demonstrating its ability to save time and aid in bilingual communication.
The art of crafting Copilot (or ChatGPT or any other AI models) prompts turns out to be a critical skill when working with Copilot. A well-phrased prompt could mean the difference between a useful response and a misfire.
Our team worked on developing a set of "out-of-the-box" prompts that yielded consistently positive results, tailored to common business scenarios. Sharing these prompts within the team helped everyone get the most out of Copilot.
Here are some examples of the prompts our team used:
Here are some more prompts that may be useful:
This prompt is helps employees quickly and professionally respond to common questions. Use it to generate concise, polite, and informative replies, saving time and ensuring communication stays consistent.
"Draft a professional response to a client asking for an update on their project status, ensuring to convey that we are on track and will meet the deadline. Include a brief overview of the completed milestones and what is scheduled for the next phase."
This example uses Copilot's capabilities to prep meeting materials. Use this prompt below to organize thoughts and data into a coherent outline and build presentations faster.
"Create a presentation outline for the quarterly project review meeting. Include slides on project objectives, progress to date, challenges faced, solutions implemented, and next steps. Use bullet points for slide content and suggest relevant graphics or charts."
This prompt is intended for data analysis tasks. Use it to sift through data you provide (in this case, sales data) to extract meaningful insights, trends, and suggestions. This can significantly speed up the process of report generation and make sure key information is not overlooked.
One of the more immediate limitations we met was Copilot's inability to perform tasks that require direct interaction with calendar tools and scheduling software.
One employee highlighted this gap:
This reflects a broader challenge with Copilot's integration capabilities, where it can provide information but cannot take actions within third-party applications or services just yet.
Our team found that the accuracy of Copilot's outputs could sometimes be hit or miss, particularly when generating action items or summarizing complex discussions.
The "Get action items" feature, for instance, was less accurate than the summarization function, indicating that Copilot sometimes struggles with context and specificity. This necessitates a degree of oversight and manual correction, which can offset some of the time-saving benefits.
In fact, in a recent Copilot readiness assessment of CoreView’s customers over 83% of respondents are concerned about Copilot generating incorrect or out-of-date information.
Over 83% of survey respondents are concerned that Copilot will generate incorrect or out-of-date information.
As an AI chat tool, Copilot is bound by its nature to be an assistant that provides suggestions and information rather than executing tasks. This means that while it can propose a draft for a presentation or an email, it cannot carry out the actual sending or presenting. This limitation emphasizes the importance of understanding Copilot's role as a supportive tool rather than a replacement for hands-on task management.
When comparing Copilot to other AI tools like our internal version of ChatGPT, CoreChat, the majority of the team members noticed a significant difference in productivity enhancement.
The sentiment was that "CoreChat is “10x” more efficient and helpful, which helped us conclude for the time being, that Copilot, while useful, may not always meet the high expectations set by other AI chat tools in certain applications. This points to a need for continuous development and fine-tuning of Copilot's algorithms to better serve its user base.
The pilot also revealed that Copilot's performance could vary in real-time scenarios. For example, when asking simple questions in Copilot Edge, the responses received were not always on par with the same queries posed to ChatGPT. This inconsistency can affect user trust and reliance on the tool for immediate, accurate information.
In a recent conversation with Microsoft 365 expert Kas Nowicka, she shared an update on Microsoft’s new feature called Restricted SharePoint Search:
The Restricted SharePoint Search feature is a welcome addition to the Copilot for Microsoft 365 tool, as it enhances the security and privacy of data in large enterprises. By limiting the search scope to only relevant and trusted sites, users can avoid exposing sensitive information to unauthorized parties or external sources.
However, this feature may also come at the cost of reducing the functionality and accuracy of Copilot, especially in real-time scenarios where users need quick and reliable answers. Therefore, users may need to weigh the trade-offs between security and convenience when using Copilot for their work tasks.
Microsoft 365 Copilot stands at the forefront of workplace evolution, an AI tool that promises to streamline our daily tasks. This pilot experience has revealed its potential and its current limitations, offering a balanced perspective on its role in an organization’s digital toolkit. As organizations continue to use Copilot, it's important to balance human creativity with technological assistance.
Ultimately though, organizations that embrace Microsoft 365 Copilot in daily operations will gain immense value.
As you navigate through all of AI tools available, remember that choosing the right technology can seriously improve your business outcomes. And, keep in mind how you will adjust your M365 governance strategy to account for this new technology.
“In an age where the pace of innovation accelerates pretty much daily, the readiness to adopt new technologies, like Microsoft 365 Copilot, demands a shift in our approach. It's no longer about 'security by obscurity,' but about embracing a proactive stance. We must lean into the future, understanding that the true measure of our adaptability lies not in resisting change, but in our willingness to evolve alongside it," reflects Kas.
While Microsoft 365 Copilot is a robust tool,complementing it with specialized solutions like CoreView can help you meet your business goals.
