Scaling Up: Managing Pagination, Throttling, and Large Tenant M365 Audits

Welcome back to the blog! If you are managing a massive Microsoft 365 environment, you already know that standard out-of-the-box reporting tools only scratch the surface. When security teams and compliance officers look at massive tenants, basic scripts and out-of-the-box admin center views simply fail under the weight of thousands of users, groups, and files. Auditing massive M365 tenants requires more than basic scripts; it demands a tactical, programmatic approach using the Microsoft Graph API. In this comprehensive guide, we are expanding directly on our recent conversation in the podcast episode Map Microsoft 365 Users, Groups, Files, and Shares with Graph. Let us dive deep into how to handle @odata.nextLink pagination, respect 429 rate limits with exponential backoff, and avoid common traps like Page 1 bias to achieve total tenant visibility.

Introduction to M365 Auditing Challenges

Enterprise environments grow organically. Users come and go, projects spin up departments overnight, and external vendors get invited into collaboration spaces. Over time, administrative visibility degrades. Default admin centers show isolated fragments of your infrastructure, leaving blind spots that malicious actors or accidental insiders can exploit. To get a true picture of your tenant security posture, you need to transition from manual point-and-click auditing to automated API-driven graph traversal. However, scaling these queries up to enterprise sizes introduces complex hurdles, including massive data payloads, aggressive throttling, and deeply nested relational structures.

Why M365 Isn't Isolated: Understanding Relational Access

One of the foundational realities of Microsoft 365 is that services like SharePoint, Teams, and OneDrive are not isolated silos. They cross-pollinate continuously via group membership and underlying access control lists. A single administrative action, such as adding a user to a security group, can ripple across dozens of team sites, document libraries, and private channels without leaving obvious audit continuity in standard views.

This creates hidden compliance risks. Guest links shared in isolated chats, files mirrored across different team libraries, and orphaned shares from long-dead projects create invisible pathways into your corporate data. Access in M365 is fundamentally relational, not location-based. To audit effectively, you must map these cascading relationships rather than treating each service as an independent island.

Footprint Mapping with Graph Explorer

Before writing enterprise-scale automation, you need to understand how to trace a user's footprint manually using Graph Explorer. Building your queries step-by-step ensures your logic is sound before writing lines of code. Here is the blueprint for mapping user touchpoints:

1. Anchor on the user: Start by fetching the core user object using GET /v1.0/users/{userPrincipalName}?$select=id,displayName,userPrincipalName.
2. Enumerate memberships: Pull the groups and roles tied to that identity using GET /v1.0/users/{id}/memberOf?$select=id,displayName.
3. Pivot to resources those groups back: For M365 Groups and Teams, target the underlying SharePoint site using GET /v1.0/groups/{groupId}/drive/root/children?$select=id,name,lastModifiedDateTime,webUrl, or enumerate site drives directly via GET /v1.0/sites/{siteId}/drives.
4. Pull item permissions and links: Inspect individual file access with GET /v1.0/drives/{driveId}/items/{itemId}/permissions?$select=id,grantedToV2,link,roles,shareId.
5. Trace who accessed what: Correlate user actions using sign-in logs or security alerts paired with file and site context.
6. Close the loop to Teams posts: Search channel messages for specific file references using GET /v1.0/teams/{teamId}/channels/{channelId}/messages?$search="filename" (keeping beta permissions and requirements in mind).

Noise-Killing Filters You'll Actually Use

Querying the Microsoft Graph without filters is a fast track to overwhelming your system with data firehoses. To surface real risk, you need precise filtering strategies. Here are the query patterns you should deploy:

• External shares in the last 7 days: Target anonymous or organization-wide links using GET /v1.0/drives/{driveId}/items/{itemId}/permissions?$filter=link/scope eq 'anonymous' or link/scope eq 'organization'.
• Only risky file types: Append filters like $filter=endsWith(resourceReference/name,'.xlsm') or endsWith(resourceReference/name,'.pst') to catch macros and archives.
• After-hours mentions of sensitive terms in Teams: Search channel messages for terms like confidential, then post-filter results by creation timestamps outside normal business hours.
• Keep payloads lean: Always include $select parameters to return only necessary identifiers, names, URLs, and timestamps.
• Cap for triage: Use $top=50 for quick spot-checks, but remove this parameter when running full exports to your data store.

Enterprise-Scale Musts: Pagination and Throttling

When running audits across a tenant with hundreds of thousands of objects, your scripts will hit walls if they do not account for API limits and pagination. You are never done pulling data until you follow the @odata.nextLink property for every single page returned by the API. Ignoring this means your scripts will capture only the first batch of results, completely missing the rest of your tenant.

Furthermore, you must respect HTTP 429 rate limit responses. Implement robust exponential backoff logic so your script pauses, waits out the cooling period, and seamlessly resumes where it left off. Batch your requests wisely by favoring fewer, larger logical requests and caching group-to-site mappings. For massive tenants, partition your crawls by site or date range and schedule them during off-peak hours to avoid impacting production performance.

Common Pitfalls and Fast Fixes

Even seasoned administrators run into roadblocks when scaling M365 audits. Here are the most common pitfalls and how to fix them:

• Pitfall: CSV exports drift as schemas change over time. Fix: Use the Graph API directly with versioned transforms and handle unknown fields gracefully.
• Pitfall: Mixed identifiers like GUIDs versus User Principal Names break database joins. Fix: Normalize everything into a clean semantic model encompassing Users, Groups, Sites, Drives, Items, and Permissions.
• Pitfall: Page 1 bias hides the majority of security incidents buried deeper in result sets. Fix: Always log total pages processed and configure alerts if expected data volume drops suddenly.
• Pitfall: Floods of irrelevant files bog down processing engines. Fix: Apply strict filters for sharing scopes, time windows, and sensitive term lists.
• Pitfall: Teams shares are disconnected from underlying files. Fix: Correlate channel message web URLs directly to drive items and store both identifiers together.

Executive-Level KPIs This Unlocks

Technical data is only valuable if you can translate it into actionable business intelligence for leadership. Building out an automated Graph auditing pipeline unlocks powerful executive KPIs:

• External exposure velocity: Measure new guest links created per week broken down by site and team.
• Sensitive file touch rate: Track unique viewers on labeled content, separating internal users from external guests.
• Group ripple risk: Measure files newly accessible within 24 hours of major group membership changes.
• Shadow circulation: Monitor sensitive files appearing across multiple teams or sites within a single sprint.
• Cleanup effectiveness: Track revoked link counts and time-to-remediation metrics over time.

Quick-Start Query Recipe

If you are ready to start building your own collection pipeline, use this copy-and-paste flow to get moving:

1. Pull user core details and memberships: /users/{upn}?$select=id,displayName and /users/{id}/memberOf?$select=id,displayName.
2. For each M365 group, enumerate drive items: /groups/{groupId}/drive/root/children?$select=id,name,webUrl,lastModifiedDateTime&$top=200, making sure to follow @odata.nextLink.
3. For each item, pull permissions: /drives/{driveId}/items/{itemId}/permissions?$select=id,roles,grantedToV2,link, filtering for anonymous or organization scopes.
4. Export rows to your designated data store, capturing tenant time, site IDs, drive IDs, item IDs, web URLs, owners, scopes, and expiration details.
5. Visualize your normalized data in Power BI for executive and operational dashboards.

Turning This Into a Living Dashboard

An audit run once is a snapshot; an audit run continuously is a security program. To turn raw Graph calls into a living dashboard, feed your data into Power Dataflows to land normalized tables for Users, Groups, Sites, Drives, Items, Permissions, and Messages. Model your data using a star schema with Items at the center, establishing clean relationships to Users and Groups.

Configure incremental refreshes based on lastModifiedDateTime fields, running intra-day updates for high-risk sites. Finally, set up automated alerts to trigger whenever new external links appear on labeled content or when group-ripple spikes indicate unauthorized access expansion.

30-Day Rollout Plan

Rome wasn't built in a day, and an enterprise audit framework takes careful staging. Follow this 30-day rollout plan to ensure a smooth deployment:

Week 1: Set up your Azure app registration, assign least-privileged Graph API permissions, and baseline your first queries using $select and $filter parameters.

Week 2: Develop your pagination scripts, normalize identifiers into consistent UPNs and device names, and run your first successful end-to-end data export.

Week 3: Build your Power BI data model and configure key performance indicators like external exposure, ripple risk, and dwell-to-revoke timelines.

Week 4: Automate data refreshes, implement anomaly alerts, and document standard operating procedures for your security investigations team.

Who Should Watch

This framework is designed specifically for M365 administrators and SecOps leads who suspect their green dashboards hide gray areas, compliance teams needing end-to-end share visibility without third-party tool sprawl, and collaboration owners accountable for maintaining safe external sharing practices.

Field Tips

As you deploy these scripts in the real world, keep these quick tips in mind: Always start your investigations from either the user or the file, then pivot in both directions to eliminate blind spots. Treat any add-to-group event as a high-risk change and review downstream access within 24 hours. Keep a living shortlist of sensitive terms to search across file names and Teams messages. Finally, log your page counts and result volumes carefully—silence from a script is often a sign of a broken job rather than a clean tenant.

Conclusion

Mastering pagination, respecting rate limits, and avoiding Page 1 bias are essential skills for anyone tasked with securing a massive Microsoft 365 tenant. By moving beyond basic administrative consoles and leveraging the full programmatic power of the Microsoft Graph API, you can transform invisible sharing risks into clear, measurable security insights.

For deeper dives, code samples, and real-world Graph recipes, make sure to listen to our complete discussion over on the podcast. Check out the related episode and catch all the details at Map Microsoft 365 Users, Groups, Files, and Shares with Graph. Stay secure, keep automating, and we will see you in the next episode!