DEV Community

Why Your EWS Impersonation Suddenly Stopped Working (And It's Probably Not Throttling)

Two months ago I picked up a ticket that looked routine: a job that reads mailbox data from Microsoft 365 through EWS, running fine for over a year, started failing on a subset of mailboxes in one tenant. Same app registration, same code path, same service account.

The error in the logs pointed at throttling, so that's where the admin had already spent three days looking. Wrong direction. The actual cause had nothing to do with throttling budgets. This mix-up happens constantly right now, and it's worth understanding why, because the fix for one problem does nothing for the other, and chasing the wrong one wastes days.

TL;DR: if your EWS failures don't scale with request volume, stop tuning throttling and go check your Application Access Policy scope groups instead.

What EWS throttling actually looks like

Exchange Online throttles EWS the same way it always has: budget-based. Every account gets a policy (the default is EwsDefaultThrottlingPolicy, but plenty of tenants layer custom ones on top) that tracks a slowly-refilling budget rather than a simple call count. When you overspend it, you get back a 503 or 429 with an X-MS-Diagnostics header telling you which budget got exhausted, usually the connection count or the concurrent-request limit.

The tell for real throttling is consistency. It scales with load, correlates with concurrency and batch size, and clears up within minutes once you back off. If you graph failure rate against request volume, you'll see a clean relationship. If you double your batch size, failures increase. If you throttle yourself proactively (respecting Retry-After, staying under EWSFindCountLimit for FindItem calls), it mostly goes away.

That correlation is the whole diagnostic test. If your failures don't scale with volume, you're not looking at a throttling problem, no matter what the error message on the surface says.

What actually changed

Over the past year or so, Microsoft tightened enforcement in two places that both produce errors easy to mistake for throttling.

First, Application Access Policies. New-ApplicationAccessPolicy has existed for years, letting admins scope which mailboxes an app registration can touch instead of granting it impersonation rights over the entire tenant. It used to be optional, something only security-conscious tenants bothered configuring. That's changed. More tenants are applying scoped access policies as a default hardening step, often as part of a broader Conditional Access or Secure Score push, and they're doing it without touching the app's actual permissions or the service account's RBAC role. The app still has ApplicationImpersonation. It just can't reach mailboxes outside its assigned scope group anymore, and mailbox group membership drifts constantly as people get added, removed, or reassigned.

Second, the long tail of Basic Auth deprecation. Most environments moved off Basic Auth for EWS years ago, but I still see tenants with a leftover service principal or a scheduled task somewhere using a cached token flow that quietly stopped working when Microsoft closed another enforcement gap. It's rarely the primary auth path failing outright. It's usually one forgotten integration that everyone assumed was already migrated.

Both of these produce intermittent, mailbox-specific failures. Neither has anything to do with request volume.

Telling them apart

Here's the sequence I actually use now before touching any throttling knobs.

1. Check whether failures correlate with specific mailboxes, not specific times.

Pull the failing mailbox list and check it against the policy's scope group:

Get-ApplicationAccessPolicy | Format-List AppId, PolicyScopeGroupId, AccessRight
Get-DistributionGroupMember -Identity "EWS-Scoped-Mailboxes" | Select-Object PrimarySmtpAddress

Compare the mailbox list this returns against the mailboxes that are actually failing. Anything missing from the group is out of scope, and that's your answer.

2. Read the actual error code, not just the HTTP status.

These two failures look almost identical if your logging only captures the status code:

# Real throttling
HTTP/1.1 503 Service Unavailable
X-MS-Diagnostics: 3006;reason="EWSMaxConcurrency budget exhausted";error_category="transient"

# Access policy scoping
HTTP/1.1 403 Forbidden
<S:Fault>
  <S:Code><S:Value>Sender</S:Value></S:Code>
  <S:Reason><S:Text>ErrorAccessDenied</S:Text></S:Reason>
</S:Fault>

If your logging layer only captures the HTTP status and not the response body, fix that first. You're throwing away the one piece of information that tells these two apart.

3. Check the Microsoft 365 audit log around the date the failures started, not the date they were noticed:

Search-UnifiedAuditLog -StartDate (Get-Date).AddDays(-30) -EndDate (Get-Date) `
  -Operations "New-ApplicationAccessPolicy","Set-ApplicationAccessPolicy","Remove-ApplicationAccessPolicy" |
  Select-Object CreationDate, UserIds, Operations

I've found the actual change was made three weeks before anyone noticed, during a routine security review nobody connected to this app.

4. Confirm the app's permission model.

Get-ManagementRoleAssignment -RoleAssignee "svc-backup-app" |
  Where-Object { $_.Role -eq "ApplicationImpersonation" }

If this returns a result, the app can impersonate any mailbox in the org by design, unless an access policy is actively narrowing that down. Worth knowing before you assume the access policy is even the right suspect.

If none of that turns anything up and failures genuinely track with request volume and clear up after backing off, then fine, it's real throttling, go tune your batching.

Fixing the actual problem

If it's an access policy issue, the fix is usually mechanical but tedious: audit the scope group, figure out why the mailbox fell out of it, and set up something that doesn't rely on someone remembering to update a static group every time a mailbox changes hands.

A dynamic distribution group tied to whatever attribute actually determines "should this app reach this mailbox" beats a manually maintained list every time:

New-DynamicDistributionGroup -Name "EWS-Scoped-Mailboxes" `
  -RecipientFilter { RecipientTypeDetails -eq "UserMailbox" -and CustomAttribute1 -eq "backup-enabled" }

Point the access policy at this group instead of a static one. Membership now updates itself whenever CustomAttribute1 changes, instead of depending on someone remembering to edit a list.

The bigger fix, though, is getting off tenant-wide impersonation entirely. Microsoft has been pushing RBAC for Applications as the replacement, and it's worth the migration even outside of this specific problem. Instead of one service account impersonating anyone, you assign granular, resource-scoped permissions directly to the app registration:

New-ServicePrincipal -AppId <your-app-id> -ObjectId <object-id> -DisplayName "Backup-App"
New-ManagementScope -Name "Backup-App-Scope" `
  -RecipientRestrictionFilter { CustomAttribute1 -eq "backup-enabled" }
New-ManagementRoleAssignment -Role "Application Mail.Read" -App <your-app-id> `
  -CustomRecipientWriteScope "Backup-App-Scope"

This replaces a single tenant-wide ApplicationImpersonation grant with a role scoped to exactly the mailboxes the management scope defines. Exact cmdlet parameters shift a bit between Exchange Online PowerShell module versions, so treat this as the shape of the migration and confirm current syntax against Microsoft's docs before running it against production.

It's more setup work up front, but it means you stop being exposed to "someone tightened a policy somewhere and now half your mailboxes silently fail" as a category of incident. Given that EWS itself is on Microsoft's retirement roadmap in favor of Graph API, this is also the direction you'd be migrating toward anyway. Doing the permission model migration first, before the API migration, splits the work into two smaller, less risky changes instead of one large one.

One more thing worth doing regardless

Separate your retry logic by error type. A lot of applications treat every failed EWS call the same way and retry with exponential backoff, which is correct for a 503 throttling response and actively harmful for a 403 access-denied response:

def should_retry(status_code, error_code):
    if status_code == 503 or error_code == "ErrorServerBusy":
        return True  # transient, back off and retry
    if status_code in (401, 403) or error_code == "ErrorAccessDenied":
        return False  # a permission problem, retrying won't fix it
    return False

Retrying an access-denied error just burns your throttling budget on calls that were never going to succeed, and can end up creating the throttling problem you originally thought you had.

The pattern underneath this

None of this is exotic. It's the same lesson as always: a generic error handler will happily lump together two unrelated failure modes under one log line, and the fix for each one does nothing for the other. The throttling ticket that isn't actually about throttling has become common enough in the last year that I've started assuming access policy scope first and traffic volume second, and I've been right more often than not.

Comments

No comments yet. Start the discussion.