No file may disappear silently: what indexing 400,000 SharePoint documents taught us about the Graph delta link

"No changes" can be the most dangerous answer an AI index gives. In a large SharePoint environment, documents can drop out of the index without any error anyone sees. In our case the fix was a rule about when not to move the Microsoft Graph delta link forward.

The setting

We run a retrieval index over a production SharePoint environment belonging to a construction group. It covers 865 sites, about 400,000 documents and four languages. Indexing everything again each day is not realistic at that size, so the indexer keeps itself current with the Microsoft Graph driveItem delta API. Each document library has one saved deltaLink, and each cycle asks only for what has changed since that link.

Microsoft documents this pattern clearly. You call delta, follow @odata.nextLink until it stops appearing, and then receive an @odata.deltaLink, which is “used to read the next set of changes in the future”.^[1] Microsoft’s guidance on scanning at scale names search indexers as a target scenario and gives one instruction: always keep the URL returned in @odata.deltaLink so you can check for changes later.^[2]

The detail that matters is easy to miss. The delta API page says that once you have received all the changes, you may apply them to your local state.^[1] The deltaLink describes what Graph has delivered. It does not say whether you managed to store any of it.

What went wrong

Between 24 August and 2 September 2026, about 30 documents and about 20 deletions never reached the index. Nothing reported it. The vector index was a SQLite-based database stored on an Azure Files SMB share, and writes to it started failing with “database is locked”. The delta job logged each failed file, skipped it and carried on. When the library was done, it saved the new deltaLink anyway.

Graph then never returned those changes again. As far as Graph knew, they had been delivered. The delta feed shows the latest state of each item, not a history of changes.^[1] An item that has been delivered once comes back only when it changes again. A document that nobody edits after that point stays missing until someone notices.

Nobody noticed it as missing documents, either. What we saw was slow cycles, taking between 20 and 87 minutes. One cycle reached the job’s replica timeout of 5,400 seconds and was killed. In Azure Container Apps jobs, replicaTimeout is the maximum time in seconds to wait for a replica to complete.^[7] The alarm said “cycle crashed”. That was true, and it pointed us at a symptom rather than the cause.

The storage cause

The first fix, on 2 September, was in the storage layer. Microsoft’s recommended mountOptions for Azure Files SMB shares include nobrl, described as disabling byte range lock requests to the server, for applications that have trouble with POSIX locks.^[4] Azure Container Apps takes mountOptions on an Azure Files volume as a comma-separated string and links to that guidance.^[5] We added the option.

We should be honest about the wider picture. Azure App Service documentation says Microsoft does not recommend storage mounts for local databases such as SQLite, or for anything else that relies on file handles and locks.^[6] The mount option removed the immediate failure. It does not make a network share a good home for a database, and the indexer must not assume its writes always succeed.

The rule: hold the cursor when a write fails

The fix that matters more was made the same day, in the indexer. If any file in a library fails for an infrastructure reason, that library’s deltaLink is not moved forward. The next cycle starts from the old link, and Graph returns the same changes again. Processing has to be safe to repeat for this to work. The delta documentation already says the same item may appear more than once in a feed and that you should use the last occurrence,^[1] so an indexer that follows the documentation is already built for this.

Getting the lost documents back took separate work. Delta would never return them, so we found the roughly 30 documents by file name through Microsoft Search and indexed them directly. If you advance a cursor too early, you have to go and find what you skipped by hand.

Not every failure should hold the cursor

The rule had a weakness of its own. If one file can never be processed, holding the deltaLink forever stalls the whole library. On 11 September we started classifying failures.

• Broken files, whose content cannot be read, go to a visible list with the reason attached, and the library moves on. They are retried when SharePoint changes them. They no longer disappear quietly, and they no longer block the files around them.
• Temporary failures, meaning HTTP 5xx, 429 and network errors, are retried in later cycles, up to 12 cycles by default. This fits Microsoft’s throttling guidance. Graph returns 429 Too Many Requests with a Retry-After header,^[3] and the SharePoint scanning guidance treats 429 and 503 alike: wait for the Retry-After duration and pause further requests while you are throttled.^[2]
• Everything else counts as infrastructure and holds the deltaLink.

The default sits on the safe side. A failure we have not classified does not get to advance the cursor.

The silent zero

The third change, on 21 September, dealt with a quieter version of the same problem. Sometimes no site answers at all. The certificate the app authenticates with has expired, or its Sites.Selected access has been removed. Sites.Selected needs an explicit grant on each resource, and Microsoft documents two ways an administrator can take that access away: delete the permission on the resource, or revoke the scope consent in Entra ID.^[8] Either way the application loses access.

Before the change, a cycle like that ended as “0 files, 0 deleted”, which looked exactly like a quiet day with no changes. Meanwhile the answering service kept answering confidently from an index that was getting older. Now a cycle in which no site answered raises an alarm. “Nothing changed” and “nothing answered” produce the same counts, but they mean very different things for anyone relying on the answers.

The general lesson

In incremental sync, “the cursor moved” and “the work was done” have to be the same event. Move the cursor only after the write has succeeded. Treat a failure you cannot classify as a reason to stay where you are. Make the difference between “nothing changed” and “nothing answered” explicit, both in code and in alerts.

For administrators evaluating a search or RAG solution over SharePoint, these points turn into questions worth asking any implementer:

• When a document fails to index, where does it show up, and who can see it?
• Is the delta position saved before or after the index write is confirmed?
• What does the system report when an expired credential or a removed site permission makes every site return nothing?
• Does an alert fire on the cause, or only on a crash that happens hours later?

None of this is exotic. Microsoft describes the delta semantics, the throttling behaviour and the storage caveats. The failure happens in the space between them, where a log line says “skipped” and the cursor moves on regardless.

How this applies to our work

Holding the delta link after an infrastructure failure, keeping broken files visible, and alarming on a silent zero are now part of how Answergrove, developed by SIA KSJ, indexes SharePoint. More about the approach is at ksj.lv/en/answergrove.

References

1. ^ a b c d “driveItem: delta – Microsoft Graph v1.0”. Microsoft Learn. Retrieved 18 October 2026.
2. ^ a b “Best practices for discovering files and detecting changes at scale – OneDrive dev center”. Microsoft Learn. Retrieved 18 October 2026.
3. ^ a “Microsoft Graph throttling guidance – Microsoft Graph”. Microsoft Learn. Retrieved 18 October 2026.
4. ^ a “Recommended and useful mountOptions settings on Azure Files – Azure”. Microsoft Learn. Retrieved 18 October 2026.
5. ^ a “Use storage mounts in Azure Container Apps”. Microsoft Learn. Retrieved 18 October 2026.
6. ^ a “Mount Azure Storage as a Local Share – Azure App Service”. Microsoft Learn. Retrieved 18 October 2026.
7. ^ a “Jobs in Azure Container Apps”. Microsoft Learn. Retrieved 18 October 2026.
8. ^ a “Overview of Selected Permissions in OneDrive and SharePoint – Microsoft Graph”. Microsoft Learn. Retrieved 18 October 2026.