Ran into this issue while I was on vacation. Dawn loved that while I'm in paradise, I break out my laptop to diagnose the issue. That took some serious explanation.
Anyways, I have an internet site deployed on SharePoint For Internet Sites (FIS). Although the site is setup for anonymous users we have a secured document library using a custom claims provider. The unfortunate thing I've found is that Office docs are not claims aware. What this means is that a user could login to the site using the custom claims provider, navigate to the doc library, click on a document, and they're hit with another authentication prompt:
But they're already logged in. What gives!
One thing to keep in mind with the double auth is that the user will only be required to do this once as it appears a cookie is cached in Office, but the particular site in question users typically only open one document at a time and then go on their merry way. So the double authentication is not gonna fly.
Workaround: use _layouts/download.aspx
I created a separate links list and created links to each file. The syntax you should use is as follows:
http://site.com/_layouts/download.aspx?SourceURL=[library name]/[file name.extension]
Using the download.aspx convention causes SharePoint to use a different web service to deliver the file to the user. It adds a few steps for the admins but I'd rather make my life harder than make the site user's life hard. If anyone knows a different way on how to solve this problem I'm all ears.