Premise

I had this code, https://realmpksharepoint.wordpress.com/2014/04/07/get-webpart-page-of-a-sharepoint-site-using-web-services-c/ which I was using to download the html content of a SharePoint page along with all the web parts. The importance of this method is that it eliminates the external file references, so in the response, you will get the schema of all the web parts of that page only.

Issue

The code was working fine until recently, when it suddenly stopped working and I started to receive the error, 403 Forbidden.

Solution-1

After a lot of digging, I figured out that I had to change, UserAgent of my HttpWebRequest. So I changed the UserAgent from,

FrontPage

to

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)

The good thing was that the 403 error was gone. The bad thing, however, was that it generated another error!

Server was unable to process request. —> Attempted to perform an unauthorized operation.

 

Solution-2

So, the previous solution worked, but not entirely. Had to figure out a different way then. What I did next was, that then I downloaded a file from SharePoint Designer and simultaneously, I also fired the Fiddler to track the request and its format. Guess what, it showed a major deviation in the format of the Cookie, that was being sent to the server. Take a look at the following screenshots.

• The following format was being sent by my code,
  

• The following format was sent by SharePoint Designer to successfully download a file
  

Once it was evident what the problem was, I changed the cookie format in my code to match the format sent by SharePoint Designer and I got my result! The code again started working.

The new cookie format now looks like the following:

 
req.CookieContainer.Add(
	new Cookie("SPOIDCRL",
		authCookieValue.TrimStart("SPOIDCRL=".ToCharArray()),
		String.Empty,
		targetSite.Authority));

Here’s the complete code snippet,

 
string webPageInfo = String.Empty; 
string webServiceUrl = ctx.Web.Url + "/_vti_bin/WebPartPages.asmx";
 
//say, we're trying to get the Home.aspx item of the List, SitePages. 
string documentName = String.Concat("SitePages/", listItem.FieldValuesAsText.FieldValues["FileLeafRef"]);
 
StringBuilder sbEnvelope = new StringBuilder();
sbEnvelope.Append("<?xml version=\"1.0\" encoding=\"utf-8\"?>");
sbEnvelope.Append("<soap:Envelope xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\">");
sbEnvelope.Append(String.Format(
    "<soap:Body>" +
        "<GetWebPartPage xmlns=\"http://microsoft.com/sharepoint/webpartpages\">" +
            "<documentName>{0}</documentName>" +
        "</GetWebPartPage>" +
    "</soap:Body>"
    , WebUtility.HtmlEncode(documentName)));
sbEnvelope.Append("</soap:Envelope>");
 
 
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(webServiceUrl);
req.Method = "POST";
req.ContentType = "text/xml; charset=\"utf-8\"";
req.Accept = "text/xml";
req.Headers.Add("SOAPAction", "\"http://microsoft.com/sharepoint/webpartpages/GetWebPartPage\"");
req.UserAgent = "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)";
req.UseDefaultCredentials = false;
 
Uri targetSite = new Uri(ctx.Web.Url);
SharePointOnlineCredentials spCredentials = (SharePointOnlineCredentials)ctx.Credentials;
 
string authCookieValue = spCredentials.GetAuthenticationCookie(targetSite);
req.CookieContainer = new CookieContainer();
req.CookieContainer.Add(
	new Cookie("SPOIDCRL",
		authCookieValue.TrimStart("SPOIDCRL=".ToCharArray()),
		String.Empty,
		targetSite.Authority));
 
using (Stream stream = req.GetRequestStream())
{
    using (StreamWriter writer = new StreamWriter(stream))
    {
        writer.Write(sbEnvelope.ToString());
    }
}
 
WebResponse response = req.GetResponse();
Stream responseStream = response.GetResponseStream();
 
XmlDocument xDoc = new XmlDocument();
xDoc.Load(responseStream);
 
if (xDoc.DocumentElement != null && xDoc.DocumentElement.InnerText.Length > 0)
{
    webPageInfo = xDoc.DocumentElement.InnerText;
 
    //webPageInfo = webPageInfo.Substring(webPageInfo.IndexOf(""));
 
    //The above commented subString code was used further 
    //to implement the logic of parsing. Since we're not 
    //concerned with that hence it's not included here.
}

Must say one thing here that, Fiddler is a real life saver. 🙂

Note: Please don’t get confuse with the second screenshot. The second image is also of the request sent from my code to fetch the web parts hence, the method is same in both the requests, WebPartPages! I just wanted to highlight the correct format.