So it has been a while since I’ve blogged, I know. I’ve got no real reason, but I can make up a couple of good excuses: writer’s block, no time, no good ideas, or maybe the dog ate my homework. The truth is, I just have been so busy that I haven’t made time in my down time to do it. My apologies, and I’ll try to do better this year.
This is a good one, though. It certainly had me banging my head against my desk for a while. I was recently on site with a client and one of the things we decided to do while I was there was apply SharePoint Server 2010 Service Pack 1 to their system. When I originally built it, it was prior to the release. So I naturally went out to Microsoft.com and downloaded the service packs. Since there was nothing in the environment yet (we are planning the migration now) and no one was using it, we decided it would be fine to go ahead and work on this during business hours. It’s a good thing, too!
So, like any good administrator, I made sure that I took a quick backup and then installed the bits. That was easy enough, and had no real issues with that. Now comes the fun part. Running the SharePoint Products Configuration Wizard… The nice thing about 2010, is you can kick it off on all of the servers (there were 4) at once and each will just wait for its turn to run. So kicked this off and then went about some other business. Later, when I came back to check on the progress, it stated there were some errors, but acted like it had finished.
Puzzled, I went to look at the PSCDiagnostics logs and found something very non-specific:
01/12/2012 22:41:42 14 INF SyncUpgradeTimerJob: Upgrade timer job failed. Return -1. 01/12/2012 22:41:42 14 ERR The exclusive inplace upgrader timer job failed.
You know exactly what that means, right? Yeah, me neither. So I went into Central Administration to see what I could find. What I found puzzled me completely.
I went to the Upgrade and Migration section of Central Administration to check the status. Clicking on “Check product and patch installation status” showed that the farm and server status were all patched up to Service Pack 1 (14.0.6029.1000). Hmmm ok. So I went back and clicked on “Review Database Status”. Oh my, that doesn’t look right! All of the Content Databases, the Central Admin database (SharePoint_AdminContent), the configuration database, and about half of the databases for the service applications showed “No action required” and had upgraded appropriately. The others, well, let’s just say they weren’t so happy!
Some of those databases stated that “Database is too old and upgrade is required” while the others stated “Database is in compatibility range and upgrade is recommended.” Um ok, they were all installed at the same time, so they should be at the same level, but they don’t show the same. Weird, right? So I do a little research and find some other commands to try to force the databases to upgrade. I tried the following command:
PSConfig.exe -cmd upgrade -inplace b2b -force -cmd applicationcontent -install -cmd installfeatures
It failed on step 5 of 6 each time… now I’m really getting frustrated. Each time, I get the same error in the logs, no further details. Finally, I’m so frustrated that there MUST be something else going on. Thankfully, the PSCDiagnostics log shows the time of the failure down to the hundredth of a second. Since I’m running all of this on a single server at this point, I can know which server is doing the work. So, I crack open the ULS logs and start looking at the times to find the EXACT time that is shown in the PSCDiagnostics log. What do I find? Well at that exact time, it is just telling me that the psconfigui.exe job is ending. Okay, so that isn’t super helpful at that hundredth of a second, so I back up one… and lo and behold if my world isn’t shaken!
01/12/2012 22:41:41.16 OWSTIMER.EXE (0x09F4) 0x0CC0 SharePoint Foundation Timer 6398 Critical The Execute method of job definition Microsoft.SharePoint.Administration.SPUpgradeJobDefinition (ID 8762c5ac-ffd3-4326-8d12-e9079efa384a) threw an exception. More information is included below. The operation cannot be performed on database <Database Name> because it is involved in a database mirroring session. ALTER DATABASE statement failed.
OH MY GOODNESS! So when we built this farm, we built it on temporary SQL hardware because the new, more robust hardware was not in yet. It is now in, and in order to prepare for the move, mirroring had been set up on all of the databases to the new server (cluster). We had also set up an alias to make this move as easy as possible. But I was in shock, mirroring is a very common DR practice, why would that affect the ability to upgrade SOME of the databases? I stress some, because some databases would upgrade and some would not. So I contacted the DBA and asked them to pause the mirroring for the upgrade. After the pause, I had the same results, it still would not upgrade. So I ended up having them completely remove the mirror from the databases, once I did this, it upgraded all databases completely.
Now, what is the point of this blog, you may ask? I am hoping for two things:
1. If someone else has this problem, they will be able to find the issue much easier and faster than I (I looked on EVERY forum and blog I could find, and no search engine even indicated this as an issue).
And 2. If there’s anyone who may have had this issue and found it or worked with someone to find out why mirroring isn’t working, or if there is a fix to make updates work with mirroring to find out that information!
I really hope there is some way to make this work, as mirroring is pretty useful as a Disaster Recovery strategy and it is not something you can turn on and off with the flip of a switch, it is a little more involved than that. I just would like to know if there is some accurate guidance somewhere that will help with this in the future. In our case, since it was to prepare for a move, we just turned it off to complete the upgrade and then turned it on again. Many thanks to Todd Klindt and Sean McDonough for listening to me rant and assuring me that it was unusual and I am not crazy (boy did they get that part wrong, huh?)!
This post is cross-posted from here.