Dell AppAssure clean core shutdown procedure...

While working at my last job we began using a new product for our backups and replication called AppAssure. Currently owned by Dell, AppAssure is a utility along the same lines as EMCs Avamar or several other vendors backup products that promise you the world but fall exceedingly short.

Our main problem lied with several of our roll up jobs not completing. While this in and of itself would not have been an issue, because the rollups were performed every night, it became increasingly problematic when the rollup jobs stopped working for roughly a year on several of our VM instances.  Even with de-duplication we sat at several TeraBytes worth up backups for roughly 50 Virtual Machines. This caused AppAssure to hang on certain processes, one of which was rollups, and on occasion stop the GUI from responding. One sure fire way of bringing the product back in order, however, was to fire up Powershell and give the core processes a much needed reboot. As for how we fixed the rollup jobs, thats a story for a whole other post: 

First we tried to manually stop all of our process through the AppAssure GUI which, as hinted at earlier, was a completed bust. After I gave up on this in about 2 minutes time I pulled up Powershell in administrator mode and got to work 

In powershell window (administrator mode), this can be accessed by right clicking on powershell and clicking run as administrator: 

NOTE: that all commands must be run on the core

• First import our powershell module that we will need to do the work 
• Import-Module AppAssurePowerShellModule
• Pause our protection on all machines
• Suspend-Snapshot –All
• Pause our replication from one core to the other (if multiple cores are at work)
• From current core to remote core: 
• Suspend-Replication -Outgoing –All
• From remote core to current core: 
• Suspend-replication -Incoming –All
• If you are using virtual standby for any of your machines stop this process as well. 
• Suspend-Vmexport –All
• Now we check to verify all of our current jobs have stopped running.
• Get-ActiveJobs –All
• If active jobs are running let them complete or try to cancel them. You can cancel any job except Rollups or Repository Checks in the Web UI. This was always funny to me because guess what one of our leading points of contention was...thats right rollups not completing. BUT DONT YOU DARE STOP THEM! 
• Once you have verified all the processes are completed by using Get-ActiveJobs -All You can now stop the core process. 
• First disable the core process. This isn't vital but can be very helpful if you restart the server and need to do work on AppAssure before the core begins to repost itself. 
• Set-Service AppAssureCore -StartupType Disabled
• Stop the Core Service 
• Stop-Service AppAssureCore
• Check the status of the core service
• Get-Service AppAssureCore
• Now this can take some time so go out have a coffee pull up netflix and watch a couple mind numbing episodes of your favorite show. Pull up youtube and tell your co workers your trying to verify bandwidth throughput. Just don't be thinking this is going to go quickly. If all else fails you can kill the process just know that you will have to sit through the mind numbing awesomeness of the repository rebuilding itself when you pull everything back up, which can take FOOOOOOOORRRREEEEEEEVVVVEEER
• Kill the process with fire 
• Stop-Process -Processname Core.Service -Force
• Check the state of the process again just to make sure its dead 
• Get-Process Core.Service
• Now you can go about your merry way. I generally ran a restart on the server at this point and any updates that it required. It really doesn't matter what you do since my problem most often was resolved by killing the Core Service in the first place. 
• Now to restart everything. First re-enable our core service with a delay. 
• Sc.exe Config AppAssureCore Start= Delayed-Auto
• Then start our core service. 
• Start-Service AppAssureCore
• From here it's basically just the opposite of what we have already done. 
• Resume our protection 
• Resume-Snapshot –All
• Resume our replications
• From current core to remote core: 
• Resume-Replication -Outgoing –All
• From remote core to current core:
•  Resume-Replication -Incoming –All
• Resume Virtual Standby:
• Resume-Vmexport –All

From here you can go back to the AppAssure GUI and verify the the repository recovery processes are being run. This part can take awhile as all the gremlins are putting all the information back into a usable state and gremlins as we know work at their own pace and take union mandated smoke breaks.  After this has been completed the process that was seemingly in a never ending loop should be dead as a doornail and things should be running again as the good lord Michael Dell had intended.