Rubrik Node Swap

 Had the "fun" of swapping out a failed Rubrik R6410 node, here are a few notes:

https://rubrik.docebosaas.com/customers/learn/course/405/play/1612/physical-replacement-r6xxx-live-demo?hash=3637a3a6fceb9dc03d7076cdf1c771ab2b26d60b&generated_by=58301

R6410 Rubrik nodes (120tb raw storage) twelve 10tb drives, apx 75tb useable

-If the node is still "functioning" remove the node from the cluster (logically), else it will have to be forcefully removed from the cluster

-Turn on UID

-Power off the node

-pull out the node

-To unlock the node from the chassis, the right pull tap ear, needs a slight downward movement to unlock it, then gripping both pull tabs pull.

-The encryption module (TPM) must be swapped, from the existing (defective node) to the replacement node.

-The SSD drive may also need to be swapped, verify the amount of ram and the NVMe SSD drive 

-The TPM module is circled in yellow.  To get to it the SSD drive tray must be unscrewed from the node and set off to the side.  The screws (which are different lengths) are pointed out in red.



-The TPM module pulls straight up.  Notice that that on the TPM module itself there is one hole for the pins that blocked out, this is so it only goes in one way and cannot be plugged in backwards.

-Once things have been cabled up, go to the front of the rack, pull off the vanity cover.  There are four sets of LED’s and buttons, one set for each node.  One set of those will be off, that will be for the node you just swapped.  Press the power button.

-Put the defective node back in the box, seal up, apply the shipping label that came with the package, and get it to the respective delivery service.  Please hold on to the defective node for at least one day before sending back, so I can be sure all is well with the swap.  


Random notes on doing a restore to a Physical computer from Rubrik v5.x

High level instructions for restoring a physical machine from a Rubrik CDM v5.x  I had some difficulties so here are my notes:

I took much of my instructions from here:

Rubrik instructions on how to make their boot ISO

Rubrik instructions to do the restore


Create an a WindowsPE Boot ISO using the info from the above links.  When creating the boot ISO, one will need both the ADK for Windows AND the Windows PE Add-On and also the Rubrik Recovery Tool Install kit.  When installing the ADK check all of the features.   

Boot the machine from that ISO.  Once there make note of the IP, if there is no IP address, assign one.

In Rubrik, find the server to be restored, choose a snapshot to "Mount", choose "No Host" , leave domain, usernames, and AD groups blank, I did put the IP's of a management station and the IP of the WinPE machine.

In Rubrik, go to the "Live Mounts" section, the "Windows Volumes", one should see the server we are working with mounted.  Make note of the "Restore Script Path".  What Rubrik has done is present a CIFS share that contains a PowerShell script and a VHDX (Hyper-V virtual hard drive).  Test this by browsing to the share (make note of the security limitations that were set during the presenting of the snapshot.  IE if the snapshot was limited to certain IP's, test the share from a machine having a matching IP.

On the machine to be restored, map the Z-drive to share off of the Rubrik, far as I can tell this is for authentication purposes.  Use: "administrator" and no password.  
net use z: \\<IP  of Rubrik>\<sharename> /user:<winpe_client_ip>\administrator *

On the machine to be restored, the next step is to launch the power shell script that will take the contents of the VHDX and dump it to the hard drive.  
Type: 
"powershell" then "set-executionPoloicy unrestricted" then "\\<IP of Rubrik>\<sharename>\with_layout\RubrikBMR.ps1

Random notes on Vmware Vcenter v6.5 & 7.0

  • Clearing out the VMware Update Manager before an upgrade will cut down on the time required to update, as by default the VCenter upgrade will drag all that data with.   Also during the migration one might see an error message saying it needs an export location because the root partition is only 4gb.  

https://www.stephenwagner.com/2020/07/22/vcsa-vcenter-upgrade-7-enter-new-export-directory-source-machine/

  1. Log in to your vCSA source appliance via SSH or console
  2. Run the applicable steps as defined in the VMware KB 2147284 to reset VUM (WARNING: commands are version specific). In my case on vCSA 6.5 I ran the following commands:
    1. shell
    2. service-control --stop vmware-updatemgr
    3. /usr/lib/vmware-updatemgr/bin/updatemgr-util reset-db
    4. rm -rf /storage/updatemgr/patch-store/*
    5. service-control --start vmware-updatemgr
  3. Open your web broswer and navigate to https://new-vcsa-IP:5480 and resume the migration. You will now notice a significant space reduction and won’t need to specify a new mount point
  •  Forgot the administrator@vsphere.local password?  SSH into the VCenter, start the shell, and type: /usr/lib/vmware-vmdir/bin/vdcadmintool
  • Need to adjust DNS settings, like the FQDN?  
  1. Access the VCSA from console or from Putty session.
  2. Login with root permission
  3. Use above command in the command prompt of VCSA : /opt/vmware/share/vami/vami_config_net
  4. Opt for option 3 (Hostname)
  5. Change the hostname to new name
  6. Reboot the VCSA appliance.

  • If there is any change to the naming of the vCenter, logon issues with using Active Directory may occur.  In my case one could logon using @vsphere.local, and clicking the check box to use local credentials worked.  However one could not type their credentials in, regardless of UPN format.  The fix for us what find the AD record of the vCenter server, go to the Attribute Editor, and change the "dNSHostName" entry.  FWIW in one instance the vCenter was using the internet name, which is different than the Active Directory name. 
  • When using the built in backup feature to backup the VCSA, when using FTP the destination folder must be empty.
  • https://www.altaro.com/vmware/backing-up-vcsa-6-5-natively-using-ftps/
  • https://sfitpro.blogspot.com/2016/11/configuring-vcsa-65-backup-lessons.html
  • When using the built in backup feature to backup the VCSA, all of the services need to be running, including ones for unused features.  Again from the shell "service-control --start --all"
  • When doing the in place upgrade, use the hostname of an ESXi server for the source of the existing vCenter server and the destination.  Things get wonky if one tries to deploy a vCenter Server on top of the old on.
  • DNS is super important (duh), my home lab kept having weird DNS lookup failures, one can use the IP addresses for the upgrade process.
  • If you have plug-ins that don't work after the upgrade and seems to uninstall them, look into using JXplorer; think of it as a regedit for vCenter. The plugins will be in the "ServiceRegistrations" section.