tag:blogger.com,1999:blog-14486117457856799132024-03-26T23:17:32.458-05:00Jungle-Information TechnologyRamblings of an IT Professional
(Aaron's Computer Services) Aaron JongbloedtJunglehttp://www.blogger.com/profile/06827257722793718797noreply@blogger.comBlogger168125tag:blogger.com,1999:blog-1448611745785679913.post-59170487278489702052024-03-15T18:40:00.013-05:002024-03-26T23:17:00.646-05:00ProxMox notes from a NOOB<p>Here are a few notes and things that stuck out to me while kicking the tires of ProxMox. Keep in mind these are coming a VMware Admin with very weak Linux knowledge. I will keep adding stuff as I learn.</p><p>Uses QEMU/KVM for virtualization</p><p>LXC for containers</p><p>Corosync Cluster Engine for server communications</p><p>Proxmox Cluster File System for cluster configuration</p><p> -If installing ProxMox v8 crashes on install, see if v7 works, if it does then do an in place upgrade</p><p>-Neither v7.4 or v8.1 seem to recognize Mellanox CX-3 40gbE/InfiniBand network cards</p><p>-The vCenter equivalent: it is just built in to the webUI of each host and works on a distrusted/cluster model, IE there is not appliance to install, no software keys, no dedicated IP. Imagine if the ESXi WebGUI had basic vCenter functions built in, ie joining and managing multiple host in one interface, vMotion, and replication.</p><p>-There are oddities about moving VMs back and forth from LVM to/from ZFS. A VM built on a ZFS volume cannot live migrate, cold migrate to a LVM volume; a template that lives on an LVM volume cannot be spawned to a LVM volume; IF there is a replication job attached.</p><p>-by default a network "bridge" is created. Network cards can be added/subtracted as necessary; very much like the "virtual switch for VMware/ESXi</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhLIzn9dzVkKaAHnxDjGqpTxbhEKORq82eWTslxDeFwxqZOWMbyctDsS-nfxHGY2EOXgLbNNNc8bA-FqPWXVelWtHbdRh-cGed0RUhC9XeIw9s1CVVpsEgTiNnnmWK1hp4MneoaD-Oy0yiBMz2ydHk54Z6A_Flw6sByaSE6C3PJG0aWDPLxpF2gkdZOabA" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="251" data-original-width="1087" height="74" src="https://blogger.googleusercontent.com/img/a/AVvXsEhLIzn9dzVkKaAHnxDjGqpTxbhEKORq82eWTslxDeFwxqZOWMbyctDsS-nfxHGY2EOXgLbNNNc8bA-FqPWXVelWtHbdRh-cGed0RUhC9XeIw9s1CVVpsEgTiNnnmWK1hp4MneoaD-Oy0yiBMz2ydHk54Z6A_Flw6sByaSE6C3PJG0aWDPLxpF2gkdZOabA=w320-h74" width="320" /></a></div><br />-The default install, will have several "nag screens" about wanting one to have a paid subscription. No judgment here: "gotta pay the bills". The default update repository is the from the paid tier, one must disable it and point updates to the "no-subscription" tier, to get updates and lessen nags. <p></p><p>-The ProxMox virtual machine tools, actually QEMU (vmtools equivalent) is a separate download. It must be installed for any "thin provisioning of VM memory. IE a Windows VM running w/o the tools set to 8gb of RAM will consume 8gb of RAM on the host. With the tools it will take some amount less. </p><p>-That same ISO (proxmox virtio-win-xxxxxx.iso) will most likely be needed for installing Windows. Things like the hard disk controller (depending which one was chosen on VM creation) will not be seen and require drivers to be installed. </p><p>-Replication jobs! If they fail and one wants to delete the job, and they seem to not go away through the GUI. Go to the shell and type: "pvesr lit" to show the jobs. Then "pvesr delete <i>JobID </i>--force"</p><p>-A template cannot migrate to a different host unless the name of the storage on both servers is the same. </p><p>-A template cannot be cloned (image spawned from) to a host that doesn't have the same storage name. If the template exists on local storage it cannot be cloned to another host, and the dropdown box for what storage to use is blank. One has to clone the machine on the same server where the template lives, then migrate the cloned VM.</p><p>-configuration files for the VMs live here: /etc/pve/qemu-server; if things go way wrong one can move the config file to another host by: </p><p>mv /etc/pve/nodes/<old-node>/qemu-server/<vmid>.conf /etc/pve/nodes/<new-node>/qemu-server</p><p>-virtual disks can be accessed here: /dev/<disk name></p><p>Rename VM LVM Storage Name VIA SSH</p><p>cd /dev/pve</p><p>lvrename /dev/pve/vm-100-disk-0 /dev/pve/vm-297-disk-0</p><p>-There does not seem to be an easy way via the GUI to rename storage.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEim2Dxq8ZkDeNFoMOylV25581AxYK1mu4Vm97C-CrUs3swjDckAOFN-1xADAUWCDeouv1hYJfM-kLXezTeIDpsV7WjShamUaLyKNoQt_SwGE4aHlkbca_cw9JzC5IztfdozamujnWURtLl4OgSQnPq-NcDGX_UAikB9TZNytMktBg60X740i5G7qJevSXw" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="143" data-original-width="105" height="200" src="https://blogger.googleusercontent.com/img/a/AVvXsEim2Dxq8ZkDeNFoMOylV25581AxYK1mu4Vm97C-CrUs3swjDckAOFN-1xADAUWCDeouv1hYJfM-kLXezTeIDpsV7WjShamUaLyKNoQt_SwGE4aHlkbca_cw9JzC5IztfdozamujnWURtLl4OgSQnPq-NcDGX_UAikB9TZNytMktBg60X740i5G7qJevSXw=w147-h200" width="147" /></a></div><span><div style="text-align: center;"> the 1st icon is running VM, the 2nd a VM that is being migrated, the third is a template. </div></span><p></p><p style="text-align: center;"><br /></p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjoYn08fW5OQELnFUHgWcqQealcGFFDw4CdqCwwcX3UyEa9gN_Rz5lbYAn-wWYnWytkYVFAkUy8LiGvnZsEgoQOlZUdlzpZ2uX5zJTmBiy7kh8vD94oWTJ6-30SBimZiqUUIIHkbbn9KoxWMZCuCvcZABtEI8iAeWxDQynxo_O2_RBUZYLNKqyN186ou3Q" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="581" data-original-width="489" height="320" src="https://blogger.googleusercontent.com/img/a/AVvXsEjoYn08fW5OQELnFUHgWcqQealcGFFDw4CdqCwwcX3UyEa9gN_Rz5lbYAn-wWYnWytkYVFAkUy8LiGvnZsEgoQOlZUdlzpZ2uX5zJTmBiy7kh8vD94oWTJ6-30SBimZiqUUIIHkbbn9KoxWMZCuCvcZABtEI8iAeWxDQynxo_O2_RBUZYLNKqyN186ou3Q=w269-h320" width="269" /></a></div><div class="separator" style="clear: both; text-align: center;">v7.4 Cluster</div><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgo5hYNCAv77yekQPbBuaRg8-aaaGrXeMZivkAVyfFV_FTl4c0PtiYxnodG9PMf_qO5z4eB8_eXK6rvMVyfKhhhRx7RMORuRyZFIGChVbsFLR_gxqPr1S-f8zIbjquwfkpXs-hsJ5oEA7L6I0efH6wZOAhpURvLTOlAqXfxtaUS79JahuI9oP1H_Suo4qc" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="853" data-original-width="1585" height="172" src="https://blogger.googleusercontent.com/img/a/AVvXsEgo5hYNCAv77yekQPbBuaRg8-aaaGrXeMZivkAVyfFV_FTl4c0PtiYxnodG9PMf_qO5z4eB8_eXK6rvMVyfKhhhRx7RMORuRyZFIGChVbsFLR_gxqPr1S-f8zIbjquwfkpXs-hsJ5oEA7L6I0efH6wZOAhpURvLTOlAqXfxtaUS79JahuI9oP1H_Suo4qc" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;">v8.1 Cluster</div><br /><br /></div><br />Things I really like about ProxMox:<p></p><div>-ability to migrate running VM's from one host to another; even without shared storage</div><div>-ability to backup VM's</div><div>-ability to replicate VM's</div><div><br /></div><div>Challenges:</div><div><br /></div><div>-I had an unexpected host failure; the VM tired to migrate to a different node; at the time there is only local storage, not even Ceph.. The local node volumes also where not the same. After the node came back online, it had the VM's virtual drive, as it couldn't migrate, another server in the cluster had the config file. Getting things back inline was a real chore. Simply moving either the virtual hard drive or the config file was not working. Sure one could blame it on me for not setting up shared storage, or not having the datastores named the same. However why is HA not doing checks before attempting a migration?</div><div><br /></div><div>-Another unexpected host failure; two nodes are disconnected from the cluster.....Nodes 1, 2, 3, and 6 are all up and joined together and report nodes 4 and 5 as offline. Nodes 4 and 5 believe they are online, and the other four nodes are off line. Removing and re-adding the nodes to a cluster is not straight forward and not doable via the GUI.</div><div><br /></div><div><p>------------------------------------</p><p>-Abbreviated: Instructions to upgrade from v7 to v8</p><p><span> -from the shell of a given node type:</span></p><p><span><span> </span>-pve7to8</span><br /></p><p><span><span> -apt update</span><br /></span></p><p><span><span> -apt dist-upgrade</span><br /></span></p><p><span><span> -pveversion</span><br /></span></p><p><span> -</span>sed -i 's/bullseye/bookworm/g' /etc/apt/sources.list</p><p><span> -</span>apt update</p><p><span> -</span>apt dist-upgrade</p><p>------------------------------------</p><div>-To remove a damaged node from a cluster:</div><div> From the damaged node:<br /></div><div> -stop and pve cluster services: systemctl stop pve-cluster<br /></div><div> -stop corosync services: systemctl stop corosync<br /></div><div> -restart in single host mode: <span> </span>pmxcfs -l </div><div><span> </span>-delete corosync config: rm /etc/pve/corosync.conf<br /></div><div> -delete corosync folder: rm -r /etc/corosync/*<br /></div><div> -delete reference to other nodes: rm -r /etc/pve/nodes/*</div><div> ***VMs living on the damaged server will be lost.....the virtual hard drive will still be there<br /></div><div><p>------------------------------------</p><p>qm list (shows VMs)</p><p>qm start/reboot/reset/stops <i>vmID </i>(starts/safe shutdown and startup/reboots/hard powers off off a VM)</p><p><br /></p><p><br /></p></div></div>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-77391003798444894252024-03-07T10:40:00.000-06:002024-03-07T10:40:23.528-06:00SAS cable connectors<p style="text-align: center;"> Just some general information that might be useful.</p><p style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjK6GwCrnyoPwI_3QCbKXOnqUIbaLyeTQEpAQddAqO_T57dNCIWPRN8_sYC18dThfkn2HYfnEmlyP44coFCniQeiTF8dCHmOJEtyBBOKMKFCB0HTSOpfimFJxbvAUV9I-N3BPmneiqhYMvMeKT5B67DYSQj95_jQ78ThwRiG5gZ3N-ZwpD_cQUgd9UCLH0/s4000/2024-01-27%2021.36.48.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4000" data-original-width="3000" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjK6GwCrnyoPwI_3QCbKXOnqUIbaLyeTQEpAQddAqO_T57dNCIWPRN8_sYC18dThfkn2HYfnEmlyP44coFCniQeiTF8dCHmOJEtyBBOKMKFCB0HTSOpfimFJxbvAUV9I-N3BPmneiqhYMvMeKT5B67DYSQj95_jQ78ThwRiG5gZ3N-ZwpD_cQUgd9UCLH0/s320/2024-01-27%2021.36.48.jpg" width="240" /></a></p><p style="text-align: center;">This is a "mini-SAS" SFF8087</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1r_fC53BPVgHPpWRggf5MrLfT6x_rlpgQyoEGEd1jELSPPsQEvAA2wMiqEz8YUxE7pAVicjjkuBAkSAbG6aXFlU5FNka7w_iv36DILz42-m4HOX4NqsOthXKxJ5Fm8RWNmpuH8kIuN3hFHgQNoID35z0ezc-2HvKI1OCNA-8oONf3Ct43nCfan1Qu15w/s1500/71ahSGaEaaL._SL1500_.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1500" data-original-width="1500" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1r_fC53BPVgHPpWRggf5MrLfT6x_rlpgQyoEGEd1jELSPPsQEvAA2wMiqEz8YUxE7pAVicjjkuBAkSAbG6aXFlU5FNka7w_iv36DILz42-m4HOX4NqsOthXKxJ5Fm8RWNmpuH8kIuN3hFHgQNoID35z0ezc-2HvKI1OCNA-8oONf3Ct43nCfan1Qu15w/s320/71ahSGaEaaL._SL1500_.jpg" width="320" /></a></div><div style="text-align: center;">This is also a "mini-SAS", SFF-8643, typically seen in SAS 12g applications.</div><div><p style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxDuJwmyTnpdg4QE1Rk6G0F6qD5jNDSNQmqxHIw3WvA7eLYxCrmA0usND9fId2sn0rJEvJnSYPG_hdRMTRw4unqJEi6XaESabbmq2VyXRq2tIW-MVlB1yXWdPOHyWN76BXydMLxeZFztKjLJ077NwlKcjJE-Gu0Bjyol_29GSu8z9OazF05l9A28PTGRg/s4000/2024-02-05%2017.16.11.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3000" data-original-width="4000" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxDuJwmyTnpdg4QE1Rk6G0F6qD5jNDSNQmqxHIw3WvA7eLYxCrmA0usND9fId2sn0rJEvJnSYPG_hdRMTRw4unqJEi6XaESabbmq2VyXRq2tIW-MVlB1yXWdPOHyWN76BXydMLxeZFztKjLJ077NwlKcjJE-Gu0Bjyol_29GSu8z9OazF05l9A28PTGRg/s320/2024-02-05%2017.16.11.jpg" width="320" /></a></p><div class="separator" style="clear: both; text-align: center;">This is an OCuLink PCI-e SAS SFF-8611 4i; as seen on a PowerEdge r640.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5Xd4JiCF0wBkRE8fdApxHroJh4z4qkL2LdnXtvSMdIT9LvbmViDE-iwMsqEIaMzTCfaujjSXWKQOGrTOLv6IKxwhjD_RVwDhgey9mz6TzO-0aGN3sSo0mxINXSHcmWROwVUHheMz3Mw6YSlXs5eu0SV6vwTuyQGuT-aZGy7B-rF1bH0cUTilIwbYWBEQ/s4000/2024-03-04%2022.21.29.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4000" data-original-width="3000" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5Xd4JiCF0wBkRE8fdApxHroJh4z4qkL2LdnXtvSMdIT9LvbmViDE-iwMsqEIaMzTCfaujjSXWKQOGrTOLv6IKxwhjD_RVwDhgey9mz6TzO-0aGN3sSo0mxINXSHcmWROwVUHheMz3Mw6YSlXs5eu0SV6vwTuyQGuT-aZGy7B-rF1bH0cUTilIwbYWBEQ/s320/2024-03-04%2022.21.29.jpg" width="240" /></a></div><div class="separator" style="clear: both; text-align: center;">The cable on the left is a "double wide" SAS cable, used on SOME HP and Dell RAID cards. As in this case an HP Smart Array P440.</div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;">The cable on the right is a PCI-e Slimiline SAS SFF-8654 8i cable</div><br /><p><br /></p></div>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-79330549770215954932024-02-25T19:04:00.002-06:002024-02-25T19:05:11.096-06:00Dell 13th gen. servers and NVMe/Bifurcation <p style="text-align: center;"> In case anyone is wondering, the Dell 13th generation servers can do PCIe Bifurcation. </p><p style="text-align: center;">So first question for those not in the the know, what the heck is PCIe Bifurcation? In simplest term is take a PCIe slot and sub-divides it into multiple slots. IE a PCIe x16 slot can be divided into two x8 or four x4 slots.</p><p style="text-align: center;">Ok great buy why would one want to do this? Well many people have found the wonders of NVMe M.2 hard drives, and run a NVMe M.2 adapter PCIe card. Sometimes the desire to run multiple M.2 cards is desired, but there simply isn't enough slots available.</p><p style="text-align: center;">A quick search on ones favorite shopping place for IT gear will show a number of cards that allow one to put two to four M.2 NVMe drives on a single card, however in order to use them, the PC must support Bifurcation, and with out it the computer will only see the first drive. For the record, there are cards that can have more than one M.2 NVMe drive even if the PC doesn't have Bifurcation support, some have a RAID controller chip on them, some have basically a PCIe switch on them. They are not very common place and are pricey.</p><p style="text-align: center;">In this case the case study is a Dell PowerEdge r930, with a $20 dual M.2 card from Amazon.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJGLOGr20zRwbripnBpWrAJn_QNuDp1kSDvJxtxTqRocDaSX7Q8uLuHbsJqknAWFEC-Tr7vIPu2oSNesCjgcpiyrSfgAPzA5UXEVeD2iYsZa38XM6PPc8ztH6gISyJ5Lk0PX24K322jIRAIw7CccQKQ4obc1o_q_ftzGwqTWF5ZhZwlt0dfPIq5PI8y3c/s4000/2024-02-25%2017.33.21.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4000" data-original-width="3000" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJGLOGr20zRwbripnBpWrAJn_QNuDp1kSDvJxtxTqRocDaSX7Q8uLuHbsJqknAWFEC-Tr7vIPu2oSNesCjgcpiyrSfgAPzA5UXEVeD2iYsZa38XM6PPc8ztH6gISyJ5Lk0PX24K322jIRAIw7CccQKQ4obc1o_q_ftzGwqTWF5ZhZwlt0dfPIq5PI8y3c/s320/2024-02-25%2017.33.21.jpg" width="240" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg3YuPtUjxjP7tznfrPpngsPrkk4WWJOZK2zpuHPZjrz87U9Q_dUUCnRuBg0Utl9aqzpyAb_gELQWSx0gggppHIRHgxzX4EXho6HsP-dJm6ma30n3rxjVFZIy190yQus4gR4IcwnrZMSl_9znD8o_n2j4-Sw7LefpX5DHArB7wPy1fx1uNO2lvNqF11ZpA/s4000/2024-02-25%2017.34.33.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4000" data-original-width="3000" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg3YuPtUjxjP7tznfrPpngsPrkk4WWJOZK2zpuHPZjrz87U9Q_dUUCnRuBg0Utl9aqzpyAb_gELQWSx0gggppHIRHgxzX4EXho6HsP-dJm6ma30n3rxjVFZIy190yQus4gR4IcwnrZMSl_9znD8o_n2j4-Sw7LefpX5DHArB7wPy1fx1uNO2lvNqF11ZpA/s320/2024-02-25%2017.34.33.jpg" width="240" /></a></div><div class="separator" style="clear: both; text-align: center;">Note on depending on the card slot six might not be usable as the card hits the screw for the heat sink.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxVg0GUf9t_WQho7-nmIgWc5FgTJvYsIYfznVHgMbEOeAv7MXSuTbZQhUqB7D5HTS2uf7zXtmR1CCLtE7gBDdO0MgcQ01Ilo_oK3ZvUMZSSw_cM9lIOcCCrHVb6AQW0LcqV_GMByXBvUYBgE6qyqbqiXVc1wQXEqHikLHugttlPV7eqBG_T3tiqxmLgeg/s4000/2024-02-25%2017.40.22.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3000" data-original-width="4000" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxVg0GUf9t_WQho7-nmIgWc5FgTJvYsIYfznVHgMbEOeAv7MXSuTbZQhUqB7D5HTS2uf7zXtmR1CCLtE7gBDdO0MgcQ01Ilo_oK3ZvUMZSSw_cM9lIOcCCrHVb6AQW0LcqV_GMByXBvUYBgE6qyqbqiXVc1wQXEqHikLHugttlPV7eqBG_T3tiqxmLgeg/s320/2024-02-25%2017.40.22.jpg" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;">The setting change...also even though this machine has something like 10 PCIe slots only six of the have the setting.</div><br /><p><br /></p>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-60167240310358705052024-02-13T22:28:00.002-06:002024-02-14T11:34:07.225-06:00Dell r720xd and VMware ESXi 8.0<p> -Even though the server or the CPU's are not on the HCL (Hardware Certified List), it does work. Even with a Xeon E-2600 v0 CPU (with the CPU bypass setting). The Perc H710 mini mono RAID card is recognized. The onboard NICs are recognized. </p><p>-Mellanox ConnectX-3 40Mbps network cards are no longer supported</p><p>-There are three different versions of Perc H710; the firmware and drivers are all the same. The third revision is actually PCIe Gen 3, however by default it is set to Gen2 mode. The change is easily done in the RAID card settings. Thanks to the <a href="https://youtu.be/di2At6GUUxw?si=IgZ6WktmsIboctjZ" target="_blank">Art of Server</a> for pointing that out.</p><p>-The XD model has twelve 3.5" hard drive bays. In order to make that work there is no real-estate for an optical drive or the LCD screen normally seen on PowerEdge Servers. There is an optional "flex-bay" kit which adds two 2.5" bays at the rear of the machine. Interestingly when one adds the flex-bay, it disables the two onboard SATA connectors, even though the drives in the flex-bay are operated by the HBA/RAID controller. </p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBiRef_LKwprFPIChpkWPdT_NmWaX-yQu6K-E6swY7Ldy3lvqDpTEy6NwF_SzpaL5DbU1Z77a3TP_QWLi5u-4P5vsV_gd5PMCp6wFJySAxolXTiOge-co2duqcupS-cKX0rYk3FrIVP0LWVBFL_B2YG-mypSErQiKDy74GVHyM2j58E0jVK4oCKXOAmIc/s500/v8.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="249" data-original-width="500" height="159" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBiRef_LKwprFPIChpkWPdT_NmWaX-yQu6K-E6swY7Ldy3lvqDpTEy6NwF_SzpaL5DbU1Z77a3TP_QWLi5u-4P5vsV_gd5PMCp6wFJySAxolXTiOge-co2duqcupS-cKX0rYk3FrIVP0LWVBFL_B2YG-mypSErQiKDy74GVHyM2j58E0jVK4oCKXOAmIc/s320/v8.JPG" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhSBQm7jRzLEWEX2JWuljo2dcFzo_7-hlPe46McydmXTN7rKsGJ-9GbSPJxHIy_qHcJAYNrTJDrXZmHzpfjU6Yr59BbJRSua3xQRqQ9VHSsnUUX9Z2I8Y0yXMsaHWbuz6clEjm-JRk-flKiKnMeIHxAPQf9ydOtp-C0jwFdqjUtdZUOpHLYSJ4u32aUSQk/s999/communityasset-a0606a8e-eddb-46a1-b6f1-99eb968c84dd-843465895.gif" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="749" data-original-width="999" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhSBQm7jRzLEWEX2JWuljo2dcFzo_7-hlPe46McydmXTN7rKsGJ-9GbSPJxHIy_qHcJAYNrTJDrXZmHzpfjU6Yr59BbJRSua3xQRqQ9VHSsnUUX9Z2I8Y0yXMsaHWbuz6clEjm-JRk-flKiKnMeIHxAPQf9ydOtp-C0jwFdqjUtdZUOpHLYSJ4u32aUSQk/s320/communityasset-a0606a8e-eddb-46a1-b6f1-99eb968c84dd-843465895.gif" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrZt3INR2qI2oH8mk4M0XUAvusP27lMJCMZTPDbErE8WSq3NJJdXix4HPiPFWFe-97fPM8TE-KIUZK4Nu5YiHsmZWgoE7ySbFha-q-sKu0lbCNvX0g5C6a5Al4GzsSOCeDsV01gVlDuds7B376qy8y-YagSUmbuGQZvaAK5FRrE7I5cv6voCL_GrnpB20/s2560/81+cb3LNEUL.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1920" data-original-width="2560" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrZt3INR2qI2oH8mk4M0XUAvusP27lMJCMZTPDbErE8WSq3NJJdXix4HPiPFWFe-97fPM8TE-KIUZK4Nu5YiHsmZWgoE7ySbFha-q-sKu0lbCNvX0g5C6a5Al4GzsSOCeDsV01gVlDuds7B376qy8y-YagSUmbuGQZvaAK5FRrE7I5cv6voCL_GrnpB20/s320/81+cb3LNEUL.jpg" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><iframe allowfullscreen="" class="BLOG_video_class" height="266" src="https://www.youtube.com/embed/2qIh4Que22c" width="320" youtube-src-id="2qIh4Que22c"></iframe></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><br /> <p></p>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-43921775723951889192024-02-02T15:14:00.005-06:002024-02-25T19:38:07.006-06:00HP Proliant DL80 Gen. 9<p> Hewlett Packard Proliant DL80 Generation 9 is an entry level server with many corners cut to make the price cheaper than the normal enterprise hardware.</p><p>Here are a few notes:</p><p>-only one Power Supply</p><p>-no dedicated ILO port</p><p>-eight memory slots, and only four can be used in single CPU configuration</p><p>-the base model has no riser cards, and can only use three PCIe cards with a single CPU installed and five with two CPU's. </p><p>-there is three? options for adding a riser card(s). One of which is a GPU kit, which includes another fan</p><p>-converting to dual CPU's will need at least two more fans, three to make it "redundant". it will operate without have the extra fans but it will complain about it at post and run the existing fans at full speed.</p><p>-the built in RAID controller is a B140i. It has two SFF-8087 ("mini-sas") connections and two normal SATA connectors. All of those connectors are controlled by the RAID card. IE if one plugs a single drive into the normal SATA port one needs to go through the HP Storage Administrator and make a RAID0 in order to use it. It only does SATA, SAS is NOT supported. RAID5 is supported! System RAM will be used for cache. Interestingly Windows will see the virtual volume as intended, however my Linux utility, Parted-Magic, as well as the VMware installer, sees all of the drives individually. The card can have it's personality switched to be a standard SATA controller.</p><p>-The standard backplane is four port, despite there being 8 drive bays. One can just add a 2nd four port backplane, the physical retention is there as is a second special power lead. There also appears to be a single 8-bay option. The standard backplane also claims to be not hot-swappable, I am not sure if that is because of the B140ii controller or the back plane. Speaking of drive bays, the 3.5" bays do not have any LED indicators. The backplane is connected to the system board via a SFF-8087 connector (at the motherboard end) to a four port SATA break out cable to the four ports of the backplane. </p><p>I ordered a HP 790487-001, which is said to work for the DL60, DL80, and DL120. This four port SAS/SATA backplane differs in that it has a SFF-8087 connector and little "fingers" to operate the LEDs for the drive sled. </p><p>-VMware ESXi v7 does see the storage and the network cards. However where as Windows does recognize RAID logical volumes, ESXi does not. In this case a four drive RAID5 was setup, and VMware only sees four individual drives. </p><p><br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKgvt2aBxUFDh9SzrgH8gUegf1BDti13C-qpqrTfxUAUUNjrVEbCoqdYNXQLEod-MqQzQ7ekaCA41-szpYVgXKszswg9z4kJsYUc_ZsxvnhiOK33-5qOJfD5RBtmV2tdwWsKdMJarp4Uv8C1eBw_cNfNxypzZElNkbDf9RPfTUwqOZlWVXz7nC4TD_Ons/s4000/2024-01-27%2021.36.42.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4000" data-original-width="3000" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKgvt2aBxUFDh9SzrgH8gUegf1BDti13C-qpqrTfxUAUUNjrVEbCoqdYNXQLEod-MqQzQ7ekaCA41-szpYVgXKszswg9z4kJsYUc_ZsxvnhiOK33-5qOJfD5RBtmV2tdwWsKdMJarp4Uv8C1eBw_cNfNxypzZElNkbDf9RPfTUwqOZlWVXz7nC4TD_Ons/s320/2024-01-27%2021.36.42.jpg" width="240" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhyphenhyphenWJdapFKtN5hP_DANA1IWhkI13AWdWBmnj4zHhY7-ApfJhE9pc67nLA9NqqEu-X_THY3a6UbefIDafoxFwoi9pn8v3QrGZZmFowVoS1b9ROAj8gcbfXXnqjpN4BjZztPcRo7pQE1uL0AABKSYJXnSg9foFT9a0Gu5rTuebfSMqCFaCZz2SsP6r62Gc4/s4000/2024-01-27%2021.36.48.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4000" data-original-width="3000" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhyphenhyphenWJdapFKtN5hP_DANA1IWhkI13AWdWBmnj4zHhY7-ApfJhE9pc67nLA9NqqEu-X_THY3a6UbefIDafoxFwoi9pn8v3QrGZZmFowVoS1b9ROAj8gcbfXXnqjpN4BjZztPcRo7pQE1uL0AABKSYJXnSg9foFT9a0Gu5rTuebfSMqCFaCZz2SsP6r62Gc4/s320/2024-01-27%2021.36.48.jpg" width="240" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGXmn2LsoRvLowOVvCslA9iwrpP8xtcq5yhEA-EtXgDh3soztKfk7tsyP3f4YR7YAY1uUBXhgyHSZIBegTajaCOhLO2gp9O8cs6EYy5O4nTkVA2OVoZBF2ctXvzEIv9qZ9OVsjTLJgrtRYYLfCdtOitOVLfp94gN5vYwSE18ABncqNyvu1uiK2Z3HU-SI/s4000/2024-01-27%2021.37.50.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3000" data-original-width="4000" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGXmn2LsoRvLowOVvCslA9iwrpP8xtcq5yhEA-EtXgDh3soztKfk7tsyP3f4YR7YAY1uUBXhgyHSZIBegTajaCOhLO2gp9O8cs6EYy5O4nTkVA2OVoZBF2ctXvzEIv9qZ9OVsjTLJgrtRYYLfCdtOitOVLfp94gN5vYwSE18ABncqNyvu1uiK2Z3HU-SI/s320/2024-01-27%2021.37.50.jpg" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyBv-Uy1RwhO6ARsqhyphenhyphenmRU2MYsNvihSIKv8xcofokjEaboM31HdQcnAhVZMA_85qLSQpW98jc6H1WaG9yguCBlSw4Z3Lpa587EAdnFMTF0zY-Enl-BvknV6Jy8hbpkb9RBQGtva94G7LxIq6xQhoSgfgOFb5y9dckVlJkOgKGIPu2xx03DVgekADC993w/s4000/2024-02-25%2017.24.00.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4000" data-original-width="3000" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyBv-Uy1RwhO6ARsqhyphenhyphenmRU2MYsNvihSIKv8xcofokjEaboM31HdQcnAhVZMA_85qLSQpW98jc6H1WaG9yguCBlSw4Z3Lpa587EAdnFMTF0zY-Enl-BvknV6Jy8hbpkb9RBQGtva94G7LxIq6xQhoSgfgOFb5y9dckVlJkOgKGIPu2xx03DVgekADC993w/s320/2024-02-25%2017.24.00.jpg" width="240" /></a></div><div class="separator" style="clear: both; text-align: center;">Original backplane on the left, upgraded one on the right</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSLI6_qEPvQ3hAsaOa6B7zbMrPO582heZ5y9qOCOnrSm2T8M8NTKLhGuE278XDgSk4W1QNSBjZVm52gv1DZ6_hx17-MEeD-1Zwg0NzHD8cvhPGBm_qrtrejNoHsxvbDLobhz0NG1rVJeGw6ytbNjlYg-u2bo-D2Xpa2oCOdJSgEbo6VVnC2KGB6OM4lmY/s4000/2024-02-25%2017.24.17.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4000" data-original-width="3000" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSLI6_qEPvQ3hAsaOa6B7zbMrPO582heZ5y9qOCOnrSm2T8M8NTKLhGuE278XDgSk4W1QNSBjZVm52gv1DZ6_hx17-MEeD-1Zwg0NzHD8cvhPGBm_qrtrejNoHsxvbDLobhz0NG1rVJeGw6ytbNjlYg-u2bo-D2Xpa2oCOdJSgEbo6VVnC2KGB6OM4lmY/s320/2024-02-25%2017.24.17.jpg" width="240" /></a></div><div class="separator" style="clear: both; text-align: center;">Notice the "fingers" to operate the drive sled LED's</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiW2u_H1Gd2EFt3uW_F6LckdWpLJyiypr5jEHAq5dXy3A5Ze-065yv_JcHlXlCxjgyXHYI3_gdyodgREu5hAYxcMtstOCEMl0DxkGOn-E31A26alvJucwH2M2ZCMXTcFlhgW24YRZ4jFCa0gudSFppJ7EoiSO5KtTuGBPqmSkHU3z0dE8eg-kXIkF74diY/s4000/2024-02-25%2017.29.21.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3000" data-original-width="4000" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiW2u_H1Gd2EFt3uW_F6LckdWpLJyiypr5jEHAq5dXy3A5Ze-065yv_JcHlXlCxjgyXHYI3_gdyodgREu5hAYxcMtstOCEMl0DxkGOn-E31A26alvJucwH2M2ZCMXTcFlhgW24YRZ4jFCa0gudSFppJ7EoiSO5KtTuGBPqmSkHU3z0dE8eg-kXIkF74diY/s320/2024-02-25%2017.29.21.jpg" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;">Both backplanes installed, note the SFF-8087 isn't installed yet.</div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-91852612809505615982024-01-13T19:57:00.003-06:002024-02-13T22:09:16.822-06:00Windows 10 Activation & Licensing (late 2023)<p>Ever since 2015 Microsoft has allowed computer users to "upgrade" to Windows 10 using the license key from Windows 7 and Windows 8. This allowed many older computer that came from manufacture with Windows 7 to have a new lease on life by legally using a newer operating system. During the late fall of 2023 Microsoft has closed this loop hole. </p><p>Today if one takes a Windows 7 computer and installs Windows 10, it will not activate. Even if one types the license key, it will not activate. So where does that leave those of us with older machines hanging around? Well...</p><p><b>YOU CAN CONTINUE TO USE A COMPUTER WITH AN UNACTIVATE COPY OF WINDOWS 10!</b></p><p><b>The computer will continue operate just fine, it will not one day stop working because of the activation status. </b></p><p><b>There is some caveats though. Here is a list of them:</b></p><p>-"Activate Windows" watermark shows up on the bottom right corner of the screen, and will be visible no mater what, even if one is watching a video, playing a game, or writing a novel. </p><p>-premium features are locked out</p><p>-optional updates are not available, but critical updates are</p><p>-desktop personalization is restricted: wall paper, taskbar settings, lock screen, themes</p><p><a href="https://royalcdkeys.com/blogs/news/unactivated-windows-10-what-are-the-disadvantages">https://royalcdkeys.com/blogs/news/unactivated-windows-10-what-are-the-disadvantages</a></p><p><a href="https://www.pcworld.com/article/2104344/its-official-upgrades-using-windows-7-and-8-keys-are-dead.html">https://www.pcworld.com/article/2104344/its-official-upgrades-using-windows-7-and-8-keys-are-dead.html</a></p><p><a href="https://softwarekeep.com/blog/what-happens-if-you-don-t-activate-windows-10">https://softwarekeep.com/blog/what-happens-if-you-don-t-activate-windows-10</a></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihNDLA_uUC3z1K2kHEEO6GvpefEhPxsQhXAU-z_ch5-Da4nCBHoGhrPZ_IXgN-evrPraSDt-IjvITUrFalNVpOdb_TVbmF2xT3cgiS14HmrlcTY1IfFJHjnaessfBYdL-OVDxiP6M3NrBwxBeyKnctMDigiIcz3KzdzwjcE6XEoxNZwji3gcKm75Eas7s/s1630/image.png.51068648bf0530965081d9cc241c3778.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="495" data-original-width="1630" height="194" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihNDLA_uUC3z1K2kHEEO6GvpefEhPxsQhXAU-z_ch5-Da4nCBHoGhrPZ_IXgN-evrPraSDt-IjvITUrFalNVpOdb_TVbmF2xT3cgiS14HmrlcTY1IfFJHjnaessfBYdL-OVDxiP6M3NrBwxBeyKnctMDigiIcz3KzdzwjcE6XEoxNZwji3gcKm75Eas7s/w640-h194/image.png.51068648bf0530965081d9cc241c3778.png" width="640" /></a></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTPpsNLL7J-TD2IBgmjjjWu8wkJZijmGxjJVxTGmtSYyVaeY8E_r05HmqvL3qZBnN5oOQD0SRaMJ9yNb-W20IXc2vOCkvjjEh4LKKl-bsL_uJZIz3DHAoNtL34OHA5V4YckHoRwlzBMSD6bWgsYy6OAoLweRqqmlPMk_E44PFNcY2zr_JDAzTqin2c1T0/s678/Activate-Windows01-678x381.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="381" data-original-width="678" height="180" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTPpsNLL7J-TD2IBgmjjjWu8wkJZijmGxjJVxTGmtSYyVaeY8E_r05HmqvL3qZBnN5oOQD0SRaMJ9yNb-W20IXc2vOCkvjjEh4LKKl-bsL_uJZIz3DHAoNtL34OHA5V4YckHoRwlzBMSD6bWgsYy6OAoLweRqqmlPMk_E44PFNcY2zr_JDAzTqin2c1T0/s320/Activate-Windows01-678x381.png" width="320" /></a></div><br />aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-62290575084520207772023-12-27T12:44:00.004-06:002024-02-12T13:24:57.823-06:00Dell PowerEdge r930 hardware guide<p> ***Work in Progress, will continue to update until the project is finished****<br /><br /></p><p>Dell PowerEdge r930 is a 4U, four CPU system. This is some notes from working on them. Finding information on them was sparse, so hopefully someone else can benefit from my research. The biggest help was this source: <a href="https://www.dell.com/support/manuals/en-us/poweredge-r930/r930om/documentation-resources?guid=guid-d2cfc267-94f6-4a0c-b813-2ec7a31ee8c6&lang=en-us" target="_blank">from Dell.</a> This <a href="https://www.dell.com/support/contents/en-ag/videos/videoplayer/tutorial-on-cabling-x24-hdd-backplane-for-poweredge-r930/6079797010001" target="_blank">link</a> as well.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgswsZSkWGDhxzzkRiU-gDM01b44IIp_7bJEfqIbkwaQceMagKqY7Ycf63Ebzu_5tZ30pt4VBrq3xeFrVwL9NZrm_ZbAEuuDVwniLBdSH_oQCBpjzHQ-ap_WD9XGU9KNljnqMHZ9FGaq_sUvLTanjWt02bdUj4M3YPDJc62yCg_vzaRNjFpl6tbpsl6TJQ/s4000/2023-12-26%2023.09.29.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4000" data-original-width="3000" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgswsZSkWGDhxzzkRiU-gDM01b44IIp_7bJEfqIbkwaQceMagKqY7Ycf63Ebzu_5tZ30pt4VBrq3xeFrVwL9NZrm_ZbAEuuDVwniLBdSH_oQCBpjzHQ-ap_WD9XGU9KNljnqMHZ9FGaq_sUvLTanjWt02bdUj4M3YPDJc62yCg_vzaRNjFpl6tbpsl6TJQ/s320/2023-12-26%2023.09.29.jpg" width="240" /></a></div><br /><p>When using four CPU's the <b><u>minimum </u></b>RAM configuration is eight memory modules. One stick of RAM in the #1 back of each of the eight memory riser cards.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgU7lU4cnsEOl6kI4leeY2Nu6VofXBirAasaj1TX9B3jpf-ZeoreEjDgKXYuGISnrSi_Vb-Qa7GWeSGrEF7MJt_2lsVVycKyofwJGMr-5dU__tsJA8HpsUCQtqzwzlcftkOLuuKYQtQws5cS1oE1zP7Ajg8yVXmNCzlOkiN5_9MDQ9XS5izX8AinXmd-F0/s4000/2023-12-26%2023.10.14.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4000" data-original-width="3000" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgU7lU4cnsEOl6kI4leeY2Nu6VofXBirAasaj1TX9B3jpf-ZeoreEjDgKXYuGISnrSi_Vb-Qa7GWeSGrEF7MJt_2lsVVycKyofwJGMr-5dU__tsJA8HpsUCQtqzwzlcftkOLuuKYQtQws5cS1oE1zP7Ajg8yVXmNCzlOkiN5_9MDQ9XS5izX8AinXmd-F0/s320/2023-12-26%2023.10.14.jpg" width="240" /></a></div><br /><p>It doesn't matter so much what hard drive controller is used, but it MUST bee in the very first slot. Remove the I/O shield put it into slot one and use the lever to secure it down. In this case it is a single Perch 730H.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguwxyJP9w42Akibf8ROPcrXuIM4ygSnqaRW4u5UUwLHsdvAzRv9bQ5H08tnTPknDvleTNcxhd8IUuc61sQ3YiVEQ2u71vgmTmWMYZ9DWLWKNom4yIcxxfooN_YNoMdEtWupJON4tsMSjWymQPK26MhCwKsTpo7WcXG4FXBFWoYb-4Svr9bcOBjyRoDz_k/s4000/2023-12-26%2023.20.27.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4000" data-original-width="3000" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguwxyJP9w42Akibf8ROPcrXuIM4ygSnqaRW4u5UUwLHsdvAzRv9bQ5H08tnTPknDvleTNcxhd8IUuc61sQ3YiVEQ2u71vgmTmWMYZ9DWLWKNom4yIcxxfooN_YNoMdEtWupJON4tsMSjWymQPK26MhCwKsTpo7WcXG4FXBFWoYb-4Svr9bcOBjyRoDz_k/s320/2023-12-26%2023.20.27.jpg" width="240" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><span style="text-align: left;">To pull out the fan tray, just remove the outer four fans, then use the release to free up the handle. In order to work on the CPU's or the back plane, all memory risers must be removed, the four fans, and the fan tray.</span></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx-FvVH51TOIk3S3mWUyRS3NNPQrK7sY6ErCsQEPR-opk1WrdWTu9GpxsvqU8iyEe2-iv9PkAkRJ41xzBZfdcnmJ0O0R6qnXbgyKCelLm8IOkXnkO1s1awoZRL6yyLZr33OWEGJL5YQoYsn5OauponmBRj5ZUce8kdYqGHjcvOGwykqCh_oobj1gILp8Q/s4000/2023-12-27%2011.20.02.jpg" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="3000" data-original-width="4000" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx-FvVH51TOIk3S3mWUyRS3NNPQrK7sY6ErCsQEPR-opk1WrdWTu9GpxsvqU8iyEe2-iv9PkAkRJ41xzBZfdcnmJ0O0R6qnXbgyKCelLm8IOkXnkO1s1awoZRL6yyLZr33OWEGJL5YQoYsn5OauponmBRj5ZUce8kdYqGHjcvOGwykqCh_oobj1gILp8Q/s320/2023-12-27%2011.20.02.jpg" width="320" /></a></p></blockquote><p>The network card is the same "form factor" as found on other r630/730's..it has the proprietary connection. There is a special riser card that takes the special NIC and converts it to PCIe. That card also has a special slot for one to install the SD memory card modules.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4umYgFgXvEafOqwBHDMHL84qGJw8CzFIZXFx_0yq1_YvSrtGxByPBGuNwp1w3J08rfVCtOK-kG4h-6ZuYQu-koOuidlOFGrAY26RB7r3bb7GpQd4cIww8XYNUmcKR-uZRfbNVkM-pi8x_xm0kClQ3e6KeDFARFVjftKG4r4DEBFObGs7nnZpi9dbfk18/s4000/2023-12-26%2023.19.58.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4000" data-original-width="3000" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4umYgFgXvEafOqwBHDMHL84qGJw8CzFIZXFx_0yq1_YvSrtGxByPBGuNwp1w3J08rfVCtOK-kG4h-6ZuYQu-koOuidlOFGrAY26RB7r3bb7GpQd4cIww8XYNUmcKR-uZRfbNVkM-pi8x_xm0kClQ3e6KeDFARFVjftKG4r4DEBFObGs7nnZpi9dbfk18/s320/2023-12-26%2023.19.58.jpg" width="240" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHPbaEhWEKpf736V-OoH1Kvx14amwaF9w0X7RLeuUggcmWsG-Y9-AcmyVUckxdgISUHFRO6jM3HVFRzxACNnUoFeYVM5xMqnmHM0VTz7N4iPHyZd344QRr7gYoHyKOl8lX-ps2cxEwhLEY4fk_mjCeTXH-O2457oeNxwpcpkpBfqgauBNzPzROpm3qpik/s4000/2023-12-26%2023.20.07.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4000" data-original-width="3000" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHPbaEhWEKpf736V-OoH1Kvx14amwaF9w0X7RLeuUggcmWsG-Y9-AcmyVUckxdgISUHFRO6jM3HVFRzxACNnUoFeYVM5xMqnmHM0VTz7N4iPHyZd344QRr7gYoHyKOl8lX-ps2cxEwhLEY4fk_mjCeTXH-O2457oeNxwpcpkpBfqgauBNzPzROpm3qpik/s320/2023-12-26%2023.20.07.jpg" width="240" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-H5pg4sz_eAcKHXaQ7AG77IRX5RyS3TBAQRK3RNcijJ45AEzO7P_W2K0OUedVc79s-F6JZqCPTTC4VILLRle5-51lwucYLJ-0JYNgimEG9wPJY2B7TOUZd-ZweRRTESboOXfc5qvXgve9RPgHLPius5cKeRlHAaKODWiiG8ZvtieJrhuMRdy5gxZkGs8/s1469/s-l1600.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1026" data-original-width="1469" height="223" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-H5pg4sz_eAcKHXaQ7AG77IRX5RyS3TBAQRK3RNcijJ45AEzO7P_W2K0OUedVc79s-F6JZqCPTTC4VILLRle5-51lwucYLJ-0JYNgimEG9wPJY2B7TOUZd-ZweRRTESboOXfc5qvXgve9RPgHLPius5cKeRlHAaKODWiiG8ZvtieJrhuMRdy5gxZkGs8/s320/s-l1600.png" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;">This is the network riser card without the NIC daughter board</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjie0xmgzWMBpa1Fgrhywt4ogAKIm64ni8ZNLFxThN6r37OIA3AmRrobBS5geOJIrrSeFqcmyNzDoTJ0Y292jaTd9HMKH7wAYhpJRokv1s47Im9eHWjMvdqILWZ05aqnhcgu1GDCiAXeo17wZCpbV86fv2HndTftCGxwUHMbQCj_XYGx8xwgnmoBNMswqI/s601/GUID-EE75DBAB-6E7F-4CD2-AB1F-613CB34424F5-low.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="571" data-original-width="601" height="304" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjie0xmgzWMBpa1Fgrhywt4ogAKIm64ni8ZNLFxThN6r37OIA3AmRrobBS5geOJIrrSeFqcmyNzDoTJ0Y292jaTd9HMKH7wAYhpJRokv1s47Im9eHWjMvdqILWZ05aqnhcgu1GDCiAXeo17wZCpbV86fv2HndTftCGxwUHMbQCj_XYGx8xwgnmoBNMswqI/s320/GUID-EE75DBAB-6E7F-4CD2-AB1F-613CB34424F5-low.jpg" width="320" /></a></div><p>ONE HBA/RAID card in a 24 drive system: the SAS cables from port A go to port A on the SAS Expander, as is the same for port B. The top SAS port from the backplane goes into the furthest port on the SAS Expander. The bottom SAS port on the backplane goes to the closest port on the SAS Expander. Use the wrong cables or in the wrong spot, the system will complain.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBAb2e4BGBpGLVQ0TBsWwAA9r6cN2kPsTpT6KEIxtvUHzwPsOlb0LZEs5JoycPUs5SSnZh2F118NaXQOuQid3TR3hhNk3mhS85WSKUuvm6e6eH9THqBB_dOPam4T8Xeyd4IsYXYpf0fKKKF21el0r4M3vOtscJkMLOVGsmbHzFd650noiHYWZ0llK1bnA/s4000/2024-01-14%2022.21.16.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3000" data-original-width="4000" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBAb2e4BGBpGLVQ0TBsWwAA9r6cN2kPsTpT6KEIxtvUHzwPsOlb0LZEs5JoycPUs5SSnZh2F118NaXQOuQid3TR3hhNk3mhS85WSKUuvm6e6eH9THqBB_dOPam4T8Xeyd4IsYXYpf0fKKKF21el0r4M3vOtscJkMLOVGsmbHzFd650noiHYWZ0llK1bnA/s320/2024-01-14%2022.21.16.jpg" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRi3JifRdWy4v3EbdZAXOUsDjHGLfQQM4y_O6QM-C4PYDVdK29jfVZ9fBIxSp9q36qXAfRZAlkBla0ANN_Idtb0d6djI6cFN5xT9UV3a6D1xo2v10qdRYJ7S1cS9boZooha986LnnstPFLXmSZ6P6twNA9RxcgpkecrhccW8KNpz1C_qLB1rPKvvBUY_Q/s717/Capture.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="717" data-original-width="713" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRi3JifRdWy4v3EbdZAXOUsDjHGLfQQM4y_O6QM-C4PYDVdK29jfVZ9fBIxSp9q36qXAfRZAlkBla0ANN_Idtb0d6djI6cFN5xT9UV3a6D1xo2v10qdRYJ7S1cS9boZooha986LnnstPFLXmSZ6P6twNA9RxcgpkecrhccW8KNpz1C_qLB1rPKvvBUY_Q/s320/Capture.JPG" width="318" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQO9F4KW0Acpo-QjJPG1z0PUduPXg2x07_gL9Dzrrs_20tznaD9p-ePNld5SE3PcCGPb3V_wK2wL1u9zAePu5d1-UiyhgzUSzNNKu_V-0_azbtuJT9694QvOCIhUc9aA__3p3wnQpa6qBLBalLSQtPrqBEjyaatSneYYpgOfIgQdKkOa7t-IrTaJJM1DI/s2000/415446049_930196768778780_2002463192310808140_n.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1500" data-original-width="2000" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQO9F4KW0Acpo-QjJPG1z0PUduPXg2x07_gL9Dzrrs_20tznaD9p-ePNld5SE3PcCGPb3V_wK2wL1u9zAePu5d1-UiyhgzUSzNNKu_V-0_azbtuJT9694QvOCIhUc9aA__3p3wnQpa6qBLBalLSQtPrqBEjyaatSneYYpgOfIgQdKkOa7t-IrTaJJM1DI/s320/415446049_930196768778780_2002463192310808140_n.jpg" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><span style="text-align: left;">These machines come standard with the ability to run four hard drives. They can be converted to run twenty four. It is a PITA, but very doable. The first one I did took a few hours, the 2nd was a breeze.</span></div><p>Parts needed:</p><p>-backplane</p><p>-SAS expander</p><p>-backplane SAS cables</p><p>-SAS cables from HBA/RAID to SAS expander</p><p>-front bezel </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCQMTC7s4FYJgtFUohIkeHgEQwtaK2H9YOPP07DJA42OUPBn_7acH9J2I_m1Yl6L6OPcHgJqFxxWfKzpOlQUaooWaIwEVbXXvethzrVCp83eZyiOoBAdHgYmjMLXu36BM5uoVDeDeeWbJZa85Wqgs5EVcz1MeAvbqmb03qATl959ih7oMljeAiFQQpV9s/s799/404910205_219647921160764_2375052875331636513_n.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="582" data-original-width="799" height="233" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCQMTC7s4FYJgtFUohIkeHgEQwtaK2H9YOPP07DJA42OUPBn_7acH9J2I_m1Yl6L6OPcHgJqFxxWfKzpOlQUaooWaIwEVbXXvethzrVCp83eZyiOoBAdHgYmjMLXu36BM5uoVDeDeeWbJZa85Wqgs5EVcz1MeAvbqmb03qATl959ih7oMljeAiFQQpV9s/s320/404910205_219647921160764_2375052875331636513_n.png" width="320" /></a></div><br /><p><br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh26k86RBZK_ikk9BAZUfNyKObZZyOO_7EMvmd2b-A4JBHjPypZVh8i8oC3cit-TVdJwd6EYEKOO8xOOhQVyXGCqdqo-WppeKd8IIHisLZkohYgpn4VBgWwGBWpQWpws3UqxvqG5WdHU082IcG0fQys5mFgAyudlKKz0_f-Ey2pCBgUzo0hX11M6F6xipk/s4000/2023-12-26%2009.56.30-1.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3000" data-original-width="4000" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh26k86RBZK_ikk9BAZUfNyKObZZyOO_7EMvmd2b-A4JBHjPypZVh8i8oC3cit-TVdJwd6EYEKOO8xOOhQVyXGCqdqo-WppeKd8IIHisLZkohYgpn4VBgWwGBWpQWpws3UqxvqG5WdHU082IcG0fQys5mFgAyudlKKz0_f-Ey2pCBgUzo0hX11M6F6xipk/s320/2023-12-26%2009.56.30-1.jpg" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEWmNA90vIt6ojJwUFngqyNalZgK9UBdgRYZ3y-MmyWP6YA8I9P5OjF6ravp0ZUe7Vpv47mbGEmSl9ermpUe8C7aiO-p6_Xoj3YIxHYooNfuxUv4Yr4ZFhXvB5H7bVmIZqVS4kp9wBGiGU0c0GlA7PXvWQ4gyltcFcuFgRZPrLTv44AlcY_TrvrIECbDI/s1005/2023-12-26%2022.48.21-1.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="754" data-original-width="1005" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEWmNA90vIt6ojJwUFngqyNalZgK9UBdgRYZ3y-MmyWP6YA8I9P5OjF6ravp0ZUe7Vpv47mbGEmSl9ermpUe8C7aiO-p6_Xoj3YIxHYooNfuxUv4Yr4ZFhXvB5H7bVmIZqVS4kp9wBGiGU0c0GlA7PXvWQ4gyltcFcuFgRZPrLTv44AlcY_TrvrIECbDI/s320/2023-12-26%2022.48.21-1.jpg" width="320" /></a></div><br /><p><br /></p><p><br /></p>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-72266400025537433982023-11-06T17:36:00.011-06:002023-12-27T12:21:07.301-06:00Quantum Lattus Part 2<p> The Lattus has two 1gbps network cards, which is all fine, but even with mechanical disks it doesn't take too many of mechanical drives to saturate a 1gb NIC. So I attempted to add a 10gb NIC. </p><p>Turns out that the S30/40/50 using the the SuperMicro system board, the placement of the PCIe slots does allow for a standard height low profile card; in either direction, going the left side of the board (away from the CPU) there isn't enough room in height. Nor going to the right as the card hits the CPU heatsink.</p><p>The S10/S20 does work either direction. If going away from the CPU one must remove the 2.5" drive bay. I didn't feel comfortable having the circuitry of the 10gb NIC basically touching the system board so I choose to run the NIC towards the CPU. Doing so meant I had to remove the PCI-e SATA card. Meaning I could only drive six hard drives, or five data, and one OS drive if running something like TrueNAS. I found a 6gbps 2-port SATA card that is super small that fit in the other side; which brought my drive controller up to 8 drives.</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzpqNnBpcBbiqhJqkMOZTz-9PMWphN4vTdDCLDCmX1m0VJFjgUa55InMnzhR7UdcmKlLVCSASBJKk9UqEeSOo0nNVUFaSlRVAo88SJayacQ8nqHBB1FvtaXUGHnMnm8Bj4DUkteAtxp7y7-jVWdFvbBGl2kGVuNP7ziGdEqZU7bng14GSIlN0wUpEKN_4/s4000/2023-10-13%2014.11.31.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4000" data-original-width="3000" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzpqNnBpcBbiqhJqkMOZTz-9PMWphN4vTdDCLDCmX1m0VJFjgUa55InMnzhR7UdcmKlLVCSASBJKk9UqEeSOo0nNVUFaSlRVAo88SJayacQ8nqHBB1FvtaXUGHnMnm8Bj4DUkteAtxp7y7-jVWdFvbBGl2kGVuNP7ziGdEqZU7bng14GSIlN0wUpEKN_4/s320/2023-10-13%2014.11.31.jpg" width="240" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2KVaIAJKNV66GpiAvmryBxXZ1mJyp7WoWO3fNujvfkfPE8bCEtjvam7rbiOSBl0e5eDTPNzd02FAkEud6obd6dGVKtyLpX4wHpMDXpYzjL__PHOcWucrGx8Yh9F9hV-i3COiDXk_vPVTACquwA4Le60n4qZBDLo-Qyl7a9lRLAyD1MpLiwrM78v3SIxM/s4000/2023-10-26%2020.03.47.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3000" data-original-width="4000" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2KVaIAJKNV66GpiAvmryBxXZ1mJyp7WoWO3fNujvfkfPE8bCEtjvam7rbiOSBl0e5eDTPNzd02FAkEud6obd6dGVKtyLpX4wHpMDXpYzjL__PHOcWucrGx8Yh9F9hV-i3COiDXk_vPVTACquwA4Le60n4qZBDLo-Qyl7a9lRLAyD1MpLiwrM78v3SIxM/s320/2023-10-26%2020.03.47.jpg" width="320" /></a></div><p></p><p></p><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggYCunhBH0DsYEuHY93CF4kFnZ2ba3EEoVSpd5hiaYiTpokjFyNJy-C9NQtF-azgMYXTI5YV4Lynk0YzTlpAyEd6-CaXJQb9x11EVCLNuSpT3PyqnHzR77bUov4G-Ds8rWjSugGxVI6sXoL64fottbSELU2JGprmyUQt0BFUXzeX9tq3S59FQJ8LPKYKs/s4000/2023-10-26%2020.03.40.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4000" data-original-width="3000" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggYCunhBH0DsYEuHY93CF4kFnZ2ba3EEoVSpd5hiaYiTpokjFyNJy-C9NQtF-azgMYXTI5YV4Lynk0YzTlpAyEd6-CaXJQb9x11EVCLNuSpT3PyqnHzR77bUov4G-Ds8rWjSugGxVI6sXoL64fottbSELU2JGprmyUQt0BFUXzeX9tq3S59FQJ8LPKYKs/s320/2023-10-26%2020.03.40.jpg" width="240" /></a><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggYCunhBH0DsYEuHY93CF4kFnZ2ba3EEoVSpd5hiaYiTpokjFyNJy-C9NQtF-azgMYXTI5YV4Lynk0YzTlpAyEd6-CaXJQb9x11EVCLNuSpT3PyqnHzR77bUov4G-Ds8rWjSugGxVI6sXoL64fottbSELU2JGprmyUQt0BFUXzeX9tq3S59FQJ8LPKYKs/s4000/2023-10-26%2020.03.40.jpg" style="margin-left: 1em; margin-right: 1em;"></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-BurTkVCLb9SVG4t-_2LDhmUK1wFkVhUsrRMKYpfEjE2Eysl4w1TjcqyZkO9JtwJ2dowAS_QSwbjgQiWOySTSWGrHW8pRM_ebP-agqObajtIi7y1ZpmuOIJEyDbQMLz9y7S5hDV4MkA8obYW1D8JKRAssnsGPx0aORNz4hLiy5bD6JSFlpz-1ADXt2_0/s4000/2023-10-21%2013.57.26.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3000" data-original-width="4000" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-BurTkVCLb9SVG4t-_2LDhmUK1wFkVhUsrRMKYpfEjE2Eysl4w1TjcqyZkO9JtwJ2dowAS_QSwbjgQiWOySTSWGrHW8pRM_ebP-agqObajtIi7y1ZpmuOIJEyDbQMLz9y7S5hDV4MkA8obYW1D8JKRAssnsGPx0aORNz4hLiy5bD6JSFlpz-1ADXt2_0/s320/2023-10-21%2013.57.26.jpg" width="320" /></a></div></div><div class="separator" style="clear: both; text-align: center;"><br /></div>Also I found these SuperMicro 10gb NICs that are based on the Intel x520 chip; they are physically smaller in nearly every dimension than the OEM Intel.<p></p><br /><br /><br /><br /><p><br /></p>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-35242275170640653842023-11-06T14:48:00.009-06:002024-02-13T15:35:26.362-06:00Quantum Lattus <p> I don't know much about the product, it was designed to be a "cheap and deep", massively scale out storage system. The nodes are "dumb" and controlled an automatically deployed by PXE boot a master node. Western Digital did much of the early R&D, that intellectual property was sold to Quantum. This particular line has reached it's End Of Life.</p><p>These servers hold 12 SATA 2.5" drives in a 1U extra deep case 33~34" long. </p><p>The Lattus S10 & S20's have an Asus CMB-a9sc2 motherboard, it has a dedicated IMPI port, dual 1gb NICs plus a dedicated IMPI. 8gb DDR3 ECC RAM. Six onboard SATA ports, a PCI eight port SATA card. It also has a 2.5" hard drive tray. Intel Xeon E3-1220L v2 CPU at 2.3ghz (2 core, 4 threads, 17 watts).</p><p><a href="https://data-us.aaeon.com/DOWNLOAD/2007%20datasheets/APD/CMB-A9SC2.pdf" target="_blank">Motherboard datasheet</a><br /></p><p>The Lattus S30, S40, & S50's have an SuperMicro x10SL7 motherboard, it has a dedicated IMPI port dual 1gb NICs. It has eight onboard SAS/SATA ports. Intel Xeon E3-1230L v3 CPU at 1.8ghz (4 core, 8 threads 25w). </p><p> The BIOS password is "Adm1n", the IMPI credentials are ADMIN/ADMIN</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJQO8CqPLsg8tmReD9QtTKZYuZeXCA-vm7J11iGLR8gj-zflJCKq-k_oK_dcjwTfgmOoXpQAzhNlmPFwE05G4wVIzwQX16il1orV9hJUzEgrVh2Vd8HYw-ncd6Q4GyDr8MSHkGUbw2FcnuT4MaEz0C7Da7G7rZ5zs-GWbZ1WOtr4_zLx4PKGs5s1KPMFM/s4032/2022-05-09%2018.08.02.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4032" data-original-width="3024" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJQO8CqPLsg8tmReD9QtTKZYuZeXCA-vm7J11iGLR8gj-zflJCKq-k_oK_dcjwTfgmOoXpQAzhNlmPFwE05G4wVIzwQX16il1orV9hJUzEgrVh2Vd8HYw-ncd6Q4GyDr8MSHkGUbw2FcnuT4MaEz0C7Da7G7rZ5zs-GWbZ1WOtr4_zLx4PKGs5s1KPMFM/s320/2022-05-09%2018.08.02.jpg" width="240" /></a></div><div class="separator" style="clear: both; text-align: center;">S10/S20 </div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhf4UZyoit7IAfISXwSLsoSoBtvfupYIugeJVUieClwDt8sJ2Szojt6uhBbX5rMRnrRw9cwtaPGZ-F5S6q2BSjL7GbeROaLSwjTR5ySnKRz5m96tuhyphenhyphenuul4v_lXRXfVFfp2uIdq63YEdHEEMSiD6dM3EaL5KukZeAxWH5C4Y3qQaYQOz3YqSsrDCXcd1As/s4000/2023-10-03%2008.17.30.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3000" data-original-width="4000" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhf4UZyoit7IAfISXwSLsoSoBtvfupYIugeJVUieClwDt8sJ2Szojt6uhBbX5rMRnrRw9cwtaPGZ-F5S6q2BSjL7GbeROaLSwjTR5ySnKRz5m96tuhyphenhyphenuul4v_lXRXfVFfp2uIdq63YEdHEEMSiD6dM3EaL5KukZeAxWH5C4Y3qQaYQOz3YqSsrDCXcd1As/s320/2023-10-03%2008.17.30.jpg" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;">S10/S20 rear</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNomkM8YDzE9R2pVJeQob9mGDl2kvNRVgU-Sk-scxqifvZYPs-mkdtnpkEV7WBK_kmgzRhMX7h47P7r84rN7TXzDISfjQxiLeIdr0IAZ0MUIlAS9xwbxD03bhGIxe7gdhqisq7RgnPxZgCETSuId8ZvMfP50x_x3kVCj4dYYhyphenhyphenRSKcc_wrtCQT6JbSVDQ/s4000/Lattus%20s10.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4000" data-original-width="3000" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNomkM8YDzE9R2pVJeQob9mGDl2kvNRVgU-Sk-scxqifvZYPs-mkdtnpkEV7WBK_kmgzRhMX7h47P7r84rN7TXzDISfjQxiLeIdr0IAZ0MUIlAS9xwbxD03bhGIxe7gdhqisq7RgnPxZgCETSuId8ZvMfP50x_x3kVCj4dYYhyphenhyphenRSKcc_wrtCQT6JbSVDQ/s320/Lattus%20s10.jpg" width="240" /></a></div><div class="separator" style="clear: both; text-align: center;">S10/S20 with the SATA controller card removed.</div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhABru0S9kmwf8e35qqfzWha3HbU8TR-T11kxy3KdymjTbaouVVKEjDhaB1FKIa4QubfbG7Fsd5MY9kPTp7dk-IBrguom-ZtHEKjdMKnO2CgwGLartHNfc6kAyQFCtBCdqhtfPaM8gNO8DzPWONpSkeEzzY7FEkGSDaMxl5W6wedOsy7RLV9clSJyTEh04/s3519/Lattus%20s40%20system%20board.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3519" data-original-width="2989" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhABru0S9kmwf8e35qqfzWha3HbU8TR-T11kxy3KdymjTbaouVVKEjDhaB1FKIa4QubfbG7Fsd5MY9kPTp7dk-IBrguom-ZtHEKjdMKnO2CgwGLartHNfc6kAyQFCtBCdqhtfPaM8gNO8DzPWONpSkeEzzY7FEkGSDaMxl5W6wedOsy7RLV9clSJyTEh04/s320/Lattus%20s40%20system%20board.jpg" width="272" /></a></div><div class="separator" style="clear: both; text-align: center;">S30/S40/S50</div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRwmcuNUhwNoZYod1f96nnCq1frfBlKXYrZuL5I-u4wEZph46ik3XxXlGtuNR-y_DSBP0i6zIFQLEzZr-dj0yfj-RsFuIx_ZWRt1BhA0uYd1uLa6DycF5Aiovk0_xDTfYFnqk4cUbkjSoFU1pcg8skvXt38kVtDk8FqGHLr6SmJ57qQf5UZi1eO_GxI-w/s1464/SM%20IMPI.JPG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="902" data-original-width="1464" height="197" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRwmcuNUhwNoZYod1f96nnCq1frfBlKXYrZuL5I-u4wEZph46ik3XxXlGtuNR-y_DSBP0i6zIFQLEzZr-dj0yfj-RsFuIx_ZWRt1BhA0uYd1uLa6DycF5Aiovk0_xDTfYFnqk4cUbkjSoFU1pcg8skvXt38kVtDk8FqGHLr6SmJ57qQf5UZi1eO_GxI-w/s320/SM%20IMPI.JPG" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5WVclrwSzDOJbUJDexL-7tlToTDYxo8OdgsN5GhyphenhyphenBOWGYR8FjrXR9XaNHmw6OlTyLd2CWldDvamWMwHfBs_e-MO-YLKWKvEtIobiiIkCfJ2XUCpufsfPlepqIT9Ef0WS737_uJ-0aS5R7rTN9nW-_IpCIVP6uiYUIVuHeWucImxoktb2EgMAabybdG_Y/s2377/Lattus%20S40%20truenas.JPG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="518" data-original-width="2377" height="70" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5WVclrwSzDOJbUJDexL-7tlToTDYxo8OdgsN5GhyphenhyphenBOWGYR8FjrXR9XaNHmw6OlTyLd2CWldDvamWMwHfBs_e-MO-YLKWKvEtIobiiIkCfJ2XUCpufsfPlepqIT9Ef0WS737_uJ-0aS5R7rTN9nW-_IpCIVP6uiYUIVuHeWucImxoktb2EgMAabybdG_Y/s320/Lattus%20S40%20truenas.JPG" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;">One of the S40's running TrueNAS</div><br /><p><br /></p>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-90682358439005702312023-10-20T13:31:00.007-05:002023-10-20T13:31:48.992-05:00Formula 1<p style="text-align: center;"> <span style="white-space-collapse: preserve;">Anyone find it odd that both VMware AND Broadcom both sponsor Formula 1 cars, but two different teams? BTW for those of you not familiar, Broadcom is in the process of taking VMware over. </span></p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUG_Komkh9MZLbhL7mu9aQmF1zKN7i2aN8bitfX2zkoT3HBZhTF9ZgRzrpeW7FX-4VQW9p4l7BScyG0WqyDK5gmxvSxhiwF3gNEtKOtatqV-WNvCEXTrpzp50n2MDTOoBPjx6OWbeMRtTayp6UPZwToHnyGxTkMCO3Frq56JXJskPR1fpf_kTbjrrCwhU/s1168/386765527_7531410566874245_7763363893241552382_n.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="659" data-original-width="1168" height="181" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUG_Komkh9MZLbhL7mu9aQmF1zKN7i2aN8bitfX2zkoT3HBZhTF9ZgRzrpeW7FX-4VQW9p4l7BScyG0WqyDK5gmxvSxhiwF3gNEtKOtatqV-WNvCEXTrpzp50n2MDTOoBPjx6OWbeMRtTayp6UPZwToHnyGxTkMCO3Frq56JXJskPR1fpf_kTbjrrCwhU/s320/386765527_7531410566874245_7763363893241552382_n.jpg" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvZccqaRjirUuYWJC8M-WEVcwo_WTeDCi2eYnJI2aKgi4U8qkWAWEeccDADASUEuU3OmQ-3XREMj9hGm5f7VwWL9S-_MCKZilWQzLpCIGfXI_rfk7q6xZbLrSmZvxacHmAWNJiSWzVJA6ocAQm7g7N4E4KP4HYS6BaNQDgnO_y3QBE8dQyOKllGL0RDWY/s1330/387752619_7531410556874246_4672345653922654500_n.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="461" data-original-width="1330" height="111" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvZccqaRjirUuYWJC8M-WEVcwo_WTeDCi2eYnJI2aKgi4U8qkWAWEeccDADASUEuU3OmQ-3XREMj9hGm5f7VwWL9S-_MCKZilWQzLpCIGfXI_rfk7q6xZbLrSmZvxacHmAWNJiSWzVJA6ocAQm7g7N4E4KP4HYS6BaNQDgnO_y3QBE8dQyOKllGL0RDWY/s320/387752619_7531410556874246_4672345653922654500_n.jpg" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;">Here is a bit how on how VMware is helping out McLaren.</div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://www.vmware.com/welcome/mclaren.html?fbclid=IwAR2oW2ftJrCkx-EAR4PV3RNBVT0O806FlEhwfqOVK2Un_7D-AQYcQ09nbMM">https://www.vmware.com/welcome/mclaren.html?fbclid=IwAR2oW2ftJrCkx-EAR4PV3RNBVT0O806FlEhwfqOVK2Un_7D-AQYcQ09nbMM</a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><span style="white-space-collapse: preserve;"><br /></span><p></p>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-10985079962196811372023-09-11T12:29:00.006-05:002023-09-11T21:53:39.231-05:0010gb vs 25gb Ethernet when concerning electrical usage<p>Controlling the amount of electricity being used in computing use has become increasingly more important over the years. Wither it is in the home lab, and the consumption directly affect one's own pocket book or the datacenter/server room has heating or power limitations. </p><p>In this case I thought it might be interesting to look at what impact of networking speed has on electrical use. One could conclude from simply looking at number of and size of the heatsink on the cards that there must be a difference. For this test my trusty Kill-A-Watt was pulled out of storage. A normal desktop was used, a Dell 3rd Gen i3, the specifics are irrelevant. Each time the machine was booted off of USB thumb drive into "Parted Magic" (a Linux Rescue Utility); I chose that OS over using Windows so other variables such as drive indexing or Windows Updates wouldn't skew the results. The machine was left to idle for 5 minutes to let things settle down before measurements where taken. Extra GBICs where removed, and the same DAC cable was used. The machine was plugged into a 10gb switch; no 25gb switch was available for testing, donations are welcome! :)</p><p>Bare machine: (no additional hardware) = 32watts</p><p>Machine w/ 10g Intel i520 NIC no cable =35watts</p><p>Machine w/ 10g Intel i520 NIC w/ cable =37watts</p><p>Machine w/ 25g Mellanox CX4 NIC no cable =40watts</p><p>Machine w/ 25g Mellanox CX4 NIC w/ cable =42watts</p><p>Machine w/ 25g Mellanox CX5 NIC w/ cable =41watts</p><p>Machine w/ 25g Broadcomm BCM51414 w/ cable =41watts</p><p>Machine w/ 25g Chelsio w/ cable =49watt (very hot to the touch!)</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhY_cXrNhnUr3f5iRqAYgkVAS00sEvwzczpUU95hJjWIbLxKtrZPLp_oImKPBs_FQ3qFIC9K8BU4c5akMCi71OOVufUs0oRR9m9DFLCzE0eHK-ieZ1faX_EqzIlkHrGqHnZ6vtqSrg0Kxa7BT3-l_i8M4LzITvNL_hTgaJow8nb6ljjLkhuoh2fRgCYcXA/s4000/2023-09-11%2012.00.48.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4000" data-original-width="3000" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhY_cXrNhnUr3f5iRqAYgkVAS00sEvwzczpUU95hJjWIbLxKtrZPLp_oImKPBs_FQ3qFIC9K8BU4c5akMCi71OOVufUs0oRR9m9DFLCzE0eHK-ieZ1faX_EqzIlkHrGqHnZ6vtqSrg0Kxa7BT3-l_i8M4LzITvNL_hTgaJow8nb6ljjLkhuoh2fRgCYcXA/w300-h400/2023-09-11%2012.00.48.jpg" width="300" /></a></div><div class="separator" style="clear: both; text-align: center;">Intel on top, Mellanox CX4 on bottom.</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgcXuas1BqniffoECouo6_nP4RX635XfGmhvMVK46Yx_GKh5ehXIxDbgnGMu4cHyZ-2ernAUdWgG7mFXvKDIe7KdT77TJsWEEEooOtudOqb92OXZ217ZK_tknJOvEW5BalW85LB1qD0EmpgHj5vVqhiBGLYMJVYhhaD9hbl_mIYH5F8s37YxI-tQzuvNgo/s4000/2023-09-11%2021.44.54.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4000" data-original-width="3000" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgcXuas1BqniffoECouo6_nP4RX635XfGmhvMVK46Yx_GKh5ehXIxDbgnGMu4cHyZ-2ernAUdWgG7mFXvKDIe7KdT77TJsWEEEooOtudOqb92OXZ217ZK_tknJOvEW5BalW85LB1qD0EmpgHj5vVqhiBGLYMJVYhhaD9hbl_mIYH5F8s37YxI-tQzuvNgo/w300-h400/2023-09-11%2021.44.54.jpg" width="300" /></a></div><div style="text-align: center;">Chelsio on top</div><div style="text-align: center;">Broadcom in the the middle</div><div><div style="text-align: center;">Mellanox CX5 on the bottom</div><div><br /></div>Another future test would be to test under load.<br /><p><br /></p></div>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-20448926361639463052023-07-14T18:21:00.002-05:002023-07-14T18:21:18.034-05:00Watchguard M270 <p> More electronic recycling. This Watchguard M270 firewall, while is still supported and the latest code works on it, it was sent out for scrap! So let's take a closer look at it.</p><p>This device had customer data on it, I didn't know the password, nor the IP scheme. Next I attached to the console port, and rebooted it into the recovery mode, I didn't find a password reset option. So a factory reset had to be done.</p><p>1. Power off the Firebox.</p><p>2. Press and hold the Reset button on the back of the Firebox.</p><p>3. While you continue to hold the Reset button, power on the Firebox.</p><p>4. Continue to press the Reset button until the Attn indicator begins to flash.</p><p>5. Release the Reset button. ...</p><p>6. Wait for the reset process to complete.</p><p>Once done, open a web browser to https://10.0.1.1:8080, use credentials admin & readwrite</p><p>Sadly the feature keys for this product are now gone. I didn't realize the reset would wipe them. The box will not download it's key from the Watchguard mothership, it complains about being expired. Now the box is pretty much useless as a Watchguard, as it will only allow one client to get through the network. I have seen some people run PFsense on this hardware, so that could be an option.</p><p>Interesting observations:</p><p>-there is a way to get into the BIOS via the console cable, but no one seem to know the password. There is a way to re-flash it. </p><p>-CPU: Intel Atom C3558 @ 2.2ghz, 4 core, 4 threads, 16watt, BenchMark 2417</p><p>-Memory: 4gb RAM DDR3 sodimm</p><p>-Hard drive: 16gb mSATA</p><p>-Marvell 2.5gbps ports? I saw mentioning of this during boot, but that might be an internal switch or something, as from the command line and the GUI all the ports report as 1Gpbs</p><p>-the power supply is a "wall wart"....the big plastic transformer/power supply is actually inside of the case, and the barrel plug fits into the system board. Interesting! I guess it makes replacement simpler, and from installation point of view I'd rather rack a full width appliance vs. a smaller one and then have to deal with securing the external transformer/power supply somewhere else in the rack.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgu1QyhQLfXyIXflM6ifUgJMhAX5A95g0hBdFqkobc08PfBKJODlBovD2KrQ7Pq_tefJbHZfY1vHye_UvO1sjQynNTzduGa7F7F4GefzuaGWFdpWYtQmZmxSzTx374J16oQO-vaMYQePV0wW6SzQD_ztQNDUHDIoms1JWaUQgljeYNC9wxb2TSuTQbJL6U/s4000/2023-07-14%2016.10.40.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4000" data-original-width="3000" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgu1QyhQLfXyIXflM6ifUgJMhAX5A95g0hBdFqkobc08PfBKJODlBovD2KrQ7Pq_tefJbHZfY1vHye_UvO1sjQynNTzduGa7F7F4GefzuaGWFdpWYtQmZmxSzTx374J16oQO-vaMYQePV0wW6SzQD_ztQNDUHDIoms1JWaUQgljeYNC9wxb2TSuTQbJL6U/s320/2023-07-14%2016.10.40.jpg" width="240" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEsPPP29XJqamxwpctaiaS8L_nbRzqLHLCT2y7GqW3UY-tgw24SrQlz4A28eDncjqOq3dUk36BF_zjpncPkKPOH6tmMbpD9kSb-IBXvmWrq8NKH-QoIwSqo9GX_zB3KXtfCmvMllb3kLszQ7YKOgDa2bWbR8WZ3v80hFfpac8NRPRE3A7WayYGvOTVIfI/s4000/2023-07-14%2016.10.55.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3000" data-original-width="4000" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEsPPP29XJqamxwpctaiaS8L_nbRzqLHLCT2y7GqW3UY-tgw24SrQlz4A28eDncjqOq3dUk36BF_zjpncPkKPOH6tmMbpD9kSb-IBXvmWrq8NKH-QoIwSqo9GX_zB3KXtfCmvMllb3kLszQ7YKOgDa2bWbR8WZ3v80hFfpac8NRPRE3A7WayYGvOTVIfI/s320/2023-07-14%2016.10.55.jpg" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXlN97GJj18DzKGGZxBwo5ZWVDp6bpFt7QLq9UlgnlbzjszM0LL22tAQvnlBTt3MT4M-oZk3Grw0v_UQZc4xYjzwnLz7xtkggdCjrdUE8iZrsY45L_kvbDqozOr1a5KzsruNK9iyi5L-u5mQjSJFA_NRetfP_e4PcX97FdChBp-14BKtDuOYJPzojg7Jg/s4000/2023-07-14%2016.11.06.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4000" data-original-width="3000" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXlN97GJj18DzKGGZxBwo5ZWVDp6bpFt7QLq9UlgnlbzjszM0LL22tAQvnlBTt3MT4M-oZk3Grw0v_UQZc4xYjzwnLz7xtkggdCjrdUE8iZrsY45L_kvbDqozOr1a5KzsruNK9iyi5L-u5mQjSJFA_NRetfP_e4PcX97FdChBp-14BKtDuOYJPzojg7Jg/s320/2023-07-14%2016.11.06.jpg" width="240" /></a></div><br /><p><br /></p><p><br /></p><p><br /></p><p><br /></p>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-81269353386434822522023-06-14T14:39:00.004-05:002023-10-02T17:41:22.442-05:00Dell PowerEdge r620 TrueNAS Build: part 2<p> Finally getting around to this project again. </p><p>TrueNAS Core 13.05 was installed onto the 64gb SSD drive. The eight Evo 840 1tb drives were set up into two groups of four RAIDz2 pools, then mirrored. The drives have a lot of hours on them, so I could arrange them better IOPs or space, but I am going error on the side of caution here and go for higher resiliency. The usable space is around 3.5tb.</p><p> One of the 1gb cards was setup with a static IP address. The Mellanox 40gb card was setup with a different IP arrange. The rack that this system will live in has a Dell Force10 switch that has six 40gb ports (two of setup in a trunk for uplink) and forty-eight 10gb ports, there also is a separate 1gb switch. </p><p>Then iSCSI service was started, bound to the 40gb interface. A zPool, portal, target, and extents were setup. For the meantime a Dell r730 running VMware ESXi v7.x with a single 10gb interface was setup for iSCSI. The connections was made, the datastore presented. Storage vMotion a Windows11 VM to the TrueNAS, and ran Atto Disk Benchmark.</p><p></p><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhFZ10Gkf_d7ITHynCqFovP0ZP4XZAOLD9g_Y7p4cubmCTHfKmWtAC3exGd6a4hMRbngODYRu1_oLaPDIimQZiCnY7FtNoZWaT_IIFu-sb3WPk-Ym9ulWUfLWPQd8SxX9BMwi8LVrCyLErEYQx613ZCiEjnEpKhdUE-YNzQ4W5Lz5Yq4SFm25s0Fr_2" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="822" data-original-width="1658" height="199" src="https://blogger.googleusercontent.com/img/a/AVvXsEhFZ10Gkf_d7ITHynCqFovP0ZP4XZAOLD9g_Y7p4cubmCTHfKmWtAC3exGd6a4hMRbngODYRu1_oLaPDIimQZiCnY7FtNoZWaT_IIFu-sb3WPk-Ym9ulWUfLWPQd8SxX9BMwi8LVrCyLErEYQx613ZCiEjnEpKhdUE-YNzQ4W5Lz5Yq4SFm25s0Fr_2=w400-h199" width="400" /></a></div><br /><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjSVnJ4jkceZXOT15wlYyDPr-K1aud_ZeJ4QCP_M6a9Nyb8hLtf2dfS1UJTfRshI-eG6RaOu5L294FjUEtqghMFQLfF4sMPXu8LphVgf-JatTIvc5y9HO7IbZYURpxEDoS1k2UajTe-lYFEhxv3hnNm6o2NVd2YgPN8rks6eQ79OuTlCh5r2KrTt9sh" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="917" data-original-width="1271" height="289" src="https://blogger.googleusercontent.com/img/a/AVvXsEjSVnJ4jkceZXOT15wlYyDPr-K1aud_ZeJ4QCP_M6a9Nyb8hLtf2dfS1UJTfRshI-eG6RaOu5L294FjUEtqghMFQLfF4sMPXu8LphVgf-JatTIvc5y9HO7IbZYURpxEDoS1k2UajTe-lYFEhxv3hnNm6o2NVd2YgPN8rks6eQ79OuTlCh5r2KrTt9sh=w400-h289" width="400" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3Gtqi8YisXUY9AFW4C7TRuMdlTlYveYr5nX1s3ZdlnDPZm9m6LfNsZRBels7vmOM34R0CEC3bpgmdyLY_dLXDxzIwitf3g7Apo3atLop8XutP-NerdTlAGogStGKw6BlMNPWPXW8Z_KNqkUxi8ygNXpQxHf0IJDQFIND0inKzBZkn2aSUScXurYsXu48/s715/TrueNAS%20Core%20Dell%20r620%20eight%20SSD%2010gb.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="715" data-original-width="478" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3Gtqi8YisXUY9AFW4C7TRuMdlTlYveYr5nX1s3ZdlnDPZm9m6LfNsZRBels7vmOM34R0CEC3bpgmdyLY_dLXDxzIwitf3g7Apo3atLop8XutP-NerdTlAGogStGKw6BlMNPWPXW8Z_KNqkUxi8ygNXpQxHf0IJDQFIND0inKzBZkn2aSUScXurYsXu48/w428-h640/TrueNAS%20Core%20Dell%20r620%20eight%20SSD%2010gb.JPG" width="428" /></a></div><div class="separator" style="clear: both; text-align: center;">Using 10gb networking, this is faster than my twenty drive SAS 7200rpm system.</div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZyEBP7s3BZ6-08OTh5yOPxd1NPTb9b9MtobP_krwdU_I3F0kbpucmKK3n0NRJjt5PAdrsy7ML81U3E8mlEkdGcP2XybOekZo1F8fXOhKAYE5-y62a1KZ6obLd1pPkYiqZtcAFmfqRAGjCIphCF0oIU3zWxHqqePOqYl62AMzf8JihJCPCv6vpaL2hxWA/s609/TrueNAS%20Core%20Dell%20r620%20eight%20SSD%2040gb.JPG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="609" data-original-width="451" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZyEBP7s3BZ6-08OTh5yOPxd1NPTb9b9MtobP_krwdU_I3F0kbpucmKK3n0NRJjt5PAdrsy7ML81U3E8mlEkdGcP2XybOekZo1F8fXOhKAYE5-y62a1KZ6obLd1pPkYiqZtcAFmfqRAGjCIphCF0oIU3zWxHqqePOqYl62AMzf8JihJCPCv6vpaL2hxWA/w474-h640/TrueNAS%20Core%20Dell%20r620%20eight%20SSD%2040gb.JPG" width="474" /></a></div><div class="separator" style="clear: both; text-align: center;">Using 40gb networking, there is some gains to be had. It appears that the system is being held back by 10gb networking.</div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><br /></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><br /><p></p>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-19258677329600316452023-04-18T17:06:00.000-05:002023-04-18T17:06:10.995-05:00Dell PowerEdge r620 TrueNAS Build: part 1<p>I have eight Samsung Evo 940 SSD 1tb drives that needed a deserving home; so I decided to build another TrueNAS to server as a VMware shared storage appliance. A Dell PowerEdge r620 server was picked out a to use, as it had eight 2.5" drive bays. The dual CPU's e5-2650's should be way more horse power needed to be an ISCSI/NFS target. Eventually they will be upgraded to v2 CPU's as they are stupid cheap on eBay and it is roughly 30% more performance with no extra electricity. </p><p>An Intel 64gb SSD was brought out of retirement, installed into a slim-CDROM to 2.5" HDD adapter. This would be where the OS will be installed. However first, I installed Windows Server to it just to test out the hardware and then update the firmware. </p><p>First issue I ran into is that even though this server has eight 2.5" drive bays, the backplane only had four drive connectors! I have seen other r620's with only four drive bays, but the other side was a metal face, this one had drive sled in it! Interesting! As luck would have it I found another r620, that had a backplane that did have all 8 drive connectors but was only two hard drives, so the back plane got swapped. </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfvNoqVrAs8oyGzSl-s2QSMPY05XKqkwqJJA2M8QHEpQV7HNXkN4uJsePv7OKrjDrt3_ABfAUlZjeoCfVVRmzU-VmtkOTeTPR-XwKGXv8svpPUkvIxDDyNrkThcusvQ4TOA4P4D9w8i2hP9QNsplwqmK4jEt7JLCQcAoYBiNbIJYOQQCHHsLu5y7M2/s4032/2023-04-17%2014.08.51.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4032" data-original-width="3024" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfvNoqVrAs8oyGzSl-s2QSMPY05XKqkwqJJA2M8QHEpQV7HNXkN4uJsePv7OKrjDrt3_ABfAUlZjeoCfVVRmzU-VmtkOTeTPR-XwKGXv8svpPUkvIxDDyNrkThcusvQ4TOA4P4D9w8i2hP9QNsplwqmK4jEt7JLCQcAoYBiNbIJYOQQCHHsLu5y7M2/s320/2023-04-17%2014.08.51.jpg" width="240" /></a></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKLDVOazCgSCbcF5HLnuQVFUg9Jg5kCmuA6nBisj8_z5aZKGfoMWIbBLyGoj1KuqkBZ_txTEyEP8QJK9LyRVRXfhLXXfHyg6eeqVH7E5qT1quB-eKPU1adgFlIeJlhqCalNNbfGjq0EjqCjPACjgBHhE-RRbM9FHnl6eQTwcy7qCBbsx1efVZAlC38/s4032/2023-04-17%2014.08.57.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3024" data-original-width="4032" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKLDVOazCgSCbcF5HLnuQVFUg9Jg5kCmuA6nBisj8_z5aZKGfoMWIbBLyGoj1KuqkBZ_txTEyEP8QJK9LyRVRXfhLXXfHyg6eeqVH7E5qT1quB-eKPU1adgFlIeJlhqCalNNbfGjq0EjqCjPACjgBHhE-RRbM9FHnl6eQTwcy7qCBbsx1efVZAlC38/s320/2023-04-17%2014.08.57.jpg" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyKLddk2cVVAm2XZB4Z8ER5H-MXY_92Qw0vOIEx-NyDPlgkYmCnlo4P8rMOPzo8NuXRGU8CT6Ipr05ySkcLxb4dqjPJMnpf3Gn8IxVVPmyQEA2mjvxmJKXAL1lqaz2re25ei8Pt8KfyV7sXSsva-moRPkptDyCMtx87_t7bAQyk-QSG_00M_At2IIp/s660/GUID-83827C27-64A5-4D65-9AD7-A6692D2C05EC-low.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="414" data-original-width="660" height="201" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyKLddk2cVVAm2XZB4Z8ER5H-MXY_92Qw0vOIEx-NyDPlgkYmCnlo4P8rMOPzo8NuXRGU8CT6Ipr05ySkcLxb4dqjPJMnpf3Gn8IxVVPmyQEA2mjvxmJKXAL1lqaz2re25ei8Pt8KfyV7sXSsva-moRPkptDyCMtx87_t7bAQyk-QSG_00M_At2IIp/s320/GUID-83827C27-64A5-4D65-9AD7-A6692D2C05EC-low.jpg" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXJ3A-s_KoljsDb6K8X6g6gl1ySlq8rEIoTiRBMCvCNS4KQ_hthAsaPKBqPuSLUi9fZ-rC1OK9_YcPr_UMMZ4gpnY5Ul0C9WaZohXr-yVTVWGzVFMlHLgwHUYSL7v-KlGik3fhykVKH4CGP_Hhjr8dtgQmvIDgWGjY7LfwH9gZLvt_BG-My-mhpu1z/s660/GUID-A982927A-9F1E-4C60-8AE2-C85C7B7C714F-low.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="429" data-original-width="660" height="208" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXJ3A-s_KoljsDb6K8X6g6gl1ySlq8rEIoTiRBMCvCNS4KQ_hthAsaPKBqPuSLUi9fZ-rC1OK9_YcPr_UMMZ4gpnY5Ul0C9WaZohXr-yVTVWGzVFMlHLgwHUYSL7v-KlGik3fhykVKH4CGP_Hhjr8dtgQmvIDgWGjY7LfwH9gZLvt_BG-My-mhpu1z/s320/GUID-A982927A-9F1E-4C60-8AE2-C85C7B7C714F-low.jpg" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;">This server has a Perc H710 mini monolithic RAID card with 512mb of battery backed cache. TrueNAS works much better using an actual hard drive controller, not a RAID card where the OS officiates the disk info. It appears that the r710 does not support non-RAID-ed disks. I attempted to pull out the RAID card, to see if the machine would see the drives; it would not. On a previous build (a Dell r320) I removed the RAID card moved the cable to to a different mini-SAS-8087 36 pin connector, thus the system ran the drives off of the onboard SATA controller. This Perc H710 uses a SAS-8654 cable? I didn't have a one into two SAS sas-8087 to go from the single onboard connector to the two back plane connectors.</div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;">Well, time to gamble a bit and attempt to flash the Perc into "IT-mode"...basically re-writing the firmware to make it think it is a some sort of LSI 2208 series SAS HBA controller. The H710 does have a faster processor than H310, and with spinning disk one probably would not max out the controller; with SSD's getting to that ceiling is a real possibility. It turns out there is not one but two different versions of the H710, one is PCI-e 2.0 vs 3.0 "capable". If one searches the web for great resource: "Art of the Server", he has lots of great information on these.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-hEAkfIv4lA7xZ6CzePNbMqTD7nJOUmwo4vBhc0GtoyVdcoLNJ3mG-1TKb7_eEVO4eIM5Jr_T4k7Pgn4RnNtU67XHUHAIncOPKkvpMvz78DNfy5atUeQw1-NHmV9SBJEOYnqhXyC3zk-HtaiBTlfMyDGh_z8YgjbllPLO5D-v6iVRgVYIrjVsLnTj/s4032/2023-04-17%2018.48.43.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4032" data-original-width="3024" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-hEAkfIv4lA7xZ6CzePNbMqTD7nJOUmwo4vBhc0GtoyVdcoLNJ3mG-1TKb7_eEVO4eIM5Jr_T4k7Pgn4RnNtU67XHUHAIncOPKkvpMvz78DNfy5atUeQw1-NHmV9SBJEOYnqhXyC3zk-HtaiBTlfMyDGh_z8YgjbllPLO5D-v6iVRgVYIrjVsLnTj/w150-h200/2023-04-17%2018.48.43.jpg" width="150" /></a></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFttAPtQb-Ju1dsWzc804MpIg9v83korLZRlshTBzJTU2GAro3-ZVX_6bZIyGpQQJndxTMNNydy5_rxMUECg60L6SbjjcooXX0VCknIdNjvMcvt4IX_ukd44hpLrn-Kg_m04h0P-yuNFloGQudE6ezJimCUYetHwGJmFmD1jtpwtYa_TWdAKeIvzS3/s4032/2023-04-17%2018.48.49.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4032" data-original-width="3024" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFttAPtQb-Ju1dsWzc804MpIg9v83korLZRlshTBzJTU2GAro3-ZVX_6bZIyGpQQJndxTMNNydy5_rxMUECg60L6SbjjcooXX0VCknIdNjvMcvt4IX_ukd44hpLrn-Kg_m04h0P-yuNFloGQudE6ezJimCUYetHwGJmFmD1jtpwtYa_TWdAKeIvzS3/w150-h200/2023-04-17%2018.48.49.jpg" width="150" /></a></div><div class="separator" style="clear: both; text-align: center;">Following the steps <a href="https://fohdeesha.com/docs/perc.html" target="_blank">here on Jon Fohdeesha's site</a> the BIOS changes were made. The RAID battery removed, the drives removed, the ROM on the Perc was erased, via the FreeDOS utility, although I did erase it three times as it would error out on the 2nd step. The new firmware was then written, the SAS identifier reprogramed. All was successful! During post, one now sees the LSI/Avago BIOS and menu screen. Also interesting is that the Dell "System setup" also see the HBA.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJ9fXGabdrzqO5ZwJx9AM7RoN7UIPbtIWUaHoeBf56_ngAMREt_87ya7jbt96x8I8E63H-6KXzKcEIeGjT97JLYuIoABP7D7JSWuP4jxRsqxOq2myRo5yVuKNB2DZI5b0j64d9hlAXTCZD0KkuMnxOCukYCS_caz2WmzSz7Kj1lHy6jkIji_7LRG48/s4032/2023-04-18%2015.49.57.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3024" data-original-width="4032" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJ9fXGabdrzqO5ZwJx9AM7RoN7UIPbtIWUaHoeBf56_ngAMREt_87ya7jbt96x8I8E63H-6KXzKcEIeGjT97JLYuIoABP7D7JSWuP4jxRsqxOq2myRo5yVuKNB2DZI5b0j64d9hlAXTCZD0KkuMnxOCukYCS_caz2WmzSz7Kj1lHy6jkIji_7LRG48/s320/2023-04-18%2015.49.57.jpg" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><p><br /></p>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-64474265787035154162023-04-04T11:52:00.002-05:002023-04-10T12:15:09.106-05:00TrueNAS & NFS (sync vs async vs sync w/ zLOG)<p> </p><div class="separator" style="clear: both; text-align: center;">Another TrueNAS (core v13.0) build! This one is mostly just a backup repository....or as I call them: "here, hold this, server". This is a <a href="https://www.supermicro.com/products/system/2U/6029/SSG-6029P-E1CR24L.cfm" target="_blank">SuperMicro SM SGG-6029p-e1cr24l</a>....which is a pretty crazy server that in a 2U configuration holds twenty-four 3.5" hard drives! Plus rear mounted two 2.5" hard drives. This one has dual Xeon 3104 CPU's at 1.7ghz, 64gb DDR4 ECC ram, a 64 SATA DOM, two Intel 480gb SSD's and soon to be 24 (currently only 22 are in service, the other two are out for warranty) Seagate 8tb 7200rpm SATA drives, driven by a SAS 9305-16i-IT controller. Network connections are the onboard 1gbps NIC which is shared for IMPI and Network (which is giving me grief, but won't go into that now) and a dual port 10gb NIC.</div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;">TrueNAS was installed onto the 64gb SSD DOM (Disk On Motherboard); yes certain people will get all bent out of shape about this. Yes the DOM has a limited lifespan, it could die as the endurance isn't as high as a normal SSD. We also don't have mirrored drives for the boot OS. Well, you know what? I don't care! TrueNAS doesn't really write that much to the boot volume, especially if since most of the OS is copied to RAM. If the drive fails, just reinstall TrueNAS, import the config file. Plus this is just an additional backup target, if it goes down for a day or so, no big deal.</div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;">The data volume was created with five groups of four drives in a ZFS RAIDz2. So basically a twenty drive RAID 10. NFS v3 was setup, and presented to a VMware host. A Windows 10 VM was migrated to it and the ATTO Disk Benchmark was ran. </div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifrpVAyZwwDpcFsv_h2raYUJWsfimK6blP2zGVx7Gs-rlMB8H-l-Uhe1kEya8ZQRxrR7Xsd4Cq1ChLcp5nlAF5auHokepfyAI2gR6QymmQHfchT58SNgnas-iIZAUBBN9h_VvrXHyVyBnvzty7C-tRueCIl7i8GnRljZfPmVEGf_b4eyXxTXt_DcRB/s749/TrueNAS%20Core%2020%20drive%20Sea%208tb%20NFS.JPG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="749" data-original-width="735" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifrpVAyZwwDpcFsv_h2raYUJWsfimK6blP2zGVx7Gs-rlMB8H-l-Uhe1kEya8ZQRxrR7Xsd4Cq1ChLcp5nlAF5auHokepfyAI2gR6QymmQHfchT58SNgnas-iIZAUBBN9h_VvrXHyVyBnvzty7C-tRueCIl7i8GnRljZfPmVEGf_b4eyXxTXt_DcRB/s320/TrueNAS%20Core%2020%20drive%20Sea%208tb%20NFS.JPG" width="314" /></a></div><div class="separator" style="clear: both; text-align: center;">This first the first test, but notice the frankly unacceptable write speed? NFS has what is commonly referred to as "synchronous writes" turned on by default. In short, the each disk write must be confirmed to be written to disk before continuing. This options can be turned off, however there is a risk. In the event of power loss or other disaster the data in cache might not be written, and therefore lost.</div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgMLJkVAoztYxHQc1fATWi9kTlV-z0wrwtaaT65P24Y3xs3FFgK4GAh14Cj8W_tYUAwShMaTKAbqH-rQhwLLOY0DjqyORwDFSDZeTKD1GjiUGWiTXp5CWnfyIMCkuxzwpiRvaj2W2JIaxPWwaj3wU2uWIjCYaqlzy80b5HwgkdMgYz4zJyCALUm17Va" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="354" data-original-width="1391" height="81" src="https://blogger.googleusercontent.com/img/a/AVvXsEgMLJkVAoztYxHQc1fATWi9kTlV-z0wrwtaaT65P24Y3xs3FFgK4GAh14Cj8W_tYUAwShMaTKAbqH-rQhwLLOY0DjqyORwDFSDZeTKD1GjiUGWiTXp5CWnfyIMCkuxzwpiRvaj2W2JIaxPWwaj3wU2uWIjCYaqlzy80b5HwgkdMgYz4zJyCALUm17Va" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;">sync or async options can be set at the dataset level.</div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgd2qesToAhnp0jmiOTMIxRQI2DK1vey5kw3s7lmU9IUI4W5IoYympWzRL33Oxu56X5C9QCJJ62104IIIGvLS9-ceAIR5BtEmawI50AFvZzYcEy3IOF8q3wlvbT4ddSP2h4ID_2csr2l9zPTTspOenaY9M-06l_NC31AgElTZcbBccdhmv4K5uPsohD/s676/TrueNAS%20Core%2020%20drive%20Sea%208tb%20NFS%20sync%20disabled.JPG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="676" data-original-width="460" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgd2qesToAhnp0jmiOTMIxRQI2DK1vey5kw3s7lmU9IUI4W5IoYympWzRL33Oxu56X5C9QCJJ62104IIIGvLS9-ceAIR5BtEmawI50AFvZzYcEy3IOF8q3wlvbT4ddSP2h4ID_2csr2l9zPTTspOenaY9M-06l_NC31AgElTZcbBccdhmv4K5uPsohD/s320/TrueNAS%20Core%2020%20drive%20Sea%208tb%20NFS%20sync%20disabled.JPG" width="218" /></a></div><div class="separator" style="clear: both; text-align: center;">Same test with sync turned off; much better.</div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgyDiyjjcuSBd2ueOrNjejrFS38D4e7bq1kcuBpxl34uy3-jQYWg6oi57DmRPjMmsV4ntKLBWJyIYj124fAbQpBJObJguUB-wpBWYM6-8Pie_Bv61TnUMA6svrmfh8mENyGkd_t8fYePbZ8fVpRndrEhDJNSw7spZHma_woJ1-zOS8Sy8DougeNdcIW/s671/TrueNAS%20Core%2020%20drive%20Sea%208tb%20NFS%20sync%20enabled%20%20NVMe%20zlog.JPG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="671" data-original-width="449" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgyDiyjjcuSBd2ueOrNjejrFS38D4e7bq1kcuBpxl34uy3-jQYWg6oi57DmRPjMmsV4ntKLBWJyIYj124fAbQpBJObJguUB-wpBWYM6-8Pie_Bv61TnUMA6svrmfh8mENyGkd_t8fYePbZ8fVpRndrEhDJNSw7spZHma_woJ1-zOS8Sy8DougeNdcIW/s320/TrueNAS%20Core%2020%20drive%20Sea%208tb%20NFS%20sync%20enabled%20%20NVMe%20zlog.JPG" width="214" /></a></div><div class="separator" style="clear: both; text-align: center;">For this test sync was turned back on and NVMe drive was added and configured to be zLOG vDev. Essentially a write cache buffer. ZFS/TrueNAS will flush the log device every 5 seconds, and with a 10Gbps NIC, doing back of the napkin math that means the system will only use about 6gb of space......yes only 6gb, so the 512gb NVMe is largely being unused. TrueNAS requires an entire disk to be allocated. So this would be a great use case for NVMe name spaces. </div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div>There is NO benefit of running a zLOG on a pool that has Sync turned off.<br /><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><p></p>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-91896725758332454662023-03-07T19:36:00.003-06:002023-03-09T12:08:03.088-06:00SuperMicro BigTwin SSG-2029-DN2R24L...and VMware ESXi Part#2<p> Decided to do a little bit of network benchmarking. </p><p>Both nodes have a VM setup on them, both VMs live on NVMe.</p><p>First test is both VMs talking to each other over the built in 10gb NICs. In the part #1 it was mentioned that this server has network backplane for the two nodes to communicate, using Intel 10gb NIC hardware.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicrRQjHuOlFqXe127rBgMDq1OwZ-xrQcuzB-U4gmrKShh6T0rRED7lFPvh-RxEywi9tVz0brg5uul05hnrt3ZiNyLR6DEwcigXFpO_OAv2jY6R2915m7i2PkSanycOhm4gANPFcBD6FXUXYUpQNVyLrRQIpH9eySiHwpwet_qBo_LJSqPgVtXP7SfB/s724/thruput%20test%20on%20internal%20nics.JPG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="557" data-original-width="724" height="493" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicrRQjHuOlFqXe127rBgMDq1OwZ-xrQcuzB-U4gmrKShh6T0rRED7lFPvh-RxEywi9tVz0brg5uul05hnrt3ZiNyLR6DEwcigXFpO_OAv2jY6R2915m7i2PkSanycOhm4gANPFcBD6FXUXYUpQNVyLrRQIpH9eySiHwpwet_qBo_LJSqPgVtXP7SfB/w640-h493/thruput%20test%20on%20internal%20nics.JPG" width="640" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;">Second test is both VMs talking to each other using a virtual switch that has a 100mb NIC.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgN6uX6BgHv-nldvVLEsHmigEuEowLNi7yv5N-cWbE1SZ9serJVA7OAW3T6p7blm6rkupLmQhwMPvFdcFgR9p-FGqO3g1SFPHxrBspWl86RgvsZ3pI6yM-lpFbwvZuU_Tc4sO2VTEtEwm1JpWyjXpBV1XuceAqUgdfK2PR2MIsow3OSwYxQ16iTTPB3/s664/thruput%20test%20on%20100gb%20nics.JPG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="510" data-original-width="664" height="493" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgN6uX6BgHv-nldvVLEsHmigEuEowLNi7yv5N-cWbE1SZ9serJVA7OAW3T6p7blm6rkupLmQhwMPvFdcFgR9p-FGqO3g1SFPHxrBspWl86RgvsZ3pI6yM-lpFbwvZuU_Tc4sO2VTEtEwm1JpWyjXpBV1XuceAqUgdfK2PR2MIsow3OSwYxQ16iTTPB3/w640-h493/thruput%20test%20on%20100gb%20nics.JPG" width="640" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;">Third test is both VMs talking to each other using a virtual switch has a 10mb NIC.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmvJQQLcGiSzpwDy4KN895mFZ5HQKmayDvl63sTQjbCk5SjLRmyh-BLd3rjkZiEUHvD9Ig9hNcGVNC4E3DuhjjAQVaYCd3hsPBOfSog2oONEtyWC3-krzAldExtpfc26SLV_btL54ENu1A2HteW7qoLR2iPZoMgmkjgCUCNTtvppT9hVHHM6tDT6U3/s684/thruput%20test%20on%2010gb%20nics.JPG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="536" data-original-width="684" height="502" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmvJQQLcGiSzpwDy4KN895mFZ5HQKmayDvl63sTQjbCk5SjLRmyh-BLd3rjkZiEUHvD9Ig9hNcGVNC4E3DuhjjAQVaYCd3hsPBOfSog2oONEtyWC3-krzAldExtpfc26SLV_btL54ENu1A2HteW7qoLR2iPZoMgmkjgCUCNTtvppT9hVHHM6tDT6U3/w640-h502/thruput%20test%20on%2010gb%20nics.JPG" width="640" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;">Fourth test: just add some perspective. This is two VM's each running on HP ML350's G8's connected via 10gb NICs on a really old Broadcomm switch.</div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFItw8EBZmqH98SvIP08v0ppCu_KVmSWzYxrJb1VZhrN6sQJ7xDFz8NvlipuUkNHivplW4Ta9jV59eROoCy7hJglFH2zmwobUlvRig2bAfL3AmGha99cYsN0Fwp2EEqLdY9vrua3DM6sIx6EcK2x-VQXjPcd-_Gpxu8S2dBErX_dZ1p_VoSe5GJxkh/s686/thruput%20test%20on%2010gb%20nics%20aaron-g8.JPG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="549" data-original-width="686" height="512" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFItw8EBZmqH98SvIP08v0ppCu_KVmSWzYxrJb1VZhrN6sQJ7xDFz8NvlipuUkNHivplW4Ta9jV59eROoCy7hJglFH2zmwobUlvRig2bAfL3AmGha99cYsN0Fwp2EEqLdY9vrua3DM6sIx6EcK2x-VQXjPcd-_Gpxu8S2dBErX_dZ1p_VoSe5GJxkh/w640-h512/thruput%20test%20on%2010gb%20nics%20aaron-g8.JPG" width="640" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><br /></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><p><br /></p>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-25615484027925610002023-02-22T17:26:00.004-06:002023-03-09T12:09:31.471-06:00SuperMicro BigTwin SSG-2029-DN2R24L...and VMware ESXi Part#1<p><br /></p><p>This 2U server holds two compute nodes and has a <b><u>shared </u></b>backplane to 24 U.2 dual path NVMe drives.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6aa472Q5NC3PFRGZWan0DoxW5Z1IAq91eMcDRbkKj8vuCe7HbMR3PnDHmBuD9sSBWgJQLYS-yWB1VP3wjSAmLuMGs5u-8cPVjjmANRi-ZxJQ8b-HYpDnZ3m86KRGZqKdfRPOi217ow1efJ-Rhqko4gxMmumb2cOvWooqZllgIAyB0UwMHNmpKLoOH/s550/2029P-DN2R24L_top.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="363" data-original-width="550" height="264" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6aa472Q5NC3PFRGZWan0DoxW5Z1IAq91eMcDRbkKj8vuCe7HbMR3PnDHmBuD9sSBWgJQLYS-yWB1VP3wjSAmLuMGs5u-8cPVjjmANRi-ZxJQ8b-HYpDnZ3m86KRGZqKdfRPOi217ow1efJ-Rhqko4gxMmumb2cOvWooqZllgIAyB0UwMHNmpKLoOH/w400-h264/2029P-DN2R24L_top.jpg" width="400" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFnwwvIOxP0S1epM_i627hKIOWsgEeihjBbVzQj1m6JSQ-Gj2fOZwnSNiQDo0-4g89K9EWUR5RF3awPZ7KWncRpvY872l00p-klLKDsgwsmvX51XKxHYdyY8Oitr6TvTuf_f_sRW3AGu5CK1e-aEsEFZVQWynK4z2q_pgaENBGc7sDNNCXMIzqg-kt/s550/2029P-DN2R24Lback.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="152" data-original-width="550" height="110" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFnwwvIOxP0S1epM_i627hKIOWsgEeihjBbVzQj1m6JSQ-Gj2fOZwnSNiQDo0-4g89K9EWUR5RF3awPZ7KWncRpvY872l00p-klLKDsgwsmvX51XKxHYdyY8Oitr6TvTuf_f_sRW3AGu5CK1e-aEsEFZVQWynK4z2q_pgaENBGc7sDNNCXMIzqg-kt/w400-h110/2029P-DN2R24Lback.jpg" width="400" /></a></div><br /><p>Each compute node has a X11DSN-ts motherboard, containing a M.2 slot, twelve DDR4 memory slots, dual LGA-3647 sockets (this one has Xeon Gold 6140 CPU's). For networking they have two Intel 1gb/10gb RJ45 NICs plus two more Intel 10gb NICs that are strictly node-to-node communication.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhn6PvvuZ54Ye2-ix7u6W9Q09kKdI60tPLcoC8O2DmjkiYYQTqWjqHNDApo0jcHxYcNFmdfSXijUjAkBxRmk2GCsQhL1binnAYcxg8mYoCy6n3za_aifuqG51hM8XneZRoqHV5Zu-TDjvDMcxb1-u4B5aBMinOEmN0zkxlGuGa-5h62DlEM_lzXvI-s/s712/Intel%20onboard%20nics.JPG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="143" data-original-width="712" height="80" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhn6PvvuZ54Ye2-ix7u6W9Q09kKdI60tPLcoC8O2DmjkiYYQTqWjqHNDApo0jcHxYcNFmdfSXijUjAkBxRmk2GCsQhL1binnAYcxg8mYoCy6n3za_aifuqG51hM8XneZRoqHV5Zu-TDjvDMcxb1-u4B5aBMinOEmN0zkxlGuGa-5h62DlEM_lzXvI-s/w400-h80/Intel%20onboard%20nics.JPG" width="400" /></a></div><div><br /></div>This system poses some questions. How will VMware react to have having two hosts accessing the very same hard drives? Is VM failover possible with out vCenter? What will vCenter think about the shared datastore? How will storage vMotion work? And what about those internal NICs? <div><br /></div><div>ESXi 7 was installed onto the onboard M.2 NVMe drive. Virtual switches were setup, one for the added in PCI-e 100gb NICs, one for the two onboard 1/10gb NICs, and a third for the two internal NICs (the ones for node-to-node communication). A single VM was created on each host, and connected to the internal switch. It was confirmed that a VM on host-A could communicate to a VM on host-B.<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3FgsqnBVQ8blo7nleU0jFAlq7WEntdRKYq0I3PPWl3ZTCa6zTX-JVId2gtXYPQz0H4GiMehmcDlpnwQzeMWt1j9BEMKtJk-XzaHgT8YC07VFfN753jWvICBka_szqXtRANUNjrTOgXyzlVWrei6BOHY4Dy_yO-4B0qwyQya-y5j4AW6H1KOQDTf7j/s2175/VMs%20communicating%20over%20private%20network.JPG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="882" data-original-width="2175" height="261" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3FgsqnBVQ8blo7nleU0jFAlq7WEntdRKYq0I3PPWl3ZTCa6zTX-JVId2gtXYPQz0H4GiMehmcDlpnwQzeMWt1j9BEMKtJk-XzaHgT8YC07VFfN753jWvICBka_szqXtRANUNjrTOgXyzlVWrei6BOHY4Dy_yO-4B0qwyQya-y5j4AW6H1KOQDTf7j/w640-h261/VMs%20communicating%20over%20private%20network.JPG" width="640" /></a></div><br /><div><p>A datastore was created using one of the U.2 NVMe drives on host-A. Host-B didn't pick up this datastore until after a reboot. I am sure there is someway of refreshing how ESXi scans for new/existing datastores that are local. Additional disks to the datastore were added as an extent on host A. Host B automatically picked up the size increase, without having to reboot.</p><p>A VM on host-A was stood up; host-B could not register that VM until it was shutdown. After registering the VM on host B, host A did not seem to care or know anything was different, other than it still showed the VM as powered off. Rebooting host A shows those VMs as invalid. VMware file locking is taking affect in the background. After shutting down the VM, waiting a few minutes, refreshing the screen, that VM could not be powered on; on host-A. It appears the only way to bring the VM back to the original host is to unregister the VM and register it.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMxOMRxXqXtpjmioM7Q7PumMQ5w6dH8-q7olvs8_Rs8A1hAJAu5MZNIWQaX20mc7LIKJnv-XGamIxnMj_B5_IibiHd6LPTqA3PN8sA6A7W-OXRAuAMf0rQvyskZbhvF4hCZsGGF46Nwvhk9Blj87CWKqM_YZutgMv6QS8nq_t-zhOt6fq2lhdifC-i/s1042/vms%20regestered%20on%20opposite%20host.JPG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="189" data-original-width="1042" height="117" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMxOMRxXqXtpjmioM7Q7PumMQ5w6dH8-q7olvs8_Rs8A1hAJAu5MZNIWQaX20mc7LIKJnv-XGamIxnMj_B5_IibiHd6LPTqA3PN8sA6A7W-OXRAuAMf0rQvyskZbhvF4hCZsGGF46Nwvhk9Blj87CWKqM_YZutgMv6QS8nq_t-zhOt6fq2lhdifC-i/w640-h117/vms%20regestered%20on%20opposite%20host.JPG" width="640" /></a></div><br /><p><br /></p><p><br /></p></div></div>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-4136421400806985732023-02-22T16:49:00.004-06:002023-03-02T09:52:44.826-06:00HP Smart Array p420 RAID card notes<p style="text-align: center;"> A couple of quick notes...</p><p style="text-align: center;">The controller WILL support larger than 4tb drives. I have used both Seagate ST6000nm0034 6tb drives and Seagate Exos 12tb SAS drives.</p><p style="text-align: center;">The ROM utility will misreport the drives, but do not worry.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrB864P242GVSVG9Hf2WwJpy1vdgJfzA7Iqi-6ErW-s3dOJ28ccO6REO8OSt4aJeCcfG85E_1tcsebH5wJptZ1GuUg32ICkY6NJ4uMp4Ul8Ft-m1XXwZtToLoCokHZhtda0Kfuiny5eIvYdfM-tLIM5yJWePtGCwpKaMokyurgMlCkEjhnN7UA_eyM/s4032/2023-01-18%2020.35.11.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3024" data-original-width="4032" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrB864P242GVSVG9Hf2WwJpy1vdgJfzA7Iqi-6ErW-s3dOJ28ccO6REO8OSt4aJeCcfG85E_1tcsebH5wJptZ1GuUg32ICkY6NJ4uMp4Ul8Ft-m1XXwZtToLoCokHZhtda0Kfuiny5eIvYdfM-tLIM5yJWePtGCwpKaMokyurgMlCkEjhnN7UA_eyM/w400-h300/2023-01-18%2020.35.11.jpg" width="400" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;">ILO however will see the drive correctly.</div><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjJWZHVuZ7EbA-AgFdPN75fvR9i-K3bTO90A1vjA33fthhWh65uufaTeJePjWfDJfLpt3siXl0r5vR2b1tYxVxTwUsKD0_wQKnpIXzx9G4Aqxb_NCGJlye-DPzZLPs5OOBOk84srs1Iy0HL8IA1gEPdWmUI2wMgzDIbqvI2tdf2b-w7pIEWT6J65lu9" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1124" data-original-width="584" height="400" src="https://blogger.googleusercontent.com/img/a/AVvXsEjJWZHVuZ7EbA-AgFdPN75fvR9i-K3bTO90A1vjA33fthhWh65uufaTeJePjWfDJfLpt3siXl0r5vR2b1tYxVxTwUsKD0_wQKnpIXzx9G4Aqxb_NCGJlye-DPzZLPs5OOBOk84srs1Iy0HL8IA1gEPdWmUI2wMgzDIbqvI2tdf2b-w7pIEWT6J65lu9=w209-h400" width="209" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;">HP Storage Administrator also reports the drives correctly.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEh45vx8jjbkFoa6_bGK6UStYugKOAyDvVsdm_lLJmfqGamPlftYDA4jChqUCu4wAd0CbTHdX9Sj5K2xlxbMq0L8br7P-TYfMQV_bhBJmKHUj7ew4G8xuHnW1WkZq4pqmSdUz52kWcTdS8PFtVuAzE0dN3aMEXME8bLhtnip4SqJmp1rVzdekFFc5pMm" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="450" data-original-width="735" height="196" src="https://blogger.googleusercontent.com/img/a/AVvXsEh45vx8jjbkFoa6_bGK6UStYugKOAyDvVsdm_lLJmfqGamPlftYDA4jChqUCu4wAd0CbTHdX9Sj5K2xlxbMq0L8br7P-TYfMQV_bhBJmKHUj7ew4G8xuHnW1WkZq4pqmSdUz52kWcTdS8PFtVuAzE0dN3aMEXME8bLhtnip4SqJmp1rVzdekFFc5pMm" width="320" /></a></div><br /><br />If for some reason one needs to re-order the drives; the RAID card will figure it out. The RAID card will notify one that on boot that the drive order was changed and has been corrected. In my case I was originally running 3.5" drives bolted into the case, and not in the drive cage. Through a bunch of trades/migration I was able to acquire a drive cage. Somehow despite taking notes of the drive position on the SAS break out cable and matching that to the drive cage, it still wasn't correct. </div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgG70EnV0kyln9nI-pjSCi0ZgucSTLLwz7uQN-WCDowTTUxCD5bXTorHo5k5PzbKDYcReLk_AhndDqT80oYOuwMub_OxL-Sc3MwuA1MaYro9Awb-icRbSu9vIfTDTOMm-K5W6bOL-sRR6esytRcRUMVG6nvnZk5KghAW75DwdsmmcnLVYa1Q28rwpQa/s750/Capture.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="267" data-original-width="750" height="114" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgG70EnV0kyln9nI-pjSCi0ZgucSTLLwz7uQN-WCDowTTUxCD5bXTorHo5k5PzbKDYcReLk_AhndDqT80oYOuwMub_OxL-Sc3MwuA1MaYro9Awb-icRbSu9vIfTDTOMm-K5W6bOL-sRR6esytRcRUMVG6nvnZk5KghAW75DwdsmmcnLVYa1Q28rwpQa/s320/Capture.JPG" width="320" /></a></div><br /> </div><div class="separator" style="clear: both; text-align: center;">Dealing with large drives can be a PAIN! This is in now way unique to HP, but when one modifies the RAID structure, or has a drive failure; doing all the recalculation takes forever. In my case, migrating a RAID1 (mirror) of two12tb 7200rpm SAS drives to a three drive RAID5, took over 24hours, and that was just the 1st step. That is correct, there is two steps, one must first recalculate the array, then another recalculation for Logical drive, which also takes forever and a day. So this is another reason why it might be better to buy more drives of a smaller size vs. fewer and larger drives. Also keep in mind the type of RAID one is using, RAID-5 will take more processing than RAID-10 and RAID-1. RAID 6 is also deserves a good look at, where available. </div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><p><br /></p>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-5496739988084497782023-02-03T18:02:00.004-06:002023-02-06T17:40:09.011-06:00NVMe namespace notes:<p> Just a bunch of notes for my future self, as I am sure I will forget, and for anyone else who may benefit. </p><p>What is a namespace? Think of it just like a partition on a normal hard drive. Except the partitioning of the drive is done at the hard drive level not at the operating system level. Thus the operating system sees the name spaces as unique hard drives. IE if one had a 1tb NVMe, and setup four 256gb name spaces, Windows would think there was four 256gb NVMe drives in the system. <br /></p><p>FROM LINUX (PartedMagic was used):</p><p><b>nvme list </b><--show what nvme name spaces are present and the model of the drive</p><p><b>nvme id-ctrl /dev/nvme0 | grep mcap </b><--shows" tnvmcap (total NVM capacity) and unvmcap (unallocated NVM capacity) attributes.</p><p><b>nvme id-ctrl /dev/nvme0 | grep cntlid</b> <--show the controller(s) ID's</p><p> </p><p><i>DELETE NAMENSPACES</i> <br /></p><p><b>nvme detach-ns /dev/nvme0 –n 1</b> <b>–controllers=0</b> <-- detaches the namespace "1" from the controller "0" </p><p></p><p><b>nvme delete-ns /dev/nvme0 -n 1</b> <--deletes the namespace "1" from the controller "0"</p><p><b>nvme ns-rescan /dev/nvme0</b> <--rescans the drive "0"</p><p> </p><p><i>CREATE NAMESPACES </i><br /></p><p><b>nvme create-ns /dev/nvme0</b> <b>-s 26214387 -c 26214387 -b 4096</b> <--creates a 100gb name space, formatted to 4k<br /></p><p><b>nvme ns-rescan /dev/nvme0</b> <--rescan<br /><b> </b></p><p><b>nvme attach-ns /dev/nvme0</b> <b>-n 1 -c 0x1</b> <--attach the namespace to the controller<br /><b> </b></p><p><b>nvme ns-rescan /dev/nvme0</b> <--rescan</p><p><br /></p><p>Ignore this stuff....just notes to me...</p><p>nvme create-ns /dev/nvme0 -s 7499000000 -c 7499000000 -b 512</p><p>11,995,709,440</p><p>3,840,755,982,336<--cap</p><p>----4096--- <br /></p><p>1,500,295,305 <-cap/256 <br /></p><p>1,000,196,870 <-cap/384 ==too big</p><p>960,188,995<-cap/400== too big</p><p>936,769,751<=cap/410==3.84<br /></p><p>923258649<-cap/416== 3.78<br /></p><p>857311603 <-cap/448 ==3.51tb<br /></p><p>750,147,652 <-cap/512 == 3.08tb<br /></p><p>93,768,456<--cap/4096 == 384gb<br /></p><p>26,214,387<--example ==100gb</p><p>---512---</p><p></p><p>93,768,456 <--cap4096 ==48gb <br /></p><p>948,334,810 <--cap/410 ==485gb</p><p></p><p>1,500,295,305 <-cap/256 ==768gb<br /></p><p>3,800,000,000<---1950gb</p><p>5,000,000,000<---2560gb </p><p>6,500,000,000<---3330gb </p><p>7,500,000,000<--3840gb <br /></p><p><br /></p><p> <br /></p><p><code><br /></code></p><p><br /></p>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-64203164771729101832023-02-02T14:52:00.003-06:002024-03-07T12:07:39.474-06:00NVMe U.2 drives in machines with out U.2 backplanes<p> I just discovered these and figured it worth the share.</p><p>NVMe drives that are in 2.5" hard drive form factor (versus the more familiar M.2 "gum stick" variety), look just like a normal SAS hard drive. However the pinout is slightly different, and one needs to have different backplane and connections to the PCIe bus than SAS. IE one can not take a server physically configured for SAS/SATA drives and utilize these U.2 drives. </p><p>This is where this adapter comes in. Mount the drive to the PCIe card, plug the card into the computer, and one can now utilize that hard drive. Another feature of this specific card is one could use a normal 2.5" SATA hard drive on this card as well. One just needs to feed a SATA cable to the card. Both are very handy options to for instance adding a single 2.5" drive to a machine that is doesn't have any extra drive bays, or the bays are configured for 3.5".</p><p><br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLzM3A_H3VMOCpulMdHHvE0isO1uELAWWwQqKz3EhN2_r3VvTuSXFPmqd_PWu4cpqIrGYMUqaXPPcr1BkAwNQkjHJ9z4pixRE12pSAAQS-A4iQKVqA4kD8tA68Di-dtLhPT_ZhQ1ImRHoYvQ0fCO9suLXpfX1ADFmpYyeHz0ReJiSwMslj6cnEMz4Y/s4032/2023-01-31%2015.29.48.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4032" data-original-width="3024" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLzM3A_H3VMOCpulMdHHvE0isO1uELAWWwQqKz3EhN2_r3VvTuSXFPmqd_PWu4cpqIrGYMUqaXPPcr1BkAwNQkjHJ9z4pixRE12pSAAQS-A4iQKVqA4kD8tA68Di-dtLhPT_ZhQ1ImRHoYvQ0fCO9suLXpfX1ADFmpYyeHz0ReJiSwMslj6cnEMz4Y/s320/2023-01-31%2015.29.48.jpg" width="240" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCI1L6YjHCp5IO9f_lVu_mM3docH_Dl63pTX_EADvv9FGY1Lk2tn0b8BoIPOdvkyXDm-s0MxiJCodOmjMgsn03hsRwcmI6S9HUOZGU6rr3sg48QutIahgQDMttE3FyBbJnPVjgmVcqm07dpE9PkycYOXU-6glutlz2tHnvnpPaVpMH9YjLNJ6aS5i8/s4032/2023-01-31%2015.30.19.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3024" data-original-width="4032" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCI1L6YjHCp5IO9f_lVu_mM3docH_Dl63pTX_EADvv9FGY1Lk2tn0b8BoIPOdvkyXDm-s0MxiJCodOmjMgsn03hsRwcmI6S9HUOZGU6rr3sg48QutIahgQDMttE3FyBbJnPVjgmVcqm07dpE9PkycYOXU-6glutlz2tHnvnpPaVpMH9YjLNJ6aS5i8/s320/2023-01-31%2015.30.19.jpg" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2shOxqt6EnUdrEFxjgTpy1TaD5--Uqo8FDxeaswCOojgnPr8R004Pyx1SZW3oK4s3z6lG0rnTxawB3nxa2UjLOFOGm-DobprFpjFGtyoBQ3AErlEsu1FTKv8EcU_hFjOFKCWxr3fw_EzSv-bstltWjk7Nckp1vVQyf8EPFvJQCeAuHfdcUcBYGwpD/s1196/1.JPG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="588" data-original-width="1196" height="157" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2shOxqt6EnUdrEFxjgTpy1TaD5--Uqo8FDxeaswCOojgnPr8R004Pyx1SZW3oK4s3z6lG0rnTxawB3nxa2UjLOFOGm-DobprFpjFGtyoBQ3AErlEsu1FTKv8EcU_hFjOFKCWxr3fw_EzSv-bstltWjk7Nckp1vVQyf8EPFvJQCeAuHfdcUcBYGwpD/s320/1.JPG" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDAswmIZ3FeN4msyaabs2VYYOHj3PDza4h0mMQhcIQEB3K-3pprFiIl1VMDbk1XGIR3IIUHyehsHFUSpS8MzoPGjfabTHVev6VYbqadTutjwYdRs4-cI7h8pj0KqACpzV9BTnz2ckbaEeVH8DL1nytH1HMjDI_S0APA_Yu7dpMBplpuX3yEX76tz8P/s686/2.JPG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="610" data-original-width="686" height="285" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDAswmIZ3FeN4msyaabs2VYYOHj3PDza4h0mMQhcIQEB3K-3pprFiIl1VMDbk1XGIR3IIUHyehsHFUSpS8MzoPGjfabTHVev6VYbqadTutjwYdRs4-cI7h8pj0KqACpzV9BTnz2ckbaEeVH8DL1nytH1HMjDI_S0APA_Yu7dpMBplpuX3yEX76tz8P/s320/2.JPG" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYqwUfu5s-0MWmDlEwC2X0pgrGrUVJ5Q9brjkgd1IrC2A2V0FGEcUvutz-YqVzBRMqJcwcfbjkIJnbDRQAC_3P2GltmM3j689GkcNICK0YdDO6xanjgKXOIzhO9Dnsz8eWej9n0l0DEBIiYWN2E_uCmhrMGVD1Rk8pPBzxi75drM5FYI3kEj2XLJczpps/s1159/Udot2%20adapters.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="619" data-original-width="1159" height="171" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYqwUfu5s-0MWmDlEwC2X0pgrGrUVJ5Q9brjkgd1IrC2A2V0FGEcUvutz-YqVzBRMqJcwcfbjkIJnbDRQAC_3P2GltmM3j689GkcNICK0YdDO6xanjgKXOIzhO9Dnsz8eWej9n0l0DEBIiYWN2E_uCmhrMGVD1Rk8pPBzxi75drM5FYI3kEj2XLJczpps/s320/Udot2%20adapters.JPG" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><p><br /></p>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-34380043588449753842023-01-30T20:13:00.008-06:002023-07-13T17:35:57.048-05:00Cable Length is relative....<p> Turns out when one order 0.5m cable it might not actually be a half meter. In this case two cables one from Mellanox the other from FS.com. Obviously someone isn't right here. Apparently one vendor includes the length of the transceiver in their overall length vs just the cable. Anyways, buyer beware as it can make a difference. </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWyiOpZBQlvOIBgHEJ1QOiM6l9fN-HIe1A5QExtsOED_xJltsaEzC-12Q5-xq_K6bE0nkpQIGbouqxvH5yz165RguchqVVU8yYZX0E5u5gKrI5AX4E09TDN512UWp5nYllbqByfja8D0GQl82065j7yrGl6pPVBrAilWeCU46RtLL3dzGYaL6bWHUP/s4032/2023-01-30%2015.55.41.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4032" data-original-width="3024" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWyiOpZBQlvOIBgHEJ1QOiM6l9fN-HIe1A5QExtsOED_xJltsaEzC-12Q5-xq_K6bE0nkpQIGbouqxvH5yz165RguchqVVU8yYZX0E5u5gKrI5AX4E09TDN512UWp5nYllbqByfja8D0GQl82065j7yrGl6pPVBrAilWeCU46RtLL3dzGYaL6bWHUP/s320/2023-01-30%2015.55.41.jpg" width="240" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjG4OAdzM6JvelQl9T7U93NwWXMtnlV3y_VA4T5Wz0BulbFPDcnckv9UXGFeg0FgPXK1q2KWC9HO53Ploes77Llin-jT6ab0Ir_e3tUcY0BKTFinDqRmC2U_ytO9MuoSDZr5p7HVVSBCoZnboBMhPALN_p6EV21IXBwuvoRHOgXs9RWtDRPwJIUbjp1/s4032/2023-01-30%2015.55.49.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4032" data-original-width="3024" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjG4OAdzM6JvelQl9T7U93NwWXMtnlV3y_VA4T5Wz0BulbFPDcnckv9UXGFeg0FgPXK1q2KWC9HO53Ploes77Llin-jT6ab0Ir_e3tUcY0BKTFinDqRmC2U_ytO9MuoSDZr5p7HVVSBCoZnboBMhPALN_p6EV21IXBwuvoRHOgXs9RWtDRPwJIUbjp1/s320/2023-01-30%2015.55.49.jpg" width="240" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjACYABJeIWx474euYgOBicSKkN9HBuhAgR05rhd4cxBOmk9feW22oCAn0UKpP0GrRUVlD1efe2fo-7Z662kul8sdvelcqc-lu_MsswB4qGCatVtk8N-NDb6xS7YJjtvFHwwNhcIP1cvSpDEXR6AemDZENizSYHjFDnVAsbPJtg13SUR-xu4dBphF49/s4032/2023-01-30%2017.08.25.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4032" data-original-width="3024" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjACYABJeIWx474euYgOBicSKkN9HBuhAgR05rhd4cxBOmk9feW22oCAn0UKpP0GrRUVlD1efe2fo-7Z662kul8sdvelcqc-lu_MsswB4qGCatVtk8N-NDb6xS7YJjtvFHwwNhcIP1cvSpDEXR6AemDZENizSYHjFDnVAsbPJtg13SUR-xu4dBphF49/s320/2023-01-30%2017.08.25.jpg" width="240" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;">Those couple of inches make a difference, especially when doing 16 cables; in this case I resorted to 1M cables.....or should I say "1M"? :)</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQjMDbmoBL7kCwGd6dg-pQbs-3y1r_KrxXU0sig8RVBPPBO2JYnG9RrY7LLM3RYo7i_-zWDyNiESA32voEhXl-qnZPyzvAGXn4T0665861nSTwSMQUI_ZM4Hje7zn4xaVtUHU3NXFqa3-TD3D9ZkfNxjJtozreqL8aIuY-FvU7KWIcjBhG_n-Bdz9Q/s4032/2023-01-31%2012.01.03.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="4032" data-original-width="3024" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQjMDbmoBL7kCwGd6dg-pQbs-3y1r_KrxXU0sig8RVBPPBO2JYnG9RrY7LLM3RYo7i_-zWDyNiESA32voEhXl-qnZPyzvAGXn4T0665861nSTwSMQUI_ZM4Hje7zn4xaVtUHU3NXFqa3-TD3D9ZkfNxjJtozreqL8aIuY-FvU7KWIcjBhG_n-Bdz9Q/s320/2023-01-31%2012.01.03.jpg" width="240" /></a></div><div class="separator" style="clear: both; text-align: center;">Amphenol 1M cables are just over 39" tip to tip.</div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvYPKQokxvxCLOaj8EEvNFhr4tSuqWNE9dZicfVNmJZ3knouTeE4CECS6sto1jX_NDnGWT8u1civYs-ycMxB7un51AR9Is1sY4p7RFCm8okwS9vF43b5R9KDtlgGi8eymLzpBd1hhnN1XfkD6zQANptOgfnTazOVFb2_XdbyXCSnA77J00qayr68YIkBg/s4000/2023-07-13%2014.39.37.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="669" data-original-width="4000" height="68" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvYPKQokxvxCLOaj8EEvNFhr4tSuqWNE9dZicfVNmJZ3knouTeE4CECS6sto1jX_NDnGWT8u1civYs-ycMxB7un51AR9Is1sY4p7RFCm8okwS9vF43b5R9KDtlgGi8eymLzpBd1hhnN1XfkD6zQANptOgfnTazOVFb2_XdbyXCSnA77J00qayr68YIkBg/w400-h68/2023-07-13%2014.39.37.jpg" width="400" /></a></div><div class="separator" style="clear: both; text-align: center;">This FS 1M cable is 40" long, just the cable, add another 6" or so for the transceivers. </div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><p><br /></p>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-40832024323009127702023-01-16T23:51:00.003-06:002023-03-06T17:07:44.997-06:00vCenter Backup/Restore and "Upstream Server not found"<p><b>I learned somethings on Virtual Center 7.x this weekend. Here are a few key take aways that hopefully helps someone else out there:</b></p><p><b><br /></b></p><p><b>Restoring Virtual Center using Veeam</b></p><p>If one needs to restore a vCenter using Veeam; and assuming that Veeam was setup where the vCenter was added to Veeam, like most environments, versus having each ESXi host added to Veeam. Veeam cannot restore vCenter the same way one would normally do a VM restore. Since during the restore process Veeam NEEDS to communicate with the now non-existent vCenter. The restore cannot happen and the process will waste great amounts of time. </p><p>The work around is to first add a single ESXi host to Veeam, add that host using the IP (assuming it the friendly name is known by Veeam; the point is we are tricking Veeam this is a new ESXi server. Then restore the vCenter backup to an alternate location...aka this "new" server. One isn't really restoring the VM to an alternate location, one can totally restore over the top of the existing VM, this is just a necessary step to get past the requirment of having to circumvent communicating with a vCenter that is currently not functioning. </p><p><br /></p><p><b>Restoring Virtual Center using built-in restore options.</b></p><p>For those you who are using vCenter's built in backup feature, the restore process works as follows. Start the vCenter installation process, choose "Restore", give it the backup repository details, and follow the rest of the prompts. What happens is a new vCenter VM is deployed and the backup information is replayed into it. HOWEVER, the backup files and the install media need to be on the very same version. For me, I attempted to use vCenter 7.0.3-209 installaton media, and my broken vCenter started out life as a v7.0.1x , then upgraded several times and currently on 7.0.3.01000. It would not restore and complained about a versioning mismatch. I did not bother trying to acquire previous versions of vCenter to attempt the recovery. Apparently, one should keep the install media around, and when patching vCenter, also acquire the full ISO to install from. </p><p><br /></p><p><b>VCenter Error when logging in: "No Healthy Upstream Server"</b></p><p>For some unknown reason my vCenter quit working and I was presented with the no healthy upstream server message when attempting to log in. The error message is very vague and somewhat of a "catch all". Hence, there is no one golden fix out there, but a dozen different fixes. Many people give up and just re-deploy from scratch. </p><p>I could log into the appliance administration page, where I attempt to start the non-running services with no luck, restarting the appliance didn't work. </p><p>My server the "VMware vCenter Server" (VMware-vpxd service) would not start, it would process for a good two minutes before failing.</p><p>A few usefull commands to type in at the shell:</p><p><span style="background-color: whitesmoke; color: #34495e; font-family: Monaco, Consolas, "Andale Mono", "DejaVu Sans Mono", monospace; font-size: 15px; white-space: pre-wrap;">service-control --list <--list all of the running services</span></p><p><span style="background-color: whitesmoke; color: #34495e; font-family: Monaco, Consolas, "Andale Mono", "DejaVu Sans Mono", monospace; font-size: 15px; white-space: pre-wrap;">#service-control --start --all <--start all services
#service-control –-start {service-name} <--start a specific service</span></p><p>DNS both forward and backwards worked, resolution with the FQDN and short name worked. Disk space was not an issue, as mentioned by dozens of others. I did attempt to change my IP settings as recommended but some sort of GUI bug prevented me of doing that. Changing the IP via console did not help. NTP did not seem to be an issue. </p><p>An attempt to upgrade vCenter, thinking the upgrade would replace/repair any broken bits, was tried. This made things way worse, and the upgrade failed; so I had to roll back.</p><p>I downloaded and ran python script from a VMware KB to check my SSL certificates, and they passed. </p><p>The logs to vmware-vpxd are found here: /storage/log/vmware/vpxd-svcs/vpxd-svcs.log</p><p>After several hours of beating on it, I found several other pages talking about certs. and a different way to check for cert. valitity. This check showed mine as being failed. Turns out the .py script I ran previously, actually lead me astray, I did indeed have expired certs.</p><pre class="ckeditor_codeblock" style="background-color: white; box-sizing: border-box; color: #0e0e0e; font-family: sfmono-regular, menlo, monaco, consolas, "Liberation Mono", "Courier New", monospace; font-size: 14px; letter-spacing: 0.08px; margin-bottom: 1rem; margin-top: 0px; overflow: auto;">for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done</pre><p> Running the Cert. Tool contained within vCenter to reset all the Certs fixed my issue. FYI it does take quite a while to run.</p><p><span face="metropolislight, Arial, sans-serif" style="background-color: white; color: #565656; font-size: 15px;">/usr/lib/vmware-vmca/bin/certificate-manager <---to launch the cert. utility; #8 to reset the cert</span></p><p><br /></p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEj1H6wGUFSJO8Axot8lZhFF699uUlmqqMD0dGAjdRnyaVxowKteZHmEfnz9r8xGnmKggHOONG7tSAe3dDKDT_-TSKfG_DzwUYOACshyyr33YOZLT2sX8R7NIBk-Ef1dqzmnpBvGfyW9rTAb8RCYQiK_U5tf6IFqucQPlCSYALXuKTl91Ys7IE3uBG-o" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="581" data-original-width="785" height="237" src="https://blogger.googleusercontent.com/img/a/AVvXsEj1H6wGUFSJO8Axot8lZhFF699uUlmqqMD0dGAjdRnyaVxowKteZHmEfnz9r8xGnmKggHOONG7tSAe3dDKDT_-TSKfG_DzwUYOACshyyr33YOZLT2sX8R7NIBk-Ef1dqzmnpBvGfyW9rTAb8RCYQiK_U5tf6IFqucQPlCSYALXuKTl91Ys7IE3uBG-o" width="320" /></a></div><div><br /></div><br /><b>Post Recovery</b><div>After all is said and done, Veeam will be broken. When a backup job is attempted to be ran it will fail. Since the SSL certs have changed, as far as Veeam is concerned vCenter is not the same machine. To fix it go into Veeam:</div><div>-Backup Infrastructure->click the vCenter, choose properties->Next->accept the dialog box to update the Certificate->Finish<br /><p></p><p><br /></p></div>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-55008133684296521302023-01-12T23:15:00.005-06:002023-02-06T16:33:41.515-06:00VMware & NVMe drives<p>The VMware HCL (Hardware Compatibility List) has always been a problem for those of us on a tight IT budget; no matter if it is for a business or home-lab. Turns out the selective approval of the HCL also applies to NVMe drives. I have been a fan of using PCIe to NVMe adapters and a NVMe gum-stick drive in older hardware for higher IOPs loads, caching tier such as vSAN, or even just for swap space. See some of my previous posts on this subject.</p><p>My last home-lab upgrade, I had allocated a specific NVMe drive to use because of the price and speed. I couldn't get that drive to work. I assumed it was because it was a Generation 4 drive and was just too new to use on my older HP G8. An older enterprise drive was purchased second hand, that I knew from previous expirence would work </p><p>Months later, almost the exact same issue cropped up on a Dell PowerEdge r730. This time after some Google-Fu and reading <a href="https://williamlam.com/2019/05/quick-tip-crucial-nvme-ssd-not-recognized-by-esxi-6-7.html" target="_blank">William Lam's blog post</a> it turns out the that the drive I was attempting to use simply wasn't in the HCL list, and therefore ESX was not going to accept it.</p><p>ESXi would see the memory controller, but not address the drive:</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiE79rdXdSP1dFWDYfUnVQxHo9YpGTghHtR3Y5mdDQFh0BSsp30DWz9JiamsQUjDxn_2niPO6sdBWhhgfprzcBAiYUSdF28UyVgeibmSqvHYHhQWNJLHMSA5M9daLjv5hY5DOUA18PSTTC0PSp22hxPdcmBVUjPxyxrmVBPauWWyLXDL_niIQo4kNw_" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="407" data-original-width="1027" height="127" src="https://blogger.googleusercontent.com/img/a/AVvXsEiE79rdXdSP1dFWDYfUnVQxHo9YpGTghHtR3Y5mdDQFh0BSsp30DWz9JiamsQUjDxn_2niPO6sdBWhhgfprzcBAiYUSdF28UyVgeibmSqvHYHhQWNJLHMSA5M9daLjv5hY5DOUA18PSTTC0PSp22hxPdcmBVUjPxyxrmVBPauWWyLXDL_niIQo4kNw_" width="320" /></a></div><br /><p></p><p>Running this command one would not see the device:</p><p>"esxcli storage core path list"</p><p>Also running this command, one would not see the device:</p><p>"esxcfg-scsidevs -l"</p><p>Running this command one could see the memory controller:</p><p style="margin: 0px;">"esxcli hardware pci list"</p><p><span class="ui-provider cnl cnm c d e f g h i j k l m n o p q r s t cnn cno w x y z ab ac ae af ag ah ai aj ak" dir="ltr"><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Web", sans-serif" style="font-size: 14px; white-space: pre-wrap;">In my case it saw the </span></span>HP EX950 as:</p><p> <span face="-apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Web", sans-serif" style="background-color: #2f2f4a; color: white; font-size: 14px;"> </span><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Web", sans-serif" style="background-color: #2f2f4a; color: white; font-size: 14px;">VMkernel Name: vmhba5</span></p><p><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Web", sans-serif" style="background-color: #2f2f4a; color: white; font-size: 14px;"> Vendor Name: Silicon Motion, Inc.</span></p><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Web", sans-serif" style="background-color: #2f2f4a; color: white; font-size: 14px;"> Device Name: SM2262/SM2262EN SSD Controller</span><br style="-webkit-font-smoothing: antialiased; background-color: #2f2f4a; box-sizing: border-box; color: white; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Web", sans-serif; font-size: 14px;" /><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Web", sans-serif" style="background-color: #2f2f4a; color: white; font-size: 14px;"> Configured Owner: VMkernel</span><br style="-webkit-font-smoothing: antialiased; background-color: #2f2f4a; box-sizing: border-box; color: white; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Web", sans-serif; font-size: 14px;" /><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Web", sans-serif" style="background-color: #2f2f4a; color: white; font-size: 14px;"> Current Owner: VMkernel</span><br style="-webkit-font-smoothing: antialiased; background-color: #2f2f4a; box-sizing: border-box; color: white; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Web", sans-serif; font-size: 14px;" /><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Web", sans-serif" style="background-color: #2f2f4a; color: white; font-size: 14px;"> Vendor ID: 0x126f</span><br style="-webkit-font-smoothing: antialiased; background-color: #2f2f4a; box-sizing: border-box; color: white; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Web", sans-serif; font-size: 14px;" /><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Web", sans-serif" style="background-color: #2f2f4a; color: white; font-size: 14px;"> Device ID: 0x2262</span><br style="-webkit-font-smoothing: antialiased; background-color: #2f2f4a; box-sizing: border-box; color: white; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Web", sans-serif; font-size: 14px;" /><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Web", sans-serif" style="background-color: #2f2f4a; color: white; font-size: 14px;"> SubVendor ID: 0x126f</span><br style="-webkit-font-smoothing: antialiased; background-color: #2f2f4a; box-sizing: border-box; color: white; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Web", sans-serif; font-size: 14px;" /><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Web", sans-serif" style="background-color: #2f2f4a; color: white; font-size: 14px;"> SubDevice ID: 0x2262</span><br style="-webkit-font-smoothing: antialiased; background-color: #2f2f4a; box-sizing: border-box; color: white; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Web", sans-serif; font-size: 14px;" /><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Web", sans-serif" style="background-color: #2f2f4a; color: white; font-size: 14px;"> Device Class: 0x0108</span><br style="-webkit-font-smoothing: antialiased; background-color: #2f2f4a; box-sizing: border-box; color: white; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Web", sans-serif; font-size: 14px;" /><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Web", sans-serif" style="background-color: #2f2f4a; color: white; font-size: 14px;"> Device Class Name: Non-Volatile memory controller</span><p><br /></p><p>In the end the HP 950 NVMe drive was pulled out and a Kioxia KXG60ZNV256G which has a Toshiba brand controller was installed. Things worked as expected. </p><p>The forums did talk about replacing a driver with an older ond from ESXi v6.7.0, but I decided to avoid all that for fear of loosing access to the drive after various updates.</p><p><br /></p><p>ALSO! It turns out ESXi does not like drives that are formatted to 4k! I had a drive that ESXi would see, but could not provision as storage. Deleting the name space and recreating it formatted to 512b and it worked just fine.</p><p><br /></p>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-76296864167429368482023-01-12T22:38:00.003-06:002023-01-13T09:36:58.011-06:00RAID comparisons with SSD's<div class="separator" style="clear: both; text-align: center;">SSD are becoming as common as dirt, and almost as cheap, well atleast in low capacity variants. Which means even for low end business customers and home-labbers can start doing entire RAIDs with SSDs. I for one now have my primary VMware server spinning-disk free!</div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;">In this case my HP Proliant G8 ML350 with a Smart Array P420 and 2gb of cache. Currently with six Samsung PM863 1tb SSD's in a RAID5.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiktQLe_N3KxzqB9U-ezktT7UHOiZIDTzr6w6shSPNSM8w7bmL1iTErucrHnYQ_YYkN1vu9tJabYTMnGzcPkdFWCvvwf-iv92Yjvc5K7_kb1GIS7oEJTw4CTXyHC5En70S06ykQfwk_g_B8A0YuoyPHfoBuN7RhlvfaKME9WbTT7g4Gs0eznk-sMe_K/s640/atto%20ml350%20g8%20samsung%20pm863%20raid5.JPG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="640" data-original-width="468" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiktQLe_N3KxzqB9U-ezktT7UHOiZIDTzr6w6shSPNSM8w7bmL1iTErucrHnYQ_YYkN1vu9tJabYTMnGzcPkdFWCvvwf-iv92Yjvc5K7_kb1GIS7oEJTw4CTXyHC5En70S06ykQfwk_g_B8A0YuoyPHfoBuN7RhlvfaKME9WbTT7g4Gs0eznk-sMe_K/w293-h400/atto%20ml350%20g8%20samsung%20pm863%20raid5.JPG" width="293" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;">Just for comparison here is the same VM, same machine, just running on a Samsung Evo 970 NVMe 1tb drive.</div><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgwoq38reafCBUGLBApCKkKlJZvPrK_On0BvAW9p0gbkBwaEe7TShj86FWUrukTHCqvuRADvrEf3-k_LX7IygHRvkcN2tzogDVR5HLTPClMRhX11duxO7ccHKGJiUiuKXGhS1mnmGS2aUMPu2NGBn7M09xO04vvtdRijrmyvN673D0QS9SlHX-6mnJu" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="658" data-original-width="470" height="400" src="https://blogger.googleusercontent.com/img/a/AVvXsEgwoq38reafCBUGLBApCKkKlJZvPrK_On0BvAW9p0gbkBwaEe7TShj86FWUrukTHCqvuRADvrEf3-k_LX7IygHRvkcN2tzogDVR5HLTPClMRhX11duxO7ccHKGJiUiuKXGhS1mnmGS2aUMPu2NGBn7M09xO04vvtdRijrmyvN673D0QS9SlHX-6mnJu=w285-h400" width="285" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;">This one is a Dell PowerEdge r730 with the entry level Perc H310 Mini RAID card. Currently with eight Intel 400gb Enterprise SSD in a RAID 10 configuration. </div><div class="separator" style="clear: both; text-align: center;"><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAbwAAAJICAYAAAAaW1FNAAAgAElEQVR4nOy9y48kyZXu95n5I16ZldnVbHI46ItLQAuBG0IAF9zOYDCAlpy/oGfdqwvupgGJdySgR4uRIN0Bes/d7FiYpQACs9CmARECuCG0I3Sbr26yK6sqMyPCX6ZFlEdHefrDzM08zN3j+wFJVnu42zn+suP2Op+4u7tTsCRNUxRFgfV6jTiO8erVKyyXS6RpijAMoZS1iVHz4sULfP75577dGIyiKCClhBDinXuaJAmiKIKU0qi8Tz75BB9++CFevXrlxL8XL14Y7f/RRx85sz0HLv39LeFzNH9CF4WsVisIIfDmzZvjf2dZhsVigSRJjCvEqfLZZ5/1Om7sFYoQAsChYvzDH/6AOI5RFAXyPIeUEkVRaJf16aefDuKj7gfHJ598Moj9KcP39xv4HM0bJwEvSRKkaYrNZnP86s/zHEopLBYLpGnqwsxsKQPKWClbeFEUIQgCvP/++xBCIAgCo3J+9rOfDeThAZ0Pji+++KL199vb2yfb7u7uevukw+3trZaNuv10j20r8+uvv+b7e8IQz5Hr+9Zlu67s0+1d/s0VJwEvjuNjF8hiscBut0McxwCA/X5/UV+Ic+T0/hVFgTAMj62BJEkQhk4eo9EwpZf/tALr6zffX32EENo9Mqf3o3p/XDxjfe/5EB9OU6HxSb69vX3nr42rqysURYH9fo80TbFcLpHn+bHLi0yfLMve+f/y3s4t2Png7u7uyTtW99+uK6SyTL6/etj0xNTdY3J+wrYvlrYvlOp+WZYdvwqzLBt9N93QlOffZ3zO5tihKANb+f9CiGPwuxROK6xq11BdV1H1/anbbuNHWYnWtfJM7VXf3/fee6/2eNPzq16fpm1jo1ovuq7Pyntl+/zYdIdfItqf59WXi3TTN2AppS7+g8EnTYGtqaLp2qft2L7UHa/ja5P/TYFT5zx0trdtGytl0Dt9F4f4CLV5fsZ8/cZI7/6ouq/MthujU4kQMgZcPZNNX+hN+1bfo6b3SgeX3WdtPTtt9k59rwuuU3j3hw52VXTuW59nYQrX+hxYDcDoXsSmQHipN+HcL9HQtL2kl3qPq5z7Ouh2cw1try5gT6G36BwtuzrGfE3mgNMR6dM+6bogpzMBZu6U3STl3xy6LnVaAJdI07PetL0aCFwFhqYJMNU/U3+7tld7gZq2jZUyyLkIdi5b6SbXbewfFufmbFPsLvmit01CKbeNcaKKCdUKbG73W+f8mvY5x7WpdhXa2DM9D53tbdvGjM37aHoP+l5f2/ur698c0A54uhdW5yZc2leHzksz1UB3is+uqk8++aRzMbAObb63TRTpOl73muiUV+0e7Pq3ri86x7vYPuZ3/xzP0envts+PyfNh4t9caQ14rr5Km75QLi3w1WGykHUK+LifH3300dltThmOudZz7udozj0iY6Ux4Ol+odTtq/tVd4k32ddg+JxhAl8zLvG90+HczxHvw3kos0MBDQFPZwo16Y/rIGdb3hwmzhBCSBNFUSCO4/qAx4Dmhr7dlaYBiAGLEELqieMY2+32kArRtzNzxabVZXrsuVt4Q6seEDI0ptp3ZLpIKZEkCdbrNQPeHDhnC28oPTtCzg0nO7lh7PMQ8jxHGIaH//ftzFxwNZ157FD4kswJTniyZ+yKGqWeZ5ZlDHgu4JciIeRSGbtqShiGRx8Z8Bzg+yuRszQJIb4IgsC3C62kaYooiiCEYMCbA0MHLKUUpJSQUiJN0+MX09i7MuYCr7895frX8lqeSv+MfQxq7Iz9+gVB8I1gtW9nyDTI8/ydCkIIMfn8n1OC198P1dmcP/7xjz15QlzAgEc6ieP4mz7wt60LIQSKogDALtGh4fU/P2Wg+/zzz2t/Z+CbJgx4M2DoCm+/3yPPcwRBcOyvF0Icu4byPB/U/tzpun+8/uflxYsXx0D32WefvfPbxx9/DOAwUa1t7L4pT3CTKDbQnvBjTOoGU86BzIDnAN+LWM/RpVVWsMBhVlZRFMdutT5jSV0Vhgm+r78tOvdvzNd/rlSDXbnNxfKjuqChEyS79j0HUw12AAOeM5q6Ps5B2bU1FEophGEIKSW22y2Wy+WxxVGubzFhiLV8Pq+/LV33bwrXfy6Urbu6YFfy4Ycf4osvvsCHH37o7aNhykHHJwx4Dml7SaZMOUkiSRL87ne/w7Nnz/Dw8ADgUFmbTEseMlMLr383zJTjj6FaZU3Sa6fbqqK7bV2kbeVVNS/rxHzbpOB8i/8y4JFOyjGmKIoQhiGeP3+Om5sb4/U3zMHZD15/0kQ1iJ7+d53uaN3vfbY3ldvmT9O/zwkDHunkdBp8Od708PBwHEciw8LrP32mNNGjSx6u5DTA6ZzbGM6/cbT79vb2yd+Q2JTf5OPQPhNC5sWPf/xj/OhHPzrOxqzD9/jdkJwGMN0gNqV6tjHgnZ6w7sn3xebrp3qD6vqPfVB+kZ/+VX9vO0anzLnh+sXRuZ59t8/9Xlw6Tcng+8zQHKJ1N4YgUx3XmwKz69IcQ7O5pG66eVtlerp/U8ojnX1I9/V0EaR4L+bJj3/8Y9zc3AB4GuA+/PBDAPb5c+uCRFPd1bavzUd+3XEmwatuEsqYGh119Ap4bTN+dGblNG3rKq96jI6fbQ9Lm99DUVaK1QqXleWBruegL6bXd4gASaZDGdDKAFfd3sXp89oWyHRp27futyb7ff0yKUNn2+RmaTaddNesnK4ZQzqzjqp+dFWMdcG2yY7LG3FaSc45oLV9EZpeT53nwJbTfJRzwOX1J+/ie5yO99YtTro0m25KW4BqCy6m/cG6FWNdJTokfbu82vYdYxBt6gbx/ULaXKtqoua6ADmWezHW60/sGeoeXuqzYR3w+gSRuvUa1d9dU2fvHF2XpnQFxrGOG42x777uWrm8ZmO6F2O8/oSMjbMLapnO7OnaR7eMrgpgDLOMfFeaTVRnJwKobfG0dT2PidOZlW3dmmPRS5vb9SfEF9YtPNMpqU0TWZrK63p5dfev7tN0nKuxItOKcgwVqwtsrl21u3uIFsuYWmVDwGBHSDOdAa/vjJu2fbqO15350/Vb1xevqR0Tql/guhnxq2W07TO3yhrQm0WmQ5/r31ROE3O/F4QCsHNjduvwqvgczNcdi+vat2sf28p2DLMVh27Nmfyus50Bbt5QAHaezD7gXUIXzxgCli2XcJ/INHAhAAsMJ9rqKy9nXw2/MYnXzj7gXQK2enimAqLMuu8XXv/zYCMAOzbRVl+M7Tow4M2APorXfaGeml94/YelTgD2dGYsMA4BWNIPBjxHNCWbPQfnHMMbq1q2z+t/TsZ6/edKdUmIq7HbLt24kq7Uil3irzrbm8qpO0aHMbdkGfAc8NFHH/l24aLh9SdzoU9qxa59u7Y32Tx3ZqpzwIDnAHZr+IXXnwzF6fIWlzNzTwOKTWBxFYRcljPW1h3AgDcLbGdpcoo9IQfK5QYff/zxcRyv+n4MPX6nGzBctsDGHKRccvbUYsQ9eZ5b/RFC3sWlAGwTXTmFAT/diH1tjr11B7CFNwvC0O422i5rIGROuBCArUu5qBMMTFLqtaV11JmEcjquZysGa+Ij1+ERK2xbaXNYuE6IS2wFYAH9AGe6TSf93tDbXfjiAwY8B1Tz7ZFudDJVEOKbPs9oV6toTAHg0mDAc0RTzr1zMLVcmlxLRuaM74A2hZaWLxjwHFKXimgOlFOz0zTFb3/7WyyXS2RZhizLEEWRUcBlphBCiC8Y8EgnRVEgCAKEYYjVaoUPPvjgGOSCINAuhzkgCSE+YcAjnZRBTQiBLMsgpcTj4yOUUsjznOv4CCGToDHgDaUj5zt79hTWioyRPM8RBAHSNAUAJEkCIQSSJEEcx569I2QYKAA7LxoD3mnC0b4r/8eCL7/qFLHrJojU/dakdt61XUc1vQ/V8oUQiKIIUkqu4yOzgwKw84RdmgNRDU6n/60TzGwys7sq55RSgqgMeFmWATiM751TnoiQoXEhANu3J8t3D9jc6R3w2lbyt8lcuLBzmnS1zv4pTS3V6rGuHyrbINN0fF1gPAelzTK4hWEIKSVbeGS22AjAAv3qlKHqI3KgV8DTlbBo2seFnbp/tyVT7SrnnA9XV9flWDnNHA/gOGHl+fPnjcfwpSVTok4AtgoFYKfLoF2aLhKfNpVhU5GeuxKutsqauhx1AmEZdKr/X7XXVY4pQRBAKXVszQVBgP1+jyiKGnPuMdiRS6YrOXRb71X136f7d33UN4nFkgEDnivpiqnftLZgV2WsY3jl8oMwDI8Bb7vdvjM70yTpLSGXTlvvVUlXr5SpWCyZ2KSVqfVru5owUuW0dXcOTluM5RheFEUIwxBpmh5/15E7IeSSaHsnTBoBOq1E0k2vgNclJ1GOidko99aVoZO5u6lrTUd3ymVFrbOEYEqUrTshxDHzSpIkUEq9k22FwY5MmToB2Cqm43dDdvnzfTOjM+DZyEn0kb7os3/bRJnq9qZjh3hw6tbEVcfcTpcqmIy9tf3uegwvTdPjmrsy00ppJ47j4xIFQuZCKQBblQdyKQB7iosPbvaudOO9S9PlpJQxCQ3qjtXpHqN73BCtxyiKjv8uM62UQe7x8ZGZVsiscCEAW6Wut6qt90qnV4rj5uZ4D3gubxJv+DAURYGiKBCGITabDfb7PaSUx9Yd1+GRuWErAGvaM1W3ratXStcO+QbvAY+MHyEEwvDwqOx2O/zbv/0b7u/vURSFsTwQIVOCArDzImRlRbooJ6oopfDP//zPkFLi4eEBm80GaZp2phYLggBCCOR5jiRJjmv6lFKQUjJgksE5zQF7OpY+xLPHgDY+ynvNFp4jykHuPoy9ws/zHHEcI4oi/OQnP8Fms8FiscDr168Rx3Gn/w8PD1iv18dZnlEUHaWGVqsVJ70QQs4CA54DPvroI98uDEr5Nbzf7/HBBx8ct11fXx/zabZRzQjz+PgIAFgul9jtdsfuUkIIGRLWNA6wzad3rgTQfSmK4tiSe3x8hJQSaZri2bNn2O12nZNWlFJIkgSLxQJJkmC1WuHNmzfHiTCEEHIOxN3d3bj70y6AsXdpAt+sIVyv19hut4ii6LhEQaeFVwbJcsxOSon9fn8cByTknFSFXW2w/WCdwvs/JOe8fgx4I2DsLbxTpfMwDLHdbrHZbI6Br+uBC8MQSZIgz3Os12vs93ssFovjkoaxnz+ZH+U6OxfYBqxLf/7Pef0Y8EbA2L/wsixDGIZYLpfHLk0p5XHmZdcDV67jWywWEEIcg99yuUSWZRf/wpNpwxaeHee8fhxAGQFjr/CXyyWAw2zLIAjeEX3V9b0MllEUIQgCxHH8TuJpQqbKpQcsW855/djCI4QQYozLcdA28jxHnue4vr7Gw8PDO/l7hRBGH81s4RFCCOnF559/bl1GVwuvnN1dFAWyLDvOGygTWpikNmTAI4QQ0psmGSWX7Pd7fPnll4jjGHmeH8f+S6kyHT799FMGPEIIIeNmsVggCAJ85zvfeSKlpsPPfvYzAGzhEUII8UhXl2QZ3ErB6fv7e+R5fjxOt4UHMOARQsg7jEVXc0g/xiQW25W4oiQMw2PwK5dFBUGAPM+1bTHgEULIW+oCgc/gcGrX1o9zn4euGoXusoRyzC7P8+PSKN1gWWK2NyGEEC9UFc7HjFH2k7dLC9r+ABzz7papCsuEFiYw4BFCiCbVgHP637e3t8e/vvv38afOZlPZdf/d5u9YEEK8o72Z53mvxPPs0iSEEEuq3YVd3Yem+5uWYbq9/LeLLs+qHNiQtAXounNhwCOEEAeYto5ctKZsyhjLpBUbmrp5m86NAY8QQmowbXWZBhAXAWdsQUt3oopLqkGv7ZpwDI8QQmromiRi+lvfsk736eomHQOnE03OlRy+rsu2DrbwCCHkLXVB7rQSPf296d+nx5juX6Vtn7Yy2nwxGVs05bR1d+7Wno7fDHiEEHJC1zo8nX83laezv+7vujbrttcFT12bU4YBjxBC0N3lOPdg4Jox6gQy4BFCCObTupnLeQwBAx4hhJBJUKoe9IUBjxBCyOj59NNPrcsQd3d34+toJYQQMmpevHiBv/mbv/HthhEMeIQQQoy5ubnx7YIx7NIkhBBizKtXr3y7YAwzrRBCCLkIGPAIIYRcBAx4hBBCLgIGPEIIIRcBAx4hhJCLgAGPEELIRdAa8G5vb9/5OxelLRcKwmPRiCKEEOKXxnV4ddnB+2YMZ6ZxQgghvjHq0mTQIoQQMlV6Z1qpU9QtW3Knqr6n+9f9pqP627afTpdlm69V3xjUCSFknvQKeNXAUBc8qv+uU/ptK0fHXt32OnRsVPcjhBAyL5zn0hwqaHDyCSGEEBsmkzyarS9CCCE2GE1aGUsry4UfZZcox+0IIeQyaGzhnU4wOd1W91tXwGiaEKJbTtN+dT7W0WZDd3yPEELItLloAVgGOEIIuRwmM4bXh67WH4MdIYRcDrMOeAxohBBCSkIhBJR6t1fzxYsXntwhZH589NFHePXqlW83yIXyk5/8BHEcY7vdIooiKKWglEIURdjv9wiCwKp8IYQjT4ensYX3+eefaxdSDZiETIkhX9hPPvlksLIJ0UFKiX/5l39BGIZIkgRxHAMA0jRFFEWevTsvrV2an3322bn8IGR0fPHFF1bHf/rpp448IaQ/eZ4jDEP8/ve/x263g5QSi8UCaZpCCGH9wTeVBs8vfvGLeY/hEWLLhx9+2Ou4n/3sZ449IaQfcRxjt9thv9/ju9/9LpIkgVIKV1dXSJIEYWgXBqYS8ABHk1aKonBRDCFekLI9/wLH38iUSZIEi8UCq9UKDw8PkFKiKAoURYH1eo3dbmdV/izG8EzoqjAIIYT4oey2lFIiTVPc3NxASontdovtdms9aUUpZZSIxCdOAt6UmrSEVJnSFyohpsRxjDRNjy293W6HoigQRRHyPLfuoXvvvfe01HN0GDoZiJOAxwqDEELGSTkbM89zLBYLBEGAJEkADNM7N+YWHvsiCSFkxpRrrYMgQJ7nyPP8uE2nsaKUgpQSQghkWXYcA9TtCj3t7iwT9tdtq9vfNZylSYgBbS/jmL9sCelLURSI4xiPj4/YbDYQQiBJEuz3e4Rh2CkCUCf+DXQLhA8BAx4hBjQpdDDYkbmyXC6x3W6xWq3w+PiIJEnw/PlzAMDDwwOCINBWnfEtMceAR4ghpvJYhEyZ7XYLIQSCIIAQAre3t3h8fESaplgsFtrlVAOhj+DHMTxCelDXFUPIHImiCHEc482bN++M211fXwPw32ozgS08QnrCYEcugXKySjnD8/7+HmEY4tWrV1iv1609Hqfi3zqC3U1i4a5gwCOEENJIOcOzKApst1tsNhsopbBcLpGmKYD2j7+uCSltk1xcw4BHCCGkkaIojksY1us1kiRBkiTYbDa4ublpPG6MPSAMeIQQMnNO192dZlbRyZJ1qqhQKiwsFgtkWTbKoNYGAx4hLVD1gMyFX/ziF4OU6zvTlklqy1EEPN+5OH3fMN/YXv+5Xj/q2ZG58PHHHw9Wtm+1HJP0aI0B75NPPrEWwCRkylCtnMyFcnLJEEypwSJevXqlqg63DUQOwZQu2BxhC294qKlH5orv99+6S/PcL+eULtgcsb3+l379CLlkpvT+j2IML89zr/YvXcDW9vpf+vUjhEyDUQQ8W8VdW6b0hTIELhSPCSFk7Izi0zzLMoRhiDRNj5pN5ZoRKSWUUlZ/YRi+I3hYrklRSnmfYTQGbK8vIYQMxek6wLK+Od1mwihaeFEUHYNdlmWIogj7/R5XV1d4fHx00gI5VfpN0xRxHGO/32Oz2SDLMkdnMk04hkd8wudv2pxjDkYZ4KqL5ouisF+W8OLFC3sPDShbc2EYYrfbQUqJKIpwf3+PZ8+eYb/fW5W/2+1wfX2NoiiQJAmur6+x3W4RxzG22y3CcBRx3xu2rVyO4REb+PxNm6F7yU4V18uPm/KeF0VhFHAba/rPP//c0k19SkXdN2/eYLlcHiXkhRDI89y6hVcGtJcvX+K9997Dbrc7dsfFceziFAghhAxA2SACcBzuKgNgURRG8aG1afPZZ5/ZeWpAmqaIouj47yAIjlHcxTqxcpwQwDv/JlyHR/zC52/aDN2lfBrsdrsdsiyr7eLUYTS1fhRFx0AUhiGEENhut4iiyElwKluLUkqEYXhsObpoQU4dVhjEJ3z+ps257p+LBtAoOr/LKF2eUHkBV6uVk2BXNnuDIHhnlk+WZRcf7AghZMyUdXZRFEepIqBfoNWKJtWCdSPsqSRF9ZhqmeUMzXLWTVEUyPP82M1pg5QSu90Oy+XyGODKJQ9ln3Cb/66ou0F1N+/UZtN20/LPRdc1a3oWura1XYeu56vvNehzLn3LdnX/deVemsrU8UPHly6/XL9bTXar1Nns8k/33G2fwy5/68pquo5tdW9f+74o/ZJSHsfuTpcpOJm0UqV64V1ciLaHrDw5VyyXy0a756LpZair/OoeVp3r7qNiabNv+nsdbdfHZH9ThgyUbfvb3H9T36oVe5s9l76c65l09T60nbvL51A3mOqeh8m5djVKfFP2yp2uozb1sVdE6WOIvMvpAzuGr6gx03R9uPDdnrbr1/XbHOqAro+nOqr7+3gOT+uOrg+oPpz7Q/lcOJu04qILqau8pq/Rpu6H6jbT7p627V1dkdXjxvDw6Fwb3fMpf687/rR10vT72NB53k6p+8LX+QLvU8F2+Vv1uc7Hc1N3ftVuqK5WbFf36hifIx/oXs+mbWN9J014/fr1cSndj370Izx79qx2Pyd9hqdNTJOXueyLLf+qFU2f8pqOq97g037gpuNNtzf5qdutUr0OfehzTU2v++nvbfuadD1U/W7bx/XL2XXeTfe/+tu5gk2bP133r9xed53bfnPpX58ySn9Mu7Oa3ofTskq/xtBi7Xtv6oKdjQ9TCoB5nuOXv/wl9vs99vs9fvnLXzYmxHc2SNbnRTl9oE0ucNtXso4ffWy2+WJbQVT9simr7Zqa+mp7fUyO13kWXFyfJvrexz7PrW/agmKfD01bX6rBrKTuntT1GnSVf+5zsqHtHdD5KNM5v7ZrPkV+/etf4+Hh4fjfDw8P+PWvf127r7OA1zd4uebcfpja8/mAjeUejZFzXZvTCnzsla9vmu6Jy8Dl8oPVFy5bqlMLgF9++SV+85vfPNn+m9/8Bl9++eWT7b0CXtdFGerhqTbbu5rx536I+9rz8bLpdgeNgancxy6qlbePiqXt3EzO22XFqNMdV/1IcFWx92kB+ggKbd38dd3GXefhsgvUF0op/OpXv2r8/Ve/+tWT89OetNI2PuF6/KJvebrH1XXj1B3fVK6tn7rltJVv+rDq+Or6PrrC1C/T69ZVvmnlalqZmN7/Nn9s7nHXddC9By5aYDrnbvM+mOK6PmryV7du8hFwx1IfnNKWOKTuN62Ap9tX3rSt7TdTm3WVka4fruyaVr6m5Xdtb3r4+pybrn91v9f9u+memz4LJn7pltn20po8N01jSDbPmc1zYTKm1bWfqR8m5bRdH5PnuS4Qmz5XXWWb3Esde3397VP+6fah3xufCCHw13/910bHjCaX5php+uryhW/7U6XvdRvb/R+bP76Z2vnTX39cTMCz+UqZ0w0n5ozt/o/NH0KmwiiSRxNCCCFD0xnwdAefTada6wy6m2w3PY4QQshl4bxLs6u7hYGJEEKIDzpbeHVTjJtW+eugVHsWAVM/mmZP1v03IYSQy8X5GF61i3HqWQwIIYTMAzn0Ys2y5eY7Lx8hhJDLRquFVxdIdAKL64DDgEYIIaQvs1uWUAZFBkNCCCGnaAe8aiDxFVgY0AghhPRhdi08gJkoCCGEPMUo4JkkfO2T5Lltu67dPuURQgiZP7Ns4RFCCCFVLibgsaVHCCGXzcUEPEIIIZcNAx4hhJCLgAGPEELIRXAxArCEEELmg1IK//7v/46Hh4fa3zebDf7qr/7qnbSWbOERQgiZJHmeG/3GgEcIIWRyCCHwgx/8oPH3H/zgB09ECxjwCCGETJJvf/vb+N73vvdk+/e+9z18+9vffrKdAY8QQshk+f73v4/NZnP8781mg+9///u1+zLgEUIIGTWniUPKf0spIaVEEAT44Q9/iMVigcVigR/+8IcIgqC2HM7SJIQQMmqqOqhKKeR5DqUUpJR49uwZ/vZv/7azHAY8Qggho6UMcFJKCCEgpewtE8eARwghZNQURQEhBIqiAIAnuqy6OAl4TMxMpozJCzNH+P6SMSOEQBgeQlXZuiv/THES8C69wiBkyvD9JWMny7J3ujKl/Ga+pUngawx4n3zyCb744gutQspmJiFT5PTluUT4/pIxE0URsixDFEXY7XbH1l4fxN3d3ZPweHNzY+UgIYQQ4oqiKCClPP5/X2pD5atXr4wKYZcImTKXPobF95dMGSddmiawS4RMmUuv8Pn+kilz9lmalz4GQqbNpbfw+P6SKcMWHiEGnH4h3t7ePvn97u7Oma2hyy9tmJTJ95dMGbbwCDHg9AuxDBSmQaNK2/HV7ba2+vhwCt9fMmXO3sK79C4hQqYM319yKfDTjhADbm9vj39126vbhrShY6+6f1+fCJkDtS28Fy9eGBXCL0QyZf7u7/5Oa79qF2H536fbT//d1J1YDTp1Zera6KLumNN/v3jxgu8vmTTGY3hCCAghkGXZ8eCPPvoIeZ436goRMhdM1p02TTopg4hOIOoaw3NhQ/f4jz76yLgsQqZGnuf413/910PAK1evl4GvTMx5f39/XN1OCGlutVVbYWO0UXe8aZIJQqZGmYMzSZJvAl4Z7MrAV24vfyPzxff99d2l1vf8q92NLoOeqY2usbm2431f/0tn6u/f2P3Psuz472OXZnlgebAQAkEQvKNBRMgQ+H5hdCmDxel/V7dXt5kGvz426o6r+lB3fMlUrj8Zhqnf/y7/y2E5KeUh4JWtuqIo3tEaKmYSigAAACAASURBVAPd1C8IIaborqFr2953W18bdZNlmv5NyKWQ5/kxvtWO4Z12a5bS6mS++O7S8v1B5fv8feP7+l86vp8/2/s/dv/LYbljCw94tzuzDHDlYB+7NOeN7wrP9/Pl+/x94/v6Xzq+nz/b+z92/6WUx78Q+KaPM8syFEVxHOQrC2ILb974/kLz/Xz5Pn/f+L7+l47v58/2/o/d/3LyZZ7nh4BXtu7CMHzHeaUUgiDwfkJk3nQ9X7nKsAmW+H2Y4VtpDIVDv3wkJO6DDJGyWysaIOveacYoNewXusqBUAACBVIhUMQL7HY73CxCiP0WaRANap+0M/X6vcv/0xaok1yahAxJjhhpovAfxQp3+ztESmEZSIRQ2IcFloVdhZle+GugRDJo+fJ6g/32EbLIoXJgWUSIwhiP2y2WqyWQ5YPaJ6Tkst90MgmuRYy92uPl7gHrmxu8ShMgkMi2GdZpjL1IrcqXYtpfuLZE2WrQ8u/uXyIIJK7Xa8gkQ/LmJeLFBsvlGg95hmGtE/INDHhk9PxhmeG78RJ/2r7C8zc73GGHq8UG8eoK+/stENsNuhcDd+mNnX08bJfuXwTP8ZgleHjYYR1HWG1WeHi4R7yID+MvOVt45Dww4JHRs0gyBPd7bG4j/LnYQ8YSX+9e4i8f7rFbFIiKa6vyBS57lmKQD9vCzV4+Il4vocJnePP4Bs9v15BFhiR5RCzjQW0TcgoDHjFGN4NIUwos0wXQ30WAN+EO2T6GVCs838b4+vYDvL57hc3qBol6aVTeE5TfWYrnUFlvK0+KYVtY0fsx7h9SXC+fYZ8kePnmHuvNAmFaQO1SqOiyE9Sf+30aAt/PsC4MeOSs9JG4ud/eQ11voPaA2CkkzyS+/YfX+N3tDkLsERR2o0BK+G/h+azIsmLYauDX0W/w/Po51CuFm3iJNyrFPksRFQKrcIndhc+StaHP+zS0LyVj8KkKAx4xpi5PZNd/2/Dm6n18cJdC3QogeImt+g1W/+G/x2bzZ2weEtiOwDW1L16N7GUdiuf/+82g5X/v6/8W/+d/+TdcbWPINMRmtcCDTJAmCpFC8w24EM79Pl0yDHhkcKoJjptESUuqL/Y6fcTDGkACABuE2EDg97h5OPx+CVMemq6Pznbf/Pb2/8UHf/yP2N8o3CeP+GB7A7l+jUWyRv4se3tfiS6275MvxvAMM+CRs9CW+Nj2a3YOeUL6qKA37acjKPsOctiF32mUQm0UgmyHWABZvoMKJUQUId3tEMoLb+L1YMj3qS9en2FNGPBIL04fyqb/N8HmgbZt4Y3hJeijgt623YjCbh1jJwJ4UPfYZCnWwQqJKJApiUgKhCoEcNnrIIFxvU998foMazKGd50Qqy/Q0LaJ53/OSid118fZV7AYto1886dniESI9WaF13evsQyuEYkISbrHYhUi5zo854ylG/OUQZ9hTRjwSG+qX5+uFL91yjidUFJY5tIcelq+a1x3Ud39pz85K6uONzdfYf11hK93Ca5uniN/SIFtiqv1Bg/pAwLBtXiA3/fp3PjyiQGPeKFJjbv8zYQosAtY+chbeDoq6G37dyHF0t7JFr7zpw+QyxCrzRKvtw+4CkPEKsJ+v0UYh1DT+t4YJS7fpyEY+hnWRdzd3SngG6FXpRR+/vOf4+///u/x8uVLqiWQ0ZNZ9siFIw94g6OGHcNbJBs8hlsEYXwQ45QKIldQokAURciyS78B5Bz8/Oc/ZwuPTJ/IMhemuvRJE3LYFt7DUkHFGeJtgZWM8SZPIcMlgkAg2+6BkLM0yXmYw4zuwZHZGhCvkMcRVLbFAjskYYoEGUJlP6U7XxRAlmOlFkjFG6QIIBEB+Q5KDjyDbgaUeo59/4Zm9PdX5cP+iT3WaYg0EUjDPfJliEA9ItwtsZfDL8Ib+v0l02EWLbyhFZuX6xx3jzFUscMK7yFXb4BiheUqh3pMre3v0i1W8i+Qqz8fyioWkPIBCgHSYoWFHLbLp1S2nyq219/2/Lvsd93fyHNqraHfn0IlKFSO9eIv8CZ9AxU9QhUBRLjHIriFHHhZxNDv79Tfn0tiFgFv6GnNj+kjxOoaS7FB/vAVwuADqPx32G+fYaF2yHO7y7iIryCRY59kQHEDhVdI0hQo3odaCOTZg6MzqedUEXiK2N5/2/Pvst91f5H7DXhDvz8CtxDiSyTFPdRig4UQiOQVHrdfQq5yKMtcqF0M/f5O/f25JCYX8P7hH/7h7DZlofDV8kt8UHwb6Zsd8ghYroG0iLFIbpDJe6vysyBF8RpYb0K8zl7i2bNnkA/v403+/yF7FmPzSInMKXPp93dfbPFMvI/Xu69xf7PDX8rneP36NdbhLdTyFYpkM6j9od/fNv7pn/5psLKJOZMLeADw2WefnddgAawC4DEDhDjMChQJgBgI8AqAXfLdHEDwtlckl0CeALHcIwsXuBYpth7GGYQAfEzOnaPdS7+/h/N/A8hrrATwsAXEEhAFkOZANPTpD/z+NvHxxx8PUi7pDyet6KCAHb5EXjwCCtjjDkH5qaDsXxZVpMjTHQAgx0vEEQC1QIE/I8Xeunzil0u/vwUegOIaBVLsxNeQAHJ8BagzBDtg8PeXTAcGPA2KAkB8i1CuAfkAgVtAPCDPcfhMtSSQEYI4BQQg1HuA2AIKKPAeJNbW5RO/XPr9ldgAAshwD8RXh+uAD1AooMCbwe0P/f6S6cCAp4GMMiCVhxy3+eawaitbIAoAKPsJB0UGQF1DiRRCKAALlCJv6aWvEZsBF39/38YUiRsgOZngIQtIDD9+OfT7S6bDMeAJIY5rkspZR2XmlUtHIUSgABUCWaiwKQAVhRDIANjnAQzCDIU41IFCKaCQSCNgmUtADn/9y1usVA6FAqoAAqSHyfLFGe6/yg+SaAUA7IG3PgDnWaN20CDNoIDD/xZl/v7MSSL/S7+/gciRBEBYSEAVwKKALAApc5xjGsHQ7+8pXKIwXoQQ/Z62ptxnp7/XSUXU7dtU5hjyvxFCyJSwEVk9V53bFj+G9r/355VO0DrdriNISEl7QgjpR1M9a7r9HOhKBbn23+kYXpmpu247IYSQ4dCtZ+uUxseMSz97t/DGJj9BCCFkGviKH066NE2awm37MogSQog9bUriOt2JQ1MXP3R0/OoEck3oFfD6Xpiui8oxPEIIsaOu7myrW30Gu1O6/NI5ry7Otg6PAYwQQobFtJ6dSr3sys9eAc+0GTmVi0oIIVNlKsHOVfwwLQew6NI0HW/TWWfHMTxCCOlPXT3bVl/7WP/c5I+pn33ikNN1eE2/6TjCAEcIIf0xqZO79h+aJtumfpqeA3NpEkIIuQgY8AghhFwEDHiEEEIuAgY8QgghF4FzbY6uTCpNCwmplkAIIXaYqAp0qd649ElXPedsaglKqaP+XamBVxQFgiDQ0sRzsbaCmVYIIaQffVQFhqxjTdVzJqWW0KSUUAeDGSGEDIvvOtaVeo7JEoYuhpcbrsBgRwgh4+DSkn2cPeC1cWkXnxBCXGKawcT3MJKp0g5gp/Zw9oBX1/d6+lsJW4KEEGKGyViX7/rVtI4/nWzTN1Z4WZbQR8eIEELIPHDVoDEtx9s6PAY9Qgjxh6/6t4+qg4tyAM9jeNXmNsfwCCGkH6YqBH3UBlxhon7gUu3BecDrm7GbAY4QQuwwncJ/LjkgXZuuVBSaYGoxQgghFwEDHiGEkIuAAY8QQshFwIBHCCHkImDAI4QQchE4naXZNcWV8kCEEGJPH7kcE5keV7jys+m33vJAtpime6E8ECGEmNNHRseknLH5ebqfTvltOJUH0oXBjBBCzoMrmZ4+dk23D+3T2cfwGOwIIYSY4CpuDJJarK9zTC1GCCHtuM5D3Ca749KGjayPK5wHvK4ToDwQIYTY0ZZPsm9ZQ9W5tsHOpV9OuzR1HaNSAiGEzB9XLbvb29tjzLCJHc4CXh8xPwY9Qggxw1W9OXT96yrYlZNZXAgNDLYOD+h2jPJAhBBihiu5n3PIAzXFBBfrrvv47yzg6XZlNm1jgCOEED36yP2ce2lCX0kik3Mw9Z+pxQghhFwEDHiEEEIuAgY8QgghF8E0A54Qhz9CCCFEk2kGPEIIIcSQs6YWozwQIYToUbdOrmsJl6vtrtGV9Rla3miQ1GK6+1EeiBBCmjFpOLja7hpdWZ9zyBt56dJkMCOEkH64qjvHVgebygb18V8Kh5M/dAKZi2An/jMApazKIISQsVPmkJxqGsaxNW4GGcPrC1OLEULIN3R12Z1ua+r+60rBdQ55oDZ/qpj6Y7K/s4BnqpRAeSBCCGmmT29Z3QSQrrGuoeSB2sprkzcy9cdkf+fyQDoSDlRKIISQ/kylQeBK1scVzgKeqYQDgx4hhDSjOzuxa3/T8l3RFBPazsuEPv57HcPrKw+kfsoJK4SQedM29lbXDWgqG3QOeaA6fMobDRLwTGQhKA9ECCH1mE7HN5XkOWeQ07E7tLwRU4sRQgi5CBjwCCGEXASjWoenTd1ieS5EJ4QQ0gJbeIQQQi4Cpy28rhkzVEsghJB6dFVmym0lY1ZLqCvfVAWij2pEE4NlWulaGEm1BEIIMV+XNhW1hKbym4K3q/3boFoCIYR4pE0NwGQqvm6dOqalYKaxwDZ2nH3SijO1hBO4EJ0QQkgXzgKei1X7VEsghBA9mlQC6rabzq9wgY5KQzWJtc3+OhwDnhAC6u3U/lIjT0p53NaFbn8q1RIIIcSeJpWAuu1tdeuQSuc28zp0/GpLt1aHlzE8Jo4mhBD/+GpYnHvsrsTbOjwGPUIIMcel2sCUe9G8qiX0GcOjWgIhhJjhUhVhyPXPQ6sfeFdLMOlvrW6b8pcGIYTY0kdlZogyXOJKvcGV2gNTixFCCLkIGPAIIYRcBPNRS2iCKgqEEELAFh4hhJALgQGPEELIRdC7S9NEsqLpGMoDEUJIe905pDxQmw1T6ur3Kl1L0IaWNzIOeKaSFW3lUB6IEHLptMn3mO7fRwbINgFI2/EmDZ9zyBsZB7ymFlnTfnXYBrOqWkIVLkwnhEyFrjVmtr1fQzccdGNC1/Gm+/c5r7OP4bHlRggh42Do+vj29vb4NwacLkugPBAhhPRn6KTK525w6MzZ0PHH1RjkYKnFKA9ECCH6nDvYDV3HtpVtKwPU9zwoD6SBAJAjhAAO/ysP24Cw/AchhPTGV8vutLtxSnVyiel18JZppa2l1wUnpRBC5oKvYDd0a+8c44Om5U9SHogQQuaE6UzMuv19yQA1YSpXZLodMD+v3gGvj9wE5YEIIeRdTOtN3bEx3fJ19+lTxpAyQJNYlkAIIYT4gAGPEELIRTB/eSATKCVECCGzhS08QgghF0GvFp6rjNxUSyCEELcqAaZKNq6UEnTLP4eKQhO91BJcZK6mWgIhhLhVDzAtf2j/62gKxudQSzDu0jSdTloHgxkhhBywrQtPl3aZLg0YajmCbvmmacV0trdx9kkrLoJdlzyQLszYQgghl4OV4jngVj2XmVYIIZdKtU7VzTCiW1e21dkuMC2/Gie6snU1lW9i1zrTimk/alsOTY7hEUIulWqd2la39qkrm+psV9iW3xVLmso3set0WYLJhZ9iZm5CCCH2+GrQGAc83ZlAXTDoEULI8LI8cy2/j13jLk2XmaurzXbdfmlONiGEzAVTlYCusS7d8l3hqvyhrwPQcwyvb+bqtuNsbkLW+8gD08yvRgiZC6ZT77vmSdjub4orf1xehzpmUdeHKCxLYIY1QgiZO7MIeMoyYA2UipoQQsiImGbAq6ol2EYsjSFB1WSHCguEEDIJphnwKuSWMWcWF4EQQkgrs6jrA98OEEIIGT1O5YFOf9fZ7koeyHbKCgMmIcQnLuR++sjutNnoQ1udruOPKzmkJpzJA9U5ZlIO0D+1GCedEEKmiiu5n6ag6KrO1jmPU0z9cSWH1IZ1l6aLnGmmPFFLEJZtNJU3//R2kbsQnJ9CCBme6trkPr1fY8tF7NKfruvTxtlzaQ5yI/Lc7o8QQi6QIboyx8wgY3h96V1uaBm3C9tRQEII6U/fuq9pfG9MwUdXBqirG7e6fx+sU4s19bO2HetcHsi2kcZBQEKIR/rUfbaBzVfrzkT2qKR3bKjgvEvzdGCyCedKCYHlHyGETAjdmfA65ejU2X3LGVtr09k6PNMIbNIqrPJELcF2MglbeISQieAqiLhqNbkq5xzB0ak8UN+yTOWBnsCARQiZKH3qVJOZm0PLA5lyDhmgJpzJA+n8PpQ8kP3S83beIMf7kNijQKwiCFEAuTwEWlnAvmc4hBQAECEQAAQQAYfu1mIWyXAIuQik7FcXmMjr9JH66Vtnm6Lrq0sZIBPfZ1Gb2qoldFFgC4ENBHZIiwgRJNLiDaJoCWAPgatB7RNCCLFnmgGvqpYwMGtc4Y/hPQpxizC4Q5oD8TKA2sVIkSFGOpjtWpUGroAnhBBjphnwKgy9ik5E91ApANwhzw/L/pJdjgA7xDIY3gFCCCHWzCLgBQO3+JJUIQ7eO6z3k1sg3CFKVhAiQaEy6qUTQsgEmEXAywfu4osjYJ++RAggL4AwWSHBFgu1gpQZ3jb/CCGEjBijgKcrU9G0fSh5IHmY0zgYeZpiEQIqWyGQWyDYYpm+D4RbZFk6j68GQshZsK1H28pyITPUB1d2XV2HJrTqalOZii45iqZtugsYn6glDN3CUldAcA8IdZhFsn8GrP6MMFgi28IoW8uTRfMdUKWBkHnguh6tludKZsgUV3ZdXYc2tIaf7u6eai2V2/swtnQzXYhcHgJdIIAUCOPXAIAs3SGQC7/OEUImQVM96svu0L7Y2u27prCNs8+3GCTYiXDQv2h9D2QA0i2AELnCYWbmEsixd3suhBDSwdQaDUNheh2cDT9VM1736Rvu35+cGexrTrIFsJQIcIUieY0ofA9J/hLYCkilUDC1GSHEMab1qG29a5NL05XdvmN4ujgJeDp9rNVg2NXUNYvcwzZUpYpQiD1UvodKgQQvIddAkYcIihsU8k+D2ieEXB66Y1pd+3fRNKnQ1s8+dtviRnV7H3+tA14fo32dLXk68WPold8SsVB4KAQikSJBAJGmENECEn+ExHcGtk8IId8wZgmec2J6HayaRraSEs408ZQc9i8HUrwEih1QRNjjHlF4mKwiFYMdIeR8lJNBbBPvO9Uk9WC3z3Vw0sKrc0Knz7Xa0usvDzTsGB7yEIieIVIhIPaI8AxQWyBbzWTpPiHEF0315dBjeH3H3oay6+o6tGFUXbetm+vat2ubzUmogaOOiPdA9naxXbFAIQHkEkEEADsAy0HtE0Lmg8lUfZ3gobv/kEsTXNl1dR2amEX7RAyueL44rMN7e7XWABAt3h62pOI6IYRMgGkGvEqy6NQyYEQaAatWpucM9tvsGsOULYSQC2aaAa9CaKuWYBkIfNsnhBDSzSwCXlHYBQyDVJijtE8IIaQb7YDXpH5QpW3G5VBqCdZ6eJYtLN/2CSHTwYXqTJWume6magamuFI/cK2iUEVbLUFnBbzO/m3b+mZaSSxnrcSW8ca3fULI+HGlHtAULPuoELg6L1O7Q25vQyvg2UZ/118UVXmgII97lwUAeZBYHT+UffVTRXkgQmaCaW+WTh3Z1fjQ2e5rAXoV13GmDqcLz01vkCvyyC5g2WYm822fEEJIN8YBrxqwdMbkdANc79X0uf6utdhO+/dtnxAyedpUC3SHhdrKsVVFaKKpe9Sm21S3YTTIGF6TE7qr6wdXSxC2agmWTSzf9gkhk0dncp9NOU3bXWA6waQNE/8GGcMzdaKLtqDXD98Bw7d9Qsgc0ZUCmgtDn1evWZpd23WwCXpP5IHU8C201skjZ7BPCJknrupXX8FxaP9d0msdHtCuitC0vUo16PUdw8stA4btwm/f9gkh06WtvjRZn2xbH/fF1K7p+bo8L+tlCSZTYdu22dyEwHPyZt/2CSHTwbRu1C2jz/au33Rx4Y/L821iFqnFIFLLAqJp2yeEENLJNANeJZVXYdlEkhr6Pm2qBUPad6aWwNXrhJALZ5oBr4KwFqSbtn1CCCHdzCLgKcsWkK2ArG/7hBBCuplFwPMdMHzbJ4QQ0o32OrwqXSvczykPpGC3LEBYLhz3bZ8QMh1s5YFMt5uk/eqbD9lUxqiuHNeyR3Vot/DacrbVbdcJiNVtumliqmoJ8D6G5tu+Bv/4br/rk8X7hJBBcSUP1Ecux8W0flP/m4JiW4oxnQXsg8sDdTln2jpzvqLedypL3/YJIaPHRW9WW7lDZymx8b8u+PZNLG1znlaK530YIn2MsFQrsJ504tk+IYS04apLcGiG9qdXl6ZJ0DLdt85eF8q2hWU7S9OzfULI9DGR7+mqK3Vl3PrW620+6QxfNaEzx8NrarE2msbpnMsD2XYJ2rawfNsnhEyepsl9bfvW7a8r4zZETs06+6bHd/3mbQyvjTbjbUFPh+qEC4WsVzklQuMytKklDGm/VaWBEEJOGJts0Nj80eqMMx1c1P1C6Tto+ZTQ8m/q9gkhU8VVPdgm0zO03TGV04Z2l6ZpX6nOTJ5qS6/vGJ7vHkHf9gkh06WpfjXdDgwvr2Piv6tyXPpvtQ6v6TeTPmM3U2qnvq6AArCEXApd8xf6bjepo3WOMSmrT/lDXYc2ptmf9kQtwQ6dcNOuljCcfWdqCccCOSBICLlMphnwKvjuUvRtnxBCSDezCHi+Bcd92yeEENINB48I9uIBSIAsKyCQYyffIEaELAdycVD7G/IPokCcAnv5gA0i5JAQaYq9igBsz3UZCCEzZxYtPCCyPD6duH07wnwD5EC0DIBcIt2HuMEeUbBAgC2gVsM6UCgkESCwwcP6EQ9Y4zqMEApghxzLYa0TQi4EZwGvbe1d2/qQPglJq2oJorALGLapwXzbN+Ifn3agLhchdmkG7AGsNpDhFYr/8Y8I/zcALwFEA3e6hgoiCaFUBvwn4Pk/3CD7pzsECgiwYJ8vmRW28kA6sj5t0js2uTR9yBi59N864LlcLNg3tZgKLGtEy5mLvu3bsv9KAt9eALs9oB5RPD4g2AF5BoTra2TJm8F9UHkGXEmEDwWyb70CdvdAtAayCFgMbp6QwXElDwS0V+5VO67kdXzJGI1GHujUwa7A1/bFYYtILQOW5VXwbd+W5SrDdl8gXgFJvkSAa2yir7G7ukLy6g5y4IATCCDdxRAyQRYCeNhgu9xilV4hWyQIEQ/rACFnwLQ3a0wpuQBzf4b2v0/5sxjDU5Fln6CyVDz3bN+WdBMD+x2SAIDaIg+3eC1uAHUHXAFFZjtG2U6xzbHYFEjVLVR+A5n/HgLvYxfdQeB2Hg8pIQ5p6+Jzm7bxvIxGHsiW0yZo003pfbLCMmBYryvwbN+SbL8D0g1QZJBBgaIAcPUKuAdE/hwq/HpYB94D9q8K4Pkd8OoOizfAAhJbXGG5LYAVJxOT+dMmr1PdbtKV1xQAbQJjH9m3ppRgVZ+qNrpSiJnIKo3q41n3JlbVEs6RaqVVtWBA++dQS9gjxWIfAXEBpDmyWOC5yPE6XxyC8dDxJgMQAJkC4jDDowKQF1BJCMkpmuRCOO3yrAtupwGgji6FmtP9ura3YaqAUHdedXZ1z6vr+rRx1oBXN9DoBN8NAN/2LYnVYWKIUgqIAwSFxCMUMgmEhXqSys05YY5EBIgLQCEHEEEJic0qhf2SD0Iuh9OgNURdOza5H1MmXlUfyCz/pm6fEDJdTOV72roDT1uAXRMKh5B90ynfpT+mnL1LU3c6rcmFDT2rFfi2TwiZLkPL4rgqB3AjPzT09jacBbyuQdOu7TbNZGUZMOxzafq1TwiZDnV1nenSBJ0g56Ic3X1d+e/6OlQZ1aQVbapjSmfI/twq0zOgfefyQE8MUC6IEHIZTDPgVcgt62zbi+DbPiGEkG5mUdcGF26fEEJIN7MIeLZTRmwDlm/7hBBCunES8LpmyrhWS6jie9KHb/u2lEOiQrwNvRLIER0eDqlxdtaZaoJDtkwJlJmihQS01+BZ2hcCyBFCAIf/leU9DR3d3PDtZYwQCADi7ZkFAIpZfHMSA0yUZfqqBOjUuTrl1B1f3d8ki0vdMX3PV0c1oooTtQTTjNVN6JZTlQeCsGwjqdzueN/2baiRCwIA/GdA/ON5XZmy3SfZfwipoJNgv/rffVQCdNe5uVJL0E31VW4fWl2hDWdqCV0MqZaA3DJg2C6D823fO74FcH3bJ6SbroXgvjFd6lCHbsvSF/PoTwktI0ZhOQrn275npLILGIVlt6Fv+4TY4krloC0xv4vyx4a3heel8bYFgoOpJdj2CNpWeL7te6awHsObtn1CzonLwNW3zq2r64dq3bWdr+lwmrOA56LJ2nsscOrTLKfdwDsouNpgu/jdt31CLOhTd9YND/UJOH3qXB/dk3Xn28cHZ7M0dYy7Ukt4MkHgDF/4rTI9A9o/hzyQtV2H59/rfB3Y93WdCQHcqRw0leOyfJtZob5xPkvTC767pHzb94yyPH/rzGye7RNig2krq2kfnZmMQwS7oXEZTJ2vwwP6JRmtlmN2Ir7VCnzb94vvgOHb/tDkAOQ+hSi7bgMJCIksSxAGEhDzmHtG3sWlSoBJ+W3YrJfusutqextnWZag039s8+XgW63At33fKEtVP2H5GPq2b0uX/4/7HKvFAgUylK+sBBCEsX13Ljk7Jg2CvioBuseZ1Lt9GjK+tjcxzU/DoRW4axhcteDcdh0OWInc8jGynPTj274tXf5fBwBUjj0CCAEUO0BmCiIUhzd4mm8xIWdnFq/K1CdZTj2XZmZ5ArYPoW/7tnT5HxZ7JFkBEa0ASCyXBQ6fQgUSFIixOoOXhEwf3++6EwLbFp9la8e3fd/4foh827el0/9ig1gCyAGVAIgO/7EVgAoWhzykhJBOpl5XAABy24A1cfvescx0AmGZGsy3fVs6/H+QwFpGQAKIBQC5x1btbHbPmgAAIABJREFUEYg1kkLpJfgmhMwj4EnPuRR92/eNsgwY9ssS/Nq3pct/IVIUKCADIJcFXql7rMUV4iREnAC4Oo+fhEyds6QWcy0P9EQtwfoL3+5w7/b70KSSUGKiHuByVUYf1QIX9j2qNMifduwTAOH+0J0pnoXIggzYCvzl6+/g//4//i98F//NWVwlbmhKy1WiK9+jW04fmZ4233X97JLvceF/W1l1OE0tZourBZIXh+9ckr6bSL7t29LlfwZkayAM1sjUI9ZbQCiF3/3FH/Aeg91kaJPtMZXLMSkH0NPM6ysPZCobZOp/W1BsK6+Os3VpDioPZL3w1m4dF+1fuH1buvxfCiBNoYoAiIHHLEYQCGC7xzLHDAaBL4M+vVkm5TTt1/d33f1N82/q+t9VTh+c5tL0Jz3hucKi/Qu3b0uH/xmAHIhFgi0AFAnyKD4EO8WIR7pxkZlljJjGnrO18E6bvs7lgSafGoz2p23flg7/0wJyFSB9EEAGyM0Vitf3KEKJr8KX+ADfOo+bZFCqKgC2qb+aMlk19bbp1rlt+5vIBlXpClwuhrlGlTxatz/5iVrCGSrM9mz6w9nXy+Lv3r6ZeoA7+/1UC+zt+1Wl6PJfYpfvsQgWyJEg2CuIWCIXGYL7FWdpzoTTrr62sTGdMk731z3GJDDV7T+ERFwbo0gerSsC6xRl+YVvPUuT9i/avi1d/hdAFCxQAAhUDBEAyHPkYYT0Cliew0dCWhhaHqitfBNbTpNHe5tZKWzHcCwvA+1ftn1bOvxXRYEgKH1UQHBoisYAEjwA2AzqHhmWoetNV8HIV1Brok/sOfub3jXtVWe/KsryNOzVEmj/ku3b0ul/BKAAlFCAkNjj0AkcJ0AcKf8nQKxwJX9jWo6r8oFhZYNc4jTg2Ug7GJ0c1RIGNGg+kCU6x6C6CrCUV/Js3xYt/8Xbx0Cpb7owIwAIgI7jU/EScfI+ilggUDkKPCIPrhBmAALReTzyHMgjvF5kWBUpCrnCIimQRxmCIuYkUUPq6ro+U/5dlONiaYKpj33KcrXEYhapxYhfcsuAYZ3L1LN9W2z970LhfUACKRJILA+pzIpH5GEEhQCiY9LWQ/iILMsQAoiQAlgBApBCgIsiyJRgwCPWBJ5bWL7t22Ltf1f5ogCyEMswBNQeIo2xXsdAEiKM/wyo91qPf4YFEEco9jkQhFASKIoEewgkAeBr9S0hpjDgEWsKyy5m61V0nu3bYut/F0IJiGUGhRRCrIAAyBOBIgCA9yFEeze23EVQC0BmAYAAWQhE4RLr/B5rtWAtQibDLB5V36kkL92+9OyBb/u22PvfQREcxgCzEIUAZPQVkF4DKkBQAJDt9rPlK7xCgvfFtwEF5CiAQEIgQJLkWLNTk0wE5+vwTKatOlNL8C3AOif7pYqCkVqCpTySPFGb6KWW4MC+R7UE8dOB9fiCFEiB5eoGmSoQ7B+x/19yLP45hLpXQNDexl2oGDIK8fJ/+C3iIkSABFF2DeQKYcBgZ0rbFPq2nMNd25tUBYZWS+hTvgsViD7+O8+0YrMmYxRr+og5kaU8Uj5x+7bY+t9VPD5AuvgKu/0rIJXIRAGk/wEq+K+HmZ5B+wXYIwV2QAwFISIIZFABUIQB9gDWg3o/H3Qyh1T/u65ONFVFaAoertQSTMt3pQLRJ/bMQy2B+CW1bGF2dKmN3r4ttv53Fb/9CvgACDKgWClcJQHe5P8VMo8gdgXyuH3SzFItsF8qiO0VEAF7GWINIECCdSGAIB7U/7nQ1iI6F64aJDrl913y4GJNXxOzGMPzkgSR9k8duHD7tlj63zXrZgPgEcgFgELhTXho0RVID3nJOiaJ7lY74E4Cq9fAfolluIDYAQ9LiUBIpjZzgCvFmbmqIrjCSWoxnYt82gR1r5Yw9WkfE6+wfU9z9G3fFt/+d9l/FeD22Yd4CWCzCJEBWAsgTgJERchkngPRJ/NIWxdf9b/7BtgmtYSufJc+0o9VmeQY3lO1hOHxm01/3HZdZrLsc74u7Pu8zqml3T4v8en5dl2/EHf4M4D39s8OxoICCDKoMEGWFgg5imeFbo7Krrq1bzqv0/JNjrFVVzBlFPJAJtQNNJLpE3pe+O3bvi2+/e+0v1viehW/Hb8DChSAiJAhwCLiLE0XmGb971O+jwTVYyt/JmN4lsf77pGcePLfzLbCnbh9W3z732VfrJaHVqA8BMaVApDnWCM8PPu+L+DEcdVD5kr9wNf+Q5cDnHEMT+f3vmN4ubCb1h3Abh2Ub/u+CT0HfN/2bfHtf6f9R2AdAigSYLEEkgwqu0d+9RwC6eSf37HiU/1gyP37jE3WzdzsU46Tb7O+Qa46gNqXILN84Syvgm/73vGtR+fbvi2+/e+w/7DZY10skMoEBRZYxjHy5XM85MCzgMHOFJNxOFP1AFfbmziHXd1r0PVbHdOsaiuZRRLLL9xY4wu7TaZnSPuDygM5mqXhW4/Ot31bfPvfZV9kBUQWQkEgWgqgAAqxw7NgCZG9BsJnlh4Qch6mGfAqRJ5n9fu2T8iQhKHAXgILdY3Doj2JWOVQAJLwGbjsnEyFWQQ8ZfmJ25EsfvT2fSOcLkyYnn1bfPvfZT/G2+xrQYZtrrAIJKSQEAUQqQyQs6hGyAUwiyfVd8Dwbd87ynYMaeL2bfHtf5d9sYWUEXZIEQWrt5OSJZArCOsZN5eHy+TIOsmmfSaPNvHTdLuO3SqzCHjKMlWF6MqtNHL73hG2/luug/Nt3xbf/nfZz1cQBZBFEssHABFQQELGOXbpFsuIE1d0cJ0cWTfZtK/k0aZ+ukqW3YZxwDP5Omk7prrdJGHoE3kg76m5fNvvyT+2NA2M5HKsJVx72nVo36c80D86vH5Gdsv/6LC/LBA/ShSywJ9++mfEKsQKz5DuMywjTljR5RzJkbs4Z/LooThLajHTr5M+9C7HQ30xKvve8X0Cvu3b4tv/DvspIPYRsnCPZ7g9rmIIFxJvcoVr733Cl4erZNNDM7Sfg63Ds/06GVIeSFjqmVlPOvFs3z/WuT4mbt8W3/532FcZghDACgAkEBxMCqRYBwtL26SkmmTZdaDoSh5tUpfX7W/bumvzp2m7aSNpHmN4ti0s21manu17R1lWuNaTLjzbt8W3/132cyCEAiSwLYCVBEQIACmCPOpUTCd61CVZrqv0h0rZ5ToZtG0qsrbrUDd2qIPTgNd2gqeDj87lgWx7hGwrDN/2feM7tZdv+7b49l/j+NdRAiTAuoxtEgCuDq09cnZMk037Su48dFJsU5wFPBcno/u1UZUHUpZdOkLjMrTJxwxpfwryQC7r6z7n68K+z+tcWNrtE+98nS95iq48UImrOROmmNoa2s8+ZToJeCZN4WFukO+eWd/2/SKUXfJsCLtp7b7t2zJ1/4kdrpIsj21/03JcbW/DuQCsD3z3SPm27x3fFa5v+7ZM3X9iRFerTWd70++m5ZjW3WPz01sLT9eJrkFOnf2eMvV1BdMe9J/6HEvf7fOp+0/IVDB+V3Qjtc4+TdNLO6moJZwj3LSpFgxpfwpqCaHlsgzbiQ++7dsydf8JmQqz+Dj03aXo275vVDD8pKEx27dl6v4TMhVm8ab4ntXt275vfFe4vu3bMnX/iRm2yaP7JIPum5y6zvch/Sl/s7k+bfBNI9b4Dvi+7dsydf+JHi6TR+ssY7BJymziZ1Nw6uOPid0+C+dnEvBsZ7lZTgv3bt8v1vJIvvUEPUeMqftP9GhqEbma5e5rhuYpNkoGQ18fYKIBr6qWIAq7gGGbGsy3/d64UksQljXu6eSZPqoFLuz7VEv4nx1ePxO7hudbTfhA/OFiPdwlMkl5oCoq8FBhjMi+d3JL/20Dvm/7tkzdf+IEk+TRbV15bcmd67bXlWHiZ1cZbXZNsV33PQt5IJFaBixbwWnP9n0jLeO99bIOz/Ztmbr/xA2nH/913YK6gaOunLbtfVOGuU423YWLJCfW8kBN+1UZUh5IRZafuMpS8dyzfd8U1plC7LqEfdu3Zer+k+kylPKCa1yVP4/OEFHY/U3dvndSy7+p27dl6v4TG9p6z4bebhJETBsprho1LoOp89RibTOCXMkDPRk8P0Oqldbs8gPan4Jagst59b3O14F9r9fZg7wU1RLGw9DJlNuSLJumhXSRbNq0nCY/+5TjJOD5HsOD9JzL0rd93/ieFu/bvi1T958YUVevuVpSYLK9Tz09tJ91v5ns28VZp0vULRZ0gbIMGPYLn/3aJ4QQ0s3E5wce8K0n5ts+IYSQbqwDXp9+3Sa0y6ksNN53+NjFQmOfNtWCIe0PqpYAcCCHEHIxWMsDNW3T+b1ujUgf4t5HusG3fULIdGgb1mlbvtW2NKxpkXrb9qbyu3yv29/H9rbzamIWXZrKsgkkLKf5+bZPCBk/XdP062at103kM8101bS/q+TRvra3XYcmZhHw7OelT90+IWTs6CbvMKHPWjoXs+hd7OcS3fOaRcDzPcvRt31CyLTpksgxKafER+AZO5MMeFW1BAjPqb1827ehSTHBSC3B4fn3UktwYN+nWsL/5OH5MThfqiSMk7pWTVsXZV1uzT4Bti6np84C8y50xyHrujJ1g7tRwHOhitC03UYtQeSWuTBt5YE82/eN7/P3bd+WqftP7HC1Ntl2IblucKpLBt0UWE3Lr9vfpUCBkVpC19eDLX1PStmmm7edtOLZvm98n79v+7ZM3X9iz2n9qVP3nXPsbsyYnpf2t6HtxWrLm2mLhLD6m7p93/g+f9/2L91/Ykc547CuVWTC0Mmdx7Z/H5wlj3Yx4NqXwnbxtK0emWf7vvF9/r7t2zJ1/8kwuErWPPT+vrb3oVfA69M8Pl1P4VwtwVoipTu1V3t2+eHsT0ItweH59ztfe/t+r/Pwz2+9XUuzpBcuEiefI4mzD7tD+28c8FwOIFbpXW5umYsysDvcu33f+D5/3/Ztmbr/hEwE41maTbMtu/YBBlRLCOxmuQlLtQPf9n3j+/x927dl6v4TMhV6z9IsGaq1Z4LvIQzf9n3ju8L1bb+LHECOrxHu34OMFXIhEWQp8jDAITHduJtoChkEQuRpgSCUyNUegQwAyMOaiEt/Achk6L0OD3ArIFgt36Rs33p0vu37RiGzOl5Yzp0av/0QEs8hI+D+4c9YXH0LQRi8fW6K0fsv0hAIcAh2ORAEC6gig5BALtijakpdw8BVkuUmG33LqSu3ur/LpNV9/Bxk4blOgTYDjkbBU5w/RAwu0+PDrqNZCyK3nOxrWWOO3X6QA4hTAAWult8CCkAlrxEuNwjwGiJ/386Bgf3fRw9IUQCIEIYBCihEIgQgkSmFwMP7OEWaZrG7SqbcZKdvObp+6gZv1+fbZ1XAJFOLVbFNzGX7herbvm8yyxOwfQjHbl8EQI47yOIGIkiwFxlWywX2iJDjvdH7v9hvsJA4fHgJIMsVikhACCAWCtPvozgPLrJK2di13U+3Bdj3fPrOGDUJfLMIeNZfmJYtHd/2feP7IRq9/QIQ8hoSMZTaIxZriAxI0h0CqRBidQ43G+nyf7dQWCIH0hxFnkMs1sgARAoQuUYB5Gz4XhNdZWzJrGfxqOa2AWvi9r1ju45MWE7LH739CFm2RywDCKUOQUIVuF49QBY3o/dfigiABMIcEjEetyniZfS2ayOHQcIm0kI1KXPbOFjfgKY7tmfiZ1PZ1X1MWn9DTYB0kjzaRVJpm2a+7LHw9l3sKhzf9n2jLCtc60lDI7efF4AMy8C2RC4BKf+I5P67CAAoSweG9j9OgCKSSEWEABLrBQ5xLgT2yLCwfv4JoJeUubrv6XbTMbi6ckzmanTtb5MibajWoHXyaJcL0XXLeSIPZP2FbHe4d/t9aZIGAszkcmwHMU8bCH1kelzYt5EH6rJfCGAVQuQp1G4BbPaQe6DIBfDPCvKnPe2W9GlgnZ5vh/9XErh/++/3X38Lv/5f/x98oP4SyCQSuTqM75FR0LQmeuglY67KH9pP6+TRps3gpv8mE0ZY/s3d/kYAjynibAnEeyCLALEAoIDH8ft/LwAZRRB74M83f8IHxYdIAiBXGa59ajnOhLbZmybbyxmT1dahy2A09P5Dj/PNYgwPwnqe2rTt+8b3+Y/dfpZBBs+Q4vXhjcsDqHB3CChK4/hOhr5+OYqiAGIBBAoKh+ydKpBcd+6A6phc0xhe1/Y26oaMTMtxtX9bOS78bKNXLs0mo7ppxWyTRz/Fd8Dwbd83vs9/5PaFwDLIsUsBZBIQO6gEQLEAFhJQ23M42UK7/2EqkMkcUAHCQiBXCqEqIGSIPYDFeZycDV0Zq2y21/3ucpzNlZ8m18DmtyrGAa+pmeyiOdq/PNtBBJeDUD7s+8b3+Y/cfgY8vtkiXi+QpHvE1yGQvY9E/RGIAezG7X8WFoddigLFG4Us2EEUAkEWIpKYwTRjcik41cProm6SSx+eygMNX2G2y6kMZ38a8kDuzr/f+drbt7vO7fYzFAjfqpJnALIsRCzfIJMrXAkgUbYRwzxgvnu+HdevOPRbZsUh2C0fV8AKQFRAIkcfeSJCfGCdPNpXwuh3UJZfyNazND3b943v8x+5fSUE0jRDFEYoxBsswmuI7BkK+TVyrABlGTAG9n8vc8QQCGUBUQhgdVhKoTJ1WJIzj5kA5AIwyqVpMtDYVk4TvcfwhO0YjuUb69u+b3yf/8jtR0UIyAxQEQJ1DSFTIIsgwucHyyP3f1EEgBQoFBCoJSALCCUhAvZl9qEr9yXQPrzjev+6Y9p8r5YxdJLoPuU3YfSmmA40du2jO7DahbJ84e3VEvza943v8x+7/ULlCMQKSiY4vnIhoJAigRy9/0IVyIoMuYywUG/lgooQkBKJOAxDkm7apuk39Z7p7Hu68Nxkfx2/qvvVBdY2f0zWbrfNC9HpXdQJ2tNsWlAt4ek+FohQ4g4FbnOBbaiwSIEEABKFNN4j6uhys74swlJeyXYtmK39jt+Dty0hgfjt/A4JhMDy7djX2P1HIBFCvm2NHs6jnKjCYKdPW4ul7/51QUF3f1N0W4C+hrict/DIPHnMJG5QAMESy3SLHEDyDMgTIIolcssKtQvrXKaW/vnumJu6/8QeX0mWx5Zsuglv6/DI/AiRQawjvHnc4jqKEGYSeL1HuAQyRAiHzqZhW+F7biHZMnX/iT19lmSZtqZcZ1zpOw7YRNs8kabrM+gYXp1BHaNDJ48mdsQLIH1MsQaALMVOhQixwC4EogQo4mH7cq1XoVl2cfsOF1P3n9jhq67rG6CGys+pO0ZY54uuP70zrXQ52Ye+5ViPYVkeP3X7u/0htWOqgGUcINvlEBmwVH9EFocI8J6lhS5sK3zfd8COqftPzo+r1l1Tsumh/fHF2bo0dabK6vJELcG3AOvE7a+iG2z/dAfcLoHdDrgGnm0zvP4v/x2WX2XYBa/s/OtCnqhN9FJLsFzHJlM7tQQb/jMgfurA/z52Dc73abIH4gpfwcK0geHLT5d2jfXwpjDASczY7u+AmxVQADeZQJooPEog/O0fsFtj+M+i3PL4yFKeyda+LVP3n1jhMjnykJj62bT/0NvbsK7KdI2errNwnzya2CAUoMItFkmAV7ECliGQKCyvcmTR8tDnOSTSsvWQWrawbe3bMnX/iREuEie73F/n9679xra9CeepxXyM4XlJNjkh+4v8Gnu8QawkkqAAQokgkchFBhEKIA+BBNhHGZAFWHydYX+zQJ7mENhZjzANz/g9bGfq/hMyDYy7NE//bRrY6lbTu8H3tI9x298XbxCvgHRfHO64KJBHxaFhsFfAMoPahYffggh5IQCZoFB/AZX8Yfy5gac+TXHq/hMyEYxyaZb4npHjYwB9GqoFDRQFICUyAGEBZDugCA5ybCmA9R7AIkOOFBILhPgjluI7+OoxwWqlcEiNPxynD2Gf83WRidLn/U0t7fYZl/B1voT45GxjeDq/cwxvIOQeXyPH82INJQT2a4UNCiBXWBYFECyAPEQQpEAqgegaASSuF8vDhAg57oXn1gvjPS/cnrr/xIy2BoPuGueht7f5Xt2/KbmzTdLnumvUJyl2lV4Br1po3yDXtJLemHn3SNrbVwu8JyWwB7II2IQKuM+RrHKoKEYMYK/+jBA3KKICAa7wiK+xk8+Rp1ssgvO18PqQ2QZMS/u2TN1/okfX7HbdNc5Db2/zT2dhuGkw6wpkpkm325jFu5ILu2ndgeUg1djtqxQoYgkZATIEkGVQcYr7YI1NIiDiPZb5+0Bw6P1U4hExnmMBQISL4edUWAb80PcHhyVT95/o0dYiclHu0AyR7szErovrNouAF2SWsyosr8LY7edxihhAFgJBXuBlsIMMV3i+O2TtR7GACrYQWEAGgFJrJEgApMBuA6xsR8m6mLgeni1T959YM9c1zn26TE2uw9nX4XmhklkksfzCjTW+sNtkeoa0ryMP1GU/AlD8/+2d748bxRnHvzOz67XvLrm7XJqjB0dSESCFCKhEVAmkCvUVL3nTN30D/wZ911f8K7y75FUl3lRqRV8UqAoKUgOkETkCgZLc+X7a3t2ZvnBsjOO1d3dmvV77+5Ei5byz831mbc/jeWbmGQO0AXhSwvM06h0BjU3A/B0GTwDxCiQArfYRiwBKNqBRg6pH1ue1TaLq5+HZUnX7yfSx2QDuSnvSHNtwmcEQqCt7Cs2l6SJJdNLrNsmj/ZJXm826voaAkgbLegWhPsKZSMIIDeHdB/Tl7rL4R2fMKKxBAYj1o5eEN7FD5WI/MhahuufmSQAIui9JYFr7XQSAGN3PsQcByN6PBG8mfi3k3eKV5X5XaR0H65qkm/Zaz56szyFP6DT3xnOXuTF75N36YCw/tMKyx551fRMIRC0DITSEB8ShQeR5kGEEX8N6H5hwsjGguvq2VN1+Yo+rJM5l4froIZd1DpL7m5LVkCIcZA9bhzH3+p3usnftdUeDbRhEtQDLsXDzC9fYzsFVXN+WqttPrCh6j7NLZ5SlnqwRv6zPIU+7nPw0LHvC1VgOUQTs9kHNun6kNeo+oMMOtASCWgPm5BiRUIjVjwhw3kofwnafnu0Qs2R9W6puPykEV8mUs76e1R5X9mclTz1O5/DGMbjvwzZ59GPHA5U+izTb+kL5MO2wO30SA+r0FPETPnAYYulPr+AkuPf4TZmOj7E+wjWnrkP9Mo8H+rPD55dJN58ajwqyI8u8V5o9zEW8nsS0dCeNANPWM0wmh5dm86INuYf2JfQXVdI3kYZX8xG3QxgfWF4ROP1fiHAJOKl/78BfF5yJZeb1bam6/YRUA2chzcH/j/PqRcSoheV5YNaLTmZc34Qxorrs9qsRcNAQEDXTndQTkQOHZ50rpeL6tlTdfkKqgZNcmj3KWl1kbEdYtqskZ12/DuA0BDxAKh+6E8OoAAglpIigrReNWHa4Vde3per2E1IRMp2W4GqiMYnc9WtLj2HbYVRBv7sZCRqP0pBFbUAwmEYIWRxyz+Flue46efTwpPk0UhGOO06lSP00x7gUoZ/l+BiX+nmOrXGhX+bxQGX8XuLxQGQRmYsdq2Vv3KX+YuvbUnX7bTGWnlcIuxBJ2fpkelT7m9Kn7GZQf7H1bam6/XaU7TDK1ifTYy6+aR3Lmaia5b4C6i+2vi1Vt5+QqiCazaYZHtJfv34db7/9NprNZklmJfPuu+8+9pqva1Z1hrJjdT/1F1vflqrbb0vZIcUi9d977z2ruok7rl+/Xr0R3qgPUH/lYU6kZdZ26i+2vi1Vt9+WWXZY09An06NyDm8UStSt7jfGbuc49Rdb35aq229L2Q6jbH0yPfoOzxgDKSWMMf0PgNYaSqmZ/0CU/YWn/mLr21J1+wmpAsYUfZS1Q1xsek9T97BGkq6tPaOy0mTVymNDmmOaXLfdVbtc6LK9xX22x322ivo8D+sW3V6X9RfZp5HRVMLhTTp81gVpzm1KSpidxZ6kI5SyamW1YdzRTUW23VW7XOmyvW7bO1jOpn5XukDx7XVR/zT6NPI4XM88Zfb390v5YJepWwbUJYQMU4kR3jRY5PDCorWd7XVf/3Cu3Wlge66mjW6R9ZPioMN7xCKHF6bV9rKe67DuNNrb6xSn3d5Ruov22Z7m9MciPM95giFNLPavtGm1fRadXZH0QsjTHvUM6xbd3ll5X3sUbcsi9xXzAB0eKZxZ6xSJW9bW1voOdpoOvixdUl3o8LDYX5ai2z5rzm4a7S2Dcat/i6Q3ohy3InSauvP6/hI39HNpDm4839nZwTvvvIO9vb2Z2Xhe1kR0UXtoRnXI07Bh2rpl7Vsqc7+Uq/qrojt43yLolv2cST52dnaq4/AIIYSQvOzs7DCkSQghZDGgwyOEELIQ0OERQghZCLjxnPyMtLkJXerlXfQzzDQn/setAk1aMZh2QcSk1aVc4EBIPujwyM8Y7KxnfdPyqNWmaVZvTtNhuMjKwUTDhLhBcgUmIfkYl8OxSOjsCMkHR3gkNcPpqkaF2YYT+o7ad5Q0Wkmqb/ierPZm1SwLF+Fd7vMiJBk6PJKJNEmJR/0/6XpSCHVUmWEmzZFNqs82VDjYriIz9w/XO87mUc97VDlCFhE6PJKbpBBeUlLfXqeb1oFNYtIcXpr6JpWxTcKc1ulMckiTnFwWOwhZVOjwSC7ydrrjOnaXnXJa+yZpprle9uiJzoyQdHAfHpkKgyMlFyOvIigz8fM4p5XFrrKSSBNSBTjCI7lI67hGlR8X8hz8uwj7hkOrrubYkih6Di/pmuvnScg8IPb39w0AJo8mhBAyt/STRxtjEEURgiBA7+QEYwyCIEAURWXbSQghhOQiDEP4vg8pZTek6fs+hBB4+PAhzpw5A6UUhBBotVqo1Wp0enOOEKJsE0iJMIJD5plGo4FmswljTDekGUURarUagiDAwcEBbty4geXlZezv76PRaLBDnHPY4S02/H6TeaY3Lae17jo8pRSOjo7ZP4lMAAAGaElEQVSwurqKOI6xsrICrXV/To/MN+zwFht+x8k8Y4xBq9VCo9HohjSNMVhaWkK73UYURYjjGO12G0EQQCkFrXXZNpMCYYe32PAHD5l34jjuTt3t7++b3h9RFEEp1XdycRxDaw2lVNn2EkIIIZnpRSqNMd0Rnud5/RBmGIb9EZ3neYjjuFRjCSGEkLy02+2+j+uHNHthrXq9jtPTU9Tr9f7qTCmZkGWeYUhzsWFIk8wzQRD0B3SeEAKhOEXdr0NohcNjhY2zGoetDiKh4NXqiKN2oQZJzQ6XkLLgDx4yz/Q+30KIRyM8tYLoIEIQAY31Olo/7EJvXEBd+/D2W+gslWovIYQQYo0HAEsh0Fr1EHsx5MNv0PnDHyEeHkKoEO1AAgUv0jRzGFGRf/u4bBMIIYQM4AGA1CE6R6eIVAyxYuAd7AF1AUiDtm7B0/VCjZBcBEoIIaRgPAA49JexFC7hrDY4FCEi3cBDEeJcXMe69nDsHRZsxhwO8QghhMwUHgDUowhtCezpDsLOEdZVByI+BpTAwVIDX159pVAjZMVPKdq+9QUah0X/KCCEEGJD19OYPQgvAJRAoH0c+zWcGIm4voTO2gbu/OZ3hRoRq2JXgRbNxrff5XZ4ow7/TDrHbdK1UdeTzkGbVG7coaRpTvhOa4cNRZ40blt3mrPoRpWZ9P4SQvLjAUDNnEXLSEQ6xlqs0Q7bWIfGUXSCjvYReseFGiFNtUd4rhns/IY7u3HX0r42XNeoci462XH1zzPDbU3zvgz+vSjPiZBpIwGg7UXwtUGgJFrSAJ6PE88gaMUwvg9pvLH/Yu3BaB9CK0SnIerCh9ISCAU81AFTg0AAgW4ZaSQ8o6AiDS9e3DydvU4uy8nhRTErdswDdFiEzCZOUqgIAyhhoJToJqEOQ0AqCOnBSAkddSChoYSAlBJaA0J5MKoGr77iwgTimEHnt7a2lugM8zrJXp3DOkl6o8on2TdcNm194+pPakOaclkZ11ZCSH6cxBI9qSGEgIkNYqkQRkDN92CUwHErxJmaBw2NKDLdxNSQ6MTAyUkbdXjwXRhRMYZDWLMa8hu0K02obvjeHuPuS9IYHAFnKT+oN+n6uNfTtHEcvfZnvX9RQ8GEFI0Th6fjEEr6iA0QRxqXr1zB6vo5KL+BBw8eYPeLT7sO0RhoABvnf4HLzz2PSANGSAidLkH17f/cxNFhEy9fez2zjUcHTdy+dbP/9+bWNp548unE8vfv3cX33+5m1pk3Bh1O1o54kpObdE+aem2d0aQy4+offDZZ51vH1UkIKQY3IzwhIZWC0QYQHq799jUsnV0DpEKzuY+vv/gMSir4SiAMQywvNfDCiy8CtQZ0rCFVkErn/r27ODps4qVXX8ts4/ff7uL2rZtYP38Bb7z5FlbOrI4t/9Krr2Hvxx/w4V//gr0ff8isl4aqhKxGdei2o9IyO/YszpcjLELmBycOT3gKRgCxEVjf2MDS2jkAAhASq6vr2Hr6Iu7tfgNPGkgpcXCwj0//9Qm8+jI6UQyFbnLPza1tbG5t4+iwif/e+vwxnePDAwDAZx//47Fry2fO4pnnryZePzpoAsDPnN3tWzf7dY6qa/38BVx7/ff44Mb7+R5MAkmdaJmd6ySbinB6k7RdlbdhUhsnhVyzQgdLSHE4cXhxZBCbCFp4eO7KCwAE7t//Du12iIuXLuHKr6/i7te70NqgXpPonBzj35/8E5GRUH4NSncAAC9fex2bW9s4PjzApx99mKg36trm1nbf4SXdu7m13Xd2H9x4f2zIcvfOV3jjzbewubWN9fMXChvluWTUSss8c2xJdY4qM84hJNU/bGea0GiW8pPIuiI1q2NPsnfUvO1wGddtJYT8hBOHJz2FKAKUX8Ozzz4PYzTu3v0GBwcHeOqpp/DLJ7fRWF5B5/QYYbsD40tAawS+QqwjFyakojdnd3TYnDg/t3vnS3TabdSCAOc23Dq8NPvixpXLUkfW+0ddn2RXkiPMqj+u3izlk+7LUp/te5G1fWm1CCH5cbItQWvAk8DFp5+G1zgDIWu48/lH+O6rzxDHBl5jFZcuXoISBtLzoY2C9HwYYyCLPophBKPCmKPYe9B1citnx8/3EUIImX2cHmW+/avLAH4aHQHor4x85spVl1KEEEJIJpw5vFoQ9OfQdu981X+95/AG588IIYSQafN/b8iNMLz5PIYAAAAASUVORK5CYII=" /></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;">Interesting resulst between the entry level Perc H310 and the Smart Array p420.</div><br /><p><br /></p>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0tag:blogger.com,1999:blog-1448611745785679913.post-45861134191761692892023-01-03T14:48:00.003-06:002023-01-03T14:56:30.570-06:00Cisco Switch Reset<p> This is one of those things I need to once in a while, and each time I have to do it, I need to re-look up how the steps. So these notes are mainly for my future self!</p><p>To factory reset a Cisco switch, when the password is unknown!</p><p>1. Obtain a console cable to connect to the switch. Typically the pale blue rj45 to serial cable, other times a Micro USB cable.</p><p>2. Open a terminal connection to the switch, on a PC I reccommend either Putty or TeraTerm. </p><p>3. Power on the Switch while holding the "Mode" button down; until the "syst" light will flash, and in the terminal window will show: "<span style="background-color: white; color: #666666; font-family: Consolas, Monaco, "Lucida Console", monospace; font-size: 12px;">The password-recovery mechanism is enabled".</span></p><p>4. In the terminal window type: <span style="background-color: white; color: #666666; font-family: Consolas, Monaco, "Lucida Console", monospace; font-size: 12px;"><b><u>del flash:config.text</u></b></span></p><p>5. Then <span style="background-color: white; color: #666666; font-family: Consolas, Monaco, "Lucida Console", monospace; font-size: 12px;"><b><u>del flash:vlan.dat</u></b></span></p><p>6. Then <span style="color: #666666; font-family: Consolas, Monaco, Lucida Console, monospace;"><span style="background-color: white; font-size: 12px;"><b><u>reset</u></b></span></span></p><p>The system will reboot and have no configuration, it will present one with the wizard to setup new passwords and basic setup.</p><p><br /></p><p><br /></p>aaronhttp://www.blogger.com/profile/10057451060414359535noreply@blogger.com0