Deploying the Veeam Hardened Repository ISO 2.0

 

 Repair mode

• Re-installs only the OS while keeping the data partitions intact.
• Please note that repair functionality cannot be used for migrations from Ubuntu or any other Linux distributions. The system will fail to boot and you would need to fix /etc/fstab manually.

Live boot
• Provides a live system for troubleshooting. It’s mainly built for use by Veeam support, however experienced Linux users can also use this for example for performance testing with fio or iperf.
• There are three scripts to mount data disk, operating system disk and collect hardware information in the home directory of vhradmin.

Fully automated installation / Zero-touch installation
• This uses regular kickstart and was designed to allow mass deployments or unattended (lab) installations. Public documentation can be created depending on demand. In general, the kickstart documentation from Red Hat can be used.
• To get “zero touch installations” working, add auto=1 to the kernel parameters in the grub bootloader. In the ks.cfg ensure to set keyboard layout, time zone and disable the cdrom installation source.

The following other changes were implemented in version 2 which are not really “features”
• System requirements change: the operating system disk must be the smallest disk. This is to ensure “repair” deletes the right disk.
• IPv6 DHCP support (UDP port 546 is open now).
• Allow “ping” with rate limit of 5 pings per second for easier troubleshooting.
• Additional warnings before installation / repair formats disks.
• Help text was adjusted.
• Network configuration is now mandatory.
• The “installation destination” button is non-clickable anymore to avoid confusion with that wizard.
• The “pre-release” warning was removed.
• the faillock configuration was changed so that a locked user will get unlocked automatically after 1min


Now you start the deployment. First you boot up your system with this ISO.

You can see the Rocky Linux 9 installation GUI.

Setup the Network & Host Name. In my case, I only setup one Ethernet adapter for this Veeam Hardened Repository. If it is the production environment, suggest to setup the network bonding with two Ethernet uplinks.

Then setup the Keyboard and Time & Date. Click the “Begin Installation“.

Click Yes.

When the installation is completed, click “Reboot System“.

Boot up the Ricky Linux.

After rebooting the system, you need to provide the default credentials vhradmin / vhradmin.

And You’ll then be forced to change the default passwords.

Then click Accept the Veeam License Agreement.

You need to start the SSH service, then it creates a single-use SSH password.

Now new VHR has been successfully deployed and is ready for use.

Now you can go to the Veeam Console > Backup Infrastructure > Backup Repositories and click the “Add Backup Repository” button and select “Direct attached storage”.

Select Linux (Hardened Repository).

Specify the IP address for our new VHR server, click Next.

Input those single-use SSH credentials that the VHR generated, click Add.

After verifying that the SSH key fingerprint matches that provided by the VHR and click Yes and then Apply.

Click Finish.

Now that the Veeam services are installed on the server and it is managed by Veeam, it’s time to deploy the VHR.

Select the Browse button. You’ll find a folder pre-created at /mnt/veeam-repository01.

Then click Next. Veeam will validate that the requirements are met for a hardened XFS repository. In my case, keep the default settings.

Next you’ll be prompted to select a mount server and then click Next.

Click Apply.

Click Next.

Click Finish.

Now I recommend disabling SSH service. Back to the Main Menu, select “Stop SSH”, confirm by selecting “Yes” and validate that the service was stopped successfully.

Finally the deployment of Veeam Hardened Repository is completed. You can now setup a Backup job to start to test the immutable space feature.

Troubleshooting Guide for FortiGate Firewalls

Here's a comprehensive list of troubleshooting commands for FortiGate firewalls, grouped by category, to help you quickly diagnose and resolve issues:

General System References:

Show FortiGate Details: Use this command to view the current firewall version, IPS/Virus/App-DB versions, serial number, and other system details.

FG-maattoos# get system status


Check CPU, Memory, and Load Usage: Monitor the system's resource utilization to ensure optimal performance.

FG-maattoos# get system performance status


Check WHO is using CPU, Memory: Identify processes or users consuming high CPU or memory.

FG-maattoos# diag sys top-all


Show Hardware Acceleration: Verify the hardware acceleration settings of your firewall.

FG-maattoos# get system npu

FG-maattoos# get system np6xlite


Check HA Status: Verify the high availability (HA) status of your FortiGate cluster.

FG-maattoos# get system ha status

FG-maattoos# diagnose sys ha status


Check Session Table: View the session table of the firewall to check the maximum sessions against used sessions.

FG-maattoos# diagnose sys session full-stat


Show Interface Settings: Check the configuration of a specific interface for issues.

FG-maattoos# diagnose hardware deviceinfo nic port1


Check IP Addresses: View all assigned IP addresses on the firewall.

FG-maattoos# diag ip address list


Show ARP Entries: Display ARP (Address Resolution Protocol) entries on the firewall.

FG-maattoos# get system arp


Get Routing Table: Check the routing table of the firewall.

FG-maattoos# get router info routing-table all


FortiGate VPN Troubleshoot Guide:

Get VPN Tunnel List: View a list of active VPN tunnels.

FG-maattoos# diagnose vpn tunnel list


Enable VPN Debugging for a Specific VPN: Debug and troubleshoot specific VPN tunnels.

diagnose debug enable

diagnose debug console timestamp enable

diagnose vpn ike log filter name <VPN-name>

diagnose debug application ike -1

diag vpn tunnel up IPSEC_PHASE2 IKE_Phase1


Authentication Debugging:

RADIUS: Test RADIUS authentication and connectivity.

#diagnose test authserver radius-direct <RADIUS_IP> <RADIUS_PORT> <RADIUS_PASSWORD>

#diagnose test authserver radius <RADIUS_NAME> <protocol-chap|pap|mschap|mschap2> <username> <password>


FSSO: Debug FSSO (Fortinet Single Sign-On) authentication.

#diag debug authd fsso summary

#diag debug enable

#diag debug authd fsso list

#diag debug authd fsso server-status


Restart FortiGate Daemons:

Restart IPS Monitor Daemon: Restart the IPS (Intrusion Prevention System) monitor daemon.

FG-maattoos# diag test application ipsmonitor 99


Packet Sniffing and Flow Monitor:

Sniffing Traffic: Capture and analyze network traffic for troubleshooting purposes.

#diag sniffer packet any <'filter'> 6 0 a


Session Flows: Monitor and analyze session flows on the firewall.

#diag debug enable

#diag debug flow filter <filter>

#diag debug flow trace start 100


Use these commands carefully and refer to Fortinet documentation for more detailed information. Happy troubleshooting!

FortiGate Firmware Update Restrictions Without Support Contract: What You Need to Know

If you're managing a FortiGate device without an active support contract, there are important limitations to consider when it comes to firmware updates. While it's understandable that firmware development requires funding, the latest FortiOS 7.4.2 update introduces some significant restrictions that go beyond the norm.

Previously, FortiGate devices without support contracts could not upgrade to higher major or minor versions but could still upgrade to higher patch builds. However, with FortiOS 7.4.2, even downgrading to previous versions within the same major version is no longer possible. This means you can't revert to an earlier version, even if you encounter issues or compatibility problems with the latest update.

Furthermore, as of February 8, 2024, upgrading within the same minor release (e.g., from 7.4.2 to 7.4.3) is also blocked for devices without active support contracts.

To work around these restrictions, you can boot from the secondary partition or format the boot device and upload a new firmware image via TFTP.

Downgrade Commands:

To boot from the secondary partition:

# exec set-next-reboot secondary

# exec reboot

Solution

 

The following CLI command lists the FortiOS image files installed in both partitions:

 

FGT # diag sys flash list
Partition    Image                       TotalSize(KB)   Used(KB)  Use%    Active
1            FGT61E-6.04-FW-build1778-201021    253920      87604   35%    Yes   
2            FGT61E-6.04-FW-build1803-201209    253920      88660   35%    No  
3            ETDB-84.00660                     3021708     200120    7%    No   
Image build at Dec  9 2020 22:27:52 for b1803

 

As per the above output, partition 1 can be seen to be active and holds the current firmware (6.4.3, while the secondary is on 6.4.4).

Backup the configuration first before reverting to the previous firmware by using the following commands through the CLI and select which firmware should be used at the next reboot:

 

FGT # execute set-next-reboot {primary | secondary} <-----In this example it will be secondary.
FGT # execute set-next-reboot secondary

Default image is changed to image# 2.

 

Primary and Secondary simply refer to partition number 1 or partition number 2 respectively. Partition number 3 can be ignored.

Once the secondary partition that is to be used to boot the device has been selected, reboot the FortiGate.

This can be done using the command:

 

FGT # execute reboot

 

The CLI command can then be used to verify that the FortiGate has rebooted from the secondary partition (see the example below):

 

FGT # diag sys flash list
Partition    Image                       TotalSize(KB)   Used(KB)  Use%    Active
1            FGT61E-6.04-FW-build1778-201021    253920      87604   35%    No   
2            FGT61E-6.04-FW-build1803-201209    253920      88660   35%    Yes  
3            ETDB-84.00660                     3021708     200120    7%    No   
Image build at Dec  9 2020 22:27:52 for b1803

 

VDOM administrators do not have permission to run this command. It must be executed by a super administrator.
After an upgrade, this will automatically change (here from 6.4.4 to 6.4.5):

 

FGT # diag sys flash list
Partition    Image                       TotalSize(KB)   Used(KB)  Use%    Active
1            FGT61E-6.04-FW-build1828-210217    253920      87396   34%    Yes   
2            FGT61E-6.04-FW-build1803-201209    253920      88660   35%    No  
3            ETDB-84.00660                     3021708     157240    7%    No   
Image build at Feb 17 2021 20:43:28 for b1828

 

Note:

  1. Rebooting the FortiGate from the other partition will cause the loss of any configuration changes that were made since the upgrade. It is preferable to use Notepad++ with Compare Plugins to quickly highlight the difference between the backup configuration before reloading and the backup of the currently running configuration. 

  2. In HA environments, the command needs to be applied to each unit in the cluster individually. This is not synchronized and will not automatically take effect on other units in the cluster.

  3. FortiToken licenses once added to any of the units, are kept and shared between the units of the cluster. Therefore, a reboot (or shutdown) of a unit in HA should not impact the operation or usage of FortiTokens through the remaining unit. When a downgrade is performed as above, the unit will load the previous configuration (with FortiTokens in the same state and assigned as they were before the last upgrade). This may be useful when the token licenses are not validated correctly following an upgrade.


To format the boot device and upload a new firmware image via TFTP, you can refer to the FortiGate documentation for detailed instructions.

Disclaimer: This information is provided for informational purposes only. The author is not responsible for any consequences that may occur. Readers should proceed with their own responsibility and refer to official FortiGate documentation for guidance.

It's essential for FortiGate users to be aware of these restrictions and plan their firmware updates accordingly, especially if they do not have an active support contract. Ensuring compatibility and stability with each update is crucial, as reverting to a previous version may not always be an option.

Important Note: Please check your secondary backup before upgrading, as it may be an old one. After downgrading and upgrading to 7.4.3, please restore the active backup to the new partition