This document will cover the steps involved in installing MAS into a client where their system is separated from the internet.
Overview
This article will not cover setting up OpenShift, the associated database (if not an OpenShift DB2 one) or how to create administrative environments. It will cover what is required to be provided by the client for the install to work.
The basic steps are as follows:
- Download Podman onto your company laptop
- Using Podman download the MAS CLI tool onto your company laptop
- Using the MAS CLI tool, download a copy of the MAS install files and any required components to your laptop
- Import the CLI tool and MAS downloaded file into the client environment
- Using the client Linux admin VM, load the MAS CLI tool into podman/docker
- Using the MAS CLI tool on the client Linux VM, mirror the imported MAS install files to the client image repository
- Using the MAS CLI tool on the client Linux VM, run the configure airgap command to configure OpenShift to use the client repository
- Using the MAS CLI tool on the client Linux VM, install MAS
Throughout this document, I will refer to podman as the tool used, but docker can be substituted at all points. It is an equivalent program and commands that start 'podman' can be started with 'docker' instead.
Throughout the document I will refer to file locations. On your company laptop I will assume you are storing the files in C:/MasFiles, and on the client Linux VM, I will assume the storage drive is mounted to u01. Anywhere you see these and they differ from yours, just replace them.
Prerequisites from the client
The client must provide:
- An OpenShift environment with the agreed number and resources of master/worker nodes
- A method for importing external files into the system
- A Windows VM with database access tools (ie SQL sutdio or equivalent) and Linux file transfer software (ie WinSCP)
- A Linux VM with podman installed and 500GB storage drive (assumed here to be mount to /u01) (size is variable based on required components) and a /var folder size of 24GB or more.
- A copy of the registry SSL certifiate (placed in the Linux VM /u01/cert folder)
## Download Podman to your company laptop
Go to the podman website and download the Podman CLI (not Podman for Windows)
Install the tool
Open command prompt and run the following commands:
- podman machine init
- podman machine start
- podman image ls
This should setup podman and list all current images (none)
Download the MAS CLI tool
Open a command prompt and run the following commands:
- podman pull quay.io/ibmmas/cli
- podman image ls Note the ImageID of the downloaded file. It will say latest so we will find the version for future tracking. Mine is 0825d1c647c7 so replace in lower command.
- podman run -ti cli:latest Note the CLI version number it shows in the opening text. My current is 13.15.2 for example
- podman tag 0825d1c647c7 localhost/cli:13.15.2
- podman tag 0825d1c647c7 localhost/cli:latest
- podman image save cli:13.15.2 -o C:/MasFiles/cli-13.15.2.tar
Download the MAS install files
Create a text file: C:\MasFiles\cli-variables.txt.
Populate it with the variables below. Most of these variables are used during the install portion so will not be needed until later, but using the single file will help ensure consistency.
IBM_ENTITLEMENT_KEY=xxxx Replace with client entitlement key
MIRROR_MODE=to-filesystem Once all files are downlaoded, change this to "from-filesystem" for running in the client environment
MIRROR_WORKING_DIR=/mnt/home
REGISTRY_PUBLIC_HOST=localhost
REGISTRY_PRIVATE_HOST=localhost
REGISTRY_PUBLIC_PORT=5000
REGISTRY_PRIVATE_PORT=5000
REGISTRY_USERNAME=user
REGISTRY_PASSWORD=password
REGISTRY_PRIVATE_CA_FILE=/mnt/cert/cert.crt Repalce with Client certificate name (not needed until later)
REDHAT_PULLSECRET=/mnt/pullsecret.txt Needed only for donwloading Redhat Operators. Need to get one from a demo openshift and put in file
OCP_RELEASE=4.14 Update with OpenShift version
SETUP_REDHAT_CATALOGS=false Set to true if importing the Redhat Operators
SETUP_REDHAT_RELEASE=false Set to true if importing the Redhat catalogs
MAS_INSTANCE_ID=dev
MAS_CONFIG_DIR=/mnt/MasFiles
MAS_WORKSPACE_ID=dev
MAS_WORKSPACE_NAME=DEV
SLS_LICENSE_ID=xxxxxxxx Taken from License File. Ask Andy W or Infra how to get it
SLS_LICENSE_FILE=/mnt/MasFiles/entitlement-license-files.lic Needs to be provided by client I think
BAS_PROVIDER=DRO
DRO_CONTACT_EMAIL=person@email.com
DRO_CONTACT_FIRSTNAME=firstname
DRO_CONTACT_LASTNAME=lastnamne
UDS_CONTACT_EMAIL=person@email.com
UDS_CONTACT_FIRSTNAME=firstname
UDS_CONTACT_LASTNAME=lastnamne
PROMETHEUS_ALTERMGR_STORAGE_CLASS=ceph-hdd GET FROM OPENSHIFT
PROMETHEUS_STORAGE_CLASS=ceph-hdd GET FROM OPENSHIFT
PROMETHEUS_USERWORKLOAD_STORAGE_CLASS=ceph-hdd
GRAFANA_INSTANCE_STORAGE_CLASS=ceph-hdd
PROMETHEUS_USERWORKLOAD_RETENTION_PERIOD=15d
PROMETHEUS_RETENTION_PERIOD=15d
MONGODB_STORAGE_CLASS=ceph-hdd
DRO_STORAGE_CLASS=ceph-hdd
CERT_MANAGER_PROVIDER=redhat
MAS_CLUSTER_ISSUER=
MAS_DOMAIN=
MAS_APP_CHANNEL=9.0.x
MAS_CHANNEL=9.0.x
MAS_CATALOG_VERSION=v9-250403-amd64
CONFIGURE_EXTERNAL_DB=true This example is for an external MSSQL database
MAS_JSBC_USER=svc_maximo
MAS_JDBC_PASSWORD=xxxxxxxxx
MAS_JDBC_URL=jdbc:sqlserver://SERVERNAME;portNumer=1433;databaseName=MASDEV;integratedSecurity=false;sendStringParametersAsUnicode=false;selectMethod=cursor;encrypt=true;trustServerCertificate=true;
MAS_CONFIG_SCOPE=system
SSL_ENABLED=true
The below are CLoudPack variables oly needed if installing Cognos
CPD_ENTITLEMENT_KEY=$IBM_LICENSE_KEY
CPD_ENTITLEMENT_USERNAME=name
CPD_INSTALL_PLATFORM=true
CPD_PRODUCT_VERSION=4.8.0 Check version number
CPD_INSTALL_COGNOS=true Cognos
CPD_INSTALL_WSL=false Waston Studio
CPD_INSTALL_WML=false Watson Machine Learning
CPD_INSTALL_SPARK=false Spark Analytics engine
CPD_INSTALL_OPENSCALE=false Watson Openscale
CPD_INSTALL_DISCOVERY=false Watson Discovery
CPD_INSTALL_SPSS=false SPSS modeller
Open a command prompt and run the below command:
podman run -ti -v C:\MasFiles\MasImages:/mnt/home:z --env-file C:\MasFiles\cli-variables.txt localhost/cli
This will run the cli tool and open as a container with a command prompt you can interact with. Run the below commands, replacing variables as required for the client reporitoy details they have provided. (if they have not yet provided these use the ones below).
The catalog and channel will be whichever version you are using.
mas mirror-images --mirror-catalog --mirror-core --mirror-manage --mirror-cfs --mirror-sls --mirror-tsm --mirror-mongo --mirror-appconnect --no-confirm
mas mirror-images --mirror-catalog --mirror-core--no-confirm
If the client is using DB2 run the below command
mas mirror-images --mirror-db2 --no-confirm
If the client is wanting Cognos, run the below command
mas mirror-images --mirror-cp4d --no-confirm
mas mirror-images --mirror-cognos --no-confirm
If the client is wanting any other add-on, run the respective below command
mas mirror-images --mirror-spark --no-confirm
mas mirror-images --mirror-wml --no-confirm
mas mirror-images --mirror-wsl --no-confirm
mas mirror-images --mirror-odf --no-confirm
mas mirror-images --mirror-assist --no-confirm
mas mirror-images --mirror-iot --no-confirm
mas mirror-images --mirror-icd --no-confirm
mas mirror-images --mirror-monitor --no-confirm
mas mirror-images --mirror-optimizer --no-confirm
mas mirror-images --mirror-predict --no-confirm
mas mirror-images --mirror-visualinspection --no-confirm
This will take quite a while and will download the images.
If you have used localhost in the adress, see the troubleshooting section for instructions on how to fix the manifest files address.
Download the operators
There are required Redhat operators. The list is below. Check with the client if these are provided in their OpenShift catalog. If not, then we will downlaod them.
TODO find list of required operators
TODO what is the PULL_SECRET for and why do we need it. Write instructions
To download them, run the following command from the CLI container (update the release to your openshift version)
mas mirror-redhat-images --mirror-operators --pullsecret $REDHAT_PULL_SECRET --release 4.14 --no-confirm
Import the CLI tool and images into the client
Discuss with the client how they would like to recieve the CLI tool and MAS images. Follow whatever procedure they require.
This will probably end up with the files being provided to a network location you can access from a Windows admin VM.
Copy the files to the Linux VM into the u01 folder.
You shoud have:
/u01/MasFiles/cli.tar
/u01/MasFiles/cli-variables.txt
/u01/MasFiles/MasImages with the Mas install images inside (3 folders, logs/manifest/v2)
Confirm that cli-variables are all updated with client values and the MIRROR_MODE is set to from-filesystem.
If incorrect registry details were used during download, see the troubleshooting section on how to fix the manifest files.
In your Linux VM, navigate to /u01/MasFiles/Masimages/v2/ibmmas/mongo/manifests
Type ls to list the files
Look at the filename of the file that starts sha256. If the character after the sha256 is a colon, continue. If not, it will probably look like a box. See troubleshooting on fixing image file names.
Import the CLI tool into the Linux VM
On the linux VM run the below command:
podman load -i /u01/MasFiles/cli.tar
podman image ls
Note the image ID of the one image (the line with <none> <none>)
Run below command replacing the imageID with the one above
podman tag imageID cli:latest
Confirm it runs by running the below command:
podman run -ti \
-v /u01/mnt:z \
-v /u01/cert/cert.crt:/etc/pki/tls/certs/cert.crt \
--env-file /u01/MasFiles/cli-variables.txt \
cli
If you get an error about running out of space or size, see the troubleshooting section on it.
## Mirror the downlaoded files to the client repository
To mirror the files to the client repository run the following commands:
mas mirror-images --mirror-catalog --mirror-core --mirror-manage --mirror-mongo --mirror-tsm --mirror-sls --mirror-appconnect --mirror-cfs --no-confirm
If the client is using DB2 run the below command
mas mirror-images --mirror-db2 --no-confirm
If the client is wanting Cognos, run the below command
mas mirror-images --mirror-cp4d --mirror-cognos --no-confirm
If the client is wanting any other add-on, run the respective below command
mas mirror-images --mirror-spark --no-confirm
mas mirror-images --mirror-wml --no-confirm
mas mirror-images --mirror-wsl --no-confirm
mas mirror-images --mirror-assist --no-confirm
mas mirror-images --mirror-iot --no-confirm
mas mirror-images --mirror-icd --no-confirm
mas mirror-images --mirror-monitor --no-confirm
mas mirror-images --mirror-optimizer --no-confirm
mas mirror-images --mirror-predict --no-confirm
mas mirror-images --mirror-visualinspection --no-confirm
If the client needed the redhat operators brought in, run the below command
mas mirror-redhat-images --mirror-operators --no-confirm
Configure OpenShift for an airgapped environment
To configure OpenShift for an airgapped environment run the below commands from your CLI container
- oc login xxxxxx Get the login command from OpenShift
- mas configure-airgap --no-confirm
If the client is using a prefix for their container regsitry, see the troubleshooting guide on the topic.
Installing MAS
To install MAS, we will use the ansible playbooks. These will take all their required values from the cli-variables.txt file.
Take this point to do a final check that all the values in there are correct.
Run the below commands as required depending on your desired components
oc login xxx Take login command from openshift.
ansible-playbook ibm.mas_devops.oneclick_core
ansible-playbook ibm.mas_devops.oneclick_add_manage
ansible-playbook ibm.mas_devops.oneclick_add_iot
ansible-playbook ibm.mas_devops.oneclick_add_assist
ansible-playbook ibm.mas_devops.oneclick_add_monitor
ansible-playbook ibm.mas_devops.oneclick_add_predict
ansible-playbook ibm.mas_devops.oneclick_add_aibroker
ansible-playbook ibm.mas_devops.oneclick_add_optimizer
ansible-playbook ibm.mas_devops.oneclick_add_visualinspection
ansible-playbook ibm.mas_devops.cp4d
See troubleshooting for known issues
Troubleshooting
Downloading files have the wrong repo address
When downloading the MAS files, you must provide the target repo address. If this is unknown (or fails due to not being able to access it), use localhost:5000 with any user and password.
Then, open the C:\MasFiles\MasImages\manifest\from-filesystem folder.
Open each text file and do a find and replace to replace localhost:5000 with your client repo address and port.
Fixing downloaded image file names
Downloading the images on a windows machine corrupts the file names of the images.
This is due to the Windows default file system not allowing : in file names.
To fix this, do the following:
- On the Linux VM, run the below commands:
- find /mnt -name sha\* > /mnt/MasFiles/filename.log
- From a Windows VM with the WinSCP tool, open the /mnt/MasFiles/filename.log
- Copy the contents into VS Code/Notepadd++/Excel or equivalent tool of your choice
- Each line is a correcupted filename. You need to add an "mv " before each and the corrected filename after so each line looks like:
- mv badFileName goodFileName
- Copy all the lines and paste them into the linux terminal.
This will 'move' all the bad files to the good filename. Effectively renaming them.
## Mirror files to client repo issue - run out of space
When pushing the imported files to the client image repository, the CLI tool runs on a space on the Linux VM in a folder called /var.
On the first client install, this /var was allocated a sapce of 8GB. During the mirror-image command, this was filling up and running out of space.
The fix was to allocate some space from our storage drive to that folder to give it more space.
To check the space, run the following command on the Linux VM
df -h
This will list all the mounts and there are probably about a dozen. Look in the last column to find the /var section. Then look at the Size column. If this is around 8GB then it is the caue of the issue. I found that it needed 24GB to have enough.
Ask the client to increase the size of this folder to 24GB.
I have updated the prerequisites portion to state this should be in the initial request from the client.
Configure Airgap issue - Client container registry has a prefix
If the client container registry has a prefix, this will cause the below files to fail:
- mas-ibm-catalog.yml
- mas-redhad-catalogs.yml If you are importing the redhat operators
This is due to the YAML files trying to create a NAME for a record in OpenShift that uses the Prefix
To fix it, get a copy of these files from the ansible devops pages below
https://github.com/ibm-mas/ansible-devops/blob/master/ibm/mas_devops/roles/ocp_idms/templates/idms/mas-ibm-catalog.yml.j2
https://github.com/ibm-mas/ansible-devops/blob/master/ibm/mas_devops/roles/ocp_idms/templates/idms/mas-redhat-catalogs.yml.j2
For mas-ibm-catalog, remove the from line 4.
For mas-redhat-catalogs, remove the from line 38.
Save these files to the MasFiles folder.
When running your CLI tool from podman, update the command to:
podman run -ti \
-v /u01/mnt:z \
-v /u01/cert/cert.crt:/etc/pki/tls/certs/cert.crt \
- /u01/MasFiles/mas-ibm-catalog.yml.j2:/mascli/ansible-devops/roles/ocp_idms/templates/idms/mas-ibm-catalog.yml.j2:z \
- /u01/MasFiles/mas-redhat-catalogs.yml.j2:/mascli/ansible-devops/roles/ocp_idms/templates/idms/mas-redhat-catalogs.yml.j2:z \
--env-file /u01/MasFiles/cli-variables.txt \
cli
Install issue - admin / all build failed
During the manage install, you may see the admin pod build fail. This is due to an IBM bug not accounting for the offline install of this pods build.
To fix this, in Openshift go to the Builds page and set the project to your manage project. You will see perhaps multiple xxx-buld-config-xx records.
Open the latest of the failed builds.
You can see in the Dockerfile section in the lower right that it is trying to reach out to ibm.icr
Go to the YAML tab. Scroll down to find the Dockerfile section.
Update the target url to insert your client registry in place of the icr.io.
In the actions in the top right, select 'Run Build'
This will re-run the build with the correct address.
If it did this for the admin build, it will do it for the EAR builds, one per server bundle type. You will need to repeat the above process for each of them.
Install issue - license operator bundle failed
This issue occurred due to the ibm-license-operator-app image files not being pushed into the client repository. The reason for this is not known. However, the fix is.
The below command can be run from the CLI tool to push all the images from a specific manifest file up to the target repository
oc image mirror -a /opt/app-root/src/.ibm-mas/auth.json -f /mnt/MasFiles/manifests/from-filesystem/xxxxxxxxxx --filter-by-os '.*' --insecure --skip-multipl-scopes --max-per-registry=1 --dir=$MIRROR_WORKING_DIR
Install issue - redirect not applied - redirect conflict
If you have run the configure airgap, but when a pod is trying to spin up and it is still pointing to the icr.io location, it may be due to a conflict between the MAS redirect policies and the client ones.
Search in the OpenShift API explorer for ImageDigestMirrorSet (IDMS) and ImageContentSourcePolicy (ICSP) records.
If there any client ones, compare then to the IDMS records created by the configure-airgap command.
If there are any conflicts, remove the redirect from the MAS one and add the target as an alternative mirror to the client record.
Missing files
skopeo login cp.icr.io -u cp -p eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJJQk0gTWFya2V0cGxhY2UiLCJpYXQiOjE2NzY5MDAwMjYsImp0aSI6IjgyZjUzM2U4NzBiNTQxZDZiMzcyYTQwYzg2YTdhMWYwIn0.RomGzcDbBul844XKBaNQM_KV3NrQ0C1kCO6XLr7nkPk
skopeo copy --preserve-digests docker://icr.io/cpopen/ibm-cpd-ws-runtimes-operator-bundle@sha256:4de17c6d3b86e52a3cc9c1bd39cf467dc4a018e9370d5ac4acb11031f35f37a3 dir://mnt/c/Users/DavidShaw/documents/ibm-cpd-ws-runtimes-operator-bundle
skopeo copy --preserve-digests docker://icr.io/cpopen/opencontent-elasticsearch-operator-bundle@sha256:a24bbe1a1eaedb1ef75c8eb41eac714a48c68eb5246196686792b958f827a849 dir://mnt/c/Users/DavidShaw/documents/opencontent-elasticsearch-operator-bundle
skopeo copy --preserve-digests docker://icr.io/cpopen/ibm-cpd-datarefinery-operator-bundle@sha256:0d51d202ed9e5e3858172c91ff8a31248d4d6409cfa62c3454035e37b0e457cc dir://mnt/c/Users/DavidShaw/documents/ibm-cpd-datarefinery-operator-bundle
skopeo copy --preserve-digests docker://icr.io/cpopen/ibm-cpd-ccs-operator-bundle@sha256:55a6a68522a56f01ca1dce57940f3a8a2e2da383fa40bf92a8c7447e6fc1e63c dir://mnt/c/Users/DavidShaw/documents/ibm-cpd-ccs-operator-bundle
icr.io/cpopen/ibm-cpd-ws-runtimes-operator-bundle@sha256:4de17c6d3b86e52a3cc9c1bd39cf467dc4a018e9370d5ac4acb11031f35f37a3
icr.io/cpopen/opencontent-elasticsearch-operator-bundle@sha256:a24bbe1a1eaedb1ef75c8eb41eac714a48c68eb5246196686792b958f827a849
icr.io/cpopen/ibm-cpd-datarefinery-operator-bundle@sha256:0d51d202ed9e5e3858172c91ff8a31248d4d6409cfa62c3454035e37b0e457cc
icr.io/cpopen/ibm-cpd-ccs-operator-bundle@sha256:55a6a68522a56f01ca1dce57940f3a8a2e2da383fa40bf92a8c7447e6fc1e63c