Harbor Container Registry¶

About this task

Harbor is an open-source container registry with a richer and fuller set of capabilities than the built-in StarlingX container registry. StarlingX end users can use Harbor to manage their own application container images. Harbor secures artifacts with policies and role-based access control, ensures images are scanned and free from vulnerabilities, and signs images as trusted. Harbor has been evolved to a complete OCI compliant cloud-native artifact registry.

With Harbor V2.0, you can manage images, manifest lists, Helm charts, CNABs, OPAs and other artifacts adhering to the OCI image specification. It supports operations such as pulling, pushing, deleting, tagging, replicating, and scanning these artifacts. Additionally, you can now sign images and manifest lists.

Harbor also supports the replication of images between registries and offers advanced security features such as user management, access control, and activity auditing.

See https://goharbor.io/docs/2.0.0/ for more details on Harbor.

Harbor Installation¶

Prerequisites

CephFS backed PVCs are recommended for Harbor such that when configuring multiple replicas, for AIO-DX or Standard configurations, both Harbor replicas can read and write to the registry.

Create a secret as described below:

Generate certificates and create secret.

A CA cert and server cert creation procedure using cert-manager is specified below:

Create the certificate for Harbor using Cert-Manager and using the local CA, system-local-ca, as the issuer. Note that the certificate should be created in the harbor-tls SECRET in the Harbor NAMESPACE.

For example:

Create the following Harbor certificate yaml configuration file:

~(keystone_admin)]$ cat <<EOF > harbor-certificate.yaml
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: harbor-certificate
  namespace: harbor
spec:
  secretName: harbor-tls
  issuerRef:
    name: system-local-ca
    kind: ClusterIssuer
  duration: 2160h # 90 days
  renewBefore: 360h # 15 days
  commonName: < oam floating IP Address or FQDN >
  subject:
    organizations:
      - ABC-Company
    organizationalUnits:
      - StarlingX-harbor
  ipAddresses:
  - < oam floating IP address >
  dnsNames:
  - < harbor dns> # e.g. harbor.yourdomian.com
  - < notary dns > # optional, required only if exposed on ingress e.g. notary.yourdomian.com
EOF

Create the Harbor namespace:

~(keystone_admin)]$ kubectl create namespace harbor

Apply the configuration:

~(keystone_admin)]$ kubectl apply -f harbor-certificate.yaml

Verify the configuration:
```
~(keystone_admin)]$ kubectl get certificate harbor-certificate -n harbor
```
After successful configuration, the certificate’s Ready status will be True.

Procedure

Locate the Harbor system application tarball in /usr/local/share/applications/helm.

For example:
```
/usr/local/share/applications/helm/harbor-<version>.tgz
```

Upload the Harbor application.

~(keystone_admin)]$ system application-upload /usr/local/share/applications/helm/harbor-<version>.tgz

Configure the Helm Overrides for Harbor.

Expose the Harbor application externally with either nodePort or Ingress.

nodePort

Create Harbor using NodePort to expose the service

Note

The instructions below assume that the NodePorts 30002, 30003, and 30004 are available (i.e., not used by any other applications). If these ports are unavailable, please choose and configure alternative ports that are not in use.

Put the following nodePort overrides in values.yaml:

expose:
  type: nodePort                                                # Type should be nodeport
  tls:
    enabled: true
    certSource: secret
    secret:                                                     # Certificate Source is secret
      secretName: "harbor-tls"          # A secret containing tls.crt and tls.key
      notarySecretName: "harbor-tls"            #  A secret containing tls.crt and tls.key

  nodePort:
    # The name of NodePort service
    name: harbor
    ports:
      http:
        # The service port Harbor listens on when serving HTTP
        port: 80
        # The node port Harbor listens on when serving HTTP
        nodePort: 30002
      https:
        # The service port Harbor listens on when serving HTTPS
        port: 443
        # The node port Harbor listens on when serving HTTPS
        nodePort: 30003
      # Only needed when notary.enabled is set to true
      notary:
        # The service port Notary listens on
        port: 4443
        # The node port Notary listens on
        nodePort: 30004
externalURL: https://harbor.yourdomian.com:30003     # URL of harbor listing on 30003 port

Ingress

Create Harbor using Ingress to expose the service.

Note

The instructions below assume that the URL harbor.yourdomain.com has been configured in the DNS server owning yourdomain.com as the OAM FLOATING IP Address of StarlingX.

Put the following Ingress overrides in values.yaml:

expose:
  type: ingress.                                                # Type should be ingress
  tls:
    enabled: true
    certSource: secret
    secret:                                                     # Certificate Source is secret
      secretName: "harbor-tls"                  # Above created secret name
      notarySecretName: "harbor-tls"            # Above created secret name
  ingress:
    hosts:
      core: harbor.yourdomian.com               # Harbor Domain name
      notary: notary.yourdomian.com             # Notary Domain name
    annotations:
      kubernetes.io/ingress.class: nginx.       # Add ingressclass name. It would be "nginx" if you are using default ingress controller.
      nginx.org/client-max-body-size: "0".    # Add this notation for nginx otherwise nginx will reject the image pull & push
externalURL: https://harbor.yourdomian.com      # URL of harbor

For AIO-DX and Standard setup, add the following storageClass and accessMode overrides for PVC used for Harbor-Jobservice and Harbor-Registry microservice.

Note

Set the registry size according to your requirements considering the number and size of images that you will have in this registry.

Example for nodePort:

persistence:
  enabled: true
  resourcePolicy: "keep"
  persistentVolumeClaim:
    registry:
      existingClaim: ""
      storageClass: "cephfs"
      subPath: ""
      accessMode: ReadWriteMany
      size: 100Gi
      annotations: {}
    jobservice:
      jobLog:
        existingClaim: ""
        storageClass: "cephfs"
        subPath: ""
        accessMode: ReadWriteMany
        size: 1Gi
        annotations: {}

Example for Ingress:

persistence:
  enabled: true
  resourcePolicy: "keep"
  persistentVolumeClaim:
    registry:
      existingClaim: ""
      storageClass: "cephfs"
      subPath: ""
      accessMode: ReadWriteMany
      size: 100Gi
      annotations: {}
    jobservice:
      jobLog:
        existingClaim: ""
        storageClass: "cephfs"
        subPath: ""
        accessMode: ReadWriteMany
        size: 1Gi
        annotations: {}

Update the Helm overrides.

~(keystone_admin)]$ system helm-override-update harbor harbor harbor  --values values.yaml

Execute Helm overrides.

~(keystone_admin)]$ system helm-override-update harbor harbor harbor  --values values.yaml

Apply/Create the Harbor system application.

~(keystone_admin)]$ system application-apply harbor

Configure LDAP Authentication for Harbor Registry¶

Prerequisites

The URL for accessing the Harbor web interface is the externalURL set in the Helm override above in the step Configure Helm Overrides for Harbor.
The default admin username is ‘admin’, and the password is ‘Harbor12345’.

To configure Harbor to use StarlingX Local LDAP for authentication, follow the instructions in Configure LDAP/Active Directory Authentication with the following values.

For StarlingX local LDAP:

LDAP URL: ldap://controller

LDAP search DN: cn=ldapadmin,dc=cgcs,dc=local

LDAP Search Password: <Password of ldapadmin>

LDAP Base DN: dc=cgcs,dc=local

LDAP UID:  cn

You can find <Password of ldapadmin> in /etc/ldap/slapd.conf.backup.

Push an Image to a <project> in Harbor¶

Note

Depending on your docker setup, you may be required to run all of the following commands with ‘sudo’.

Docker Login.
```
docker login <harbor_address> -u <username>
```
Where <harbor-address> is either:
- for ‘Ingress’ expose: harbor.yourdomian.com
- for ‘NodePort’ expose: https:// <oam-floating-ip>:30003 and <username> is your actual username

Tag the image.

docker tag redis:latest <harbor_address>/<project>/redis:latest

Push the image.

docker push <harbor_address>/<project>/redis:latest

Pull an Image from Harbor¶

Use command to pull an image:

docker pull <harbor_address>/<project>/ redis:latest

Where <harbor-address> is either:

for 'Ingress' expose: harbor.yourdomian.com
for 'NodePort' expose: https:// <oam-floating-ip>:30003

Push a Helm Chart as an OCI Object¶

helm package <Chart_Path>

helm push <chart_package> oci://<harbor_address>/<project>/

Pull a Helm Chart¶

helm pull oci://<chart_package/<project>/harbor

Project Management¶

A project in Harbor contains all the repositories (images) of a particular application; e.g. the StarlingX project in Harbor would contain all the images from StarlingX.

For more details on creating and configuring a project see: Create Projects and Project Configuration.

Use Images from a Remote Registry¶

A project in a local Harbor registry can be used as a pull-through cache for a remote registry. For example, a Harbor registry project on a StarlingX Subcloud can be used as a pull-through cache of a Harbor registry project on its StarlingX system controller. This can be achieved using a proxy project which acts as a proxy to the remote registry, or using a replication of the remote registry.

Add a proxy project

Proxy cache allows you to use Harbor to proxy and cache images from a target public or private registry. It does not allow to push the image by the user. Below are the steps to create a proxy project:

Create a registry endpoint. See Creating Replication Endpoints.
Create a new project using the above created registry endpoint. See Configure Proxy Cache.

Add Replication Rule

Setting up pull based replication replicates the images from the remote registry based on a trigger. Trigger can be Manual, scheduled, or event based as shown below.

Create a replication endpoint. See Creating Replication Endpoints.
Create a new replication rule with replication mode as pull based as describe in Creating a Replication Rule.

Add Users to the Project¶

You can add individual users to a project, for more details see Assign Users to a Project.

LDAP/AD users can be added to the project.
LDAP/AD groups to projects and assign a role to the group.
You can see the various roles and user permission associated with the roles in User Permissions By Role.

Configure Signed Images Support¶

For more details see: Implementing Content Trust.

Select cosign or notary for content trust. Harbor will then only allow verified images to be pulled from the project.
Enable content trust by setting the following environment variables on the machine on which you run the Docker client.
```
export DOCKER_CONTENT_TRUST=1

export DOCKER_CONTENT_TRUST_SERVER=https://<harbor_address>:<notary port[30004]>
```
If you push the image for the first time, you will be asked to enter the root key passphrase.

Configure Trivy Scanner Plugin¶

Trivy is installed and configured as a default scanner.

../../_images/interrogation-services.png

Configure Size of Registry DB¶

Registry DB size can be configured by setting following in values.yaml under:

persistence:
    registry:
      size: 5Gi
    jobservice:
      jobLog:
        size: 1Gi

Set the the value (Default set to 5Gi).

system helm-override-update harbor harbor harbor --values values.yaml

Apply the change:
```
system application-apply harbor
```

Enforcement of Image Security Policies Using Portieris¶

Portieris allows to configure trust policies for an individual namespace or cluster-wide and checks the image against a signed image list on a specified notary server to enforce the configured image policies. An Administrator can enforce StarlingX image security policies using the Portieris admission controller.

It is required to pull from a registry using a docker-registry secret. Enforcing trust for anonymous image pulls is not supported.

To use portieris, an administrator needs to follow below steps:

Install portieris as specified in Install Portieris.

Configure image policy to allow images from Harbor registry + notary as specified Portieris ClusterImagePolicy and ImagePolicy Configuration. Below example shows the policy allowing image from Harbor registry.

apiVersion: portieris.cloud.ibm.com/v1
kind: ImagePolicy
metadata:
 name: allow-custom
  namespace: harbor
spec:
repositories:
    - name: "<harbor-address>:30003/*"
    policy:
        trust:
        enabled: true
        trustServer: "https://<notary address>:30004" # Optional, custom trust server for repository

Create a SECRET with a Harbor username and password, to use as an ImagePullSecret in a POD spec.
```
kubectl create secret docker-registry \
-n harbor harbor-registry-secret \
--docker-server=<harbor-address>:port \
--docker-username=admin \
--docker-password=Test@123
```
Note

If the pod creation with the above secret fails, the user should try with new secret with --docker-server as <harbor-address>.

Pull a signed image from Harbor registry in a pod using harbor-secret created above. Please note that image policy and pod should be created in the same namespace.

apiVersion: v1
kind: Pod
metadata:
  name: test-pod-public
spec:
  containers:
  - command:
    - sleep
    - '3600'
    image: <harbor-address>:30003/public-demo/redis:latest
    imagePullPolicy: Always
    name: test-pod
  tolerations:
  - key: "node-role.kubernetes.io/master"
    operator: "Exists"
    effect: "NoSchedule"
  imagePullSecrets:
  - name: harbor-registry-secret

Limitation¶

Harbor application cannot be deployed during bootstrap due to the bootstrap deployment dependencies such as early availability of storage class.