Harbor Project - The Maintainers Session - Orlin Vasilev & Vadim Bauer

Harbor Project - The Maintainers Session

Speakers: Orlin Vasilev, Harbor Community Manager & Maintainer; Vadim Bauer

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=1UHZT_v0rts

Overview

This talk, delivered by Orlin Vasilev and Vadim Bauer at KubeCon EU, provides a comprehensive update on the Harbor project, a graduated cloud-native container registry within the CNCF ecosystem. The session primarily focuses on the upcoming Harbor 2.13 release, highlighting new features such as enhanced audit logging and first-class support for Artificial Intelligence (AI) models. A significant portion of the presentation is dedicated to a deep dive into Harbor Satellite, an ambitious new sub-project designed to address the complex challenges of artifact distribution across multi-site and edge computing environments.

Orlin Vasilev, the Harbor Community Manager and a maintainer, emphasizes the importance of community engagement and outlines areas where contributions are actively sought. Vadim Bauer, a key contributor to Harbor Satellite, meticulously details the architecture, capabilities, and future vision for this innovative solution. The talk underscores Harbor's continued evolution as a critical component in the cloud-native landscape, adapting to emerging trends like AI model management and the growing demands of distributed systems. The introduction of Harbor Satellite, in particular, signals a strategic move to simplify and secure artifact delivery at scale, addressing a pain point for organizations deploying applications to hundreds or thousands of remote locations.

The importance of this talk lies in its dual focus: providing practical updates for existing Harbor users and unveiling a forward-looking solution for a pervasive industry challenge. For developers, operators, and security professionals managing cloud-native deployments, understanding Harbor's latest features and the potential of Harbor Satellite is crucial. It offers insights into how to better secure, manage, and distribute container images, Helm charts, and now AI models across complex, distributed infrastructures, ultimately enhancing the reliability and efficiency of modern software supply chains.

Background

▶ Watch: Welcome and Harbor 2.13 RC1 release announcement. (0:00)

Harbor stands as a foundational component in the cloud-native ecosystem, recognized as one of the oldest and a graduated project within the Cloud Native Computing Foundation (CNCF). It functions as an open-source, cloud-native registry that stores, signs, and scans container images and other cloud-native artifacts. While Harbor excels as a central repository, the landscape of modern application deployment increasingly involves highly distributed environments, often encompassing edge computing, IoT devices, and multiple cloud regions or on-premises data centers. This presents significant challenges for artifact distribution.

Historically, delivering software artifacts to these multi-site locations has been fraught with difficulties. Key problems include unreliable network connectivity, varying network topologies (from high-bandwidth data centers to intermittent, low-bandwidth edge devices), and the sheer scale of managing updates across hundreds or thousands of disparate sites. Traditional methods, such as direct pulls from a central registry or custom scripting, often prove inefficient, insecure, or unmanageable at scale. Previous attempts to solve this, like projects such as Dragonfly or Kraken, aimed to introduce peer-to-peer distribution mechanisms. However, as Vadim Bauer notes, these solutions often proved to be "really, really complicated things and to operate," especially when constrained by the limited resources of small edge devices.

The need for a simpler, more robust, and centrally manageable solution became evident. This problem is further exacerbated by the increasing size and complexity of artifacts, such as large AI models, which can easily exceed 10 gigabytes. Distributing such payloads efficiently and securely to numerous remote locations without overwhelming network infrastructure or requiring extensive local management is a critical requirement.

In parallel, the Harbor project itself continues to evolve, with maintainers actively seeking community involvement. Orlin Vasilev highlighted specific areas of need, including UI/UX contributions, project management and release coordination, CI/CD pipeline development (migrating to GitHub Actions and CNCF cluster infrastructure), and enhancing test coverage and functionality verification. This ongoing development and community engagement underscore Harbor's commitment to adapting to new challenges and expanding its feature set to meet the demands of the cloud-native community. The introduction of Harbor 2.13, with its focus on improved auditability and support for new artifact types, lays the groundwork for more advanced capabilities like Harbor Satellite.

Key Findings

▶ Watch: Call for adopter organizations to share use cases. (3:50)

The talk presented several significant updates and new directions for the Harbor project, with the most prominent being the release of Harbor 2.13 RC1 and the detailed unveiling of Harbor Satellite.

For the core Harbor registry, the 2.13 release (with a targeted General Availability date around April 11, post-conference) brings several enhancements. A major improvement is the extension of audit logs, which now capture a broader range of events—currently tracking "14-15 events." This functionality is configurable, allowing users to enable or disable specific event types, providing greater control over monitoring and compliance efforts. Crucially, Harbor 2.13 introduces AI models as first-class citizens. This means users can now upload and manage large AI models (e.g., "10 gigs") within Harbor, viewing associated metadata and details, and downloading them directly. This feature positions Harbor as a more versatile artifact management solution beyond just container images.

Updates to Harbor's sub-projects were also discussed:

• The Terraform provider remains healthy and actively supported, with contributions from OVH Cloud.
• The Pulumi provider is currently lacking resources, presenting an opportunity for community contribution.
• A Crossplane provider is in development, with a company internally building it and planning to donate it to the Harbor project for future support.
• The Harbor Operator is under review for potential deprecation and archiving due to low reported usage and inactivity.
• The Harbor CLI, developed through the Linux Foundation mentorship program, is nearing its GA release and is already in a usable state.
• Harbor Satellite is actively being developed, also benefiting from the Linux Foundation mentorship program.

The most substantial "key finding" of the talk is the comprehensive overview of Harbor Satellite. This new sub-project is presented as a "centrally managed artifact distribution solution for multi-site locations." It aims to simplify the delivery of artifacts to diverse environments, including IoT, edge devices, and different cloud providers, by ensuring consistent image distribution across "hundreds and thousands of satellite locations." Satellite distinguishes itself from prior complex solutions like Dragonfly and Kraken by focusing on simplicity of operation. Its core features include:

• State managed in OCI: The desired state and configuration of each satellite are stored as versioned, signable, and roll-backable artifacts within an OCI registry.
• Reconciliation loop: Satellites continuously reconcile their local state with the desired state defined centrally.
• Outbound-only connections: Edge satellites initiate connections back to the central Harbor instance, simplifying network configurations and improving security posture.
• No database dependency on the edge: Satellites are designed to run without complex external databases like PostgreSQL, reducing their operational footprint.
• Minimal configuration: Initial setup requires only a token and remote URL; all other configurations can be managed centrally.
• Scalability: The architecture is designed to scale to "more than 10,000 satellites."

Looking to the future, Vadim Bauer hinted at a particularly innovative use case for Harbor Satellite on Kubernetes leveraging Spiegel. Spiegel, described as a peer-to-peer registry interface, aims to use the container runtime's storage (where image layers are typically stored) as the registry storage. This would enable the deployment of a "fully stateful, high available container registry on a completely stateless cluster" at the edge, representing a significant advancement for resilient edge workloads.

Technical Deep Dive

▶ Watch: Brainstorming future features for Harbor 2.14+. (4:50)

The technical core of the presentation revolved around the enhancements in Harbor 2.13 and the detailed architecture of Harbor Satellite.

Harbor 2.13 Core Enhancements

1. Extended Audit Logs: Harbor 2.13 significantly expands its auditing capabilities. The system now tracks a more comprehensive set of user and system events, currently encompassing "14 to 15 different event types." This provides administrators with finer-grained visibility into actions performed within the registry. A key feature is the ability to disable or enable different event types, allowing organizations to tailor their audit trails to specific compliance requirements or operational monitoring needs. This flexibility is crucial for improving the traceability of actions, aiding in security investigations, and demonstrating adherence to regulatory standards.

1. AI Models as First-Class Citizens: A notable addition to Harbor's artifact support is the native handling of AI models. Previously, Harbor primarily managed container images and Helm charts. With 2.13, AI models, which can often be very large (e.g., "10 gigs"), are treated as distinct artifact types. The user interface now provides specific views for these models, displaying details and metadata. This integration allows organizations to leverage Harbor's existing security features, such as vulnerability scanning, signing, and immutability, for their AI/ML pipelines, ensuring the integrity and provenance of critical machine learning assets throughout their lifecycle.

Harbor Satellite: Architecture and Workflow

Harbor Satellite is designed as a robust, scalable solution for artifact distribution to distributed environments. Its architecture comprises several key components and interaction patterns:

1. Central Components (West Side):

• Harbor Registry: The primary, central Harbor instance where all artifacts (images, charts, AI models) are initially stored and managed.
• Ground Control: A new component responsible for the centralized management of all satellite instances. Ground Control handles site registration, allowing individual satellites to securely enroll. It also provides Fleet Management capabilities, enabling administrators to group satellites logically. This grouping allows for layered artifact distribution, where a base layer of common images can be applied across all sites, followed by site-specific, customer-specific, or application-specific layers. This hierarchical approach significantly simplifies the management of artifact sets across "hundreds and thousands of sites" with a manageable number of policies (e.g., "10-30 groups").

1. Edge Components (East Side):

• Satellite Edge: The core binary running on each edge location. It is responsible for fetching state and configuration from Ground Control/Harbor and reconciling its local state.
• Registry: A lightweight OCI-compliant registry baked into the Satellite Edge binary. While a default registry is provided, users have the flexibility to swap it out for any other OCI-compliant registry if specific requirements dictate.

1. Communication and State Management:

• Device Registration: Satellites register with Ground Control, obtaining a token that authenticates them and dictates what information they should receive.
• OCI for Artifacts and State: The primary communication channel between Satellite and the central Harbor/Ground Control for fetching artifacts and state is OCI (Open Container Initiative).
• State File: A critical innovation is the management of the satellite's desired state as a versioned OCI artifact. This "state file" is transferred via OCI, allowing for:
• Version control: Every change to the desired state is versioned.
• Signing: The state file can be cryptographically signed, ensuring its integrity and authenticity.
• Rollback capability: Administrators can easily roll back a satellite's configuration or artifact set to a previous, known-good state. This also extends to the state config, which manages the satellite's configuration and is similarly versioned and signable via OCI.
• Outbound-Only Connections: A fundamental security and operational design choice is that satellites only initiate outbound connections (east to west, from the edge to the central Harbor). This simplifies firewall configurations, enhances security by reducing the attack surface on edge devices, and accommodates environments behind restrictive firewalls or with intermittent connectivity.
• No Database on the Edge: To minimize resource requirements and operational complexity, Harbor Satellite is designed to run without an external database (like PostgreSQL) on the edge. It stores its operational state and image metadata alongside the application, making it lightweight and easy to deploy.
• Minimal Configuration: Edge deployment is streamlined; initially, only a token and the remote URL for Ground Control are required. All subsequent configuration changes can be managed and pushed downstream from Ground Control.

1. Eventing System:

• Recognizing the vast array of unique edge use cases (e.g., different behaviors for 4G vs. 2G connectivity, or no connection), Satellite incorporates a flexible Eventing System.
• When a state change occurs, an event is generated. Instead of trying to implement all bespoke logic within Satellite, the system can "spin up a container that you tell us that we should spin up." This container receives state parameters and can then execute custom logic. It can trigger further actions, such as pausing, resuming, or delaying replication, or feeding back information to Ground Control. This externalized approach makes Satellite highly adaptable without increasing its core complexity.

1. Runtime Component:

• To simplify the developer experience at the edge, a Runtime Component is included. Its purpose is to automatically modify the container runtime configuration (e.g., Docker, Containerd) on the edge device.
• This modification directs the runtime to pull images from the local Satellite registry instead of the central Harbor. The design ensures that whatever is pushed to a specific namespace in central Harbor will correspond to the same namespace on the satellite, maintaining consistency, though the domain name will differ.

Workflow Summary:

A developer pushes artifacts to the central Harbor. Using Harbor's existing replication policies, administrators define which artifacts belong to which satellite groups. Ground Control then generates an OCI State artifact based on these policies. The Satellite Edge instances fetch this OCI State and continuously reconcile their local artifact set and configuration to match the desired state. During this process, the Eventing System can be invoked for custom actions. Finally, the local container runtime, configured by the Runtime Component, pulls artifacts directly from the local Satellite registry.

Future and Roadmap: Satellite on Kubernetes with Spiegel

A compelling future direction for Harbor Satellite is its integration with Kubernetes at the edge, leveraging a project called Spiegel.

• Spiegel is described as a "peer-to-peer registry" interface that operates at a very low level. It enables nodes within a Kubernetes cluster to share information about the image blobs they possess.
• The innovative aspect of Spiegel is its ability to use the container runtime's storage (where image layers are cached) as the backend for the local registry.
• This combination would allow for running a "fully stateful, high available container registry on a completely stateless cluster." This concept is revolutionary for edge deployments, where resources are often limited, and resilience is paramount. It means that even if individual nodes or the underlying cluster state is lost, the registry can maintain its state and availability by leveraging the distributed, peer-to-peer nature of Spiegel and the inherent storage of the container runtime.

Demo / Proof of Concept

▶ Watch: Harbor now supports AI models as first-class citizens. (7:00)

The talk, being a maintainer session and project update, did not include a live demonstration or proof of concept of the new features or Harbor Satellite in action. Orlin Vasilev did briefly show a screenshot of the user interface for managing AI models within Harbor 2.13, illustrating how metadata and details for these models would be displayed.

Regarding Harbor Satellite, Vadim Bauer indicated that "quite a few workloads are working" and that "a lot of things are working there's no release yet but it's all already in a usable usable state." While the Eventing System component was noted as not yet implemented, the core Satellite and Ground Control functionalities, along with the necessary changes to the central Harbor, are progressing well. The discussion focused on the architectural design and capabilities rather than a live operational demonstration.

Defensive Implications

▶ Watch: Updates on Terraform, Pulumi, and Crossplane providers. (8:00)

The advancements in Harbor 2.13 and the introduction of Harbor Satellite carry significant defensive implications for organizations managing cloud-native software supply chains, particularly in distributed and edge environments.

For Core Harbor Users (2.13 and beyond):

1. Enhanced Observability with Extended Audit Logs: The expanded audit logging capabilities in Harbor 2.13 are a crucial defensive tool. By capturing "14-15 different event types" and allowing selective enablement/disablement, security teams gain granular visibility into activities within the registry. This enables:

• Improved Incident Response: Faster detection and investigation of unauthorized access, suspicious artifact modifications, or policy violations.
• Compliance Adherence: Easier demonstration of controls for regulatory requirements by providing a comprehensive, tamper-evident record of actions.
• Threat Hunting: The ability to analyze audit trails for patterns indicative of reconnaissance, privilege escalation, or data exfiltration attempts. Defenders should configure audit logs to capture critical events and integrate them with SIEM (Security Information and Event Management) systems for centralized monitoring and alerting.

1. Securing AI/ML Supply Chains with First-Class AI Model Support: Treating AI models as first-class citizens in Harbor allows organizations to extend existing software supply chain security practices to machine learning assets. This means:

• Integrity and Authenticity: AI models, often large and critical, can now be signed, scanned for vulnerabilities (if applicable, e.g., for embedded code or dependencies), and managed with immutability policies within Harbor. This helps prevent tampering and ensures that only trusted models are deployed.
• Provenance and Traceability: Harbor can serve as a single source of truth for AI models, providing a clear audit trail of who published which model version and when, crucial for debugging, auditing, and compliance in MLOps workflows.
• Vulnerability Management: While direct scanning of models might be nascent, the ability to store and manage them securely provides a foundation for future vulnerability scanning specific to AI model components or dependencies.

For Multi-Site and Edge Deployments (Harbor Satellite):

1. Centralized Control with Decentralized Execution: Harbor Satellite's "centrally managed artifact distribution solution" offers a powerful defensive posture. A single Ground Control instance can manage artifact distribution policies for thousands of edge sites. This reduces the administrative overhead and potential for misconfiguration that often plagues highly distributed systems. Security policies, such as which images are allowed on which groups of edge devices, can be enforced from a single point.

1. Enhanced Security via Outbound-Only Connections: The design principle of "satellites always kind of east-west and never west-east" significantly improves the security model for edge deployments. Edge devices, often in less secure or physically exposed environments, do not need to expose inbound network ports. This:

• Reduces Attack Surface: Limits the ways attackers can initiate connections to compromise edge devices.
• Simplifies Firewall Rules: Makes network segmentation and firewall configuration much easier for edge networks, as only outbound connections to the central Harbor are required.
• Mitigates Lateral Movement: If an edge device is compromised, the lack of inbound connectivity makes it harder for attackers to leverage it as a pivot point into the central network.

1. Integrity and Rollback with OCI-based State: Storing the desired state and configuration of satellites as versioned, signable OCI artifacts is a robust defensive mechanism.

• Tamper Detection: Cryptographic signing ensures that any unauthorized modification to the desired state or configuration can be immediately detected.
• Disaster Recovery and Rollback: The versioning allows for quick and reliable rollbacks to previous, known-good configurations in case of accidental misconfiguration, faulty updates, or even successful attacks that alter the desired state. This is critical for maintaining operational continuity and resilience in highly distributed environments.

1. Resilience through No Database on the Edge: By eliminating the need for a complex external database on edge devices, Satellite reduces the operational burden and the potential attack surface. Databases are often targets for attackers, and managing them securely at scale on resource-constrained edge devices is challenging. Satellite's lightweight design improves the overall security and stability of edge deployments.

1. Flexible Defensive Responses with the Eventing System: The externalized Eventing System provides a powerful hook for custom security responses. When state changes occur (e.g., an image fails to pull, connectivity issues arise), defenders can trigger custom containers that:

• Initiate Alerts: Notify security teams of anomalies.
• Implement Remediation: Automatically pause replication, switch to a backup artifact source, or trigger local security checks.
• Adapt to Conditions: Respond differently based on network conditions or local security posture, allowing for context-aware defensive actions.

1. Future-Proofing with Stateless Edge Registry (Spiegel): The planned integration of Satellite on Kubernetes with Spiegel offers an advanced defensive capability. Running a "fully stateful, high available container registry on a completely stateless cluster" means that edge nodes can be treated as ephemeral and easily replaced without losing registry state. This significantly enhances resilience against node compromise or failure, as a compromised node can be quickly decommissioned and replaced without impacting artifact availability or integrity.

In summary, Harbor's evolution, particularly with Satellite, provides a comprehensive set of tools and architectural patterns for securing the cloud-native software supply chain from the data center to the furthest edge. Defenders should prioritize leveraging these features to enhance visibility, control, integrity, and resilience across their distributed environments.

Key Takeaways

• Harbor 2.13 Enhances Observability and AI Model Management: The latest Harbor release introduces extended audit logs with configurable event types, significantly improving visibility and traceability. Crucially, it establishes AI models as first-class citizens, enabling secure storage, management, and distribution of large machine learning assets.
• Harbor Satellite Addresses Multi-Site/Edge Artifact Distribution Challenges: This new sub-project provides a centrally managed solution for distributing container images and other artifacts to thousands of remote locations, overcoming issues of unreliable connectivity and complex topologies with a focus on simplicity.
• OCI is Key to Satellite's Integrity and Manageability: Harbor Satellite leverages OCI for managing its desired state and configuration as versioned, signed, and roll-backable artifacts. This ensures the integrity, authenticity, and recoverability of edge deployments.
• Secure and Lightweight Edge Architecture: Satellite features outbound-only connections from edge devices, simplifying network security and firewall rules. It operates without a complex database on the edge, reducing its footprint and operational overhead, making it ideal for resource-constrained environments.
• Flexible Eventing and Future Innovations: The Eventing System allows for highly customizable reactions to state changes at the edge, adapting to unique operational requirements. Future plans include integrating with Spiegel to enable highly available, stateful registries on completely stateless Kubernetes clusters for enhanced edge resilience.
• Community Contribution is Actively Sought: The Harbor project is an open-source initiative that actively seeks community involvement in areas such as UI/UX, project management, CI/CD pipeline development, and test coverage to continue its growth and evolution.

About the Speaker(s)

Orlin Vasilev serves as the Harbor Community Manager and is a Maintainer of the Harbor project. His role involves fostering the growth and engagement of the Harbor community, ensuring its health, and driving adoption. He is instrumental in facilitating contributions, organizing community events, and communicating project updates.

Vadim Bauer is a key contributor to the Harbor project, with a specific and active focus on the development of the Harbor Satellite sub-project. His expertise lies in addressing the complex technical challenges of artifact distribution in multi-site and edge computing environments, leading the architectural design and implementation of solutions like Harbor Satellite.