Skip to content
Private S3 Package Repo

Private S3 Package Repo

Publish rpm and deb trees to private S3, rebuild indexes, replicate Syd → Akl, and install with dnf and apt through regional gateway endpoints.
Get Started

Why this lab

Section titled “Why this lab”

You want the same private yum and apt trees in Sydney and Auckland — packages plus repodata / dists, not plain objects. A full Pulp stack can host those repos, but you own the servers, upgrades, and network path. Here S3 holds the rpm and deb layout, a Lambda rebuilds indexes after publish, and each consumer VPC reads its local bucket through a gateway VPC endpoint
S3 gateway VPC endpoint. Route-table target that keeps S3 traffic on the AWS network; no hourly charge for the gateway itself.
.

A separate public catalog viewer
Public static website on the UI bucket. Shows catalog.json only; packages stay on the private S3 buckets and are not downloaded from this page. lists what was published. It is read-only metadata (catalog.json), not a package download CDN and not Pulp’s UI.

Once packages and indexes have replicated, Auckland keeps serving Auckland if Sydney is down (and the other way around). That is not full multi-Region HA
High availability. Design goal of staying usable through failures; multi-Region HA usually needs failover routing, not only a replica bucket.. You still publish to Sydney, replication is async so RPO
Recovery point objective. How much data you can afford to lose after a failure, measured as time. Asynchronous replication means RPO is greater than zero. is not zero, and anything in a failed Region is still down.

Pulp-style yum/apt repo versus regional S3 package trees Same job on both sides: Sydney and Auckland package consumers. Left keeps one HTTP repo (Pulp-style) and a network path that reaches both Regions. Right publishes rpm/deb trees to Sydney S3, rebuilds indexes, replicates repos/ to Auckland, and each consumer installs via dnf/apt through a local gateway VPCE. HTTP repo (Pulp-style) Icon-Resource/General-Resource/Res_Client_48_Light Publisher Pulp / yum+apt one Region Icon-Architecture/64/Arch_AWS-Transit-Gateway_64 Network path you operate this too Icon-Resource/General-Resource/Res_Client_48_Light dnf / apt Sydney Icon-Resource/General-Resource/Res_Client_48_Light dnf / apt Auckland This lab (S3 pkgs) Icon-Resource/General-Resource/Res_Client_48_Light Publisher Icon-Architecture/64/Arch_Amazon-Simple-Storage-Service_64 S3 pkgs Sydney + indexes Icon-Resource/Storage/Res_Amazon-Simple-Storage-Service_S3-Replication_48 CRR repos/ Icon-Architecture/64/Arch_Amazon-Simple-Storage-Service_64 S3 pkgs Auckland Icon-Resource/Networking-and-Content-Delivery/Res_Amazon-VPC_Endpoints_48 Gateway Syd VPCE Icon-Resource/General-Resource/Res_Client_48_Light dnf / apt Sydney Icon-Resource/Networking-and-Content-Delivery/Res_Amazon-VPC_Endpoints_48 Gateway Akl VPCE Icon-Resource/General-Resource/Res_Client_48_Light dnf / apt Auckland

What you will do

Section titled “What you will do”

Everything goes through ./scripts/demo.sh.

Shared buckets + CRR

Package buckets ps3p-pkgs-…-syd/akl, public UI bucket ps3p-ui-…-syd, replication filtered to repos/.

Dual-OS consumers

AL2023 + Ubuntu 24.04 probes per Region, each VPC locked with aws:SourceVpce
Bucket policy condition key aws:SourceVpce. Limits which VPC endpoint IDs can access the bucket..

Publish + rebuild

Put sample rpm/deb under repos/, then Lambda indexes (createrepo_c
createrepo_c builds yum/dnf repodata (repomd.xml and friends) for an RPM tree so dnf can resolve packages. / apt-ftparchive) and catalog.json.

Prove dnf / apt

SSM Run Command installs the samples on both OS probes in Syd and Akl.

Lab architecture: private S3 package repo Publish rpm/deb to a private S3 primary in Sydney. EventBridge invokes a rebuild Lambda (ECR image) that signs repo metadata with a GPG key from Secrets Manager, writes indexes and catalog.json, and publishes the public key. CRR copies repos/ to Auckland. Each Region has a lab VPC with S3 gateway and SSM interface endpoints for dual-OS probes. Syd pkgs → EventBridge/Lambda (+GPG) → Akl replica · regional S3 + SSM ap-southeast-2 (Sydney) ap-southeast-6 (Auckland) publish S3 pkgs primary rpm + deb + signed indexes Replication repos/ S3 pkgs replica same tree S3 gateway + SSM interfaces AL2023 + Ubuntu lab VPC · dnf / apt S3 gateway + SSM interfaces AL2023 + Ubuntu lab VPC · dnf / apt EventBridge Rebuild Lambda indexes + GPG sign Secrets Mgr lab GPG key ECR image Catalog UI public metadata + pubkey IAM: CRR · publisher · Lambda · Secrets · probe EC2 · CloudWatch via Lambda basic exec
Expectation Notes
Same package keys Different regional hostnames
Cross-region installs Via the replica tree, not cross-Region PrivateLink
Catalog viewer Public website; packages stay private
Cost while running S3 + CRR + Lambda/ECR + 4 probe EC2 + SSM endpoints

Out of scope

Section titled “Out of scope”
  • Full Pulp (or other long-lived repo servers)
  • Public package download from the catalog viewer
  • CodeBuild / schedule-first index rebuild

Read in this order

Section titled “Read in this order”
  1. Architecture
  2. Package formats
  3. Lab findings: New Zealand
  4. Gateway VPCE
  5. Replication Syd to Akl
  6. Deploy and operate

References

Section titled “References”