S3 Storage (Amazon S3 and compatible services)

Event Gallery can keep your images at Amazon S3 or at any storage provider which offers the S3 API, for example Cloudflare R2, Backblaze B2, Wasabi or Hetzner Object Storage. See the section called “Setting up your storage provider” for the setup of each of them.

Basically, S3 files behave like local files. There is just one important difference. While thumbnails for local files are generated on demand, we need to calculate the thumbnails for images stored in an S3 bucket in advance. For this, we use two buckets. One bucket contains the original files. The folder and file structure is the same as for local files. No nesting is supported. Create a folder in the bucket and upload your images into it. Example: event1/myimage.jpg. The second bucket is used to store the generated thumbnails. Browsers need access to this bucket in order to load the images. You just need to create the bucket; everything else is handled by Event Gallery. Make sure you’re using the same region for both buckets.

Once you uploaded your images and created an S3 account for your buckets, you need to let the database know about your new files. Use the Sync Database button and sync your folders to start the sync process. Once the new files appear, hit the Start Sync button of step 3, Synchronize Images, to read in the necessary data. Check the Sync Database process documentation for information on how to generate thumbnails. You can upload files using Event Gallery, too.

S3 accounts

An S3 account tells Event Gallery where your images are stored and how to sign in. You find the accounts under Event Gallery → Accounts - S3 Storage. You can have as many accounts as you like, for example one per customer or one per storage provider, at Amazon S3 or at any service which offers the S3 API.

Figure 3.56. The list of S3 accounts

s3 accounts

  • Name, Description: what you see when you choose an account for an event.
  • Published: an unpublished account is not offered for new events. Events which already use it keep working.
  • Provider: choose your storage provider, and Event Gallery fills endpoint, region, path style and the ACL switch with the values which are known to work for it. A note below the field tells you what is special about the provider. You can change every value afterwards; the provider itself has no further effect.
  • Endpoint: the host name of the S3 API of your provider without the bucket name, for example s3.eu-central-1.wasabisys.com. Leave it empty for Amazon S3.
  • Region: the region of your buckets, for example eu-central-1.
  • Path style addressing: No puts the bucket into the host name (https://bucket.endpoint/file), Yes into the path (https://endpoint/bucket/file). Most hosted providers work with No, self hosted services usually need Yes.
  • Access key, Secret key: the credentials Event Gallery signs in with. They need permission to list the buckets and to read, write and delete files in them.

Figure 3.57. An S3 account

s3 account

  • Bucket for original files: holds your original files. It has to be private.
  • Bucket for thumbnails: Event Gallery writes the thumbnails here. Its files have to be publicly readable. Leave it empty to use one bucket for everything.
  • Make thumbnails public by ACL: Yes uploads every thumbnail with the access control list public-read; the bucket has to allow ACLs. No sends no ACL, and the policy of the bucket makes the thumbnails readable. Choose No for providers without ACLs and for Amazon S3 buckets with the default setting Bucket owner enforced.
  • Public URL of the thumbnails: optional. The address your visitors load the thumbnails from if it is not the bucket itself, for example a CloudFront distribution or a custom domain.
Check the connection

Once an account is saved, its form offers the button Check the connection. Event Gallery signs in, writes a small test file as an original and as a thumbnail, and then looks at your buckets the way a stranger would: without credentials. The test files are removed again.

The result tells you two things. First, whether the account works: signing in, writing, deleting, and whether your visitors can load a thumbnail. Second, and more important, whether your original files are really private. A bucket which is public by mistake works perfectly well, so you would never notice. If the check reports Your original files are public!, remove the public access from that bucket at your storage provider and run the check again.

Figure 3.58. An account which passed the check

s3 account check

Run the check whenever you changed something at your storage provider.

One bucket instead of two

Two buckets are the default, because they keep private and public files apart in the most obvious way. If you prefer one bucket, leave Bucket for thumbnails empty. Event Gallery then stores the original files below originals/ and everything your visitors load below resized/:

originals/event1/myimage.jpg
resized/event1/s1600/myimage.jpg

Upload your images to originals/<folder>/. The bucket has to stay private as a whole; only the prefix resized/ may be publicly readable. Set Make thumbnails public by ACL to No and give the bucket a policy like this one, with your bucket name:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-bucket/resized/*"
    }
  ]
}

Decide before you upload: switching an account from two buckets to one, or back, does not move any files.

Figure 3.59. The buckets of an account

s3 account buckets

When you create an event, choose S3 Storage as the image source, pick the account and enter the folder name. If the sync finds a new folder in a bucket, it creates the event for the published account which has that folder; if several have it, for the one which was created first. A folder name can only be used once across all accounts, because it identifies the event.

Figure 3.60. A new event with the image source S3 Storage

event s3

An account which events still use can not be deleted. Assign another account to those events first; deleting an account never touches the files in the buckets.

Your original files never leave the private bucket through a link. Event Gallery downloads an original with the credentials of the account and hands it out itself, after it checked that the visitor may have it.

If you update from an older version, Event Gallery creates an account named Amazon S3 from the former options of the component and assigns it to all existing S3 events. It keeps Make thumbnails public by ACL switched on, because that is how your buckets were set up.

Deleting images

When you delete images in the image list of an Amazon S3 event with Delete images, Event Gallery deletes them in Amazon S3 as well: the original file in the bucket of the original images, and all thumbnails of it in the thumbnail bucket. For a video, the copy which Event Gallery keeps in the thumbnail bucket for playback is deleted, too.

Deleting a whole event is different. It removes the event and its images from Event Gallery, but leaves the files in both buckets untouched, on purpose. If you want to get rid of the files, delete the folder in both buckets in the AWS console. If you create an event with the same folder name again, a sync finds the files and imports them again.

Always keep in mind that reading from and writing to an Amazon S3 bucket is not free. You have to pay for each request. So don’t perform a sync or a thumbnail creation if it is not necessary.

[Note]Note

If you encounter any errors, increase your error reporting level. The thumbnail generator will output error messages which might help with invalid Amazon S3 credentials.

Technical Details

The table #__eventgallery_file has two new fields: s3_etag and s3_etag_thumbnails. The field s3_etag contains the S3 hash of the original file. It is used to detect image changes. The field s3_etag_thumbnails contains a JSON object. It stores the thumbnail sizes and the hash values of each thumbnail. While creating the thumbnails this data is used to detect thumbnail changes. Once a change of the original file is detected, the field s3_etag_thumbnails is cleared. The next thumbnail creation run will find those thumbnails as missing and will recreate them. Make sure you uncheck the Refresh thumbnail hashes option. Otherwise, the s3_etag_thumbnails field will be populated with the hash values of existing thumbnails.

Figure 3.61. Thumbnail Creator

s3 thumbnail creator

Setting up your storage provider

Event Gallery works with Amazon S3 and with every service which offers the S3 API. Whatever provider you choose, the setup is always the same four steps:

  1. Create a private bucket for the original files and a bucket for the thumbnails. Both in the same region.
  2. Make the files of the thumbnail bucket publicly readable, but not the list of its files. The bucket of the originals stays private.
  3. Create an access key which may list both buckets and read, write and delete files in them, and nothing else.
  4. Create the S3 account in Event Gallery, choose your provider, enter the values and save. Then press Check the connection. It tells you whether everything works and whether your original files are really private.

Bucket names should consist of lower case letters, digits and hyphens only. Avoid dots.

[Note]Note

The values below are taken from the documentation of the providers at the time of writing. Providers change their offers. If something does not match what you see at your provider, follow your provider and let the connection check be the judge. Event Gallery itself is tested against Amazon S3 and against a self hosted S3 service.

Table 3.1. What the form fills in when you choose a provider

ProviderEndpointRegion, for examplePath styleACLRemarks

Amazon S3

(empty)

eu-central-1

No

No

Publish the thumbnails with a bucket policy. Details

Cloudflare R2

<ACCOUNT_ID>.r2.cloudflarestorage.com

auto

Yes

No

Needs a public URL. Two buckets only. Details

Backblaze B2

s3.<region>.backblazeb2.com

eu-central-003

No

No

Two buckets only. Details

Wasabi

s3.<region>.wasabisys.com

eu-central-2

No

No

Public access may have to be enabled by their support. Details

Hetzner Object Storage

<region>.your-objectstorage.com

fsn1

No

No

Details

IONOS Object Storage

s3.<region>.ionoscloud.com

eu-central-3

No

No

Details

DigitalOcean Spaces

<region>.digitaloceanspaces.com

fra1

No

Yes

Publishes single files by ACL. Details

Scaleway Object Storage

s3.<region>.scw.cloud

fr-par

No

No

Details

OVHcloud Object Storage

s3.<region>.io.cloud.ovh.net

gra

No

No

Details

Self hosted (MinIO, Garage, SeaweedFS, Versity …)

https://<YOUR_HOST>

us-east-1

Yes

No

Details


Providers with bucket policies

This is the way for Amazon S3, Wasabi, Hetzner, IONOS, Scaleway, OVHcloud and most self hosted services. Leave Make thumbnails public by ACL at No and give the thumbnail bucket this policy, with the name of your bucket. It allows everybody to read a file and nobody to list the bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PublicReadThumbnails",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-thumbnail-bucket/*"
    }
  ]
}

Most providers have a field for the bucket policy in their web console. If yours has none, any S3 tool can set it, for example the AWS command line:

aws s3api put-bucket-policy --endpoint-url https://<endpoint> --bucket my-thumbnail-bucket --policy file://policy.json

The bucket of the originals gets no policy. If you use one bucket for everything, the resource of the policy is arn:aws:s3:::my-bucket/resized/*.

Providers which publish files by ACL

DigitalOcean Spaces, and Amazon S3 buckets which were set up for an older version of Event Gallery, publish single files by an access control list. Set Make thumbnails public by ACL to Yes. Event Gallery then uploads every thumbnail as public-read and every original as private, and the buckets themselves stay private. The access key needs the permission to set ACLs (s3:PutObjectAcl at Amazon S3).

Cloudflare R2
  • Create two buckets. Never use one bucket with R2: R2 can only publish a whole bucket, and with it your original files.
  • R2 never serves public files from its S3 endpoint. Open the settings of the thumbnail bucket and connect a custom domain to it (or, for a test, enable the r2.dev address). Enter that address as Public URL of the thumbnails, for example https://images.example.org. Leave the bucket of the originals without any public access.
  • Create an API token of the type Object Read & Write, restricted to the two buckets. It gives you the access key and the secret key.
  • In the account form replace <ACCOUNT_ID> in the endpoint with your Cloudflare account id. The region is auto.
Backblaze B2
  • Create the bucket of the originals as Private and the thumbnail bucket as Public. Never use one bucket with B2: B2 can only publish a whole bucket.
  • Create an application key which is restricted to the two buckets, with read and write access. The keyID is the access key, the applicationKey the secret key.
  • The region is part of the endpoint B2 shows for your bucket, for example eu-central-003 in s3.eu-central-003.backblazeb2.com.
  • Depending on its settings a public B2 bucket may also reveal the list of its files. The connection check tells you so with a warning. If the folder names of your events are confidential, put a CDN in front of the bucket and enter it as Public URL of the thumbnails.

Quick Steps to set up AWS S3 for Event Gallery

  • Create two buckets in the region you like. One will contain the original files, the other the resized thumbnails. Leave Block all public access switched on for the bucket of the originals.
  • For the thumbnail bucket, switch Block all public access off and add the bucket policy which makes its files readable. ACLs can stay disabled, which is the default of AWS.
  • Create a new user in IAM. This lets you restrict the user’s access to the buckets only.
  • Create a new policy. You can use the JSON example below. Just replace the names of the buckets.
  • Assign the policy to the user
  • Create an S3 account in Event Gallery with the provider Amazon S3, the access key of the user, the region and the two buckets, and press Check the connection.
  • Upload your images in a simple folder structure. No nesting! folder/image.jpg
  • Use the Sync Database process to detect the new files
  • Run the thumbnail creator to generate the thumbnails
[Note]Note

Sites which were set up with an older version of Event Gallery publish their thumbnails by ACL instead of a bucket policy. That keeps working: the account which the update created has Make thumbnails public by ACL switched on. You can move to a bucket policy at any time by adding the policy and switching the option off.

Access Policy

You need to specify credentials so Event Gallery can access the S3 buckets you created. The following policy allows reading from and writing to those buckets. The line s3:PutObjectAcl is only needed if the account makes thumbnails public by ACL; you can remove it otherwise. You just need to change the name of those buckets to the ones you created.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::eventgallery-images-test-original",
                "arn:aws:s3:::eventgallery-images-test-resized"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::eventgallery-images-test-original/*",
                "arn:aws:s3:::eventgallery-images-test-resized/*"
            ]
        }
    ]
}

Extended steps to set up Event Gallery for AWS S3

The screenshots show the setup with ACLs, which older versions of Event Gallery required. It still works. If you publish the thumbnails with a bucket policy instead, skip the step Allow ACLs for both buckets and add the bucket policy to the thumbnail bucket.

Log into the AWS console and switch to the S3 overview page. There, you need to create two new buckets.

Figure 3.62. Create the bucket for the original files

01 create original bucket

Figure 3.63. Create the bucket for the calculated thumbnails

02 create resized bucket

Figure 3.64. Two new buckets are created

03 buckets created

Once you have the two buckets, you need to change the permissions on the thumbnail/resized bucket. It needs to be publicly accessible.

Allow ACLs for both buckets.  By default, items in this bucket are not accessible. When Event Gallery creates thumbnails or uploads files, it sets the ACL to make the files/thumbnails public. The buckets need to support this. Allowing the usage of an ACL helps here.

03 buckets allow acl

Figure 3.65. Change bucket permission for the thumbnail bucket

04 bucket permission change

Figure 3.66. Allow public access to the thumbnails bucket

05 bucket permission change

Figure 3.67. Confirm permission change

06 bucket permission change

The buckets are ready. To upload images and calculate thumbnails, you need to create a new user. Don’t use your AWS root credentials! For every Event Gallery installation, I recommend creating a new IAM user. This user has access only to the S3 buckets and to nothing else. The needed permissions are granted by an access policy. We will create both in the next steps.

Figure 3.68. Create a new IAM user

07 IAM

Figure 3.69. Allow programmatic access

08 IAM new user

Figure 3.70. Create a new access policy

09 IAM create policy

Figure 3.71. Copy and modify the access policy JSON from above

10 IAM policy json

Figure 3.72. Add a name and description for the new policy

11 IAM policy review

Figure 3.73. Assign the policy to the user

12 IAM user creation policy selection

Figure 3.74. Copy the access credentials for the user

13 IAM user created

Once you have the access credentials, bucket names and the bucket region, you can configure Event Gallery and start using AWS S3 there.

All prices include VAT. The gross price will vary depending on the selected shipping country.