Archive Recent creates compressed archives containing files that have recently changed. Archive Recent is designed to simplify the creation of incremental back ups and copies of modified files.

Archive Recent can create encrypted archives suitable for uploading off site storage and archiving services.

Using archiverecent

To get started with Miln Archive Recent, download and expand the executable file onto your computer.

Archive Recent is a single executable file. It does not need to be installed, can be run from any directory, and can be removed by deleting the file.

Let’s walk through an example use of archiverecent. Consider this command line call:

./archiverecent -from Photos -since 2025-01-31 -output latest-photos.tgz

The flags for this call to archiverecent are:

-from Photos
The directory to read files from. Photos is the relative path to a directory.
-since 2025-01-31
The date from which to include modified files. Every file within the Photos directory, that has been modified on or after 31st January 2025, will be included in the archive.
-output latest-photos.tgz
The filename of the archive to create.

What Happens

So what happens when this command is issued?

archiverecent does the following:

The created file is a gzipped tar archive (.tar.gz/.tgz). This is a widely supported file format that will be recognised and accessible for decades to come. This makes it ideal for long term storage.

If no files have been modified, the final archive will still be created but will be empty.

Archive Recent will overwrite any existing archive file with the same name.

Why I wrote this tool

I wrote Archive Recent to help me prepare incremental archives of photos for backing-up.

Each month I like to back-up my new and changed photos to a long term store on a remote server. I used to prepare an archive manually by exporting photos from an application and then running a series of command line tools to finalise the archive.

The manual process was fine for many years. Sometimes it felt a little time consuming but never a real burden. Then the photos application I used changed and obscured the file layout I had been relying on for easily extracting recently changed files.

That application’s change was the first in a series of changes that pushed me away. I exported all my photos and metadata to a network drive.

I am now manually managing my photos as “just a bunch of images” on disk. Thankfully a welcome number of photo applications work well with this approach. As a bonus my reliance on a single photo application is behind me.

Without the ability to ask an authoritative photo application for recently changed photos, incremental back-ups remained a challenge.

I searched for various approaches to solving this problem. Many good solutions exist in the form of custom shell scripts using find and rsync. This should have been good enough but I worried my script would be fragile.

So rather than write my own shell script, I wrote archiverecent. The problem felt common enough that it deserved a more general tool.

Today, with archiverecent, this is the command I am using to back up our photos:

sudo archiverecent
	-from "/var/services/homes/./megan/Photos:/var/services/homes/./graham/Photos/"
	-since /var/services/homes/graham/photos-since.txt
	-output '/var/services/homes/graham/miln-photos-${now}-${since}-${uuid}.tgz'
	-passphrase -

The flags above are:

-from <directory paths>
Two directories containing our photos. The /./ path segment truncates the archived directory parent path to just our names, e.g. graham/photo1.jpg rather than var/services/homes/graham/Photos/photo1.jpg.
-since <file path>
Path to a text file containing the last since date.
-output <templated path>
Output path with variable substitutions of the time now, since, and a unique identifier.
-passphrase -
Encrypt the archive using a passphrase provided interactively.

This command can be called once a month or whenever we have a need to create an archive of recently added photos. The since date is dynamic and ensures everything modified after the last call is included in the new archive.

This tool has solved my incremental back-up problem. I expect it will be useful for you too.

If you have any suggestions for improvements or need additional options, please get in touch.

Examples

Below are examples of alternative uses of Archive Recent.

Last 24 hours

Archive documents that have changed in the last 24 hours:

./archiverecent -since 24h -from ~/Documents

Multiple Sources

Archive changed files in the last 30 days, from multiple directories, into one archive:

./archiverecent -since 720h -from Photos:Documents/projects:Documents/legal

Flags

archiverecent accepts the following command line flags:

Usage of ./archiverecent:
  -config string
    	File path to configuration.
  -dry-run
    	list files that would be archived but do not create the archive
  -exclude string
    	exclude directories and files with these names (default "@eaDir")
  -f string
    	paths to archive files from (shorthand)
  -from string
    	paths to archive files from
  -h	Show this help message and exit. (shorthand)
  -help
    	Show this help message and exit.
  -l string
    	Directory path to licence certificate files (PEM encoded) (shorthand) (default "~/.miln/")
  -legal
    	Show legal notices and exit.
  -licence string
    	Directory path to licence certificate files (PEM encoded) (default "~/.miln/")
  -o string
    	path to create archive at (shorthand) (default "${now}-archive-since-${since}-${uuid}.tgz")
  -output string
    	path to create archive at (default "${now}-archive-since-${since}-${uuid}.tgz")
  -passphrase string
    	passphrase source: empty for no encryption, 'env' for env ARCHIVERECENT_PASSPHRASE, '-' interactive, or a path to a text file.
  -s string
    	date or duration since file modification to archive (shorthand)
  -show-licence
    	Show licence details and exit.
  -since string
    	date or duration since file modification to archive
  -v	Show version details and exit. (shorthand)
  -version
    	Show version details and exit.

Qualifying as Recently Changed

A file qualifies for inclusion in the final archive, if the file’s modifed date is more recent than the since date. If no since date is provided, all files qualify.

Since Dates and Time Periods

The since value can been provided in a number of formats, including a dynamic value managed by a file.

Absolute Dates

A precise date with second accuracy and time zone can be passed as an RFC 3339 encoded value, i.e. 2006-01-02T15:04:05Z07:00.

./archiverecent -since 2012-02-07T09:00:00 -from ~/documents

A day accurate date can be passed in the unambigous numeric format YYYY-MM-DD, i.e. 2006-01-02.

./archiverecent -since 2023-11-01 -from ~/documents

Relative Dates

A relative time period can be passed to calculate the since date, i.e. 60s, 30m, 2h45m, 48h. The largest unit of time is h for hour.

./archiverecent -since 24h -from ~/projects

Dynamic Dates

Passing a file path for since will read the file contents for the date to use and, on completion, updated with the current date. This allows Archive Recent to be called periodically without needing to update the since value manually.

Each time the command below is called, a new archive will be created containing the files from the photos directory that have changed since the last archive was created:

./archiverecent -since ./lastarchive.txt -from photos

The initial contents of the file must be empty, a date, or a relative time.

If the file at the file path is empty or does not exist, Archive Recent will use the default since value. This will cause the first archive to include all existing files. The subsequent archives will be incremental.

The file path must be absolute, beginning with a file path separator (/), or relative with an intitial period (.). Any other format is treated as a date or duration.

Archiving Multiple Directories

Multiple directories can be passed to Archive Recent for inclusion in the archive. Any qualifying files in the directories will be included in the final archive.

Pass multiple from paths to be archived using the path separator (:). The command below passes two directories for inclusion:

./archiverecent -from "/homes/alice/photos:/homes/bob/documents"

Pass a glob pattern as the from path:

./archiverecent -from "/homes/*/photos"

Isolating Multiple Directories

When multiple directories are used for the from paths, there is the possibility of file name collisions. Two paths can contain files with the same name, and when stored in a single archive they would conflict. Archive Recent uses the tar archive file format. While this format allows conflicting file names, the last written file will frequently be the only file that is extracted.

To avoid this problem, Archive Recent includes the unique parent directory path when multiple from paths are used. Consider the single directory being archived below:

./archiverecent -from /homes/alice/photos

In the example above, Alice’s photos will appear at the top of the archive:

If we modify the command to add a second directory, also called photos, the layout of the archive changes:

./archiverecent -from /homes/alice/photos:/homes/bob/photos

This time the archive contains two isolating directories before the included contents:

Merging and Modifying Parent Directories

Archive Recent defaults to avoiding merging the contents of multiple directories. If multiple from directories should be merged into a single directory in the archive, this is possible. from paths can use a special prefix directory segment (/./) to denote where the parent directory should begin.

./archiverecent -from /homes/./alice/photos

In the example above, Alice’s photos will be embedded in the directory alice/photos/ within the archive:

Had the directory segment /./ not been included, the contents of the photos directory would have been the top most items in the archive.

The use of the prefix directory segment (/./) disables the use expansion of glob patterns.

When passing multiple paths to from, the prefix segment can be used to merge the contents of matching directories.

./archiverecent -from /homes/alice/./photos:/homes/bob/./photos

In the multiple path from example above, both Alice and Bob’s photos will be combined into a single directory in the archive:

Note the duplicate photo1.jpg entries in the list above. Both Alice and Bob had a recently modified file called photo1.jpg in their respective photos directories. Archive Recent will write both files to the archive. The tar archive format permits this. Which file is extracted depends on the tar extraction tool used.

Archiving Immich

Archive Recent can archive assets from Immich. Immich is open source software for managing your photos and videos. Immich publish recommendations about backing up; consider Archive Recent as an additional layer beyond these recommendations.

./archiverecent -from 'https://immich:MRRvaIZ7IiH6gMC2KQhht11uXISWx7TzdwYzR1dKV6@localhost:2283/api'

To archive assets from Immich, pass a URL to the Immich API end point with an API key, using the format:

In the example URLs above, the text apikey must be replaced with a real API key from your Immich account.

To identify the service type, the URL must begin with either https://immich: or http://immich:. This approach uses the embedded URL credentials to store the service name as the user, and the API key as the password.

Immich API Key

An Immich API key is required for Archive Recent to be able to access assets. Each key is associated with a single Immich user.

Archive Recent requires the following Immich permissions:

Recently Changed in Immich

How Archive Recent determines what has recently changed in Immich is worth understanding.

The aim of Archive Recent is to copy files that have been added to Immich within the since period.

We want to include photos and videos taken before the since period but only recently added to Immich. This covers this situation where photos have remained on the camera for a while before being uploaded into Immich.

Archive Recent includes Immich assets that have been created within since period. Within Immich, the createdAt time is defined as:

The UTC timestamp when the asset was originally uploaded to Immich.

In addition to assets created within the since period, Archive Recent checks for assets that have been modified during same period. Immich defines an asset’s fileModifiedAt time as:

The UTC timestamp when the file was last modified on the filesystem. This reflects the last time the physical file was changed, which may be different from when the photo was originally taken.

Between these two searches, Archive Recent should find and include both newly added and recently modified assets within Immich.

Multiple Immich Accounts

To combine the assets of multiple Immich users in a single archive, you will need an API key from each user. With each API key, pass the Immich URL using an additional from flag:

./archiverecent -from 'https://immich:apikey1@photos.example.com/api' -from 'https://immich:apikey2@photos.example.com/api' -from 'https://immich:apikey3@photos.example.com/api'

Assets can appear multiple times across multiple accounts because they are shared or accessible as a partner. To avoid this duplication, Archive Recent stores a single copy of each changed asset within a single archive.

Combining File System and URL Sources

Archive Recent can create combined archives containing recently changes files from the file system and from URLS.

Use a separate from flags for file paths and URLs. File paths can be combined using your platform’s path separator (: on Unix). URLs can be combined with white space.

The single from flag passing two URLs, -from 'https://immich:apikey1@photos.example.com/api https://immich:apikey2@photos.example.com/api', is equivalent to two from flags -from 'https://immich:apikey1@photos.example.com/api' -from 'https://immich:apikey2@photos.example.com/api'

Prefer a separate from flag per source to clarify intent.

./archiverecent -from 'https://immich:apikey1@photos.example.com/api' -from '${home}/photos'

Encryption

Archive Recent can encrypt the archive using a cryptographically secure and well known algorithm.

Use the passphrase flag to set the secret passphrase needed to decrypt the archive. The passphrase can be provided in a few different ways:

If the passphrase is forgotten or lost, it can not be recovered.

The algorithm used is aes-256-cbc pbkdf2 salted sha256. This was chosen because the resulting archive can be decrypted with the open source and widely available openssl.

Decrypting the Archive

To decrypt an archive with openssl, use the following flags:

openssl aes-256-cbc -pbkdf2 -salt -d -in archive.tgz.aes -out archive.tgz

Archive Recent does not support decrypting archives. This role is left to openssl and other tools.

Archive Name

The default archive name makes use of variables to ensure it differs every time an archive is created:

${now}-archive-since-${since}-${uuid}.tgz

Variables can be used to insert dynamic elements into the archive path. Environment variables are available, in addition to the named values below:

now
now date as yyyy-mm-dd, i.e. 2025-01-28
now_ymd
now date as yyyymmdd, i.e. 20250128
now_y
now year as yyyy, i.e. 2025
now_m
now month as mm, i.e. 01
now_Month
now month as English string, i.e. January
now_d
now day of the month as dd, i.e. 28
now_Day
now day of the month as English string, i.e. Tuesday
since
since date as yyyy-mm-dd, i.e. 2025-01-28
since_ymd
since date as yyyymmdd, i.e. 20250128
since_y
since year as yyyy, i.e. 2025
since_m
since month as mm, i.e. 01
since_Month
since month as English string, i.e. January
since_d
since day of the month as dd, i.e. 28
since_Day
since day of the month as English string, i.e. Tuesday
uid
Current user identifer
gid
Current user’s primary group identifier
username
Current user short name
name
Current user name
home
Current user’s home directory
uuid
Universally unique lexicographically sortable identifier (ulid); see https://github.com/ulid/spec for the properties of this value.

Excluded Files and Directories

Archive Recent does not include irregular files or empty directories. Irregular files include devices, sockets, and symbolic links. Only directories containing qualifying files are included.

Files beginning with period (.) are excluded. This avoids including macOS’s .DS_Store files.

Additionally, Archive Recent has the option to exclude files and directories with given names. The exclude flag defaults to @eaDir, which is the name of the Synology DiskStation Manager’s (DSM) supporting directories.

./archiverecent -from photos -exclude '@eaDir:Cache'

If a file matches an excluded name, the file is skipped and not be included in the archive.

If a directory matches an excluded name, the directory is skipped and the contents of the directory are also skipped.

Configuration Files

If you find passing lengthy command line arguments obtuse or unwieldy, Archive Recent supports configuration files through the config flag:

./archiverecent -config archive-config.txt

The configuration file must be a text file with one flag per line. The flag and value are separated by an equals (=):

from=photos
verbose = true

# Back-up any changed files from the last 24 hours
since=24h

White space is trimmed. Blank lines are permitted. Lines beginning with # are ignored as comments.

Balanced quotes (") can be used to set values with leading or trailing whitespace:

key = "   white space surrounded value    "

If the flag has already been parsed, only non-default values will be set; this ensures command line options override configuration file options.

Download

Latest downloads for Archive Since.

Legal

Legal notices associated with Archive Since.