Download internals

Drop uses a special method to download over HTTP to ensure fast and reliable downloads, while achieving the goals of the Depot API. Here’s how they work

Game manifest

On importing a version, Drop generates a “game manifest”. You can read more about it on the Depot API section, but what it means for non-developer is that downloads stay the same speed no matter how many files there are, or how big they are.

Typical file transfer protocols, like FTP or HTTP, make one request for each file. This is great if you have just one, large file, but if you have lots of little ones, you lose a lot of bandwidth to overhead. Drop minimises this by packing smaller files into larger chunks.

Drop also splits larger files into several, smaller chunks. This is to ensure proper balancing of downloads, and make the chunks easier to manage (since they all are roughly the same size).

Note

Due to limitations with 7zip, this is not true for archive-backed versions. Drop cannot read parts of files with 7zip, so large files must stay in their own, equally large chunk.

Tip

What’s torrential?

If you’ve built the Drop server from source, or checked out the logs in the container, you may have noticed an applcation called torrential. torrential is a download server that’s embedded into Drop, and serves the chunks required for the Depot API, without actually assembling them ahead of time.

Delta versions or “Update mode”

While called “update mode” in the UI, internally they are called “delta versions”. In practice, they apply files on top of each other, essentially patching previous versions.

However, if you’ve been paying attention, we don’t actually need to do this physically on disk, we can just create a “pseudo-manifest” that pulls chunks from both versions.

The catch is, we can’t control what files are packed into each chunk (we can, but we won’t know what files will be patched ahead of time, when we’re importing). So while we can drop chunks here and there, if and only if every file has been patched by another version, we usually end up downloading the entirety of the base version + the patch.