Skip to content

Allow multiple manifests to be downloaded from the same depot #707

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
scarletcafe wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Allow multiple manifests to be downloaded from the same depot #707

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
3ef4352
Allow multiple manifests to be specified
scarletcafe Apr 29, 2026
fbe48cf
Allow specified output directory to use variables
scarletcafe Apr 29, 2026
2014c90
Document multiple manifest downloading
scarletcafe Apr 29, 2026
644b23b
Place configs in 0-substituted directory
scarletcafe Apr 29, 2026
73bc294
Rewrite deduplication code to be conscious of the output directory
scarletcafe Apr 29, 2026
ffaf7be
Fix error from Config.InstallDirectory being null with no -dir argument
scarletcafe May 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 39 additions & 15 deletions DepotDownloader/ContentDownloader.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ private sealed class DepotDownloadInfo(
public byte[] DepotKey { get; } = depotKey;
}

static bool CreateDirectories(uint depotId, uint depotVersion, out string installDir)
static bool CreateDirectories(uint depotId, uint depotVersion, ulong manifestId, out string installDir)
{
installDir = null;
try
Expand All @@ -78,9 +78,14 @@ static bool CreateDirectories(uint depotId, uint depotVersion, out string instal
}
else
{
Directory.CreateDirectory(Config.InstallDirectory);
// Allow substitution in the specified install directory
installDir = Config.InstallDirectory
.Replace("%(depot_id)", depotId.ToString())
.Replace("%(depot_version)", depotVersion.ToString())
.Replace("%(manifest_id)", manifestId.ToString())
;

installDir = Config.InstallDirectory;
Directory.CreateDirectory(installDir);

Directory.CreateDirectory(Path.Combine(installDir, CONFIG_DIR));
Directory.CreateDirectory(Path.Combine(installDir, STAGING_DIR));
Expand Down Expand Up @@ -422,7 +427,7 @@ public static async Task DownloadUGCAsync(uint appId, ulong ugcId)

private static async Task DownloadWebFile(uint appId, string fileName, string url)
{
if (!CreateDirectories(appId, 0, out var installDir))
if (!CreateDirectories(appId, 0, 0, out var installDir))
{
Console.WriteLine("Error: Unable to create install directories!");
return;
Expand Down Expand Up @@ -456,10 +461,14 @@ public static async Task DownloadAppAsync(uint appId, List<(uint depotId, ulong
cdnPool = new CDNClientPool(steam3, appId);

// Load our configuration data containing the depots currently installed
var configPath = Config.InstallDirectory;
if (string.IsNullOrWhiteSpace(configPath))
var configPath = DEFAULT_DOWNLOAD_DIR;
if (!string.IsNullOrWhiteSpace(Config.InstallDirectory))
{
configPath = DEFAULT_DOWNLOAD_DIR;
configPath = Config.InstallDirectory
.Replace("%(depot_id)", "0")
.Replace("%(depot_version)", "0")
.Replace("%(manifest_id)", "0")
;
}

Directory.CreateDirectory(Path.Combine(configPath, CONFIG_DIR));
Expand Down Expand Up @@ -568,7 +577,7 @@ public static async Task DownloadAppAsync(uint appId, List<(uint depotId, ulong
throw new ContentDownloaderException(string.Format("Couldn't find any depots to download for app {0}", appId));
}

if (depotIdsFound.Count < depotIdsExpected.Count)
if (!depotIdsExpected.All(depotIdsFound.Contains))
{
var remainingDepotIds = depotIdsExpected.Except(depotIdsFound);
throw new ContentDownloaderException(string.Format("Depot {0} not listed for app {1}", string.Join(", ", remainingDepotIds), appId));
Expand Down Expand Up @@ -639,7 +648,7 @@ static async Task<DepotDownloadInfo> GetDepotInfo(uint depotId, uint appId, ulon

var uVersion = GetSteam3AppBuildNumber(appId, branch);

if (!CreateDirectories(depotId, uVersion, out var installDir))
if (!CreateDirectories(depotId, uVersion, manifestId, out var installDir))
{
Console.WriteLine("Error: Unable to create install directories!");
return null;
Expand Down Expand Up @@ -724,18 +733,33 @@ private static async Task DownloadSteam3Async(List<DepotDownloadInfo> depots)
cts.Token.ThrowIfCancellationRequested();
}

// If we're about to write all the files to the same directory, we will need to first de-duplicate any files by path
// If we're about to write multiple manifests to the same directory, we will need to first de-duplicate any files by path
// This is in last-depot-wins order, from Steam or the list of depots supplied by the user
if (!string.IsNullOrWhiteSpace(Config.InstallDirectory) && depotsToDownload.Count > 0)
var installLocations = depotsToDownload
.GroupBy((depot) => depot.depotDownloadInfo.InstallDir)
.ToDictionary((group) => group.Key, (group) => group.Count());

if (installLocations.Any((pair) => pair.Value > 1) && depotsToDownload.Count > 0)
{
var claimedFileNames = new HashSet<string>();
var claimedFileNames = new Dictionary<string, HashSet<string>>();

for (var i = depotsToDownload.Count - 1; i >= 0; i--)
{
// For each depot, remove all files from the list that have been claimed by a later depot
depotsToDownload[i].filteredFiles.RemoveAll(file => claimedFileNames.Contains(file.FileName));
var depot = depotsToDownload[i];

claimedFileNames.UnionWith(depotsToDownload[i].allFileNames);
if (!claimedFileNames.TryGetValue(depot.depotDownloadInfo.InstallDir, out var claimedSet))
{
// If this is the first depot to be downloaded into this directory, it gets to claim the full list
claimedFileNames.Add(depot.depotDownloadInfo.InstallDir, depot.allFileNames);
}
else
{
// Remove files that have been claimed by a later depot
depot.filteredFiles.RemoveAll(file => claimedSet.Contains(file.FileName));

// Add files owned by this depot to the claimed set
claimedSet.UnionWith(depot.allFileNames);
}
}
}

Expand Down
16 changes: 12 additions & 4 deletions DepotDownloader/Program.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -293,14 +293,22 @@ ex is ContentDownloaderException
var manifestIdList = GetParameterList<ulong>(args, "-manifest");
if (manifestIdList.Count > 0)
{
if (depotIdList.Count != manifestIdList.Count)
if (depotIdList.Count == 1 && manifestIdList.Count > 1)
{
Console.WriteLine($"Only one depot ID was provided, but multiple manifest IDs were provided. Assuming all manifests are part of depot {depotIdList[0]}");
var zippedDepotManifest = manifestIdList.Select((manifestId) => (depotIdList[0], manifestId));
depotManifestIds.AddRange(zippedDepotManifest);
}
else if (depotIdList.Count == manifestIdList.Count)
{
var zippedDepotManifest = depotIdList.Zip(manifestIdList, (depotId, manifestId) => (depotId, manifestId));
depotManifestIds.AddRange(zippedDepotManifest);
}
else
{
Console.WriteLine("Error: -manifest requires one id for every -depot specified");
return 1;
}

var zippedDepotManifest = depotIdList.Zip(manifestIdList, (depotId, manifestId) => (depotId, manifestId));
depotManifestIds.AddRange(zippedDepotManifest);
}
else
{
Expand Down
19 changes: 18 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,23 @@ By default it will use anonymous account ([view which apps are available on it h
To use your account, specify the `-username <username>` parameter. Password will be asked interactively if you do
not use specify the `-password` parameter.

### Downloading multiple manifests

You can specify additional depot+manifest pairs by adding multiple arguments to `-depot` and `-manifest`.
```powershell
./DepotDownloader -app <id> -depot <id1> <id2> -manifest <id1> <id2>
[-username <username> [-password <password>]] [other options]
```

For example: `./DepotDownloader -app 730 -depot 731 732 -manifest 7617088375292372759 1021596294116939367`

The first manifest ID is downloaded from the first depot ID, and so on.
So for this example, it would download manifest `7617088375292372759` from depot `731` and manifest `1021596294116939367` from depot `732`.

If you only specify one depot ID and multiple manifest IDs, it will download those manifests from the same depot ID.
To prevent them being downloaded into the same folder, you can use variable substitution with `-dir`:
`./DepotDownloader -app 730 -depot 731 -manifest 7617088375292372759 4899579212072880822 -dir "depots/%(depot_id)/%(manifest_id)"`

### Downloading a workshop item using pubfile id
```powershell
./DepotDownloader -app <id> -pubfile <id> [-username <username> [-password <password>]]
Expand Down Expand Up @@ -95,7 +112,7 @@ Parameter | Description
`-all-languages` | download all language-specific depots when `-app` is used.
`-language <lang>` | the language for which to download the game (default: english)
`-lowviolence` | download low violence depots when `-app` is used.
`-dir <installdir>` | the directory in which to place downloaded files.
`-dir <installdir>` | the directory in which to place downloaded files. You can use `%(depot_id)`, `%(depot_version)` and `%(manifest_id)` to substitute these values into the path.
`-filelist <file.txt>` | the name of a local file that contains a list of files to download (from the manifest). prefix file path with `regex:` if you want to match with regex. each file path should be on their own line.
`-validate` | include checksum verification of files already downloaded.
`-manifest-only` | downloads a human readable manifest for any depots that would be downloaded.
Expand Down
Loading