OpamRepositoryRoot tgz folding with memoisation

master_changes.md

@@ -71,6 +71,7 @@ users)
 ## Repository
   * No longer call tar tool to create archives, use tar library instead [#6945 @kit-ty-kate]
   * [BUG] Do not fail on directories named `opam` when scanning the `packages` directory of a repository during `opam repo add` or `opam init` (worked on subsequent `opam update`) [#6941 @kit-ty-kate @rjbou]
+  * Speedup repository operations on Windows & BSD by changing its storage in the opam root from plain directory to archive, for HTTP repositories, or non-VCS one if `OPAMREPOSITORYTARRING` is enabled [#6625 @rjbou @kit-ty-kate @arozovyk - fix #5741 #5648 #5484 #5559 #3050]
 
 ## Lock
   * [BUG] Fix `undefined variable` error when a lock file filter contains an undefined variables: fail gracefully with strict mode, continue and default the variable to false on normal mode [#6947 @rjbou - fix #6946]
@@ -92,6 +93,7 @@ users)
   * Fix switch and repo format upgrade on Windows. A block occurred because the global lock fd was reopened instead of using the one already opened.  [#6839 @rjbou]
   * Add opam root format upgrade conditional mechanism for hard upgrades [#6949 @rjbou]
   * Stop the opam 1.2 to 2.0 repository upgrade process from downloading packages without checksums to add a non-trusted md5 [#6978 @kit-ty-kate]
+  * Add conditional upgrade to opam root 2.6. It is triggered by an already present repository archive (from `OPAMREPOSITORYTARRING` usage, or 2.1 opamroot) [#6625 @rjbou]
 
 ## Sandbox
   * Allow the macOS sandbox to write in the `/var/folders/` and `/var/db/mds/` directories as it is required by some of macOS core tools [#4797 @kit-ty-kate - fix #4389 #6460]
@@ -189,6 +191,11 @@ users)
   * Add a lock test for undefined variables in a lock file [#6947 @rjbou - fix #6946]
   * Add a test showing the behaviour of `opam repo add` and `opam update` when faced with a repository containing an `opam` directory [#6995 @kit-ty-kate]
   * Add a tests for the several layouts of packages in a repository [#6941 @rjbou]
+  * Add a `opam repo set-url` case in repository-http [#6625 @rjbou]
+  * Add in `repository-http` a test case for switching from directory to archive format, automatically [#6625 @rjbou]
+  * Add in `repository` test cases for switching automatically from directory to archive format & vice versa [#6625 @rjbou]
+  * Add in `repository` test cases for upgrade opam root from 2.5 with repo tarring or 2.1 to 2.6, with `OPAMREPOSITORYTARRING` enabled (trigger upgrade) [#6625 @rjbou]
+  * Add 2.6 root test cases in opamroot-versions [#6625 @rjbou]
 
 ### Engine
   * Add `http-server` to launch a minimal http server [#6939 @rjbou]
@@ -247,6 +254,14 @@ users)
   * `OpamRepositoryRoot` was added [#6680 @kit-ty-kate @rjbou]
   * `OpamTar`: add module to manipulate tar gz archive. It handles only files, not directories [#6945 @kit-ty-kate @rjbou]
   * `OpamRepositoryCommand.update_with_auto_upgrade`, `OpamUpdate.repository`: no longer call an external process to create an archive [#6945 @kit-ty-kate]
+  * `OpamTar`: add `patch` function to patch files in an tar gz archive [#6625 @rjbou]
+  * `OpamTar.create`: add `?flat` argument to do not integrate the target root directory in the archive [#6625 @rjbou]
+  * `OpamTar.create`: add `?except_vcs` argument exclude VCS files from archive creation [#6625 @rjbou]
+  * `OpamTar`: when an archive is opened, the first step is to check and normalise canonical paths (no `/../`, remove `./`, etc.) [#6625 @rjbou]
+  * `OpamRepositoryRoot`: add `remove_prefix` and `remove_prefix_dir` [#6625 @rjbou]
+  * `OpamRepositoryRoot`: add `read_file` that reads an `OpamFile.t` using its pp from the archive [#6625 @rjbou]
+  * `OpamRepositoryBackend.get_diff`: now computes the diff between two repository roots (dir, archive), instead of only dirs [#6625 @rjbou]
+  * `OpamRepositoryRoot`: add `Tgz` module for tar gz archive repository root support [#6625 @rjbou @kit-ty-kate]
 
 ## opam-state
   * `OpamStateConfig.t`: replace `no_depexts` fields that contains disabling informations by `depexts` field that returns if the depexts mechanism is enabled. This field is automatically update by global config value in `OpamStateConfig.load_defaults` [#6489 @rjbou]
@@ -270,7 +285,10 @@ users)
   * `OpamRepositoryState`: add `syspkgs_available` that returns the stored depext availability status in repository state [#6489 @rjbou]
   * `OpamSysInteract`: add `available_packages_and_family` that returns availability status and the os family [#6489 @rjbou]
   * `OpamRepositoryState.load_opams_from_dir`: now sorts files and directories read from disk before processing them [#6941 @rjbou]
-
+  * `OpamFileTools`: add new `lint_repo_package` that lints a file from a repository root [#6625 @rjbou]
+  * `OpamRepositoryState`: add `load_opams_from_tgz` [#6625 @rjbou]
+  * `OpamRepositoryState`: add `load_opams` that operates from a repository root [#6625 @rjbou]
+  * `OpamStateTypes.repos_state`: remove `repos_tmp` field [#6625 @kit-ty-kate @rjbou]
 
 ## opam-solver
 
@@ -289,6 +307,9 @@ users)
   * `OpamRepositoryPath.{root,repo,packages_dir,packages,opam,files,descr,url}: have been moved to a new `OpamRepositoryPath.Make` functor [#6680 @rjbou @kit-ty-kate]
   * `OpamFilter.expand_interpolations_in_file`: changed argument type from `basename` to `filename` [#6910 @NathanReb]
   * `OpamFile.OPAM.print_errors`: now prints the repository if the informations is available [#6971 @rjbou]
+  * `OpamFile.*.read_from_string`: add optional `?loc` string argument to propagate location information when available (path on disk, archive, etc.), and add logging with level 3 that displays it [#6625 @rjbou]
+  * `OpamFile.*`: add `safe_read_from_string` [#6625 @rjbou]
+  * `OpamRepositoryPath.tar`: renamed to `repo_tarring`, deprecated, no longer used [#6625 @rjbou]
 
 ## opam-core
   * `OpamCmdliner` was added. It is accessible through a new `opam-core.cmdliner` sub-library [#6755 @kit-ty-kate]
@@ -315,5 +336,16 @@ users)
   * `OpamSystem.{command,commands,read_command_output}`: add a `?dir: dirname` optional arg to launch the command in a specific directory [#6910 @NathanReb]
   * `OpamFilename.Unix`: add `starts_with` [#6945 @rjbou]
   * `OpamCompat.Seq`: add `to_dispenser` [#6945 @kit-ty-kate]
-  * `OpamSystem.directories_with_links`, `OpamSystem.rec_files`, `OpamFilename.rec_files`: add optional `?except_vcs` that default to false to exclude VCS directories [#6945 @kit-ty-kate @rjbou]
+  * `OpamSystem.directories_with_links`, `OpamSystem.rec_files`, `OpamFilename.rec_files`: add optional `?except_vcs` that default to false to exclude VCS directories [#6625 @kit-ty-kate @rjbou]
   * `OpamFilename.make_tar_gz{_job}`: rename `make_tar_gz_job` into `make_tar_gz` as it no longer need an external process call [#6945 @kit-ty-kate]
+  * `OpamSystem`: add `open_in` and `open_in_bin` that checks is the file is a directory and raise `Sys_error` in that case [#6941 @rjbou]
+  * `OpamFilename.{open_in,open_in_bin}`: now errors when the filename is a directory, in all platforms [#6941 @rjbou]
+  * `OpamFilename.Unix`: add `basename` and `dirname` [#6625 @rjbou]
+  * `OpamFilename.Unix`: add `basename`, `dirname` and `remove_prefix` [#6625 @rjbou]
+  * `OpamFilename.Unix`: add `basename`, `dirname`, `remove_prefix` and `add_extension` [#6625 @rjbou]
+  * `OpamFilename.Unix`: add `basename`, `dirname`, `remove_prefix`, `add_extension`, `check_canonical` [#6625 @rjbou]
+  * `OpamFilename.Unix`: add `basename`, `dirname`, `remove_prefix`, `add_extension`, `check_canonical`, `root_dir` [#6625 @rjbou]
+  * `OpamFilename.Unix.Dir`: add `basename` and `dirname` [#6625 @rjbou]
+  * `OpamFilename.Unix.{to_dir,to_filename}`: now translate `/` into filesystem directory separator [#6625 @rjbou]
+  * `OpamPatch.patch`: no longer patch a file on disk, but take as argument a filesystem abstraction `FS_ABSTR` that delivers the needed functions [#6625 @rjbou]
+  * `OpamPatch.parse_patch`: no longer take `~dir` the directory to translate the patch in as argument, it now takes `~translate` argument that is a string option (directory option), if we want to perform a translation in that directory [#6625 @rjbou]

src/client/opamAdminCommand.ml

@@ -1112,17 +1112,12 @@ let get_virtual_switch_state repo_root env =
     global_state_to_upgrade = { gtc_repo = false; gtc_switch = false; };
   } in
   let singl x = OpamRepositoryName.Map.singleton repo.repo_name x in
-  let repos_tmp =
-    let t = Hashtbl.create 1 in
-    Hashtbl.add t repo.repo_name (lazy repo_root); t
-  in
   let rt = {
     repos_global = gt;
     repos_lock = OpamSystem.lock_none;
     repositories = singl repo;
     repos_definitions = singl repo_def;
     repo_opams = singl opams;
-    repos_tmp;
     repos_syspkgs_available = None;
   } in
   let gt =

src/client/opamArg.ml

@@ -224,10 +224,10 @@ let environment_variables =
       "enables option --no-checksums when available.";
       "REPOSITORYTARRING", cli_from cli2_2,
       (fun b -> REPOSITORYTARRING (env_bool b)),
-      "internally store the repositories as tar.gz files. This can be much \
-       faster on filesystems that don't cope well with scanning large trees \
-       but have good caching in /tmp. However this is slower in the \
-       general case.";
+      "internally store all non-vcs repositories as tar.gz files (by default, \
+       http repositories are stored as archive).This can be much faster on \
+       filesystems that don't cope well with scanning large trees. All \
+       operation are done in-memory, no I/O for scanning the repository.";
       "REQUIRECHECKSUMS", cli_original,
       (fun v -> REQUIRECHECKSUMS (env_bool v)),
       "Enables option `--require-checksums' when available \

src/client/opamCommands.ml

@@ -2524,32 +2524,31 @@ let repository cli =
       in
       OpamGlobalState.with_ `Lock_none @@ fun gt ->
       OpamRepositoryState.with_ `Lock_write gt @@ fun rt ->
-      OpamFilename.with_tmp_dir @@ fun tmp_dir ->
+      OpamFilename.with_tmp_dir @@ fun inn ->
+      let repo_root =
+        OpamRepositoryState.get_repo_root rt
+          (OpamRepositoryState.get_repo rt name)
+      in
+      if not (OpamRepositoryRoot.exists repo_root) then
+        OpamConsole.error_and_exit `Internal_error
+          "Repository not found, consider running 'opam update %s' \
+           to retrieve a consistent state."
+          (OpamRepositoryName.to_string name);
       let rt0 = rt in
-      let backup =
-        let tar = OpamRepositoryPath.tar gt.root name in
-        if OpamFilename.exists tar then
-          (let target = OpamFilename.create tmp_dir (OpamFilename.basename tar) in
-           OpamFilename.copy ~src:tar ~dst:target;
-           fun () -> OpamFilename.copy ~src:target ~dst:tar)
-        else
-          (let dir = OpamRepositoryRoot.Dir.Path.root gt.root name in
-           if not (OpamRepositoryRoot.Dir.exists dir) then
-             OpamConsole.error_and_exit `Internal_error
-               "Repository not found, consider running 'opam update %s' \
-                to retrieve a consistent state."
-               (OpamRepositoryName.to_string name);
-           let target = OpamRepositoryRoot.Dir.backup ~inn:tmp_dir dir in
-           OpamRepositoryRoot.Dir.copy ~src:dir ~dst:target;
-           fun () -> OpamRepositoryRoot.Dir.copy ~src:target ~dst:dir)
+      let backup = OpamRepositoryRoot.backup ~inn repo_root in
+      OpamRepositoryRoot.copy ~src:repo_root ~dst:backup;
+      let restore_backup () =
+        OpamRepositoryRoot.copy ~src:backup ~dst:repo_root
       in
       let rt = OpamRepositoryCommand.set_url rt name url trust_anchors in
       let failed, rt =
         OpamRepositoryCommand.update_with_auto_upgrade rt [name]
       in
       OpamRepositoryState.drop rt;
-      if failed <> [] then
-        (let repo = OpamRepositoryState.get_repo rt0 name in
+      (match failed with
+       | [] -> `Ok ()
+       | _ ->
+         let repo = OpamRepositoryState.get_repo rt0 name in
          OpamConsole.error
            "Fetching repository %s with %s fails, reverting to %s"
            (OpamRepositoryName.to_string name)
@@ -2558,10 +2557,9 @@ let repository cli =
          let rt =
            OpamRepositoryCommand.set_url rt0 name repo.repo_url repo.repo_trust
          in
-         backup ();
+         restore_backup ();
          OpamRepositoryState.drop rt;
-         OpamStd.Sys.exit_because `Sync_error);
-      `Ok ()
+         OpamStd.Sys.exit_because `Sync_error)
     | Some `set_repos, names ->
       let names = List.map OpamRepositoryName.of_string names in
       OpamGlobalState.with_ `Lock_none @@ fun gt ->
@@ -2740,7 +2738,7 @@ let switch cli =
         lists installed switches, with one switch argument, defaults to \
         $(b,set).";
     `P (Printf.sprintf
-         "Switch handles $(i,SWITCH) can be either a plain name, for switches \
+        "Switch handles $(i,SWITCH) can be either a plain name, for switches \
          that will be held inside $(i,~%s.opam), or a directory name, which in \
          that case is the directory where the switch prefix will be installed, as \
          %s. Opam will automatically select a switch by that name found in the \
@@ -2764,7 +2762,7 @@ let switch cli =
     mk_subdoc ~cli ~defaults:["","list";"SWITCH","set"]
       ~extra_defaults:[
         cli_from cli2_2, "-",
-        "Switches back to the previous non-local switch (similar to $(b,git switch -)). \
+         "Switches back to the previous non-local switch (similar to $(b,git switch -)). \
          To set the current switch to a switch named $(i,-) use $(b,--cli=2.1) \
          or lower, or use $(b,opam switch set -)"
       ] commands
@@ -3999,19 +3997,26 @@ let lint cli =
              | None -> raise Not_found
            else
              let opam = OpamSwitchState.opam st nv in
-             match OpamPinned.orig_opam_file st (OpamPackage.name nv) opam with
+             match OpamPinned.orig_opam_file st name opam with
              | None -> raise Not_found
              | Some file ->
-               let label =
-                 let reponame =
+               let repo, label =
+                 let repo, reponame =
                    match OpamFile.OPAM.metadata_dir opam with
-                   | None | Some (None, _) -> "repo"
-                   | Some (Some repo, _) -> OpamRepositoryName.to_string repo
+                   | None | Some (None, _) -> None, "repo"
+                   | Some (Some repo, _) ->
+                     Some repo,
+                     OpamRepositoryName.to_string repo
                  in
+                 repo,
                  Printf.sprintf "<%s>/%s"
                    reponame (OpamPackage.to_string nv)
                in
-               [`pkg (file, label)]
+               match repo with
+               | None -> [`pkg (file, label)]
+               | Some repo ->
+                 let repo_root = OpamRepositoryState.get_root st.switch_repos repo in
+                 [`repopkg (repo_root, file, label)]
          with Not_found ->
            OpamConsole.error_and_exit `Not_found "No opam file found for %s%s"
              (OpamPackage.Name.to_string (fst pkg))
@@ -4045,6 +4050,10 @@ let lint cli =
                 OpamFileTools.lint_file ~check_upstream ~handle_dirname:false
                   file,
                 Some label
+              | `repopkg (repo_root, file, label) ->
+                OpamFileTools.lint_repo_package repo_root ~check_upstream
+                  ~handle_dirname:false file,
+                Some label
               | `stdin ->
                 OpamFileTools.lint_channel ~check_upstream ~handle_dirname:false
                   stdin_f stdin,
@@ -4129,9 +4138,9 @@ let clean cli =
   let download_cache =
     mk_flag ~cli cli_original ["c"; "download-cache"]
       (Printf.sprintf
-        "Clear the cache of downloaded files (\\$OPAMROOT%sdownload-cache), as \
-         well as the obsolete \\$OPAMROOT%sarchives, if that exists."
-        OpamArg.dir_sep OpamArg.dir_sep)
+       "Clear the cache of downloaded files (\\$OPAMROOT%sdownload-cache), as \
+        well as the obsolete \\$OPAMROOT%sarchives, if that exists."
+       OpamArg.dir_sep OpamArg.dir_sep)
   in
   let repos =
     mk_flag ~cli cli_original ["unused-repositories"]
@@ -4364,7 +4373,8 @@ let clean cli =
            rmdir
              (OpamRepositoryRoot.Dir.to_dir
                 (OpamRepositoryRoot.Dir.Path.root root r));
-           rm (OpamRepositoryPath.tar root r))
+           rm (OpamRepositoryRoot.Tgz.to_file
+                 (OpamRepositoryRoot.Tgz.Path.root root r)))
          unused_repos;
        let repos_config =
          OpamRepositoryName.Map.filter

src/client/opamListCommand.ml

@@ -875,11 +875,13 @@ let info st ~fields ~raw ~where ?normalise ?(show_empty=false)
            let repo_dir =
              OpamRepositoryRoot.Dir.Path.root st.switch_global.root repo
            in
-           let tar = OpamRepositoryPath.tar st.switch_global.root repo in
-           if OpamFilename.exists tar &&
+           let tgz =
+             OpamRepositoryRoot.Tgz.Path.root st.switch_global.root repo
+           in
+           if OpamRepositoryRoot.Tgz.exists tgz &&
               not (OpamRepositoryRoot.Dir.exists repo_dir) then
              Printf.sprintf "<%s>%s%s"
-               (OpamFilename.to_string tar)
+               (OpamRepositoryRoot.Tgz.to_string tgz)
                Filename.dir_sep
                rdir
            else