Skip to content
Snippets Groups Projects
  1. Oct 24, 2019
    • Gregory Szorc's avatar
      packaging: consolidate CLI functionality into packaging.py · 081a77df
      Gregory Szorc authored
      Consolidating functionality for invoking code in the hgpackaging
      package through a single CLI entry point will make things simpler
      when we add more complexity to that package. For example, it will
      allow us to run things out of a virtualenv with third party
      packages.
      
      This commit consolidates functionality from the Inno and WiX
      build.py scripts into a new packaging.py script. That script
      simply creates a virtualenv and runs the CLI functionality in
      it.
      
      The new virtualenv is populated with jinja2 because I felt
      it easier to incorporate requirements file processing in this
      commit and we will soon use jinja2 in an upcoming commit.
      
      The unified CLI functionality will also make it easier to
      script other packaging workflows going forward. e.g. RPM, Debian,
      and macOS packaging.
      
      Differential Revision: https://phab.mercurial-scm.org/D7156
      081a77df
  2. Oct 29, 2019
  3. Oct 14, 2019
  4. Oct 06, 2019
  5. Oct 05, 2019
  6. Sep 06, 2019
    • Gregory Szorc's avatar
      automation: implement "publish-windows-artifacts" command · 92593d72
      Gregory Szorc authored
      The new command and associated functionality can be used to
      automate the publishing of Windows release artifacts. It
      supports uploading wheels to PyPI (using twine) and copying
      the artifacts to mercurial-scm.org and updating the latest.dat
      file to advertise them via the website.
      
      I ran `automation.py publish-windows-artifacts 5.1.1` and it
      appeared to "just work." But the real test will be to do this
      on the next release...
      
      Differential Revision: https://phab.mercurial-scm.org/D6786
      92593d72
  7. Apr 27, 2019
    • Gregory Szorc's avatar
      automation: initial support for running Linux tests · 65b3ef16
      Gregory Szorc authored
      Building on top of our Windows automation support, this commit
      implements support for performing automated tasks on remote Linux
      machines. Specifically, we implement support for running tests
      on ephemeral EC2 instances. This seems to be a worthwhile place
      to start, as building packages on Linux is more or less a solved
      problem because we already have facilities for building in Docker
      containers, which provide "good enough" reproducibility guarantees.
      
      The new `run-tests-linux` command works similarly to
      `run-tests-windows`: it ensures an AMI with hg dependencies is
      available, provisions a temporary EC2 instance with this AMI, pushes
      local changes to that instance via SSH, then invokes `run-tests.py`.
      
      Using this new command, I am able to run the entire test harness
      substantially faster then I am on my local machine courtesy of
      access to massive core EC2 instances:
      
      wall: 16:20 ./run-tests.py -l (i7-6700K)
      wall: 14:00 automation.py run-tests-linux --ec2-instance c5.2xlarge
      wall:  8:30 automation.py run-tests-linux --ec2-instance m5.4xlarge
      wall:  8:04 automation.py run-tests-linux --ec2-instance c5.4xlarge
      wall:  4:30 automation.py run-tests-linux --ec2-instance c5.9xlarge
      wall:  3:57 automation.py run-tests-linux --ec2-instance m5.12xlarge
      wall:  3:05 automation.py run-tests-linux --ec2-instance m5.24xlarge
      wall:  3:02 automation.py run-tests-linux --ec2-instance c5.18xlarge
      
      ~3 minute wall time to run pretty much the entire test harness is
      not too bad!
      
      The AMIs install multiple versions of Python. And the run-tests-linux
      command specifies which one to use:
      
      automation.py run-tests-linux --python system3
      automation.py run-tests-linux --python 3.5
      automation.py run-tests-linux --python pypy2.7
      
      By default, the system Python 2.7 is used. Using this functionality,
      I was able to identity some unexpected test failures on PyPy!
      
      Included in the feature is support for running with alternate
      filesystems. You can simply pass --filesystem to the command to
      specify the type of filesystem to run tests on. When the ephemeral
      instance is started, a new filesystem will be created and tests
      will run from it:
      
      wall:  4:30 automation.py run-tests-linux --ec2-instance c5.9xlarge
      wall:  4:20 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem xfs
      wall:  4:24 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem tmpfs
      wall:  4:26 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem ext4
      
      We also support multiple Linux distributions:
      
      $ automation.py run-tests-linux --distro debian9
      total time: 298.1s; setup: 60.7s; tests: 237.5s; setup overhead: 20.4%
      
      $ automation.py run-tests-linux --distro ubuntu18.04
      total time: 286.1s; setup: 61.3s; tests: 224.7s; setup overhead: 21.4%
      
      $ automation.py run-tests-linux --distro ubuntu18.10
      total time: 278.5s; setup: 58.2s; tests: 220.3s; setup overhead: 20.9%
      
      $ automation.py run-tests-linux --distro ubuntu19.04
      total time: 265.8s; setup: 42.5s; tests: 223.3s; setup overhead: 16.0%
      
      Debian and Ubuntu are supported because those are what I use and am
      most familiar with. It should be easy enough to add support for other
      distros.
      
      Unlike the Windows AMIs, Linux EC2 instances bill per second. So
      the cost to instantiating an ephemeral instance isn't as severe.
      That being said, there is some overhead, as it takes several dozen
      seconds for the instance to boot, push local changes, and build
      Mercurial. During this time, the instance is largely CPU idle and
      wasting money. Even with this inefficiency, running tests is
      relatively cheap: $0.15-$0.25 per full test run. A machine running
      tests as efficiently as these EC2 instances would cost say $6,000, so
      you can run the test harness a >20,000 times for the cost of an
      equivalent machine. Running tests in EC2 is almost certainly cheaper
      than buying a beefy machine for developers to use :)
      
      # no-check-commit because foo_bar function names
      
      Differential Revision: https://phab.mercurial-scm.org/D6319
      65b3ef16
  8. Mar 15, 2019
    • Gregory Szorc's avatar
      automation: perform tasks on remote machines · b05a3e28
      Gregory Szorc authored
      Sometimes you don't have access to a machine in order to
      do something. For example, you may not have access to a Windows
      machine required to build Windows binaries or run tests on that
      platform.
      
      This commit introduces a pile of code intended to help
      "automate" common tasks, like building release artifacts.
      
      In its current form, the automation code provides functionality
      for performing tasks on Windows EC2 instances.
      
      The hgautomation.aws module provides functionality for integrating
      with AWS. It manages EC2 resources such as IAM roles, EC2
      security groups, AMIs, and instances.
      
      The hgautomation.windows module provides a higher-level
      interface for performing tasks on remote Windows machines.
      
      The hgautomation.cli module provides a command-line interface to
      these higher-level primitives.
      
      I attempted to structure Windows remote machine interaction
      around Windows Remoting / PowerShell. This is kinda/sorta like
      SSH + shell, but for Windows. In theory, most of the functionality
      is cloud provider agnostic, as we should be able to use any
      established WinRM connection to interact with a remote. In
      reality, we're tightly coupled to AWS at the moment because
      I didn't want to prematurely add abstractions for a 2nd cloud
      provider. (1 was hard enough to implement.)
      
      In the aws module is code for creating an image with a fully
      functional Mercurial development environment. It contains VC9,
      VC2017, msys, and other dependencies. The image is fully capable
      of building all the existing Mercurial release artifacts and
      running tests.
      
      There are a few things that don't work. For example, running
      Windows tests with Python 3. But building the Windows release
      artifacts does work. And that was an impetus for this work.
      (Although we don't yet support code signing.)
      
      Getting this functionality to work was extremely time consuming.
      It took hours debugging permissions failures and other wonky
      behavior due to PowerShell Remoting. (The permissions model for
      PowerShell is crazy and you brush up against all kinds of
      issues because of the user/privileges of the user running
      the PowerShell and the permissions of the PowerShell session
      itself.)
      
      The functionality around AWS resource management could use some
      improving. In theory we support shared tenancy via resource
      name prefixing. In reality, we don't offer a way to configure
      this.
      
      Speaking of AWS resource management, I thought about using a tool
      like Terraform to manage resources. But at our scale, writing a
      few dozen lines of code to manage resources seemed acceptable.
      Maybe we should reconsider this if things grow out of control.
      Time will tell.
      
      Currently, emphasis is placed on Windows. But I only started
      there because it was likely to be the most difficult to implement.
      It should be relatively trivial to automate tasks on remote Linux
      machines. In fact, I have a ~1 year old script to run tests on a
      remote EC2 instance. I will likely be porting that to this new
      "framework" in the near future.
      
      # no-check-commit because foo_bar functions
      
      Differential Revision: https://phab.mercurial-scm.org/D6142
      b05a3e28
  9. Mar 08, 2019
    • Gregory Szorc's avatar
      wix: functionality to automate building WiX installers · 4371f543
      Gregory Szorc authored
      Like we did for Inno Setup, we want to make it easier to
      produce WiX installers. This commit does that.
      
      We introduce a new hgpackaging.wix module for performing
      all the high-level tasks required to produce WiX installers.
      This required miscellaneous enhancements to existing code in
      hgpackaging, including support for signing binaries.
      
      A new build.py script for calling into the module APIs has been
      created. It behaves very similarly to the Inno Setup build.py
      script.
      
      Unlike Inno Setup, we didn't have code in the repo previously
      to generate WiX installers. It appears that all existing
      automation for building WiX installers lives in the
      https://bitbucket.org/tortoisehg/thg-winbuild repository - most
      notably in its setup.py file. My strategy for inventing the
      code in this commit was to step through the code in that repo's
      setup.py and observe what it was doing. Despite the length of
      setup.py in that repository, the actual amount of steps required
      to produce a WiX installer is actually quite low. It consists
      of a basic py2exe build plus invocations of candle.exe and
      light.exe to produce the MSI.
      
      One rabbit hole that gave me fits was locating the Visual Studio
      9 C Runtime merge modules. These merge modules are only present
      on your system if you have a full Visual Studio 2008 installation.
      Fortunately, I have a copy of Visual Studio 2008 and was able
      to install all the required updates. I then uploaded these merge
      modules to a personal repository on GitHub. That is where the
      added code references them from. We probably don't need to
      ship the merge modules. But that is for another day.
      
      The installs from the MSIs produced with the new automation
      differ from the last official MSI in the following ways:
      
      * Our HTML manual pages have UNIX line endings instead of Windows.
      * We ship modules in the mercurial.pure package. It appears the
        upstream packaging code is not including this package due to
        omission (they supply an explicit list of packages that has
        drifted out of sync with our setup.py).
      * We do not ship various distutils.* modules. This is because
        virtualenvs have a custom distutils/__init__.py that automagically
        imports distutils from its original location and py2exe gets
        confused by this. We don't use distutils in core Mercurial and
        don't provide a usable python.exe, so this omission should be
        acceptable.
      * The version of the enum package is different and we ship
        an enum.pyc instead of an enum/__init__.py.
      * The version of the docutils package is different and we
        ship a different set of files.
      * The version of Sphinx is drastically newer and we ship a
        number of files the old version did not. (I'm not sure why
        we ship Sphinx - I think it is a side-effect of the way the
        THG code was installing dependencies.)
      * We ship the idna package (dependent of requests which is a
        dependency of newer versions of Sphinx).
      * The version of imagesize is different and we ship an
        imagesize.pyc instead of an imagesize/__init__.pyc.
      * The version of the jinja2 package is different and the sets
        of files differs.
      * We ship the packaging package, which is a dependency for Sphinx.
      * The version of the pygments package is different and the sets
        of files differs.
      * We ship the requests package, which is a dependency for Sphinx.
      * We ship the snowballstemmer package, which is a dependency for
        Sphinx.
      * We ship the urllib3 package, which is a dependency for requests,
        which is a dependency for Sphinx.
      * We ship a newer version of the futures package, which includes a
        handful of extra modules that match Python 3 module names.
      
      # no-check-commit because foo_bar naming
      
      Differential Revision: https://phab.mercurial-scm.org/D6097
      4371f543
  10. Mar 07, 2019
    • Gregory Szorc's avatar
      packaging: extract py2exe functionality to own module · a2e191a9
      Gregory Szorc authored
      py2exe builds are shared between Inno Setup and WIX. We'll
      want the logic for performing py2exe builds to be reusable
      across the code for both installers.
      
      This commit extracts the py2exe-specific functionality into
      its own module.
      
      There's definitely room to customize things further. This will
      be done in future commits, as necessary. (I'm not even sure what
      customizations WIX will require yet. Presumably a lot.)
      
      Differential Revision: https://phab.mercurial-scm.org/D6091
      a2e191a9
    • Gregory Szorc's avatar
      packaging: move Inno Setup core logic into a module · dc7827a9
      Gregory Szorc authored
      Aspects of building the Inno Setup and WIX installers are shared.
      It will make sense for them to share code.
      
      Plus, having code in a reusable library (as opposed to a standalone
      script) is just a better approach.
      
      This commit moves the core logic to build the Inno Setup installer
      into the hgpackaging package. inno/build.py is now a simple frontend
      script that calls into a module to do the bulk of the work.
      
      As part of this change, I also found a typo in build() where it was
      referencing "iscc" instead of "iscc_exe." Because "iscc" was in
      the global scope via the only caller, things just happened to work
      before. Another benefit of always using functions and not putting
      global code for __main__ in the same file as library code.
      
      Differential Revision: https://phab.mercurial-scm.org/D6087
      dc7827a9
    • Gregory Szorc's avatar
      packaging: split downloading code into own module · c2237fe1
      Gregory Szorc authored
      As we will introduce more code to support packaging, it will be
      useful to have download code in its own module.
      
      Differential Revision: https://phab.mercurial-scm.org/D6084
      c2237fe1
    • Gregory Szorc's avatar
      packaging: establish hgpackaging package · 9da97f49
      Gregory Szorc authored
      Previously, contrib/packaging behaved as a root to a
      package directory and we had a "packagingutil" module. As I
      work more on packaging code, we'll want to have more code
      shared between different packaging tools. I think it makes
      sense to have a single package containing multiple modules
      than multiple top-level modules.
      
      This commit establishes an "hgpackaging" package by moving
      the existing packagingutil code to it.
      
      Differential Revision: https://phab.mercurial-scm.org/D6083
      9da97f49
  11. Mar 04, 2019
    • Gregory Szorc's avatar
      inno: script to automate building Inno installer · d7dc4ac1
      Gregory Szorc authored
      The official Inno installer build process is poorly documented.
      And attempting to reproduce behavior of the installer uploaded
      to www.mercurial-scm.org has revealed a number of unexpected
      behaviors.
      
      This commit attempts to improve the state of reproducibility
      of the Inno installer by introducing a Python script to
      largely automate the building of the installer.
      
      The new script (which must be run from an environment with the
      Visual C++ environment configured) takes care of producing an
      Inno installer. When run from a fresh Mercurial source checkout
      with all the proper system dependencies (the VC++ toolchain,
      Windows 10 SDK, and Inno tools) installed, it "just works."
      The script takes care of downloading all the Python
      dependencies in a secure manner and manages the build
      environment for you. You don't need any additional config
      files: just launch the script, pointing it at an existing
      Python and ISCC binary and it takes care of the rest.
      
      The produced installer creates a Mercurial installation with
      a handful of differences from the existing 4.9 installers
      (produced by someone else):
      
      * add_path.exe is missing (this was removed a few changesets ago)
      * The set of api-ms-win-core-* DLLs is different (I suspect this
        is due to me using a different UCRT / Windows version).
      * kernelbase.dll and msasn1.dll are missing.
      * There are a different set of .pyc files for dulwich,
        keyring, and pygments due to us using the latest versions of
        each.
      * We include Tcl/Tk DLLs and .pyc files (I'm not sure why these
        are missing from the existing installers).
      * We include the urllib3 and win32ctypes packages (which are
        dependencies of dulwich and pywin32, respectively). I'm not
        sure why these aren't present in the existing installers.
      * We include a different set of files for the distutils package.
        I'm not sure why. But it should be harmless.
      * We include the docutils package (it is getting picked up as
        a dependency somehow). I think this is fine.
      * We include a copy of argparse.pyc. I'm not sure why this was
        missing from existing installers.
      * We don't have a copy of sqlite3/dump.pyc. I'm not sure why. The
        SQLite C extension code only imports this module when
        conn.iterdump() is called. It should be safe to omit.
      * We include files in the email.test and test packages. The set of
        files is small and their presence should be harmless.
      
      The new script and support code is written in Python 3 because
      it is brand new and independent code and I don't believe new
      Python projects should be using Python 2 in 2019 if they have
      a choice about it.
      
      The readme.txt file has been renamed to readme.rst and overhauled
      to reflect the existence of build.py.
      
      Differential Revision: https://phab.mercurial-scm.org/D6066
      d7dc4ac1
  12. Feb 04, 2019
  13. Oct 12, 2018
  14. Aug 10, 2018
  15. Apr 20, 2018
  16. Feb 26, 2018
    • Augie Fackler's avatar
      http: drop custom http client logic · 23d12524
      Augie Fackler authored
      Eight and a half years ago, as my starter bug on code.google.com, I
      investigated a mysterious "broken pipe" error from seemingly random
      clients[0]. That investigation revealed a tragic story: the Python
      standard library's httplib was (and remains) barely functional. During
      large POSTs, if a server responds early with an error (even a
      permission denied error!) the client only notices that the server
      closed the connection and everything breaks. Such server behavior is
      implicitly legal under RFC 2616 (the latest HTTP RFC as of when I was
      last working on this), and my understanding is that later RFCs have
      made it explicitly legal to respond early with any status code outside
      the 2xx range.
      
      I embarked, probably foolishly, on a journey to write a new http
      library with better overall behavior. The http library appears to work
      well in most cases, but it can get confused in the presence of
      proxies, and it depends on select(2) which limits its utility if a lot
      of file descriptors are open. I haven't touched the http library in
      almost two years, and in the interim the Python community has
      discovered a better way[1] of writing network code. In theory some day
      urllib3 will have its own home-grown http library built on h11[2], or
      we could do that. Either way, it's time to declare our current
      confusingly-named "http2" client logic and move on. I do hope to
      revisit this some day: it's still garbage that we can't even respond
      with a 401 or 403 without reading the entire POST body from the
      client, but the goalposts on writing a new http client library have
      moved substantially. We're almost certainly better off just switching
      to requests and eventually picking up their http fixes than trying to
      live with something that realistically only we'll ever use. Another
      approach would be to write an adapter so that Mercurial can use pycurl
      if it's installed. Neither of those approaches seem like they should
      be investigated prior to a release of Mercurial that works on Python
      3: that's where the mindshare is going to be for any improvements to
      the state of the http client art.
      
      0: http://web.archive.org/web/20130501031801/http://code.google.com/p/support/issues/detail?id=2716
      1: http://sans-io.readthedocs.io/
      2: https://github.com/njsmith/h11
      
      Differential Revision: https://phab.mercurial-scm.org/D2444
      23d12524
  17. Nov 30, 2017
  18. Nov 22, 2017
  19. Jan 24, 2015
  20. Sep 26, 2017
    • David Demelier's avatar
      doc: rename README to README.rst · 1b59287a
      David Demelier authored
      Many hosting services consider README without extension as plain text. By using
      .rst extension, we bring better formatting on many services (e.g. bitbucket).
      1b59287a
  21. Oct 01, 2017
  22. Jul 21, 2017
  23. Jul 15, 2017
  24. Jul 02, 2017
  25. Jun 28, 2017
    • Adam Simpkins's avatar
      tests: use the system hg for examining the local repository · 6c113a7d
      Adam Simpkins authored
      Most test scripts use "hg" to interact with a temporary test repository.
      However a few tests also want to run hg commands to interact with the local
      repository containing the mercurial source code.  Notably, many of the
      test-check-* tests want to check local files and commit messages.
      
      These tests were previously using the version of hg being tested to query the
      source repository.  However, this will fail if the source repository requires
      extensions or other settings not supported by the version of mercurial being
      tested.  The source repository was typically initially cloned using the system
      hg installation, so we should use the system hg installation to query it.
      
      There was already a helpers-testrepo.sh script designed to help cope with
      different requirements for the source repository versus the test repositories.
      However, it only handled the evolve extension.  This new behavior works with
      any extensions that are different between the system installation and the test
      installation.
      6c113a7d
  26. May 06, 2017
  27. Apr 26, 2017
  28. May 01, 2017
  29. Apr 26, 2017
  30. May 04, 2017
    • Phil Cohen's avatar
      demandimport: add urwid.command_map to ignore list · c939fdce
      Phil Cohen authored
      The useful pudb debugger can be used with Mercurial, but its import of urwid
      fails when demandimport is enabled. Add urwid.command_map to the ignore list so
      pudb can be used with hg without disabling all of demandimport.
      c939fdce
  31. Apr 14, 2017
    • Gregory Szorc's avatar
      tests: add tests for poorly behaving HTTP server · c85f19c6
      Gregory Szorc authored
      I've spent several hours over the past few weeks investigating
      networking failures involving hg.mozilla.org. As part of this, it
      has become clear that the Mercurial client's error handling when
      it encounters network failures is far from robust.
      
      To prove this is true, I've devised a battery of tests simulating
      various network failures, notably premature connection closes. To
      achieve this, I've implemented an extension that monkeypatches the
      built-in HTTP server and hooks in at the socket level and allows
      various events to occur based on config options. For example, you
      can refuse to accept() a client socket or you can close() the socket
      after N bytes have been sent or received. The latter effectively
      simulates an unexpected connection drop (and these occur all the
      time in the real world).
      
      The new test file launches servers exhibiting various "bad" behaviors
      and points a client at them. As the many TODO comments in the test
      call attention to, Mercurial often displays unhelpful errors when
      network-related failures occur. This makes it difficult for users
      to understand what's going on and difficult for server administrators
      to pinpoint root causes without packet tracing.
      
      Upcoming patches will attempt to fix these error handling
      deficiencies.
      c85f19c6
    • Kostia Balytskyi's avatar
      windows: add win32com.shell to demandimport ignore list · 3e03a4b9
      Kostia Balytskyi authored
      Module 'appdirs' tries to import win32com.shell (and catch ImportError as an
      indication of failure) to check whether some further functionality should
      be implemented one or another way [1]. Of course, demandimport lets it down, so
      if we want appdirs to work we have to add it to demandimport's ignore list.
      
      The reason we want appdirs to work is becuase it is used by setuptools [2] to
      determine egg cache location. Only fairly recent versions of setuptools depend
      on this so people don't see this often.
      
      
      [1] https://github.com/ActiveState/appdirs/blob/master/appdirs.py#L560
      [2] https://github.com/pypa/setuptools/blob/aae0a928119d2a178882c32bded02270e30d0273/pkg_resources/__init__.py#L1369
      3e03a4b9
  32. Apr 07, 2017
  33. Apr 06, 2017
  34. Apr 03, 2017
Loading