1. 01 Dec, 2017 12 commits
  2. 29 Nov, 2017 1 commit
    • Bob Van Landuyt's avatar
      Create fork networks for forks for which the source was deleted. · 8202f0452dd9
      Bob Van Landuyt authored
      That way we can join forks-of-forks into the same network even if
      their original source was deleted.
      Consider the following:
        `awesome-oss/badger` is forked to `coolstuff/badger`, which is
        forked to `user-a/badger` which in turn is forked to
        When `awesome-oss/badger` is deleted, we will now create a fork
        network with `coolstuff/badger` as the root. The `user-a/badger`
        and `user-b/badger` projects will be added to the network.
  3. 23 Nov, 2017 1 commit
    • Yorick Peterse's avatar
      Added Rubocop config for background migrations · 160529ae7ce9
      Yorick Peterse authored
      This adds a Rubocop configuration file specific to
      lib/gitlab/background_migrations. This configuration will be used to
      (hopefully) make reviewing background migrations easier by enforcing
      stricter rules compared to the rest of GitLab. Because this
      configuration is directory specific it will only affect background
  4. 17 Nov, 2017 1 commit
  5. 14 Nov, 2017 1 commit
    • Bob Van Landuyt's avatar
      Don't try to create fork network memberships for forks of forks · 8de51094b5f5
      Bob Van Landuyt authored
      In case the root project of a Fork-of-fork is deleted, the ForkNetwork
      and the membership for that fork network is never created. In this
      case we shouldn't try to create the membership, since the parent
      membership will never be created.
      This means that these fork networks will be lost.
  6. 17 Oct, 2017 1 commit
  7. 09 Oct, 2017 1 commit
  8. 07 Oct, 2017 4 commits
  9. 05 Oct, 2017 2 commits
  10. 29 Sep, 2017 2 commits
    • Sean McGivern's avatar
      Make MR diff background migration less likely to time out · 12590f27c94e
      Sean McGivern authored
      This version does not use transactions, but individual statements. As we have
      unique constraints on the target tables for the inserts, we can just ignore
      uniqueness violations there (as long as we always insert the same batch size, in
      the same order).
      This means the spec now must use truncation, not a transaction, as the
      uniqueness violation means that the whole transaction for that spec would be
      invalid, which isn't what we'd want. In real-world use, this isn't run in a
      transaction anyway.
      This commit also wraps unhandled exceptions, for easier finding in Sentry, and
      logs with a consistent format, for easier searching.
    • Sean McGivern's avatar
      Insert at most 1,000 rows at once in MR diff background migration · 18726e90c42a
      Sean McGivern authored
      We were hitting the statement timeout for very large MR diffs. Now we insert at
      most 1,000 rows to `merge_request_diff_commits` in a single statement, or 100
      rows to `merge_request_diff_files`.
  11. 14 Sep, 2017 3 commits
  12. 18 Aug, 2017 1 commit
    • Sean McGivern's avatar
      Fix MR diffs background migration when file modes are missing · b50aaed65772
      Sean McGivern authored
      Guess the modes based on the following:
      1. If the file didn't exist, it's zero.
      2. If the diff contains 'Subproject commit', it might be a submodule, so 0600.
      3. Otherwise, it's 0644.
      This isn't perfect, but it doesn't have to be - it won't change file modes in
      the repository.
  13. 11 Aug, 2017 2 commits
  14. 10 Aug, 2017 1 commit
    • Yorick Peterse's avatar
      Migrate events into a new format · 20d17d1129f6
      Yorick Peterse authored
      This commit migrates events data in such a way that push events are
      stored much more efficiently. This is done by creating a shadow table
      called "events_for_migration", and a table called "push_event_payloads"
      which is used for storing push data of push events. The background
      migration in this commit will copy events from the "events" table into
      the "events_for_migration" table, push events in will also have a row
      created in "push_event_payloads".
      This approach allows us to reclaim space in the next release by simply
      swapping the "events" and "events_for_migration" tables, then dropping
      the old events (now "events_for_migration") table.
      The new table structure is also optimised for storage space, and does
      not include the unused "title" column nor the "data" column (since this
      data is moved to "push_event_payloads").
      == Newly Created Events
      Newly created events are inserted into both "events" and
      "events_for_migration", both using the exact same primary key value. The
      table "push_event_payloads" in turn has a foreign key to the _shadow_
      table. This removes the need for recreating and validating the foreign
      key after swapping the tables. Since the shadow table also has a foreign
      key to "projects.id" we also don't have to worry about orphaned rows.
      This approach however does require some additional storage as we're
      duplicating a portion of the events data for at least 1 release. The
      exact amount is hard to estimate, but for GitLab.com this is expected to
      be between 10 and 20 GB at most. The background migration in this commit
      deliberately does _not_ update the "events" table as doing so would put
      a lot of pressure on PostgreSQL's auto vacuuming system.
      == Supporting Both Old And New Events
      Application code has also been adjusted to support push events using
      both the old and new data formats. This is done by creating a PushEvent
      class which extends the regular Event class. Using Rails' Single Table
      Inheritance system we can ensure the right class is used for the right
      data, which in this case is based on the value of `events.action`. To
      support displaying old and new data at the same time the PushEvent class
      re-defines a few methods of the Event class, falling back to their
      original implementations for push events in the old format.
      Once all existing events have been migrated the various push event
      related methods can be removed from the Event model, and the calls to
      `super` can be removed from the methods in the PushEvent model.
      The UI and event atom feed have also been slightly changed to better
      handle this new setup, fortunately only a few changes were necessary to
      make this work.
      == API Changes
      The API only displays push data of events in the new format. Supporting
      both formats in the API is a bit more difficult compared to the UI.
      Since the old push data was not really well documented (apart from one
      example that used an incorrect "action" nmae) I decided that supporting
      both was not worth the effort, especially since events will be migrated
      in a few days _and_ new events are created in the correct format.
  15. 03 Aug, 2017 1 commit
    • Sean McGivern's avatar
      Migrate MR commits and diffs to new tables · 543e4e36bc49
      Sean McGivern authored
      Previously, we stored these as serialised fields - `st_{commits,diffs}` - on the
      `merge_request_diffs` table. These now have their own tables -
      `merge_request_diff_{commits,diffs}` - with a column for each attribute of the
      serialised data.
      Add a background migration to go through the existing MR diffs and migrate them
      to the new format. Ignore any contents that cannot be displayed. Assuming that
      we have 5 million rows to migrate, and each batch of 2,500 rows can be
      completed in 5 minutes, this will take about 7 days to migrate everything.
  16. 18 Jul, 2017 2 commits
  17. 11 Jul, 2017 1 commit
  18. 10 Jul, 2017 1 commit
  19. 07 Jul, 2017 2 commits