1. 01 Nov, 2018 1 commit
  2. 17 Oct, 2018 1 commit
  3. 23 Sep, 2018 1 commit
    • gfyoung's avatar
      Enable more frozen string in app/controllers/ · 30174479c9f2
      gfyoung authored
      Enables frozen string for the following:
      
      * app/controllers/dashboard/**/*.rb
      * app/controllers/explore/**/*.rb
      * app/controllers/google_api/**/*.rb
      * app/controllers/groups/**/*.rb
      * app/controllers/import/**/*.rb
      * app/controllers/instance_statistics/**/*.rb
      * app/controllers/ldap/**/*.rb
      * app/controllers/oauth/**/*.rb
      * app/controllers/profiles/**/*.rb
      
      Partially addresses #47424.
      30174479c9f2
  4. 11 Sep, 2018 1 commit
  5. 06 Jun, 2018 1 commit
    • Stan Hu's avatar
      Show a more helpful error for import status · eb9c377d9044
      Stan Hu authored
      Importing a project from GitHub for a project namespace that already exists
      would show an unhelpful error, "An error occurred while importing project."
      We now add the base message from Projects::CreateService when this fails.
      
      Closes #47365
      eb9c377d9044
  6. 04 May, 2018 1 commit
  7. 07 Mar, 2018 1 commit
  8. 09 Feb, 2018 1 commit
  9. 07 Nov, 2017 1 commit
    • Yorick Peterse's avatar
      Rewrite the GitHub importer from scratch · 86c05c535691
      Yorick Peterse authored
      Prior to this MR there were two GitHub related importers:
      
      * Github::Import: the main importer used for GitHub projects
      * Gitlab::GithubImport: importer that's somewhat confusingly used for
        importing Gitea projects (apparently they have a compatible API)
      
      This MR renames the Gitea importer to Gitlab::LegacyGithubImport and
      introduces a new GitHub importer in the Gitlab::GithubImport namespace.
      This new GitHub importer uses Sidekiq for importing multiple resources
      in parallel, though it also has the ability to import data sequentially
      should this be necessary.
      
      The new code is spread across the following directories:
      
      * lib/gitlab/github_import: this directory contains most of the importer
        code such as the classes used for importing resources.
      * app/workers/gitlab/github_import: this directory contains the Sidekiq
        workers, most of which simply use the code from the directory above.
      * app/workers/concerns/gitlab/github_import: this directory provides a
        few modules that are included in every GitHub importer worker.
      
      == Stages
      
      The import work is divided into separate stages, with each stage
      importing a specific set of data. Stages will schedule the work that
      needs to be performed, followed by scheduling a job for the
      "AdvanceStageWorker" worker. This worker will periodically check if all
      work is completed and schedule the next stage if this is the case. If
      work is not yet completed this worker will reschedule itself.
      
      Using this approach we don't have to block threads by calling `sleep()`,
      as doing so for large projects could block the thread from doing any
      work for many hours.
      
      == Retrying Work
      
      Workers will reschedule themselves whenever necessary. For example,
      hitting the GitHub API's rate limit will result in jobs rescheduling
      themselves. These jobs are not processed until the rate limit has been
      reset.
      
      == User Lookups
      
      Part of the importing process involves looking up user details in the
      GitHub API so we can map them to GitLab users. The old importer used
      an in-memory cache, but this obviously doesn't work when the work is
      spread across different threads.
      
      The new importer uses a Redis cache and makes sure we only perform
      API/database calls if absolutely necessary.  Frequently used keys are
      refreshed, and lookup misses are also cached; removing the need for
      performing API/database calls if we know we don't have the data we're
      looking for.
      
      == Performance & Models
      
      The new importer in various places uses raw INSERT statements (as
      generated by `Gitlab::Database.bulk_insert`) instead of using Rails
      models. This allows us to bypass any validations and callbacks,
      drastically reducing the number of SQL queries and Gitaly RPC calls
      necessary to import projects.
      
      To ensure the code produces valid data the corresponding tests check if
      the produced rows are valid according to the model validation rules.
      86c05c535691
  10. 14 Aug, 2017 1 commit
  11. 08 Aug, 2017 1 commit
  12. 19 Dec, 2016 3 commits
  13. 20 Sep, 2016 1 commit
  14. 31 Aug, 2016 2 commits
  15. 30 Jun, 2016 2 commits
    • Rémy Coutable's avatar
      Make GH one-off auth the default again for importing GH projects · 2e5fe182b784
      Rémy Coutable authored
      
      
      Advertise the PAT as an alternative unless GH import is not configured.
      Signed-off-by: default avatarRémy Coutable <remy@rymai.me>
      2e5fe182b784
    • Eric K Idema's avatar
      Import from Github using Personal Access Tokens. · de147ece3102
      Eric K Idema authored
      This stands as an alternative to using OAuth to access a user's Github
      repositories.  This is setup in such a way that it can be used without OAuth
      configuration.
      
      From a UI perspective, the how to import modal has been replaced by a full
      page, which includes a form for posting a personal access token back to the
      Import::GithubController.
      
      If the user has logged in via GitHub, skip the Personal Access Token and go
      directly to Github for an access token via OAuth.
      de147ece3102
  16. 19 Oct, 2015 1 commit
  17. 13 Oct, 2015 1 commit
  18. 23 Aug, 2015 1 commit
    • Stan Hu's avatar
      Remove user OAuth tokens stored in database for Bitbucket, GitHub, and GitLab · 1f548c79b42a
      Stan Hu authored
      and request them each session. Pass these tokens to the project import data.
      
      This prevents the need to encrypt these tokens and clear them in case they
      expire or get revoked.
      
      For example, if you deleted and re-created OAuth2 keys for Bitbucket, you would get
      an Error 500 with no way to recover:
      
      ```
      Started GET "/import/bitbucket/status" for x.x.x.x at 2015-08-07 05:24:10 +0000
      Processing by Import::BitbucketController#status as HTML
      Completed 500 Internal Server Error in 607ms (ActiveRecord: 2.3ms)
      
      NameError (uninitialized constant Import::BitbucketController::Unauthorized):
        app/controllers/import/bitbucket_controller.rb:77:in `rescue in go_to_bitbucket_for_permissions'
        app/controllers/import/bitbucket_controller.rb:74:in `go_to_bitbucket_for_permissions'
        app/controllers/import/bitbucket_controller.rb:86:in `bitbucket_unauthorized'
      ```
      
      Closes #1871
      1f548c79b42a
  19. 20 Apr, 2015 1 commit
  20. 31 Mar, 2015 1 commit
  21. 12 Mar, 2015 1 commit
  22. 24 Feb, 2015 1 commit
  23. 06 Feb, 2015 2 commits
  24. 05 Feb, 2015 4 commits
  25. 03 Feb, 2015 1 commit
  26. 20 Jan, 2015 1 commit
  27. 18 Jan, 2015 1 commit
    • Steven Burgart's avatar
      Fix various typos · 046df0a8b967
      Steven Burgart authored
      signe-in -> signed-in
      go_to_gihub_for_permissions -> go_to_github_for_permissions
      descendand -> descendant
      behavour -> behaviour
      recepient_email -> recipient_email
      generate_fingerpint -> generate_fingerprint
      dependes -> depends
      Cant't -> Can't
      wisit -> visit
      notifcation -> notification
      sufficent_scope -> sufficient_scope?
      levet -> level
      046df0a8b967
  28. 13 Jan, 2015 1 commit
  29. 10 Jan, 2015 1 commit