Gitlab

更新後數據庫遷移的Gitlab docker錯誤(重啟循環)

  • July 12, 2020

我剛剛更新了我的 gitlab docker 實例,在主機上使用 docker-compose 執行。我遇到以下錯誤:

Recipe: gitlab::database_migrations
 * bash[migrate gitlab-rails database] action run
   [execute] rake aborted!
             PG::ConnectionBad: could not connect to server: No such file or directory
               Is the server running locally and accepting
               connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
             /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:48:in `block (3 levels) in <top (required)>'
             /opt/gitlab/embedded/bin/bundle:23:in `load'
             /opt/gitlab/embedded/bin/bundle:23:in `<main>'
             Tasks: TOP => gitlab:db:configure
             (See full trace by running task with --trace)
   
   ================================================================================
   Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
   ================================================================================
   
   Mixlib::ShellOut::ShellCommandFailed
   ------------------------------------
   Expected process to exit with [0], but received '1'
   ---- Begin output of "bash"  "/tmp/chef-script20200707-23-1sorsih" ----
   STDOUT: rake aborted!
   PG::ConnectionBad: could not connect to server: No such file or directory
       Is the server running locally and accepting
       connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
   /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:48:in `block (3 levels) in <top (required)>'
   /opt/gitlab/embedded/bin/bundle:23:in `load'
   /opt/gitlab/embedded/bin/bundle:23:in `<main>'
   Tasks: TOP => gitlab:db:configure
   (See full trace by running task with --trace)
   STDERR: 
   ---- End output of "bash"  "/tmp/chef-script20200707-23-1sorsih" ----
   Ran "bash"  "/tmp/chef-script20200707-23-1sorsih" returned 1
   
   Resource Declaration:
   ---------------------
   # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
   
    55: bash "migrate gitlab-rails database" do
    56:   code <<-EOH
    57:     set -e
    58:     log_file="#{node['gitlab']['gitlab-rails']['log_directory']}/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log"
    59:     umask 077
    60:     /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}
    61:     STATUS=${PIPESTATUS[0]}
    62:     chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} ${log_file}
    63:     echo $STATUS > #{db_migrate_status_file}
    64:     exit $STATUS
    65:   EOH
    66:   environment env_variables unless env_variables.empty?
    67:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
    68:   dependent_services.each do |svc|
    69:     notifies :restart, svc, :immediately
    70:   end
    71:   not_if "(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)"
    72:   only_if { node['gitlab']['gitlab-rails']['auto_migrate'] }
    73: end
   
   Compiled Resource:
   ------------------
   # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:55:in `from_file'
   
   bash("migrate gitlab-rails database") do
     action [:run]
     default_guard_interpreter :default
     command nil
     backup 5
     interpreter "bash"
     declared_type :bash
     cookbook_name "gitlab"
     recipe_name "database_migrations"
     code "    set -e\n    log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n    umask 077\n    /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n    STATUS=${PIPESTATUS[0]}\n    chown git:git ${log_file}\n    echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-8411f47246f\n    exit $STATUS\n"
     domain nil
     user nil
     not_if "(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-8411f47246f) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-8411f47246f | grep -Fx 0)"
     only_if { #code block }
   end
   
   System Info:
   ------------
   chef_version=14.14.29
   platform=ubuntu
   platform_version=16.04
   ruby=ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
   program_name=/opt/gitlab/embedded/bin/chef-client
   executable=/opt/gitlab/embedded/bin/chef-client
   

Running handlers:
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 55) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20200707-23-1sorsih" ----
STDOUT: rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
       Is the server running locally and accepting
       connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:48:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR: 
---- End output of "bash"  "/tmp/chef-script20200707-23-1sorsih" ----
Ran "bash"  "/tmp/chef-script20200707-23-1sorsih" returned 1

Running handlers complete
Chef Client failed. 9 resources updated in 18 seconds
Thank you for using GitLab Docker Image!
Current version: gitlab-ce=13.1.3-ce.0

Configure GitLab for your system by editing /etc/gitlab/gitlab.rb file
And restart this container to reload settings.
To do it use docker exec:

 docker exec -it gitlab vim /etc/gitlab/gitlab.rb
 docker restart gitlab

For a comprehensive list of configuration options please see the Omnibus GitLab readme
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md

If this container fails to start due to permission problems try to fix it by executing:

 docker exec -it gitlab update-permissions
 docker restart gitlab

我已經看到了這個問題,但沒有一個答案解決了我的問題。

如果需要更多資訊,請隨時發表評論。

更新:

在@fuero 的回答之後,我遇到了一些問題:

  1. 嘗試執行 docker 命令時出現 tty 錯誤(可能不相關,但無論如何我找到了解決方法)。我必須先拆下容器。
  2. /assets/wrapper 使機器崩潰了,所以我忽略了這一步,因為它使用 docker-compose up 執行良好。
  3. 儲存庫中不存在 postgresql 包,因此我必須手動添加它們。
  4. 檢查時出現權限錯誤,因為我在 /。移動到 /tmp 解決了這個問題。
  5. 我現在仍然在 –check 另一個錯誤:Old cluster data and binary directories are from different major versions.我還找不到解決這個問題的方法。

遵循@fuero 的回答會產生一個工作安裝,但是gitlab.rb必須手動更新以停止在不需要遷移時發生的數據庫遷移。

執行這些步驟的Old cluster data and binary directories are from different major versions錯誤是由於數據庫已經遷移了一半。修復方法是對新舊二進製文件都使用版本 11。

這裡也一樣。查看它包含的 Postgresql 日誌:

The data directory was initialized by PostgreSQL version 10, which is not compatible with this version 11.7.

似乎 docker 版本無法正確升級其數據庫。

我現在已經恢復到舊版本 (12.7) - 如果沒有手動數據庫升級,13.x 似乎壞了。以下是有人這樣做的方式,包括作為要保留的引用的內容:

在 gitlab-ce 13.1.2 容器中將 PostgreSQL 10 升級到 11

使用 docker-compose 啟動我們的容器將在其中啟動 gitlab。由於數據庫故障,容器每分鐘重啟一次。我們不希望在我們的遷移工作中使用它。所以我們手動啟動它並訪問其中的 bash。

請用您的自定義配置替換(至少)主機名和其他選項!

docker run --rm \
--hostname gitlab.example.org \
--publish 8143:443 --publish 8180:80 \
--env GITLAB_OMNIBUS_CONFIG="external_url 'https://gitlab.example.org'; letsencrypt['enabled'] = false" \
--name gitlab_custom \
--volume /srv/gitlab/config:/etc/gitlab \
--volume /srv/gitlab/logs:/var/log/gitlab \
--volume /srv/gitlab/data:/var/opt/gitlab \
-it gitlab/gitlab-ce:latest /bin/bash

現在我們在 docker 容器中。

首先執行預設的入口腳本來檢查它是否真的不起作用!

/assets/wrapper

如果配方中出現故障:gitlab::database_migrations 部分,並且至少出現錯誤消息 Error execution action ‘run’ on resource ‘bash

$$ migrate gitlab-rails database $$’,繼續遷移數據庫。 我們需要安裝預設的(舊的和新的)PostgreSQL 版本。

apt update
apt install postgresql-10 postgresql-11

為遷移的數據庫創建目標文件夾並將目錄所有者更新為 gitlab-psql。我們假設,“舊”(版本 10)數據庫數據位於 /var/opt/gitlab/postgresql/data。

mkdir -p /tmp/11/data
chown -R gitlab-psql /tmp/11 /var/opt/gitlab/postgresql/data

現在我們需要初始化新的數據庫結構。

sudo -u gitlab-psql /usr/lib/postgresql/11/bin/initdb -D /tmp/11/data

查看 /var/opt/gitlab/postgresql/data/postgresql.conf 以獲取正確的埠(並使用 -p 進行設置)。

在執行真正的遷移之前,我們應該檢查數據庫的訪問和模式。如果出現錯誤,可以多次執行此命令。

sudo -u gitlab-psql /usr/lib/postgresql/11/bin/pg_upgrade -D /tmp/11/data -d /var/opt/gitlab/postgresql/data/ -v -b /usr/lib/postgresql/10/bin -B /usr/lib/postgresql/11/bin/ -p 5432 --check

如果在檢查過程中出現錯誤,您可能需要更新一些其他目錄權限。

如果檢查成功,請使用以下命令進行數據庫遷移。這可能需要一些時間,具體取決於您的 gitlab 實例的大小。

sudo -u gitlab-psql /usr/lib/postgresql/11/bin/pg_upgrade -D /tmp/11/data -d /var/opt/gitlab/postgresql/data/ -v -b /usr/lib/postgresql/ 10/bin -B /usr/lib/postgresql/11/bin/ -p 5432

停止所有 gitlab 服務,備份舊數據庫並將遷移的數據移動​​到原始路徑。

gitlab-ctl stop
rm -f /var/opt/gitlab/postgresql/.s.PGSQL.5432*
mv /var/opt/gitlab/postgresql/data /var/opt/gitlab/postgresql/data_old_10
mv /tmp/11/data /var/opt/gitlab/postgresql/data

現在執行預設的入口腳本來完成內置的 gitlab 遷移。

/assets/wrapper

如果可以(載入 gitlab 元件後會出現一些日誌輸出,當然網站可以正常工作),您可以按 Ctrl+C,然後輸入 exit 離開並刪除目前容器。

如果仍然存在數據庫錯誤,…哎呀。

如果出現這樣的錯誤,

PG::ConnectionBad: could not connect to server: No such file or directory
     |                Is the server running locally and accepting
     |                connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?

確保沒有 postgresql 程序正在執行,刪除套接字文件並重試。

之後,離開容器。

引用自:https://serverfault.com/questions/1024377