• warning: Illegal string offset 'files' in /data/web/1/000/027/003/273448/htdocs/panticz.de/modules/upload/upload.module on line 281.
  • warning: Illegal string offset 'files' in /data/web/1/000/027/003/273448/htdocs/panticz.de/modules/upload/upload.module on line 281.
  • warning: Illegal string offset 'files' in /data/web/1/000/027/003/273448/htdocs/panticz.de/modules/upload/upload.module on line 281.
  • warning: Illegal string offset 'files' in /data/web/1/000/027/003/273448/htdocs/panticz.de/modules/upload/upload.module on line 281.
  • warning: Illegal string offset 'files' in /data/web/1/000/027/003/273448/htdocs/panticz.de/modules/upload/upload.module on line 281.
  • warning: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead in /data/web/1/000/027/003/273448/htdocs/panticz.de/includes/unicode.inc on line 349.
  • warning: Illegal string offset 'files' in /data/web/1/000/027/003/273448/htdocs/panticz.de/modules/upload/upload.module on line 281.
  • warning: Illegal string offset 'files' in /data/web/1/000/027/003/273448/htdocs/panticz.de/modules/upload/upload.module on line 281.
  • warning: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead in /data/web/1/000/027/003/273448/htdocs/panticz.de/includes/unicode.inc on line 349.
  • warning: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead in /data/web/1/000/027/003/273448/htdocs/panticz.de/includes/unicode.inc on line 349.
  • warning: Illegal string offset 'files' in /data/web/1/000/027/003/273448/htdocs/panticz.de/modules/upload/upload.module on line 281.
  • warning: Illegal string offset 'files' in /data/web/1/000/027/003/273448/htdocs/panticz.de/modules/upload/upload.module on line 281.
  • warning: Illegal string offset 'files' in /data/web/1/000/027/003/273448/htdocs/panticz.de/modules/upload/upload.module on line 281.
  • warning: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead in /data/web/1/000/027/003/273448/htdocs/panticz.de/includes/unicode.inc on line 349.
  • warning: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead in /data/web/1/000/027/003/273448/htdocs/panticz.de/includes/unicode.inc on line 349.
  • warning: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead in /data/web/1/000/027/003/273448/htdocs/panticz.de/includes/unicode.inc on line 349.
  • warning: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead in /data/web/1/000/027/003/273448/htdocs/panticz.de/includes/unicode.inc on line 349.
  • warning: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead in /data/web/1/000/027/003/273448/htdocs/panticz.de/includes/unicode.inc on line 349.
  • warning: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead in /data/web/1/000/027/003/273448/htdocs/panticz.de/includes/unicode.inc on line 349.
  • warning: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead in /data/web/1/000/027/003/273448/htdocs/panticz.de/includes/unicode.inc on line 349.

GitLab: Backup to S3

Configure Git
# /etc/gitlab/gitlab.rb
gitlab_rails['backup_upload_connection'] = {
'provider' => 'AWS',
'region' => 'ew-west-1',
'aws_access_key_id' => 'KEY123',
'aws_secret_access_key' => 'PASS124',
'endpoint' => 'https://s3.example.com'
}
gitlab_rails['backup_upload_remote_directory'] = 'backups'

gitlab-ctl reconfigure

Test
gitlab-rake gitlab:backup:create

Configure periodic backup
# crontab -e
0 2 * * * /opt/gitlab/bin/gitlab-rake gitlab:backup:create CRON=1

Links

GitLab: LFS on S3

Configure GitLab
...
gitlab_rails['lfs_enabled'] = true
gitlab_rails['lfs_object_store_enabled'] = true
gitlab_rails['lfs_object_store_proxy_download'] = true
gitlab_rails['lfs_object_store_remote_directory'] = "lfs"
gitlab_rails['lfs_object_store_connection'] = {
'provider' => 'AWS',
'aws_access_key_id' => 'KEY123',
'aws_secret_access_key' => 'PASS1234',
'endpoint' => 'https://s3.example.com',
}

gitlab-ctl reconfigure

Install (on client)
apt-get install -y git git-lfs

# create LFS testfile

GitLab: Artifacts on S3

Create bucket
s3cmd mb s3://artifacts

Configure GitLab
/etc/gitlab/gitlab.rb
...
nginx['client_max_body_size'] = '1024m'
gitlab_rails['artifacts_enabled'] = true
gitlab_rails['artifacts_object_store_enabled'] = true
gitlab_rails['artifacts_object_store_remote_directory'] = "artifacts"
gitlab_rails['artifacts_object_store_connection'] = {
'provider' => 'AWS',
'region' => 'us-west-1',
'aws_access_key_id' => 'KEY1234',
'aws_secret_access_key' => 'PASS1234
'endpoint' => 'https://s3.example.com'
}

gitlab-ctl reconfigure

S3: mount as filesystem under Linux

Install
sudo apt install -y s3fs

Configuration
# cat ~/.passwd-s3fs
AWS Access Key ID:AWS Secret Access Key

Mount
s3fs backup /media/backup -o url=https://s3.example.com,allow_other,umask=0000
# -o passwd_file=/etc/passwd-s3fs
# -o use_cache=/tmp/cache

/etc/fstab
mybucket1.mydomain.org /mnt/mybucket1 fuse.s3fs _netdev,allow_other,passwd_file=/home/ftpuser/.passwd-aws-s3fs,default_acl=public-read,uid=1001,gid=65534 0 0

Links

php-fpm under Nginx

Install
sudo apt install -y php-fpm nginx #mariadb-server php-mysql php-gd php-curl

Configure php-fpm
# /etc/php/7.2/fpm/pool.d/www.conf
[www]
user = www-data
group = www-data
listen = /run/php/php7.2-fpm.sock
;listen = 127.0.0.1:9000
listen.owner = www-data
listen.group = www-data
pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3

Configure Nginx
# /etc/nginx/sites-enabled/default
server {
...
# pass PHP scripts to FastCGI server
location ~ \.php$ {

Vault

Install
wget -P /tmp/
apt install -y unzip
unzip -d /usr/sbin /tmp/vault_0.11.5_linux_amd64.zip
vault -autocomplete-install

Linux

Duplicity with S3

Install
sudo apt install -y duplicity python-boto

SSH
duplicity incr --log-file /tmp/duplicity.log /etc/ scp://foo@10.0.1.123//media/etc

S3
vi ~/.boto
[Credentials]
aws_access_key_id = KEY_ID
aws_secret_access_key = SECRET_ACCESS_KEY

# create backup to s3
duplicity full --log-file /tmp/duplicity.log /etc/ s3://s3.example.com/backup/host.examp.eocm/etc/

duplicity list-current-files s3+http://bucket/folder

WebDav
export FTP_PASSWORD="pass1234"

GitLab registry with S3 storage

Configure CEPH
touch /tmp/placeholder
s3cmd mb s3://gitlab
s3cmd put /tmp/placeholder s3://gitlab
s3cmd ls s3://gitlab
s3cmd du s3://gitlab

Configure GitLab
# /etc/gitlab/gitlab.rb
...
registry_external_url 'https://registry.example.com'
registry_nginx['enable'] = true
registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/registry.example.com.crt"
registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/registry.example.com.key"
#registry['log_level'] = "debug"
#registry['storage_delete_enabled'] = true
...
registry['storage'] = {

S3 stoage under Linux / Ubuntu

Install CLI client
apt install -y s3cmd

Configuration file
s3cmd --configure
${HOME}/.s3cfg

Commands
s3cmd mb s3://buk1
s3cmd put /tmp/1m.img s3://buk1
s3cmd ls s3://buk1
s3cmd du
s3cmd rm s3://buk1/1m.img
s3cmd rb s3://buk1
s3cmd rb s3://buk1 --recursive

s3cmd rm s3://lfs/ --recursive --force

# acl
s3cmd setacl s3://bucket/path/to/file --acl-public
s3cmd info s3://bucket/path/to/file
s3cmd setacl s3://bucket/path/to/file --acl-private

Ansible

Syndicate content