(Many thanks, Andreas Barth.)
Buildd is the distribute build system for Debian. There are two buildd network, 'official buildd' and 'experimental buildd' (plus 'security buildd'). 'official buildd' is coordinated by James Troup (elmo) and Ryan Murray (neuro). 'experimental buildd' is coordinated by Andreas Barth (aba).
I'm ARM experimental/non-free/volatile/testing-secure buildd maintainer.
If you'd like to join the buildd team, attending experimental buildd team is good starting point, in my opinion.
Although this guide is for experimental buildd, it will be applied for official buildd also.
You need:
buildd will work 24 hours, 365 days.
Because buildd has a root privilege, I recommend you to setup buildd system in chroot.
proc /chroot/buildd/proc proc defaults 0 0 /tmp /chroot/buildd/tmp none bind 0 0 devpts /chroot/buildd/dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
You'd better to setup schroot or dchroot to enter buildd chroot again easily.
OK, let's start to install buildd packages.
deb http://ftp.jp.debian.org/debian sid main deb http://farmarchive.turmzimmer.net/ etch-buildds main
buildd send logs by mail. So you have to setup local mail server on this chroot. You can use any your favorite SMTP server. I prefer postfix.
relayhost = mail.example.org mynetworks = 127.0.0.0/8 mailbox_command = procmail -a "$EXTENSION" mailbox_size_limit = 0 recipient_delimiter = + message_size_limit=50000000
Next, let's make buildd account.
Defaults env_keep+="*" ... buildd ALL=(ALL) NOPASSWD:ALL
There are many distributions to build. The build processes will be done in each chroots. So let's make the distribution chroot for what you'd like to build.
deb http://ftp.jp.debian.org/debian sid main contrib non-free deb-src http://ftp.jp.debian.org/debian sid main contrib non-free
Then setup /etc/fstab of your root environment and do "mount -a".
proc /chroot/buildd/org/buildd/chroots/sid/proc proc defaults 0 0 devpts /chroot/buildd/org/buildd/chroots/sid/dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
deb http://ftp.jp.debian.org/debian sid main contrib non-free deb-src http://ftp.jp.debian.org/debian experimental main contrib non-free
Then setup /etc/fstab of your root environment and do "mount -a".
proc /chroot/buildd/org/buildd/chroots/experimental/proc proc defaults 0 0 devpts /chroot/buildd/org/buildd/chroots/experimental/dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
deb http://ftp.jp.debian.org/debian etch main contrib non-free deb http://ftp.jp.debian.org/debian-volatile etch main contrib non-free deb http://security.debian.org/ etch/updates main contrib non-free deb-src http://volatile.debian.net/debian-volatile etch-proposed-updates/volatile main contrib non-free
Then setup /etc/fstab of your root environment and do "mount -a".
proc /chroot/buildd/org/buildd/chroots/etch-volatile/proc proc defaults 0 0 devpts /chroot/buildd/org/buildd/chroots/etch-volatile/dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
deb http://ftp.jp.debian.org/debian etch main contrib non-free deb http://secure-testing.debian.net/debian-secure-testing etch-proposed-updates/security-updates main contrib non-free deb http://security.debian.org/ etch/updates main contrib non-free deb-src http://secure-testing.debian.net/debian-secure-testing etch-proposed-updates/security-updates main contrib non-free deb-src http://farmarchive.turmzimmer.net/ etch-buildd main
Then setup /etc/fstab of your root environment and do "mount -a".
proc /chroot/buildd/org/buildd/chroots/etch-secure/proc proc defaults 0 0 devpts /chroot/buildd/org/buildd/chroots/etch-secure/dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
deb http://ftp.jp.debian.org/debian etch main contrib non-free deb http://security.debian.org/ etch/updates main contrib non-free deb http://www.backports.org/buildd/ etch-backports main contrib non-free deb-src http://www.backports.org/buildd/ etch-backports main contrib non-free
Then setup /etc/fstab of your root environment and do "mount -a".
proc /chroot/buildd/org/buildd/chroots/etch-bpo/proc proc defaults 0 0 devpts /chroot/buildd/org/buildd/chroots/etch-bpo/dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
@take_from_dists = qw(etch-farm etch-secure etch-bpo unstable experimental) # Target distributions $sshcmd = "/usr/bin/ssh -l wanna-build wanna-build.farm.ftbfs.de"; # Wanna-build server $uploaddir = sub { # Upload directories my $changes = shift; my @dists = main::get_dists_from_changes( $changes ); foreach (@dists) { return "upload-bpo" if /bpo$/; return "upload-debian" if /^unstable$/; return "upload-debian" if /^experimental$/; return "upload-volatile" if /volatile$/; return "upload-secure" if /secure$/; return "upload-ganneff" if /ganneff$/; return "upload-farm" if /-farm$/; } return "upload-unknown"; }; %uploadtargets = ( # dupload name 'upload-debian' => 'anonymous-ftp-master', 'upload-volatile' => 'volatile', 'upload-secure' => 'secure', 'upload-ganneff' => 'ganneff', 'upload-bpo' => 'bpo', 'upload-farm' => 'farmarchive' ); $ENV{'LC_ALL'} = 'C'; # don't remove this, Perl needs it 1;
vi /etc/sbuild.conf.local
... $stalled_pkg_timeout = 10080; # although default is 90, it's too early for ARM ...
require "/etc/buildd.conf.global"; require "/etc/buildd.conf.farm"; $wanna_build_user = "kmuto"; # personal account of wanna-build $wanna_build_dbbase = "arm/build-db"; # wanna-build DB name $admin_mail='kmuto_builddarm@example.jp'; # address to send an admin log # don't remove this, Perl needs it 1;
$mailto = 'kmuto_buildd@example.com,build@ftbfs.de'; # address to send a build log $maintainer_name = 'arm Build Daemon - Kenshi Muto <kmuto_buildd@example.com>'; # Maintainer field of built package $fakeroot = '/usr/bin/fakeroot';
Now, you're ready to build, but you don't have a permission to access wanna-build server. Send your information and ~buildd/.ssh/id_dsa.pub to the buildd coordinator.
When coordinator adds your key, you can start buildd.
$ buildd
Check ~/daemon.log.
You can watch the build progress by using buildd-vlog. If you'd like to stop current sbuild, use buildd-abort.
buildd/.forward
| buildd-mail-wrapper
buildd sends a log file to administrator's address. Administrator's reply mail will be processed by buildd-mail.
aba has nice interface for mutt MUA.
set from="kmuto@debian.org" my_hdr From: Kenshi Muto <kmuto@debian.org> send-hook '~t buildd ~s success' "set signature='' pgp_autosign=yes indent_string='' edit_headers=yes editor='~/bin/buildd-script.sh' fast_reply=yes pgp_create_traditional=yes include=yes pgp_sign_as=kmuto@debian.org" send-hook '!(~t buildd ~s success)' "set pgp_autosign=no indent_string='> ' editor=vim fast_reply=no pgp_create_traditional=no include=yes set signature=''" set editor="vim"
#!/bin/bash +e cat $1 > ~/Mail/buildd/orig egrep -v '^Fcc:' $1 | sed -e '9,/\.changes\:$/d' -e '/^\*/,$d' > ~/Mail/buildd/changes cat ~/Mail/buildd/changes > $1
$ cd build $ sbuild -d unstable -v hello_2.1.1-4
(-d defines which chroot do you use, -v is verbose)
Log will be sent.
When you build a package for someone's request,
# cd ~buildd # chroot chroot-unstable # (Fix them)
# chroot chroot-unstable debfoster (Answer 'n' to all questions, and remove them)
# cd ~buildd # buildd-update-chroots (It creates NO-DAEMON-PLEASE file. When buildd-watcher find this file, buildd will shutdown. Then, buildd-update-chroots run apt-get upgrade each chroots.)
$ cd build $ sbuild --make-binNMU="Rebuild for erase unwelcomed dependency" --binNMU=1 -d unstable -v hello_2.1.1-4
MAILTO=<admin mail address> 30 0,6,12,18 * * * /usr/bin/buildd-uploader 20 0,6,12,18 * * * * /usr/bin/buildd-watcher
On my system, buildd-mail-farm-wrapper fails during fork. I disabled fork routine:
sed -e 's/\*p = 0;/exit(0);\n *p = 0;/' buildd-mail-wrapper.c.old > buildd-mail-wrapper.c
And run buildd-mail by cron.
10 * * * * /usr/bin/buildd-mail
For example:
ssh -l wanna-build odin.fnb.maschinenbau.tu-darmstadt.de wanna-build --database=arm/build-db --dist=experimental --list=building
Following texts/scripts have some obsolete contexts. Under constructing...
http://www.nl.debian.org/devel/buildd/setting-up これ。微妙に情報が足りなかったり間違ってたり…。
さらにabaからいろいろ追加情報をもらって、動作中。
proc /chroot/buildd/proc proc defaults 0 0 /tmp /chroot/buildd/tmp none bind 0 0
deb http://byhand.turmzimmer.net/ woody-byhand mainも追加する。
buildd ALL=(ALL) NOPASSWD:ALL
$mailto = 'kmuto,build@ftbfs.de'; # (build〜はログに残すサイト。でもどうやって参照するんだろう?) $maintainer_name = 'arm Build Daemon - Kenshi Muto <kmuto_arm@...>'; $fakeroot = '/usr/bin/fakeroot';
(mips, mipselの場合には修正版fakerootが必要)
ssh -l wanna-build odin.fnb.maschinenbau.tu-darmstadt.de wanna-build --database=arm/build-db --dist=experimental --list=building
*p = 0; DPRINTF( "forking\n" );の中間に、exit(0);を突っ込んで終了させてしまうことにした。buildd-mailを手動で動かすとmqueue/の中身が自動的に全部処理され、削除される。builddデータベースも更新される。
/.muttrc
set from="kmuto@debian.org" my_hdr From: Kenshi Muto <kmuto@debian.org> send-hook '~t buildd ~s success' "set signature='' pgp_autosign=yes indent_string='' edit_headers=yes editor='~/bin/buildd-script.sh' fast_reply=yes pgp_create_traditional=yes include=yes pgp_sign_as=kmuto@debian.org" send-hook '!(~t buildd ~s success)' "set pgp_autosign=no indent_string='> ' editor=vim fast_reply=no pgp_create_traditional=no include=yes set signature=''" set editor="vim"
/bin/buildd-script.sh
#!/bin/bash +e cat $1 > ~/Mail/buildd/orig if [ -n "$(grep Subject $1 | grep '(')" ]; then egrep -v '^Fcc:' $1 | sed -e '9,/\.changes\:$/d' -e '/^\*/,$d' > ~/Mail/buildd/changes else egrep -v '^Fcc:' $1 | sed -e '10,/\.changes\:$/d' -e '/^\*/,$d' > ~/Mail/buildd/changes fi cat ~/Mail/buildd/changes > $1
#!/bin/sh MIRROR=http://ftp.jp.debian.org/debian apt-get install debootstrap dchroot mkdir -p /chroot/buildd debootstrap sid /chroot/buildd $MIRROR cat >> /etc/fstab <<EOT proc /chroot/buildd/proc proc defaults 0 0 /tmp /chroot/buildd/tmp none bind 0 0 EOT mount /chroot/buildd/proc mount /chroot/buildd/tmp cat > /chroot/buildd/etc/apt <<EOT deb $MIRROR sid main contrib non-free deb-src $MIRROR sid main contrib non-free EOT chroot /chroot/buildd apt-get update chroot /chroot/buildd apt-get install build-essential echo "buildd /chroot/buildd" >> /etc/dchroot.conf
#!/bin/sh ARCH=$(dpkg --print-architecture) ADMINMAIL="kmuto" FULLNAME="Kenshi Muto" SIGNMAIL="buildd-$ARCH@topstudio.co.jp" MIRROR=http://ftp.jp.debian.org/debian cat >> /etc/apt/sources.list <<EOT deb http://db.debian.org debian-admin/ EOT adduser --disabled-password --home /org/buildd buildd apt-get update apt-get install buildd sbuild sudo cvs quinn-diff debootstrap ssh cd /org/buildd install -d build mqueue -o buildd -g buildd install -d chroots -m 755 install -d logs stats -o buildd -g buildd install -d secret -o buildd -g adm -m 2770 install -d /var/debbuild/srcdep-lock -o buildd install -d /var/lib/sbuild/srcdep-lock -o buildd install -d bin -o buildd -g adm -m 775 chown buildd /var/debbuild chmod u+w /etc/sudoers cat >> /etc/sudoers <<EOT buildd ALL=(ALL) NOPASSWD:ALL EOT chmod u-w /etc/sudoers cat > ~buildd/.sbuildrc <<EOT \$mailto = '$ADMINMAIL'; \$maintainer_name = '$ARCH Build Daemon - $FULLNAME <$SIGNMAIL>'; \$fakeroot = '/usr/bin/fakeroot'; %dist_order = ( 'oldstable-security' => 0, stable => 1, 'stable-security' => 1, testing => 2, 'testing-security' => 2, unstable => 3, 'woody-backports.org' => 5, 'sarge-backports.org' => 6, experimental => 7 ); @take_from_dists = qw(experimental unstable); EOT chown buildd:buildd ~buildd/.sbuildrc cd ~buildd install -d chroots/sid/var/debbuild/srcdep-lock -o buildd install -d chroots/sid/build/buildd -m 777 debootstrap --variant=buildd sid chroots/sid $MIRROR cp /etc/passwd /etc/group chroots/sid/etc/ ln -s source-dependencies-unstable /etc/source-dependencies-sid-nonfree ln -s /org/buildd/chroots/sid/ build/chroot-unstable grep -v db.debian.org /etc/apt/sources.list > chroots/sid/etc/apt/sources.list chroot chroots/sid apt-get update chroot chroots/sid apt-get install fakeroot build-essential sudo debfoster install -d chroots/experimental/var/debbuild/srcdep-lock -o buildd install -d chroots/experimental/build/buildd -m 777 debootstrap --variant=buildd sid chroots/experimental $MIRROR cp /etc/passwd /etc/group chroots/experimental/etc/ ln -s source-dependencies-unstable-experimental /etc/source-dependencies-experimental-nonfree ln -s source-dependencies-unstable /etc/source-dependencies-experimental ln -s /org/buildd/chroots/experimental/ build/chroot-experimental grep -v db.debian.org /etc/apt/sources.list > chroots/experimental/etc/apt/sources.list echo "deb-src $MIRROR ../project/experimental main contrib non-free" >> chroots/experimental/etc/apt/sources.list chroot chroots/experimental apt-get update chroot chroots/experimental apt-get install fakeroot build-essential sudo debfoster chroot chroots/sid debfoster chroot chroots/experimental debfoster
#!/bin/sh ARCH=$(dpkg --print-architecture) ADMINMAIL="kmuto" CHANGESMAIL="kmuto" MIRROR=http://ftp.jp.debian.org/debian HOSTNAME=$(hostname) cd ~ cvs -d :pserver:anon@cvs.linux-m68k.org:/CVS login cvs -d :pserver:anon@cvs.linux-m68k.org:/CVS co wanna-build cd wanna-build mv buildd-mail-wrapper.c buildd-mail-wrapper.c.old sed -e 's/\*p = 0;/exit(0);\n *p = 0;/' buildd-mail-wrapper.c.old > buildd-mail-wrapper.c make buildd-mail-wrapper mv wanna-build wanna-build.old sed -e "s/\$| = 1;/@distributions = qw(oldstable-security stable testing unstable stable-security testing-security experimental);\n%dist_order = ( 'oldstable-security' => 0, stable => 1, 'stable-security' => 1, testing => 2, 'testing-security' => 2, unstable => 3, experimental => 7 );\n\$| = 1;/" wanna-build.old > wanna-build chmod 755 wanna-build cp /usr/bin/sbuild sbuild.old sed -e "s/@main::toolchain_pkgs = ();/%main::dist_order = ( 'oldstable-security' => 0, stable => 1, 'stable-security' => 1, testing => 2, 'testing-security' => 2, unstable => 3, experimental => 7 );\n@main::toolchain_pkgs = ();/" sbuild.old > sbuild cp do-merge-packages do-merge-packages.old sed -e 's/@dists = qw(stable frozen unstable);/@dists = qw(stable frozen unstable experimental);/' -e "s/my \$f = \"\$conf::pkgs_source\/dists\/\$dist\/\$sect\/\$bindir\/Packages.gz\";/my \$f = \"\$conf::pkgs_source\/dists\/\$dist\/\$sect\/\$bindir\/Packages.gz\";\n\$f = \"\$conf::pkgs_source\/project\/\$dist\/\$sect\/\$bindir\/Packages.gz\" if (\$dist eq 'experimental');/" -e "s/my \$f = \"\$conf::pkgs_source\/dists\/\$dist\/\$sect\/sources\/Sources.gz\";/my \$f = \"\$conf::pkgs_source\/dists\/\$dist\/\$sect\/source\/Sources.gz\";\n\$f = \"\$conf::pkgs_source\/project\/\$dist\/\$sect\/source\/Sources.gz\" if (\$dist eq 'experimental');/" do-merge-packages.old > do-merge-packages cp do-merge-quinn do-merge-quinn.old sed -e 's/@dists = qw(stable frozen unstable);/@dists = qw(stable frozen unstable experimental);/' do-merge-quinn.old > do-merge-quinn cp wanna-build-statistics wanna-build-statistics.old sed -e 's/qw(stable frozen unstable)/qw(stable frozen unstable experimental)/' wanna-build-statistics.old > wanna-build-statistics sudo cp sbuild buildd-mail-wrapper wanna-build wanna-build-mail.pl wanna-build-statistics do-merge-packages do-merge-quinn do-push check-old-builds /usr/bin sudo cp FetchFile.pm /usr/lib/perl5 cp wanna-build.conf.local wanna-build.conf.local.old sed -e "s#^1;#\$basedir = \"\/org\/buildd\";\n\$db_maint = '$ADMINMAIL';\n\$notforus_maint = '$ADMINMAIL';\n\$log_mail = '$CHANGESMAIL';\n\$stat_mail = '$CHANGESMAIL';\n\$web_stats = \"\/org\/buildd\/stats\/stats.txt\";\n@sections = qw(main contrib non-free);\n\$pkgs_source = \"$MIRROR\";\n\$quinn_source = \"\/org\/buildd\/quinn.diff\";\n1;#" wanna-build.conf.local.old > wanna-build.conf.local sudo cp wanna-build.conf wanna-build.conf.local /etc cp abort-current-build abort-current-build.old sed -e 's/\/bin\/sh dpkg-buildpackage/\/bin\/sh \/usr\/bin\/dpkg-buildpackage/' abort-current-build.old > abort-current-build cp vlog abort-current-build finish-build ~/bin mkdir ~/old-logs ~/lib touch ~/build-db-stable ~/build-db-frozen ~/build-db-testing ~/build-db-unstable ~/build-db-experimental do-merge-packages mkdir ~/quinn.diff mkdir -p ~/quinn.diff/$ARCH/main/ mkdir -p ~/quinn.diff/$ARCH/contrib/ mkdir -p ~/quinn.diff/$ARCH/non-free/ HOST=$(echo $MIRROR | sed -e 's/http:\/\///') cat > ~/bin/getfiles <<EOT #!/bin/sh cd ~/quinn.diff host=$HOST for a in $ARCH; do for v in unstable testing stable; do for m in main contrib non-free; do wget -m http://\$host/dists/\$v/\$m/binary-\$a/Packages.gz wget -m http://\$host/dists/\$v/\$m/source/Sources.gz zcat \$host/dists/\$v/\$m/binary-\$a/Packages.gz > Packages zcat \$host/dists/\$v/\$m/source/Sources.gz > Sources quinn-diff -p Packages -s Sources > \$a/\$m/\$v done done for m in main contrib non-free; do wget -m http://\$host/project/experimental/\$m/binary-\$a/Packages.gz wget -m http://\$host/project/experimental/\$m/source/Sources.gz zcat \$host/project/experimental/\$m/binary-\$a/Packages.gz > Packages zcat \$host/project/experimental/\$m/source/Sources.gz > Sources quinn-diff -p Packages -s Sources > \$a/\$m/experimental done done EOT chmod 755 ~/bin/getfiles ~/bin/getfiles do-merge-quinn cd ~ zcat /usr/share/doc/buildd/examples/buildd.conf.gz > buildd.conf.old sed -e "s#^1;#@take_from_dists = qw(experimental unstable);\n\$admin_mail = '$ADMINMAIL';\n\$statistics_mail = '$ADMINMAIL';\n\$wanna_build_user = \"buildd_$ARCH-$HOSTNAME\";\n\$wanna_build_dbbase = \"build-db\";\n\$sshcmd = \"\";\n1;#" buildd.conf.old > buildd.conf crontab - <<EOT MAILTO=$ADMINMAIL # 0 1 * * * /usr/bin/buildd-uploader 10 2 * * * /org/buildd/bin/getfiles && /usr/bin/do-push 10 * * * * /usr/bin/buildd-mail 20 * * * * /usr/bin/buildd-watcher EOT echo "| \"buildd-mail-wrapper\"" > ~/.forward ssh-keygen -t dsa cd build exec buildd
| "dchroot -c buildd buildd-mail-wrapper"
exit(0);を追加。
$basedir="/org/buildd"; $db_maint = "kmuto"; $notforus_maint = "kmuto"; $log_mail = "kmuto"; # 本当はm68k-changes@nocrew.orgなど $stat_mail = "kmuto"; # 同上 $web_stats = "/org/buildd/stats/stats.txt"; # /var/www以下のファイルにしてwebから見られるようにしてもよい @sections = qw(main contrib non-free); $pkgs_source = "http://ftp.jp.debian.org/debian"; #$quinn_source = "http://ftp.uni-erlangen.de/pub/Linux/LOCAL/Debian-support/quinn-diff"; #http://buildd.debian.org/quinn-diff/にあるのだが古い。アカウント貰わないとだめ $quinn_source = "/org/buildd/quinn.diff" 1;
正式builddにするには、buildd.debian.orgから取ってくるようにしたほうがよいすな。
たとえばこんなスクリプトとか(quinn.diffの中に置いて実行)。
#!/bin/sh host=ftp.jp.debian.org/debian for a in arm; do # アーキテクチャ名を入れる for v in unstable testing stable; do for m in main contrib non-free; do wget -m http://$host/dists/$v/$m/binary-$a/Packages.gz wget -m http://$host/dists/$v/$m/source/Sources.gz zcat $host/dists/$v/$m/binary-$a/Packages.gz > Packages zcat $host/dists/$v/$m/source/Sources.gz > Sources quinn-diff -p Packages -s Sources > $a/$m/$v done done for m in main contrib non-free; do wget -m http://$host/project/experimental/$m/binary-$a/Packages.gz wget -m http://$host/project/experimental/$m/source/Sources.gz zcat $host/project/experimental/$m/binary-$a/Packages.gz > Packages zcat $host/project/experimental/$m/source/Sources.gz > Sources quinn-diff -p Packages -s Sources > $a/$m/experimental done done