(Many thanks, Andreas Barth.)
Buildd is the distribute build system for Debian. There are two buildd network, 'official buildd' and 'experimental buildd' (plus 'security buildd'). 'official buildd' is coordinated by James Troup (elmo) and Ryan Murray (neuro). 'experimental buildd' is coordinated by Andreas Barth (aba).
I'm ARM experimental/non-free/volatile/testing-secure buildd maintainer.
If you'd like to join the buildd team, attending experimental buildd team is good starting point, in my opinion.
Although this guide is for experimental buildd, it will be applied for official buildd also.
You need:
buildd will work 24 hours, 365 days.
Because buildd has a root privilege, I recommend you to setup buildd system in chroot.
proc /chroot/buildd/proc proc defaults 0 0 /tmp /chroot/buildd/tmp none bind 0 0 devpts /chroot/buildd/dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
You'd better to setup schroot or dchroot to enter buildd chroot again easily.
OK, let's start to install buildd packages.
deb http://ftp.jp.debian.org/debian sid main deb http://farmarchive.turmzimmer.net/ etch-buildds main
buildd send logs by mail. So you have to setup local mail server on this chroot. You can use any your favorite SMTP server. I prefer postfix.
relayhost = mail.example.org mynetworks = 127.0.0.0/8 mailbox_command = procmail -a "$EXTENSION" mailbox_size_limit = 0 recipient_delimiter = + message_size_limit=50000000
Next, let's make buildd account.
Defaults env_keep+="*" ... buildd ALL=(ALL) NOPASSWD:ALL
There are many distributions to build. The build processes will be done in each chroots. So let's make the distribution chroot for what you'd like to build.
deb http://ftp.jp.debian.org/debian sid main contrib non-free deb-src http://ftp.jp.debian.org/debian sid main contrib non-free
Then setup /etc/fstab of your root environment and do "mount -a".
proc /chroot/buildd/org/buildd/chroots/sid/proc proc defaults 0 0 devpts /chroot/buildd/org/buildd/chroots/sid/dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
deb http://ftp.jp.debian.org/debian sid main contrib non-free deb-src http://ftp.jp.debian.org/debian experimental main contrib non-free
Then setup /etc/fstab of your root environment and do "mount -a".
proc /chroot/buildd/org/buildd/chroots/experimental/proc proc defaults 0 0 devpts /chroot/buildd/org/buildd/chroots/experimental/dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
deb http://ftp.jp.debian.org/debian etch main contrib non-free deb http://ftp.jp.debian.org/debian-volatile etch main contrib non-free deb http://security.debian.org/ etch/updates main contrib non-free deb-src http://volatile.debian.net/debian-volatile etch-proposed-updates/volatile main contrib non-free
Then setup /etc/fstab of your root environment and do "mount -a".
proc /chroot/buildd/org/buildd/chroots/etch-volatile/proc proc defaults 0 0 devpts /chroot/buildd/org/buildd/chroots/etch-volatile/dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
deb http://ftp.jp.debian.org/debian etch main contrib non-free deb http://secure-testing.debian.net/debian-secure-testing etch-proposed-updates/security-updates main contrib non-free deb http://security.debian.org/ etch/updates main contrib non-free deb-src http://secure-testing.debian.net/debian-secure-testing etch-proposed-updates/security-updates main contrib non-free deb-src http://farmarchive.turmzimmer.net/ etch-buildd main
Then setup /etc/fstab of your root environment and do "mount -a".
proc /chroot/buildd/org/buildd/chroots/etch-secure/proc proc defaults 0 0 devpts /chroot/buildd/org/buildd/chroots/etch-secure/dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
deb http://ftp.jp.debian.org/debian etch main contrib non-free deb http://security.debian.org/ etch/updates main contrib non-free deb http://www.backports.org/buildd/ etch-backports main contrib non-free deb-src http://www.backports.org/buildd/ etch-backports main contrib non-free
Then setup /etc/fstab of your root environment and do "mount -a".
proc /chroot/buildd/org/buildd/chroots/etch-bpo/proc proc defaults 0 0 devpts /chroot/buildd/org/buildd/chroots/etch-bpo/dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
@take_from_dists = qw(etch-farm etch-secure etch-bpo unstable experimental) # Target distributions
$sshcmd = "/usr/bin/ssh -l wanna-build wanna-build.farm.ftbfs.de"; # Wanna-build server
$uploaddir = sub { # Upload directories
my $changes = shift;
my @dists = main::get_dists_from_changes( $changes );
foreach (@dists) {
return "upload-bpo" if /bpo$/;
return "upload-debian" if /^unstable$/;
return "upload-debian" if /^experimental$/;
return "upload-volatile" if /volatile$/;
return "upload-secure" if /secure$/;
return "upload-ganneff" if /ganneff$/;
return "upload-farm" if /-farm$/;
}
return "upload-unknown";
};
%uploadtargets = ( # dupload name
'upload-debian' => 'anonymous-ftp-master',
'upload-volatile' => 'volatile',
'upload-secure' => 'secure',
'upload-ganneff' => 'ganneff',
'upload-bpo' => 'bpo',
'upload-farm' => 'farmarchive'
);
$ENV{'LC_ALL'} = 'C';
# don't remove this, Perl needs it
1;
vi /etc/sbuild.conf.local
... $stalled_pkg_timeout = 10080; # although default is 90, it's too early for ARM ...
require "/etc/buildd.conf.global"; require "/etc/buildd.conf.farm"; $wanna_build_user = "kmuto"; # personal account of wanna-build $wanna_build_dbbase = "arm/build-db"; # wanna-build DB name $admin_mail='kmuto_builddarm@example.jp'; # address to send an admin log # don't remove this, Perl needs it 1;
$mailto = 'kmuto_buildd@example.com,build@ftbfs.de'; # address to send a build log $maintainer_name = 'arm Build Daemon - Kenshi Muto <kmuto_buildd@example.com>'; # Maintainer field of built package $fakeroot = '/usr/bin/fakeroot';
Now, you're ready to build, but you don't have a permission to access wanna-build server. Send your information and ~buildd/.ssh/id_dsa.pub to the buildd coordinator.
When coordinator adds your key, you can start buildd.
$ buildd
Check ~/daemon.log.
You can watch the build progress by using buildd-vlog. If you'd like to stop current sbuild, use buildd-abort.
buildd/.forward
| buildd-mail-wrapper
buildd sends a log file to administrator's address. Administrator's reply mail will be processed by buildd-mail.
aba has nice interface for mutt MUA.
set from="kmuto@debian.org" my_hdr From: Kenshi Muto <kmuto@debian.org> send-hook '~t buildd ~s success' "set signature='' pgp_autosign=yes indent_string='' edit_headers=yes editor='~/bin/buildd-script.sh' fast_reply=yes pgp_create_traditional=yes include=yes pgp_sign_as=kmuto@debian.org" send-hook '!(~t buildd ~s success)' "set pgp_autosign=no indent_string='> ' editor=vim fast_reply=no pgp_create_traditional=no include=yes set signature=''" set editor="vim"
#!/bin/bash +e cat $1 > ~/Mail/buildd/orig egrep -v '^Fcc:' $1 | sed -e '9,/\.changes\:$/d' -e '/^\*/,$d' > ~/Mail/buildd/changes cat ~/Mail/buildd/changes > $1
$ cd build $ sbuild -d unstable -v hello_2.1.1-4
(-d defines which chroot do you use, -v is verbose)
Log will be sent.
When you build a package for someone's request,
# cd ~buildd # chroot chroot-unstable # (Fix them)
# chroot chroot-unstable debfoster (Answer 'n' to all questions, and remove them)
# cd ~buildd # buildd-update-chroots (It creates NO-DAEMON-PLEASE file. When buildd-watcher find this file, buildd will shutdown. Then, buildd-update-chroots run apt-get upgrade each chroots.)
$ cd build $ sbuild --make-binNMU="Rebuild for erase unwelcomed dependency" --binNMU=1 -d unstable -v hello_2.1.1-4
MAILTO=<admin mail address> 30 0,6,12,18 * * * /usr/bin/buildd-uploader 20 0,6,12,18 * * * * /usr/bin/buildd-watcher
On my system, buildd-mail-farm-wrapper fails during fork. I disabled fork routine:
sed -e 's/\*p = 0;/exit(0);\n *p = 0;/' buildd-mail-wrapper.c.old > buildd-mail-wrapper.c
And run buildd-mail by cron.
10 * * * * /usr/bin/buildd-mail
For example:
ssh -l wanna-build odin.fnb.maschinenbau.tu-darmstadt.de wanna-build --database=arm/build-db --dist=experimental --list=building
Following texts/scripts have some obsolete contexts. Under constructing...
http://www.nl.debian.org/devel/buildd/setting-up これ。微妙に情報が足りなかったり間違ってたり…。
さらにabaからいろいろ追加情報をもらって、動作中。
proc /chroot/buildd/proc proc defaults 0 0 /tmp /chroot/buildd/tmp none bind 0 0
deb http://byhand.turmzimmer.net/ woody-byhand mainも追加する。
buildd ALL=(ALL) NOPASSWD:ALL
$mailto = 'kmuto,build＠ftbfs.de'; # (build〜はログに残すサイト。でもどうやって参照するんだろう？) $maintainer_name = 'arm Build Daemon - Kenshi Muto <kmuto_arm@...>'; $fakeroot = '/usr/bin/fakeroot';
(mips, mipselの場合には修正版fakerootが必要)
ssh -l wanna-build odin.fnb.maschinenbau.tu-darmstadt.de wanna-build --database=arm/build-db --dist=experimental --list=building
*p = 0;
DPRINTF( "forking\n" );
の中間に、exit(0);を突っ込んで終了させてしまうことにした。buildd-mailを手動で動かすとmqueue/の中身が自動的に全部処理され、削除される。builddデータベースも更新される。
/.muttrc
set from="kmuto@debian.org" my_hdr From: Kenshi Muto <kmuto@debian.org> send-hook '~t buildd ~s success' "set signature='' pgp_autosign=yes indent_string='' edit_headers=yes editor='~/bin/buildd-script.sh' fast_reply=yes pgp_create_traditional=yes include=yes pgp_sign_as=kmuto@debian.org" send-hook '!(~t buildd ~s success)' "set pgp_autosign=no indent_string='> ' editor=vim fast_reply=no pgp_create_traditional=no include=yes set signature=''" set editor="vim"
/bin/buildd-script.sh
#!/bin/bash +e
cat $1 > ~/Mail/buildd/orig
if [ -n "$(grep Subject $1 | grep '(')" ]; then
egrep -v '^Fcc:' $1 | sed -e '9,/\.changes\:$/d' -e '/^\*/,$d' > ~/Mail/buildd/changes
else
egrep -v '^Fcc:' $1 | sed -e '10,/\.changes\:$/d' -e '/^\*/,$d' > ~/Mail/buildd/changes
fi
cat ~/Mail/buildd/changes > $1
#!/bin/sh MIRROR=http://ftp.jp.debian.org/debian apt-get install debootstrap dchroot mkdir -p /chroot/buildd debootstrap sid /chroot/buildd $MIRROR cat >> /etc/fstab <<EOT proc /chroot/buildd/proc proc defaults 0 0 /tmp /chroot/buildd/tmp none bind 0 0 EOT mount /chroot/buildd/proc mount /chroot/buildd/tmp cat > /chroot/buildd/etc/apt <<EOT deb $MIRROR sid main contrib non-free deb-src $MIRROR sid main contrib non-free EOT chroot /chroot/buildd apt-get update chroot /chroot/buildd apt-get install build-essential echo "buildd /chroot/buildd" >> /etc/dchroot.conf
#!/bin/sh ARCH=$(dpkg --print-architecture) ADMINMAIL="kmuto" FULLNAME="Kenshi Muto" SIGNMAIL="buildd-$ARCH@topstudio.co.jp" MIRROR=http://ftp.jp.debian.org/debian cat >> /etc/apt/sources.list <<EOT deb http://db.debian.org debian-admin/ EOT adduser --disabled-password --home /org/buildd buildd apt-get update apt-get install buildd sbuild sudo cvs quinn-diff debootstrap ssh cd /org/buildd install -d build mqueue -o buildd -g buildd install -d chroots -m 755 install -d logs stats -o buildd -g buildd install -d secret -o buildd -g adm -m 2770 install -d /var/debbuild/srcdep-lock -o buildd install -d /var/lib/sbuild/srcdep-lock -o buildd install -d bin -o buildd -g adm -m 775 chown buildd /var/debbuild chmod u+w /etc/sudoers cat >> /etc/sudoers <<EOT buildd ALL=(ALL) NOPASSWD:ALL EOT chmod u-w /etc/sudoers cat > ~buildd/.sbuildrc <<EOT \$mailto = '$ADMINMAIL'; \$maintainer_name = '$ARCH Build Daemon - $FULLNAME <$SIGNMAIL>'; \$fakeroot = '/usr/bin/fakeroot'; %dist_order = ( 'oldstable-security' => 0, stable => 1, 'stable-security' => 1, testing => 2, 'testing-security' => 2, unstable => 3, 'woody-backports.org' => 5, 'sarge-backports.org' => 6, experimental => 7 ); @take_from_dists = qw(experimental unstable); EOT chown buildd:buildd ~buildd/.sbuildrc cd ~buildd install -d chroots/sid/var/debbuild/srcdep-lock -o buildd install -d chroots/sid/build/buildd -m 777 debootstrap --variant=buildd sid chroots/sid $MIRROR cp /etc/passwd /etc/group chroots/sid/etc/ ln -s source-dependencies-unstable /etc/source-dependencies-sid-nonfree ln -s /org/buildd/chroots/sid/ build/chroot-unstable grep -v db.debian.org /etc/apt/sources.list > chroots/sid/etc/apt/sources.list chroot chroots/sid apt-get update chroot chroots/sid apt-get install fakeroot build-essential sudo debfoster install -d chroots/experimental/var/debbuild/srcdep-lock -o buildd install -d chroots/experimental/build/buildd -m 777 debootstrap --variant=buildd sid chroots/experimental $MIRROR cp /etc/passwd /etc/group chroots/experimental/etc/ ln -s source-dependencies-unstable-experimental /etc/source-dependencies-experimental-nonfree ln -s source-dependencies-unstable /etc/source-dependencies-experimental ln -s /org/buildd/chroots/experimental/ build/chroot-experimental grep -v db.debian.org /etc/apt/sources.list > chroots/experimental/etc/apt/sources.list echo "deb-src $MIRROR ../project/experimental main contrib non-free" >> chroots/experimental/etc/apt/sources.list chroot chroots/experimental apt-get update chroot chroots/experimental apt-get install fakeroot build-essential sudo debfoster chroot chroots/sid debfoster chroot chroots/experimental debfoster
#!/bin/sh
ARCH=$(dpkg --print-architecture)
ADMINMAIL="kmuto"
CHANGESMAIL="kmuto"
MIRROR=http://ftp.jp.debian.org/debian
HOSTNAME=$(hostname)
cd ~
cvs -d :pserver:anon@cvs.linux-m68k.org:/CVS login
cvs -d :pserver:anon@cvs.linux-m68k.org:/CVS co wanna-build
cd wanna-build
mv buildd-mail-wrapper.c buildd-mail-wrapper.c.old
sed -e 's/\*p = 0;/exit(0);\n *p = 0;/' buildd-mail-wrapper.c.old > buildd-mail-wrapper.c
make buildd-mail-wrapper
mv wanna-build wanna-build.old
sed -e "s/\$| = 1;/@distributions = qw(oldstable-security stable testing unstable stable-security testing-security experimental);\n%dist_order = ( 'oldstable-security' => 0, stable => 1, 'stable-security' => 1, testing => 2, 'testing-security' => 2, unstable => 3, experimental => 7 );\n\$| = 1;/" wanna-build.old > wanna-build
chmod 755 wanna-build
cp /usr/bin/sbuild sbuild.old
sed -e "s/@main::toolchain_pkgs = ();/%main::dist_order = ( 'oldstable-security' => 0, stable => 1, 'stable-security' => 1, testing => 2, 'testing-security' => 2, unstable => 3, experimental => 7 );\n@main::toolchain_pkgs = ();/" sbuild.old > sbuild
cp do-merge-packages do-merge-packages.old
sed -e 's/@dists = qw(stable frozen unstable);/@dists = qw(stable frozen unstable experimental);/' -e "s/my \$f = \"\$conf::pkgs_source\/dists\/\$dist\/\$sect\/\$bindir\/Packages.gz\";/my \$f = \"\$conf::pkgs_source\/dists\/\$dist\/\$sect\/\$bindir\/Packages.gz\";\n\$f = \"\$conf::pkgs_source\/project\/\$dist\/\$sect\/\$bindir\/Packages.gz\" if (\$dist eq 'experimental');/" -e "s/my \$f = \"\$conf::pkgs_source\/dists\/\$dist\/\$sect\/sources\/Sources.gz\";/my \$f = \"\$conf::pkgs_source\/dists\/\$dist\/\$sect\/source\/Sources.gz\";\n\$f = \"\$conf::pkgs_source\/project\/\$dist\/\$sect\/source\/Sources.gz\" if (\$dist eq 'experimental');/" do-merge-packages.old > do-merge-packages
cp do-merge-quinn do-merge-quinn.old
sed -e 's/@dists = qw(stable frozen unstable);/@dists = qw(stable frozen unstable experimental);/' do-merge-quinn.old > do-merge-quinn
cp wanna-build-statistics wanna-build-statistics.old
sed -e 's/qw(stable frozen unstable)/qw(stable frozen unstable experimental)/' wanna-build-statistics.old > wanna-build-statistics
sudo cp sbuild buildd-mail-wrapper wanna-build wanna-build-mail.pl wanna-build-statistics do-merge-packages do-merge-quinn do-push check-old-builds /usr/bin
sudo cp FetchFile.pm /usr/lib/perl5
cp wanna-build.conf.local wanna-build.conf.local.old
sed -e "s#^1;#\$basedir = \"\/org\/buildd\";\n\$db_maint = '$ADMINMAIL';\n\$notforus_maint = '$ADMINMAIL';\n\$log_mail = '$CHANGESMAIL';\n\$stat_mail = '$CHANGESMAIL';\n\$web_stats = \"\/org\/buildd\/stats\/stats.txt\";\n@sections = qw(main contrib non-free);\n\$pkgs_source = \"$MIRROR\";\n\$quinn_source = \"\/org\/buildd\/quinn.diff\";\n1;#" wanna-build.conf.local.old > wanna-build.conf.local
sudo cp wanna-build.conf wanna-build.conf.local /etc
cp abort-current-build abort-current-build.old
sed -e 's/\/bin\/sh dpkg-buildpackage/\/bin\/sh \/usr\/bin\/dpkg-buildpackage/' abort-current-build.old > abort-current-build
cp vlog abort-current-build finish-build ~/bin
mkdir ~/old-logs ~/lib
touch ~/build-db-stable ~/build-db-frozen ~/build-db-testing ~/build-db-unstable ~/build-db-experimental
do-merge-packages
mkdir ~/quinn.diff
mkdir -p ~/quinn.diff/$ARCH/main/
mkdir -p ~/quinn.diff/$ARCH/contrib/
mkdir -p ~/quinn.diff/$ARCH/non-free/
HOST=$(echo $MIRROR | sed -e 's/http:\/\///')
cat > ~/bin/getfiles <<EOT
#!/bin/sh
cd ~/quinn.diff
host=$HOST
for a in $ARCH; do
for v in unstable testing stable; do
for m in main contrib non-free; do
wget -m http://\$host/dists/\$v/\$m/binary-\$a/Packages.gz
wget -m http://\$host/dists/\$v/\$m/source/Sources.gz
zcat \$host/dists/\$v/\$m/binary-\$a/Packages.gz > Packages
zcat \$host/dists/\$v/\$m/source/Sources.gz > Sources
quinn-diff -p Packages -s Sources > \$a/\$m/\$v
done
done
for m in main contrib non-free; do
wget -m http://\$host/project/experimental/\$m/binary-\$a/Packages.gz
wget -m http://\$host/project/experimental/\$m/source/Sources.gz
zcat \$host/project/experimental/\$m/binary-\$a/Packages.gz > Packages
zcat \$host/project/experimental/\$m/source/Sources.gz > Sources
quinn-diff -p Packages -s Sources > \$a/\$m/experimental
done
done
EOT
chmod 755 ~/bin/getfiles
~/bin/getfiles
do-merge-quinn
cd ~
zcat /usr/share/doc/buildd/examples/buildd.conf.gz > buildd.conf.old
sed -e "s#^1;#@take_from_dists = qw(experimental unstable);\n\$admin_mail = '$ADMINMAIL';\n\$statistics_mail = '$ADMINMAIL';\n\$wanna_build_user = \"buildd_$ARCH-$HOSTNAME\";\n\$wanna_build_dbbase = \"build-db\";\n\$sshcmd = \"\";\n1;#" buildd.conf.old > buildd.conf
crontab - <<EOT
MAILTO=$ADMINMAIL
# 0 1 * * * /usr/bin/buildd-uploader
10 2 * * * /org/buildd/bin/getfiles && /usr/bin/do-push
10 * * * * /usr/bin/buildd-mail
20 * * * * /usr/bin/buildd-watcher
EOT
echo "| \"buildd-mail-wrapper\"" > ~/.forward
ssh-keygen -t dsa
cd build
exec buildd
| "dchroot -c buildd buildd-mail-wrapper"
exit(0);を追加。
$basedir="/org/buildd"; $db_maint = "kmuto"; $notforus_maint = "kmuto"; $log_mail = "kmuto"; # 本当はm68k-changes@nocrew.orgなど $stat_mail = "kmuto"; # 同上 $web_stats = "/org/buildd/stats/stats.txt"; # /var/www以下のファイルにしてwebから見られるようにしてもよい @sections = qw(main contrib non-free); $pkgs_source = "http://ftp.jp.debian.org/debian"; #$quinn_source = "http://ftp.uni-erlangen.de/pub/Linux/LOCAL/Debian-support/quinn-diff"; #http://buildd.debian.org/quinn-diff/にあるのだが古い。アカウント貰わないとだめ $quinn_source = "/org/buildd/quinn.diff" 1;
正式builddにするには、buildd.debian.orgから取ってくるようにしたほうがよいすな。
たとえばこんなスクリプトとか(quinn.diffの中に置いて実行)。
#!/bin/sh
host=ftp.jp.debian.org/debian
for a in arm; do # アーキテクチャ名を入れる
for v in unstable testing stable; do
for m in main contrib non-free; do
wget -m http://$host/dists/$v/$m/binary-$a/Packages.gz
wget -m http://$host/dists/$v/$m/source/Sources.gz
zcat $host/dists/$v/$m/binary-$a/Packages.gz > Packages
zcat $host/dists/$v/$m/source/Sources.gz > Sources
quinn-diff -p Packages -s Sources > $a/$m/$v
done
done
for m in main contrib non-free; do
wget -m http://$host/project/experimental/$m/binary-$a/Packages.gz
wget -m http://$host/project/experimental/$m/source/Sources.gz
zcat $host/project/experimental/$m/binary-$a/Packages.gz > Packages
zcat $host/project/experimental/$m/source/Sources.gz > Sources
quinn-diff -p Packages -s Sources > $a/$m/experimental
done
done