Monday, October 13, 2008

Extending Debian with customizable packages

In my previous post I drafted a rough idea how to add a feature to Debian that would bridge the gap between Debian as a Binary Distribution and any Source Distribution. The feature in question is giving users the chance to build customized Debian packages for specific feature sets. The possible use cases are simple:

Lets say Anthony User needs super-duper-tool with postgresql support. However this is a not so common use case and so the archive only has postgresql support. If he now wants to build a custom package, the process is "as simple" as:

apt-get source super-duper-tool
cd super-duper-tool-*
apt-get build-dep super-duper-tool OR sudo mk-build-deps -r -i
(possibly try out which ./configure options are supported by super-duper-tool)
vi debian/rules
edit debian/rules
dpkg-buildpackage -us -uc
sudo debi

That is, to be honest lot of work. And the biggest problem about it is that you need to know all this, which is not really user knowledge.

The basic idea is that Anthony can do something like
export DEB_BUILD_OPTIONS=nomysql,postgresql
deb-build-tool build super-duper-tool

and is done.
So the question is how could this be achieved?

  1. Define a set of common options (options should be consistent through all source packages, otherwise it makes not that much sense) that every package should support, if it is a supported feature by the package in question. This includes flags like mysql, postgresql, ldap and its no-equivalents (nomysql, nopostgresql) to negate it.
  2. Enable use of this common options in the package build process and that is really the hardest part. Defining a lot of ifneq..findstring constructs for 2-10 options per source package in any available source package is.... not.... a good approach. Luckily a lot of packages uses autoconf where enabling and disabling features is as easy as adding a --with-foo or --enable-foo option to the ./configure parameters. So we could write a wrapper that would handle these options. The debhelper scripts already have dh_auto_configure: Maybe this could get enhanced.
  3. If it does not exist: Write a tool to auto-build packages with DEB_BUILD_OPTIONS set by the user. Should automatically get the source, (optional: setup/update a pbuilder environment) and build it and optional install/upload it (to a user repository for example) .
Still, the idea is only a rough draft. There are still a lot of open questions/issues, for example:
  • How to handle these feature flags in packages that don't use autoconf
  • How to enable these feature flags for packages that don't use debhelpers
  • Shall a central place for setting DEB_BUILD_OPTIONS exist (like it does for Gentoo, FreeBSD etc.) and if so: Where?
  • How should packages define new build options, if those defined Debian-wide, aren't enough?
  • What about support? Do we support such custom builds of packages or will this be a support-on-good-will thing
Is anybody interested in pushing this idea forward? I'm happily interested in it and I know people who like the idea, too. But I can't push this forward on my own all alone. Certainly the work won't affect Lenny anymore, but considerable the time frame that is planned for Lenny+1 by the release team, this could be a feature for Lenny+1.

6 comments:

Simon said...

already in progress

however, we're not reinventing Gentoo, and different feature sets will continue to have different package names.

everything else would be madness.

Anonymous said...

First things first: I like this idea very much because sometimes it would be really handy to have an own set of configuration options, while most of the time there is no need for it.

So my answers/ideas to your questions would be:
1. Common options: as functionality can only be integrated if a package exists, which offers the requested stuff, my idea would be, that the base name of said package becomes the common option name. This means, that we would need sometimes an alias (I think most users would know pidgin for an example but have no idea what libpurple is).
2. Using/Respecting such flags must be handled in an common/unified way, otherwise, we will see a lot of breakage (I guess). Therefore I would recommend to create a debhelper (or enhance an existing one), which handles such flags.
3. Packages, that don't use debhelper: for maintainence sake I would recommend a policy change, that makes it mandatory to use debhelper, if there is something, that the user can change (if there are only options, which aren't useable in Debian due to missing dependencies, then one might still build a package without debhelper).
4. I would like to have such an central place for options. And because only root can install packages, a systemwide config would suffice.
5. Package-specific options: first the »base-name-rule« should be applied and if this is not an option, packages should offer an unified interface to display all available flags with an brief explanation, what this changes if (not) set.
6. Support should be provided, if the user has only used official flags from the source package. If the user has further modified the build, then there is no support. This means, that the maintainer has to check, if everything stays buildable.

Especially the last point calls for a way of defining such flags in which a maintainer may restrict certain flags in certain combinations or on certain platforms. Much like the Depends in the kernel.

Greetings,
Drizzt

Patrick Schoenfeld said...

simon,

what exact progress are you talking about? I haven't heard about it. Yes, we have noopt, nostrip and nodoc is hitting us, but nothing else or did I miss that?

And: Hu? Who talked about reinventing Gentoo? Gentoo is still NOT comparable to Debian. Why should it be? Just because we give some more choice to our users? Still, we are a binary distribution. Still, we have an installer to install our system. Still installing Debian does not take days, because its not neccessary to build your whole system. All it is about is making something easier, that a lot of our users already _do_.

If "different feature sets will continue to have different package names" was never discussed, so I don't understand why you point that out.
But the point is interesting anyway. Package names could get mad with such a system, if a package name needs to reflect the feature sets. Lets say we have a package foo, which usually includes ldap and mysql support. Now a user would compile it with noldap,nomysql and postgresql. What would the package name be?

Helmut said...

I think it should go even further: For common options there should be multiple flavored builds (example: exim4-daemon-*, vim-*), only that these flavors will automatically be selected based on an options file. This should also greatly reduce installation size. Also libc6-i686 could go in such a flavor (and all these number crunching applications could join in).

Israel Miranda said...

Man, I read you previous blog post about it, and I feel the same need too. I have some old computers(pentium4, athlonxp) and I'd like to rebuild my system from source to gain a little speed, but also to change some build options(like thunderbird that no longer comes with movemail UNIX mailbox support).
But what I really feel the need is a graphical tool for that. I agree that debhelper use has to be mandatory and maybe apt-get should be a bit tweaked to pass this options forward and allow all of these new options to be accessed via apt-get.

Some times I use apt-build to rebuild the world, but it doesn't fit the purpose of changing build options.

If you wanna talk about it reach me at
izzy_miranda at yahoo dot com
I am not a DD or a experienced developer, but I am also an IT administrator for a development company, maybe I can help you out.

jhr said...

rource rpms have some similar posibility using the --with $option parametr for rpmbuild. this could be nice inspiration. as i remember, this could be used for packages like mplayer to have some option enabled only on alternative repositories, but have the package maintained as a single source.