Wednesday, July 22, 2009

Upgrading Flex from source RPM to compile PostgreSQL from CVS

This week I'm working on reviewing a patch that's part of the current PostgreSQL CommitFest, the periodic points where all outstanding patches are looked at and committed if ready. The patch I'm looking at requires some performance testing, and all my servers I'd do that on are running CentOS 5, the popular RedHat Enterprise clone. There's a fun surprise waiting for anyone else who tries this: as of last week, you can't build PostgreSQL from the development CVS or GIT repositories anymore on that platform without updating the Flex package from the default one.

The problem started when I ran "make" to build everything and saw this:

make[3]: Entering directory `/home/gsmith/pgproject/pgsql.tpgbench/src/backend/bootstrap'
ERROR: `flex' is missing on your system. It is needed to create the
file `bootscanner.c'. You can either get flex from a GNU mirror site
or download an official distribution of PostgreSQL, which contains
pre-packaged flex output.

I knew I had flex on my system so this was kind of confusing. Usually there's more detail about dependency failures in the config.log file, and sure enough it had the details:

configure:6793: WARNING:
*** The installed version of Flex, /usr/bin/lex, is too old to use with PostgreSQL.
*** Flex version 2.5.31 or later is required, but this is /usr/bin/lex version 2.5.4.
configure:6807: result: no
configure:6815: WARNING:
*** Without Flex you will not be able to build PostgreSQL from CVS nor
*** change any of the scanner definition files. You can obtain Flex from
*** a GNU mirror site. (If you are using the official distribution of
*** PostgreSQL then you do not need to worry about this because the Flex
*** output is pre-generated.)

A quick check shows this change was made to allow some more powerful scanning capabilities in the PostgreSQL language parser, followed by inserting that warning.

Now, while it's possible to just build quickly from source and install directly over top of the existing flex in this case, I don't like to do that on production systems (or even development ones). It's not a good idea to mix the RPM packages on the system with stuff installed that way. It's better if you can create a new flex RPM package based on newer source code and upgrade to that one, then everything stays managed consistently with RPM.

You can locate SRPMs with a newer version of flex, the one released with Fedora 9, from the following locations:

Those were the first I found with a new enough version number that they should work. You could easily try to substitute the flex SRPM that comes with Fedora 10 or Fedora 11 instead, haven't tested that here myself yet to say how that goes; probably fine.

I used the link, downloaded and installed like this:

$ wget
$ sudo rpm -i flex-2.5.35-1.fc9.src.rpm

This dumps the source into /usr/src/redhat:

$ cd /usr/src/redhat/


You can then build like this:

$ cd /usr/src/redhat/SPECS/
$ sudo rpmbuild -bb flex.spec

That produces the new RPM we want:

$ ls -l /usr/src/redhat/RPMS/x86_64/
-rw-r--r-- 1 root root 322619 Jul 22 17:14 flex-2.5.35-1.x86_64.rpm
-rw-r--r-- 1 root root 283675 Jul 22 17:14 flex-debuginfo-2.5.35-1.x86_64.rpm

Now install it:

$ cd /usr/src/redhat/RPMS/x86_64/
$ ls
flex-2.5.35-1.x86_64.rpm flex-debuginfo-2.5.35-1.x86_64.rpm
$ rpm -qa flex
$ sudo rpm -Uvh flex-2.5.35-1.x86_64.rpm
Preparing... ########################################### [100%]
1:flex ########################################### [100%]
$ rpm -qa flex

And then cleanup:

$ cd /usr/src/redhat/SPECS/
$ sudo rpmbuild --clean --rmsource flex.spec
Executing(--clean): /bin/sh -e /var/tmp/rpm-tmp.99970
+ umask 022
+ cd /usr/src/redhat/BUILD
+ rm -rf flex-2.5.35
+ exit 0

After that, I had to run "configure" again to pick up the change, ran "make again", and now my build against CVS checkout of the future PostgreSQL 8.5 in progress compiles without any problems.

Note that normally, when you substitute a package like this you need to be careful you keep up with security patches to it because you're not going to get them automatically anymore. Since flex is a pretty low-level tool used only for developing software, I'm not too concerned about the security implication of my running a custom version here.


Andrew said...

It's generally considered better practice not to build RPMs as root. There's less danger when you get the SRPM from a reasonably trusted source, but bad mistakes can still happen.

Greg Smith said...

While true in general, in this case regular users can't write to the standard /usr/src/redhat/ directory the packages you install via SRPM go into. I could have led people through copying that whole area into their own directory structure and then building there, but that seemed like an unnecessarily painful diversion.

Andrew said...

well, rpmdevtools makes it easy - all you have to do is run rpmdev-setuptree and you're off to the races. Pity that's not shipped with RHEL5/CentOS5, but you can grab that from fedora too.

Devrim Gündüz said...

@Andrew: IIRC it is shipped with EPEL repository. (It = rpmdevtools stuff)

Devrim Gündüz said...

@Greg: FWIW, even Fedora rawhide SRPM can be built on CentOS 5.3 easily -- and it works. See buildfarm member orca.

Greg Smith said...

One other problem I ran into when doing this on a really minimal CentOS install...the flex package lists this as its build requirements:

BuildRequires: gettext bison m4

But when you actually build the RPM, it will execute some tests that are written in C++, and all of these will fail if g++ isn't available to compile them. You'll see errors like

Executing test test-c++-basic
Test test-c++-basic FAILED. See test-c++-basic/OUTPUT for details.

And then at the end:

Tests succeeded: 41
Tests FAILED: 5
RPM build errors:
Bad exit status from /var/tmp/rpm-tmp.25443 (%check)

Solution is simple: before running rpmbuild, make sure that you can run g++, and if you can't install it:

yum install gcc-c++

Gurjeet said...

Thanks for this post...

Just an FYI that apart from installing gcc-c++ I had to install rpm-build package too.

Gurjeet said...

Also, since was no longer available, I tried fc13 through fc11 and all of those source RPMs failed with MD5 error.

So, I gave fc10 a try from the archives and that worked.