Saturday, December 10, 2011

Apache BigTop SUSE build instructions

Apache SUSE build instructions. These instructions create the rpm and srpm files for the Hadoop Components in Apache Bigtop. The listing below is from the bigtop readme. I only installed the first half of the required tools, from Java JDK1.6 to libfuse-dev. I didn't install the rest as listed in the README. There are some differences between what the README says to do and what I did to get this working.

Apache BigTop openSUSE build instructions:

Building Bigtop requires the following tools:

* Java JDK 1.6
* Apache Forrest 0.8 (requires 32bit version of Java JDK 1.5)
* Apache Ant
* Apache Maven
* git
* subversion
* autoconf
* automake
* liblzo2-dev
* libz-dev
* sharutils
* libfuse-dev

On openSUSE 11.4+, in addition, one also needs to ensure the following is installed:


relaxngDatatype
docbook-utils docbook-simple
asciidoc
fuse-devel
docbook5
docbook5-xsl-stylesheets
libxml2-devel
xmlformat
xmlto
libxslt
libopenssl-devel




## Building packages

$ make [component-name]-[rpm|deb]

## Building local YUM/APT repositories

$ make [component-name]-[yum|apt]


1) Start an SUSE 64 bit AMI







2) Login, and update packages
sudo zipper update
3) Install the prereqs as listed in the README
Make sure we are above openSUSE 11.4
ip-10-118-183-252:~/bigtop-0.2.0-incubating # more /etc/SuSE-release
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 1
Which CPU do we have?
ip-10-118-183-252:~/bigtop-0.2.0-incubating # uname -a
Linux ip-10-118-183-252 2.6.32.46-0.3-ec2 #1 SMP 2011-09-29 17:49:31 +0200 x86_64 x86_64 x86_64 GNU/Linux
ip-10-118-183-252:~ # zypper install findutils findutils-locate
ip-10-118-183-252:~ # updatedb
//download apache-forrest from the apache website. Use version 0.8
ip-10-118-183-252:~ # unzip apache-forrest-0.8.zip
ip-10-118-183-252:~/bigtop-0.2.0-incubating # ./check-env.sh
Found all necessary env variables.
ip-10-118-183-252:~/bigtop-0.2.0-incubating # zypper install ant

4) Warning: See below for correct error message. This is kept for documentation.
Install ANT, note after the install it installs another Java version java-1_5_0-gcj-compat-devel. We have to fix the symlinks after we are finished with the prereqs.
ip-10-118-183-252:~/bigtop-0.2.0-incubating # ant -version
/usr/bin/build-classpath: error: Could not find xml-commons-apis Java extension for this JVM
/usr/bin/build-classpath: error: Some specified jars were not found
Apache Ant version 1.7.0 compiled on February 25 2009
ip-10-118-183-252:~/bigtop-0.2.0-incubating # java -version
java version "1.5.0"
gij (GNU libgcj) version 4.3.4 [gcc-4_3-branch revision 152973]

Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Zypper installs ANT under /usr/share/ant and /usr/bin.

The above instructions don’t work for ANT, install the latest version for ANT 1.8.2, install a symlink under /usr/bin/ant to point to ant 1.8.2 and set environment ANT_HOME to the installation directory. The error message cant find xml-commons-apis do not mean the xml-commons-apis are not installed on the system.
5) Install GIT,
ip-10-118-183-252:~ # zypper git install
ip-10-118-183-252:~ # git --version
git version 1.6.0.2
6) Install subversion
ip-10-118-183-252:~ # zypper install subversion subversion-devel
ip-10-118-183-252:~ # svn --version
svn, version 1.5.7 (r36142)
compiled Jun 7 2011, 12:23:36

Copyright (C) 2000-2008 CollabNet.
Subversion is open source software, see http://subversion.tigris.org/
This product includes software developed by CollabNet (http://www.Collab.Net/).

The following repository access (RA) modules are available:

* ra_neon : Module for accessing a repository via WebDAV protocol using Neon.
- handles 'http' scheme
- handles 'https' scheme
* ra_svn : Module for accessing a repository using the svn network protocol.
- with Cyrus SASL authentication
- handles 'svn' scheme
* ra_local : Module for accessing a repository on local disk.
- handles 'file' scheme

7) Install autoconf and automake
ip-10-118-183-252:~ # zypper install autoconf autoconf213 automake

7) install libfuse and sharutils(already installed), we will get to libz-dev and liblzo2-dev later
ip-10-118-183-252:~ # zypper install fuse fuse-devel libfuse2

8) Install lzo
ip-10-118-183-252:~ # zypper search lzo
Refreshing service 'susecloud'.
Loading repository data...
Reading installed packages...

S | Name | Summary | Type
--+-----------------+--------------------------------------+-----------
i | liblzo2-2 | A Real-Time Data Compression Library | package
| liblzo2-2-32bit | A Real-Time Data Compression Library | package
| lzo | A Real-Time Data Compression Library | srcpackage
| lzo-devel | Development files for lzo | package
| lzo-devel-32bit | Development files for lzo | package
ip-10-118-183-252:~ # zypper install liblzo2-2 lzo lzo-devel lzo-devel-32bit liblo2-2-32bit
No update candidate for 'liblzo2-2'.
'liblzo2-2' providing 'lzo' is already installed.
'liblo2-2-32bit' not found.
Resolving package dependencies...

The following NEW packages are going to be installed:
liblzo2-2-32bit lzo-devel lzo-devel-32bit

The following packages are not supported by their vendor:
lzo-devel lzo-devel-32bit

3 new packages to install.
Overall download size: 96.0 KiB. After the operation, additional 336.0 KiB will be used.
Continue? [y/n/?] (y): y
Retrieving package liblzo2-2-32bit-2.03-3.18.x86_64 (1/3), 49.0 KiB (134.0 KiB unpacked)
Retrieving: liblzo2-2-32bit-2.03-3.18.x86_64.rpm [done]
Installing: liblzo2-2-32bit-2.03-3.18 [done]
Retrieving package lzo-devel-2.03-3.18.x86_64 (2/3), 43.0 KiB (202.0 KiB unpacked)
Retrieving: lzo-devel-2.03-3.18.x86_64.rpm [done]
Installing: lzo-devel-2.03-3.18 [done]
Retrieving package lzo-devel-32bit-2.03-3.18.x86_64 (3/3), 4.0 KiB (0 B unpacked)
Retrieving: lzo-devel-32bit-2.03-3.18.x86_64.rpm [done]
Installing: lzo-devel-32bit-2.03-3.18 [done]

9) install libz
ip-10-118-183-252:~ # zypper search zlib
Refreshing service 'susecloud'.
Loading repository data...
Reading installed packages...

S | Name | Summary | Type
--+------------------+-------------------------------------------------------+-----------
| php5-zlib | PHP5 Extension Module | package
| zlib | Data Compression Library | srcpackage
i | zlib | Data Compression Library | package
i | zlib-32bit | Data Compression Library | package
i | zlib-devel | Include Files and Libraries mandatory for Development | package
| zlib-devel-32bit | Include Files and Libraries mandatory for Development | package
ip-10-118-183-252:~ # zypper install zlib zlib-devel zlib-devel-32bit zlib-32bit
Refreshing service 'susecloud'.
Loading repository data...
Reading installed packages...
'zlib' is already installed.
No update candidate for 'zlib'.
'zlib-devel' is already installed.
No update candidate for 'zlib-devel'.
'zlib-32bit' is already installed.
No update candidate for 'zlib-32bit'.
Resolving package dependencies...

The following NEW package is going to be installed:
zlib-devel-32bit

The following package is not supported by its vendor:
zlib-devel-32bit

1 new package to install.
Overall download size: 84.0 KiB. After the operation, additional 213.0 KiB will be used.
Continue? [y/n/?] (y): y
Retrieving package zlib-devel-32bit-1.2.3-106.34.x86_64 (1/1), 84.0 KiB (213.0 KiB unpacked)
Retrieving: zlib-devel-32bit-1.2.3-106.34.x86_64.rpm [done]
Installing: zlib-devel-32bit-1.2.3-106.34 [done]

10) Install other components, from an error message when you run make rpm
ip-10-118-183-252:~/bigtop-0.2.0-incubating # zypper install ant-trax ant-nodeps gcc-c++

11) Change permissions on forrest
chmod 777 /usr/local/apache-forrest-0.8/bin/forrest
ip-10-118-183-252:~/bigtop-0.2.0-incubating # chmod 777 /usr/local/apache-forrest-0.8/tools/ant/bin/ant
12) install xml-commons for flume
zypper install xml-commons xml-commons-apis xml-commons-apis-bootstratp xml-commons-resolver xml-commons-respolver-bootstrap xml-commons-which xml-commons-which-bootstrap
13) run Build!!!!!
ip-10-118-183-252:~/bigtop-0.2.0-incubating # make rpm

this is a screenshot of what happens when the build stops:



>updatedb
> locate rpm


We get a bunch of files which end in .rpm which don’t show up under the Ubuntu distro.

No comments:

Post a Comment