OneClickInstall for Yum

Spent the whole day today learning Python ElementTree. And the product, a yum plugin for parsing OpenSuSE's One Click Install metadata, though, its still quite incomplete.

I've always wanted something like OCI, and I've been using RPM MetaPackages for most of my previous use. However, MetaPackages have some limitations, one of which it might conflict with existing RPMs, and if I use a lot of metapackages, I'll end up with a rpm database filled with them and each of them must have a unique name. That is good enough for sysadmins, but it can be a hassle for the so-called 'users' in the long term.

OpenSuSE's implementation looks more suitable for me, so, after waiting for quite some time for someone to implement it on fedora (and nobody has - or I didn't notice it), I decided to jump and write my own. However, the implementation on OpenSuSE is only for YaST (which I guess it beats the purpose of having the XML format multi-distro), so, I need to write my own parser.

I have experience writing yum plugins for adding features into yum for specific purposes, so what I needed is to figure out how to extract data from the XML file and install them. Gladly, ElementTree serve that purpose very well. Python Rocks!!.

For those interested to try it out, the code is here - http://kagesenshi-private.googlecode.com/svn/trunk/yum-oneclickinstall/. It works ok through command line, though for GUI, I guess it'll need some work.

To use, first, install it as you would to install any yum plugin (I won't help here, coz I don't want any 'users' to use it at this moment). Get a OneClickInstall XML file, theres an example in the code folder or write your own (OpenSuSE specification is good) and run:

# to install packages
yum oci-install oneclickinstallxmlfile

# to query metadata for packages
yum oci-query packages oneclickinstallxmlfile

# to query metadata for details
yum oci-query details oneclickinstallxmlfile

# to query metadata for repositories
yum oci-query repositories oneclickinstallxmlfile

(Of course, thats nowhere near being One-click)

Installing packages kindof works, but I'm still working on the ability to install 3rd party repository (the code is there, just need to think of some way to make it safe). Adding 3rd party repo feature is controversially not safe, as mixing repositories might make user end up with dependencies and compatibility problems. So, need to think of a good way to handle that.

Anyway, happy trying it. :D

Exporting Plone3.0 Memberdata and Passwords

I need to export plone accounts from Inigo Intranet to LDAP. To accomplish that, first I need some way to export the data I need. Plone itself does not have such tool for it (that I know of).

Not so long ago, Kaeru pointed me to a zope script for such purpose, however, the script fails to extract passwords from Plone3.0 mainly due to passwords are now managed by the PluggableAuthService - which made getPassword() to return None and _getPassword() to raise NotImplementedError. I don't know whether somebody forgot to implement those functions into PAS or it was purposely done. Googling lead me to this page in plone.org and from there, to this other script. Again, none of them able to extract the password hashes. I lost hope with google, and to the source I went.

After a whole night digging through the plone source, at last, I managed to found the method to extract the hash. So, heres the External Script which I wrote to extract stuff I want.

# Memberdata export script for Plone 3.0
#
# based on:
#    http://transcyberia.info/archives/23-howto-sync-mailman-from-plone.html
#    http://www.zopelabs.com/cookbook/1140753093
#    http://plone.org/documentation/how-to/export-member-data-to-csv
#
# desc:
#    None of the scripts above can extract password hashes on Plone3.0, 
#    BUT THIS ONE CAN!!!
#
#    Execute this as normal External Script, and DON'T make it public accessible 
#    (unless you don't mind people having your hashes). You have been warned. 
#    Have fun (^,^)
#


from StringIO import StringIO
import csv
import time

def getMembersCSV(self):

    request = self.REQUEST
    text = StringIO()
    writer = csv.writer(text)

    # core properties (username/password)
    core_properties = ['member_id','password']

    # extra portal_memberdata properties
    extra_properties = ['fullname',
                        'email',
                        'location',
                        'home_page',
                        'description']

    properties = core_properties + extra_properties

    writer.writerow(properties)

    membership=self.portal_membership
    passwdlist=self.acl_users.source_users._user_passwords

    for memberId in membership.listMemberIds():
        row = []
        for property in properties:
            if property == 'member_id':
               row.append(memberId)
            elif property == 'password':
               row.append(passwdlist[memberId])
            else:
               member = membership.getMemberById(memberId)
               row.append(member.getProperty(property))

        writer.writerow(row)


    request.RESPONSE.setHeader('Content-Type','application/csv')
    request.RESPONSE.setHeader('Content-Length',len(text.getvalue()))
    request.RESPONSE.setHeader('Content-Disposition',
                               'inline;filename=members-%s.csv' %
                               time.strftime("%Y%m%d-%H%M%S",time.localtime()))

    return text.getvalue()

Have fun (^.^)

12 YUM Tips and Tricks

YUM Update Manager is the default package manager used in Fedora and Fedora derivatives. It is written in Python and have tonnes of features which average Joe might not know about. These additional features helps a lot on maintaining your system, improving your experience using the package manager and become more productive. Granted, yum depsolving and querying is quite slow compared with APT, but the flexibility and power yum grant to its users, its worth it.

To my comrade, Surface, here, as you requested.

Tips 1: Fastest Mirror

This is usually the very first yum plugin I install in a new Fedora deployment. What does it do?, well, the name tell it all. The plugin will time all the servers in the replied mirrorlist from mirrors.fedoraproject.org, and pick the one which is the fastest for you. Everything is done automagically, and considering its a YumCore class of plugin, it works with whatever GUI you use for yum.

To install this plugin, use

yum install yum-fastestmirror

Unlike APT which you had to pick the fastest mirror yourself, yum handles this more elegantly

The plugin, after it has checked which server is the fastest, it will store a metadata in /var/cache/yum/timedhosts.txt . That metadata will expire after some time. However, if somehow, the cached timedhosts are no longer fast, you can easily force yum to recheck for fastestmirror by deleting that file.

Tips 2. Security Only Update

This tips helps a lot for users who can't keep up with Fedora updates. Fedora is well known for its very fast updating "updates" repository. Things get old quite fast in Fedora. Some people however, are afraid of updating their system to the latest updates packages, mainly because sometimes, these updates, suddenly break stuff as Fedora updates are not backport of fixes but rather moving forward to a newer version. One way to get around this overly sized and numbered updates is by only updating the system with security updates. Theres a plugin for yum for this purpose, and you can install it through

yum install yum-security

To use this plugin to update your computer to the latest security fixes, just use

yum --security update

I have a detailed post about this plugin here

Tips 3: Presto

If you are on a budget internet connection with very limited bandwidth, yum-presto is for you. The presto plugin will download deltarpms, which are the changes of the current version installed in your computer and the one on the internet. This will drastically reduce your download size and you can get your updates faster.

Work is being done to integrate Presto to Fedora's base repositories, however, the developers are still facing problem trying to integrate Presto with Koji and Bodhi, the buildsystem and the updates pushing system of Fedora. Therefore you will need to configure Presto after installation by following the details at http://fedorahosted.org/presto/

To install this plugin, use:

yum install yum-presto

Tips 4: Do NOT Mix Repositories!

Mixing repositories .. is BAD. There are a lot of repositories out there for you to choose from if you google enough. However, these repositories doesn't necessarily compatible with Fedora base repositories or between each other. I would only recommend users to install only Livna and small repositories which provide a small number of packages like Adobe Linux flash repository or Google's Apps repositories. Livna is the best because of their packages follow closely with Fedora's Packaging Guideline, so, you can expect that their quality of packages are as good as the real thing.

Good news that Livna, Freshrpms and Dribble are merging to form RPMFusion. Until RPMFusion is launched, just stick with Livna.

Tips 5: Broken dependencies? Skip them

Once you start installing 3rd party repositories, broken dependencies is a risk you will need to take on. This usually happens when the base repository and the 3rd party repository's packages are not in sync with each other. There is a yum plugin to help you automatically skip these broken dependencies together with packages which are affected by it. Its called yum-skipbroken. In rawhide F9, skip-broken have been merged with the core yum. I am not sure about F8.

To use it, just use the --skip-broken option parameter on the command line.

yum --skip-broken update

If the option is not available for your Fedora installation, try installing the plugin through

yum install yum-skipbroken

Tips 6: Using proprietary drivers? lock your Xorg and Kernel version!

This plugin is very useful for users who are using proprietary drivers or kernel modules and would like certain packages to remain as it is to avoid breaking the drivers. I found this useful when I installed Fedora in a friend's computer and he/she needed to use a proprietary driver. I would lock his/her kernel and xorg from upgrading (yes, somehow this approach feels ugly, but it helps preventing them away from updating things which might break the driver)

To install, use:

yum install yum-versionlock

Documentation on how to use it is available in /usr/share/doc/yum-versionlock-%{version}/README

Tips 7: Keep the cache for future use

Yum by default does not keep it cached downloads. You can change this behaviour through editing the yum config and change keepcache=1.

The yum cache directory, /var/cache/yum can be useful if you are going to play around a lot to the point that you might need to reinstall you Fedora installation frequently. It helps saving some time to redownload all those packages all over again. Just copy the rpm packages in that directory out, and you can reuse it later. You can also create a repository out of it using createrepo.

Tips 8: For organizations, create local mirrors!

It easy to create local mirrors for YUM repositories. There are several way of doing it - Rsync, or reposync (provided by yum-utils package). In normal situation I would recommend using rsync, but for poor me who in a college that blocks rsync, I uses reposync. Reposync downloads through http, but a bit harder to use to setup a usable repository.

A documentation on how to start contributing as a mirror is available here: http://fedoraproject.org/wiki/Infrastructure/Mirroring.

After you've get your mirror synced, get yourself a Fedora Account, and register yourself with Fedora's MirrorManager. The beauty here is, from the MirrorManager, you can set, for example, any requests for mirrorlist from your organization's public IP, will be replied with the URL to the mirror in your LAN IP. Your users won't know that they are using a LAN mirror! It JustWorks!. This is something APT won't give you :P. You can also set your mirrors to be available to certain range of IP/countries or not.

Tips 9: Detect problems early. Use package-cleanup

Sometimes, you might get your installation interrupted or some crazy happening which your installed package database goes crazy with duplicate packages or broken dependencies. There is one tool which helps a lot in tracing the problem and its called package-cleanup. This tool is provided by the yum-utils package, so to acquire it,

yum install yum-utils

Several common commands I usually use:

package-cleanup --dupes # list out duplicates
package-cleanup --cleandupes # clean up duplicates
package-cleanup --problems # list out packages with broken dependencies

More options? .. invoke the --help parameter

Tips 10: Download packages for usage later, together with dependencies

Sometime, you might want to download packages with its dependencies but without installing it and you want to have it easily copyable to another computer with exactly the same installed package list with your computer. yumdownloader (provided by yum-utils package) tool is great for this. You can use it to, for example, build reusable packs for a bunch of identical computers, and distribute them. You can also use it for grabbing the source rpm of a package, just in case you want to hack some stuff from the package.

My usual use of yumdownloader:


# download source package
yumdownloader --source packagename

# download a package, together with
# dependencies needed on my computer
yumdownloader --resolve packagename

# download a package, together with
# dependencies needed on a different
# chroot/mounted Fedora install
yumdownloader --resolve packagename --root /path/to/chroot


Theres another trick of using the yumdownloader with chroot, by copying /var/lib/rpm folder, you can make use of the rpm profile to download packages for a friend's computer and pass it in a CD. This concept is what being used in opyum (a similar tool like APTonCD) to build what debarashi call as YumPacks. Opyum is cool, but somehow I feel creating the set manually is more comfortable for me. Telling a user to pass me the /var/lib/rpm folder is much easier and quicker rather than telling them to install opyum and guide them through how to create a profile. Though, that is my personal preference, opyum is still a great tool to use for "users" who don't know how to mess with the CLI tricks.

Tips 11: Clean caches first if somehow yum behave weirdly

Sometimes the files cached in your computer might be broken because of some reason or another (I usually getting this problem in my university, with my hackish Fedora repository mirror - so, the metadata tend to be a crazy broken metadata once in a while). Try cleaning the cache first and retry. Chances are the problem is from there. To clean whatever cached files in your yum cache, use:

yum clean all

Tips 12: When yum fails miserably for everything, fear not, APT and Smart are there to save the day

Fedora have APT too!!!. Those from debian roots who can't seem to get away from APT, just install apt-rpm. You'll get your familiar package manager again~ :D. Another available package manager for Fedora is smart.

One beauty about apt-rpm and smart on Fedora, they all uses the generic repository metadata similar to yum. That means, all yum repositories will work with apt-rpm and smart effortlessly!. Nice eh?. hehe

EOTipsList

Thats all the tips that I can recall during the writing of this post. I might have other tricks behind my head somewhere, but could not recall it at this moment. The only next time I might remember it is when I faced a problem which made me recalled it.

Enjoy with yum. Its a fun, yummy package manager once you get the hang of it. A final tips, not useful for "users", but useful for python hackers, Yum plugins are easy to write, and it can make use any python libraries. If you need a particular special-case feature for your own use, you can get more information how to write your own custom plugin here: http://wiki.linux.duke.edu/WritingYumPlugins. I myself wrote one hackish plugin for people in my university to get to my repository. My university network layout and restrictions sucks bigtime, so, I had to write something to automatically create a tunnel to my repository and let users yum through the tunnel - seamlessly. The users doesn't even need to learn how to tunnel, it JustWorks.

Have Fun~ :D

PIDA: An IDE which loves you!

I discovered PIDA while reading around Fedora mailing lists (forgot which thread), and decided to try it out. yum'ed it from the Fedora repository, and launched it.

My first impression - Coool!.. its gvim + version control + shell/terminal + customizable IDE, all in one!.

I usually do development with very basic tools - gvim and a bunch of shells/terminals. I don't like most IDE because of the over-complexity of them. I do use geany or gedit once in a while when I want to have a bit of non-vim'ish behaviour of text editor. But PIDA, its a whole different level. It looks very promising to be my next best-friend.

Not so long after, that I started using it for continuing my current project with Inigo, and quickly, it took over my daily use of gnome-terminal, and my frequent need to change directories and restart Zope. The basic version control feature it have is wonderful. Unlike Eclipse or other IDEs where it is quite hard to get flexibility on how the project is laid out because of the "smartness" of the IDE, PIDA's basic, "stupid", yet useful, version control integration JustWorks.

So far, I have fell in love with this IDE. I haven't tried its other features yet, but I'm quite sure its going to be fun. Its written in Python, so it should be easily extendable. Going to use this a bit more longer and see how my comfort with it - its already replaced my habit of using gnome-terminal at this point of time.

Some screenies:

Hacks: find_changed_rpms.py

Sometimes, we might be facing certain problems which we could not find what might have caused it. One of the annoying cause is a broken file somewhere in the filesystem due to some reason.

My practise, is that I regularly run rpm -Va to verify and list out broken or changed files. Then I would reinstall any rpm which I think have broken files. Now that I'm working, and being on rawhide, its rather tedious and time consuming to do this regularly. So I just wrote a python script for helping out with the extracting changed files list and querying which rpm the file belongs to.

Those who interested to use it, you can grab it here: find_changed_rpms.py

 Usage: ./find_changed_rpms.py [OPTIONS]

 This script will automatically find for packages which 
 one or more of its files does not match the information
 stored in the RPMdb. 

 -h : show this menu
 -v : be verbose

 This script excludes packages which only have its config files
 changed and packages which its files only does not match the
 stored mtime.

NOTE: Its not bugproof!.

If anybody out there have some cool hacks related to Fedora. How about start sharing them now and help Greg build his Fedora weekend hacks catalog.

Tee'ing Python subprocess.Popen output

A little hack for python coders out there who wanted to have a functionality similar to the unix's tee command for redirecting output to multiple places.

import sys
from subprocess import Popen,PIPE
p = Popen(['put','command','and','arguments','here'],stdout=PIPE)

while True:
   o = p.stdout.readline()
   if o == '' and p.poll() != None: break
   # the 'o' variable stores a line from the command's stdout
   # do anything u wish with the 'o' variable here
   # this loop will break once theres a blank output 
   # from stdout and the subprocess have ended

Zenoss on Fedora 8

I've wasted 1.5 day trying to get zenoss from their RPM to run on my F8 laptop using compat-python24 from livna. I've tried lots of ways like recompiling, export PYTHON and PYTHONPATH env variables etc but without much luck. I solved it just now after looking into the python files and noticed that they are all hardcoded with /usr/bin/python instead of /usr/bin/env python. There are even some that uses /opt/zenoss/bin/python. Here's my little hackfix to make zenoss starts cleanly.

Before running this, I recommend existing zenoss install to be removed, rm -rf /opt/zenoss, and reinstall it clean.


#!/bin/sh

ZENOSS_DIR="/opt/zenoss"
PYTHON="/usr/bin/python2.4"
cd $ZENOSS_DIR

find . -type f -exec sed -i "s|/usr/bin/python|${ZENOSS_DIR}/bin/python|g" '{}' ';'

cd $ZENOSS_DIR/bin
rm python
ln -s $PYTHON python


A little description about the above script. It simply change all /usr/bin/python to /opt/zenoss/bin/python and replace the symlink /opt/zenoss/bin/python to point to /usr/bin/python2.4

Now Zenoss can be started using /sbin/service zenoss start.

Happy that I have a test instance now, time to figure out how to install it on freebsd qemu. >.<

Micropolis-Activity RPM

Following the post I've made about the MicropolisCore package, I've packaged micropolis-activity.

The package was applied a patchset by Miriam Ruiz which was acquired from debian's svn:
http://svn.debian.org/wsvn/pkg-games/packages/trunk/micropolis-activity/debian/patches/. (Thanks to Rahul Sundaram for notifying me about this).

The (S)RPMS are here http://izhar.fedorapeople.org/micropolis-activity/ (again, WARNING: Hackish Package!)

The game is running but i didn't test it much. There are some segfaults when loading certain maps. The dependency list might also not be complete. Here a screenie of it running a saved city from the the source tarball.



Happy hacking. ^-^

A month with FreeBSD, Zope and Plone

For the past month, I have been working as an intern at Inigo Tech. The company focuses on Zope and Plone deployment and customization and I'm the sysadmin of the company server.

FreeBSD

The company server uses FreeBSD as the OS with several BSD Jails on top of it. BSD Jails is one of the virtualization technology on the OS level. Unlike Xen, Qemu and VirtualBox, Jails is a very lightweight virtualization where the Guest environment does not run any kernel on it. While Xen, Qemu and VirtualBox allows different OSes running as Guest, Jails can only support another BSD installations.

I found that Jails is very useful for those who only needs better process and environment separation and doesn't need another full blown OS features. Interested to know what available on the Linux world for this, I went googling and found the Linux-Vserver Project and the OpenVZ Project. However, the RPM packages provided by both of them (especially the kernel) are quite not in sync with the current Fedora repositories. So, if anybody thinking of trying them out, I would recommend installing on CentOS or a Debian derivative instead as the packages does not being updated as fast as Fedora. Btw, if anybody successfully deployed a OpenVZ guest, can you post me an easy to follow guide??. As I'm quite confused with the OpenVZ documentations (the templates etc).

But so far, I think thats the only thing I liked from the BSD world. Everything else, are kindof PITA and not elegant in my point of view. Perhaps because I have been pampered by RPM way too much. Ports, while have quite a large collection of packages, is not as elegant as RPM or DEB in package management. If only for installation and removal of packages, Ports works quite well, but if I want to do extra stuff to the installed files, everything went messy. Dependency hell, I dont know how many times i've faced that in Ports. Updating and removing packages feels quite scary when it might causes another app, hidden somewhere, to have a library problem. For the filesystem hierarchy and file placements, is another messy thing as trying to separate stuff that maintained by package manager and stuff that are done by local user is not as clear as in RPM distros. But for this, I might be biased as I come from the Red Hat Linux / Fedora Linux origins and still not that experienced in BSD. Perhaps after exploring and learning more about the BSD way of doing stuff, I might change my opinion on this.

Zope and Plone

In Inigo Tech, I was introduced for the first time to Zope and Plone. I have heard about it before, and followed the debate on Fedora mailing list about why Fedora 7 could not support Zope and Plone due to Python2.5, but I had never tried using it firsthand. What I know about during that time Zope was that its something like Tomcat but in Python, with Plone as one of the most popular app running on Zope.

After this whole month of using Zope and Plone, I grown to like it and the ideas it brought together. I have been a fan of proper separation of services and environments since I learned about virtualization. So, I feel Zope is a great platform. A virtual filesystem on an ORDB? Cool!. Full separation of the web environment and the host server? Great!. Using Instances instead of a full copy of files? Yay!. Plugins architechture, of which different instances can have different set of Plugins? Superb!. All in all, I love Zope, except for maybe, the ZMI which looks kind of cluttered, but looks good or not is a matter of who looking at it :).

As for Plone, Inigo sees it as an alternative to Microsoft SharePoint. As I had never tried out SharePoint, I couldn't comment much about their differences/similarities. But one thing I know, Plone offers more features that SharePoint in term of document management and sharing, Plone is easier to use than SharePoint, Plone can work nicely with commodity softwares while SharePoint requires you to upgrade to IE7, Vista, Office 2007 etc, and best of all, Plone is Free! and SharePoint is dictated by Microsoft (>.<) . So, that makes it Plone is better than SharePoint right? So if your company want to buy SharePoint, how about you look at Plone first before deciding ;).

======

All in all, working with Inigo is fun with all these. I'm glad I took Kaeru's offer and not other internship offers with other companies. Plus, I got to telecommute. Saved some hassle of transportation, food, etc :D

Wiki , Compiz fusion 20070716git, and package review

It has been 2 weeks+ since I last post. Here goes

Compiz Fusion 20070716git
A little update on the compiz fusion packages:

• fusion-icon is now separated into 3 packages. fusion-icon, fusion-icon-gtk,fusion-icon-qt


• I have included in 20070716git compiz package my rewrite of desktop-effects to make use of compiz's CCP plugin. Desktop-effects menu is back, but might be buggy, I'm quite noobie in coding, help greatly appreciated

New Wiki
Just created a wiki to keep certain stuff in/from this blog more organized http://wiki.kagesenshi.org/. Compiz Fusion's wiki page is here :
http://wiki.kagesenshi.org/moin.fcgi/CategoryPackages/CompizFusion

Compiz Fusion package review
I have submitted Fedora package reviews for libcompizconfig, compizconfig-python, and compiz-bcop in Fedora bugzilla.

compiz-bcop: #247405
libcompizconfig: #247406
compizconfig-python: #247408

and a RFE for compiz : #247409

I havent send the other packages yet, I would like these core packages to pass review first.

Using Glade to write GUI applications

A few useful links for people who want to learn on how to write GTK applications using Glade

• Python - http://video.google.com/videoplay?docid=5838951374743244232


• Java - http://people.redhat.com/overholt/nativeeclipse/index.html


• Ruby - http://robrohan.com/2007/05/21/linux-gui-programming-with-ruby/

The video for using Glade with Python is not very clear, so I'll explain a little bit the process below.

1. Build your GUI

2. Open a .py file, and import gtk and gtk.glade

#!/usr/bin/python
import gtk
import gtk.glade

3. define your handlers method

def hander_method(widget):
    do_something_here
    do_more

4. load the glade XML

xml = gtk.glade.XML("/path/to/glade/file")

5. connect handlers to its methods/functions

dict = { 'handler_name' : handler_method,
'handler_name2' : handler_method2
 }

xml.signal_autoconnect(dict)

6. run gtk instance

gtk.main()

I also have written a similar script to the ruby glade template generator in Video 3. It'll generate a very basic Python template for use. ( I couldn't find where to download GladeGen )

pyglade-gentemplate.py

VMXGenerator.py : Python program to generate VMware configuration

Ok, I'm bored during this holidays ..

While surfing around and playing with Xen and VMWare, I stumbled upon this http://sanbarrow.com/vmxbuilder.html and saw a DOS Shell script to generate VMX. The crazy me took over, I started converting the script to Python~ :P. I have reimplemented most of the basic functionalities of the script in Python classes. Below is the script.

vmxgenerator.py

If youre interested to improve it, Sanbarrow.com provides lots of useful docs related to VMX parameters here - http://sanbarrow.com/vmx.html