Hacks: Replace yum's downloader with Axel

/!\ This is an ugly hack, use it at your own risk, 
I am not responsible for any breakage

P/S, your sysadmin might not gonna like this

Disclaimer aside, I really love Python.

So, I'm behind such a bad network which keep timing out, and downloading stuff using yum in this network is a real PITA. And so, I had enough of it and decided to hack around.

Axel is a CLI download accelerator thats lightweight and pretty fast. I've been using it quite frequently to get a bit more speed for my downloads. I think, why not just replace this urllib2 grabber with axel, so, I did. (I am such an evil guy)

After a few hours looking through yum code to find what to modify, I ended up with this

# file: /usr/lib/python2.5/site-packages/urlgrabber/customgrabber.py

import grabber
import os

class AxelGrabber(grabber.URLGrabber):
def urlgrab(self, url, filename=None, **kwargs):
"""grab the file at and make a local copy at
If filename is none, the basename of the url is used.
urlgrab returns the filename of the local file, which may be
different from the passed-in filename if copy_local == 0.

opts = self.opts.derive(**kwargs)
(url,parts) = opts.urlparser.parse(url, opts)
(scheme, host, path, parm, query, frag) = parts

def retryfunc(opts, url, filename):
if os.environ.has_key('http_proxy'):
os.environ['HTTP_PROXY'] = os.environ['http_proxy']
os.system('/usr/bin/axel -a -o %s %s' % (filename,url))
return filename

return self._retry(opts, retryfunc, url, filename)

Make sure you put the file above at its path unless you are sure what you are doing. To make it usable with yum, some modification is needed in one of yum's source file.


from urlgrabber.grabber import URLGrabber

from urlgrabber.customgrabber import AxelGrabber as URLGrabber

and you are good to go. Do it at YOUR own risk remember.


sweemeng said…
Can't help to think it is really cool. Could be annoying to the server.......

p.s Your blog is a bit cluttered
Michael said…
Generally you should never use os.system and the shell to invoke something when you know the full path and arguments to the binary.

Use rc = subprocess.call([args], shell=False)

Subprocess also allows for capturing the output, piping between commands, and doing other nice things.

I'm not sure of other performance or technical implications of this, but that was worth pointing out.
Ganeshkumar said…
Thank you for posting.

Very useful and handy. Good Job!
hi there,
i used this way to change the yum's default downloader to axel, but there's a problem, when i use this hack, yum can't download files from my local repos, can you tell me how can i fix it?

Popular posts from this blog

Tee'ing Python subprocess.Popen output

Consolidated community site infrastructure on Plone

Adding simple popup to Plone frontpage