|
Logilab.org - enNews from Logilab and our Free Software projects, as well as on topics dear to our hearts (Python, Debian, Linux, the semantic web, scientific computing...)
|
I have been wondering for some time why debsign would not use the DEBSIGN_KEYID environment variable that I exported from my bashrc. Debian bug 444641 explains the trick: debsign ignores environment variables and sources ~/.devscripts instead. A simple export DEBSIGN_KEYID=ABCDEFG in ~/.devscripts is enough to get rid of the -k argument once and for good.
First of all, I've to say that pylint bugs day wasn't that successful in term of 'community event': I've been sprinting almost alone. My Logilab's felows were tied to customer projects, and no outside people shown up on jabber. Fortunatly Tarek Ziade came to visit us, and that was a nice opportunity to talk about pylint, distribute, etc ... Thank you Tarek, you saved my day ;)
As I felt a bit alone, I decided to work on somethings funnier than bug fixing: refactoring!
First, I've greatly simplified the command line: enable-msg/enable-msg-cat/enable-checker/enable-report and their disable-* counterparts were all merged into single --enable/--disable options.
I've also simplified "pylint --help" output, providing a --long-help option to get what we had before. Generic support in `logilab.common.configuration of course.
And last but not least, I refactored pylint so we can have multiple checkers with the same name. The idea behind this is that we can split checker into smaller chunks, basically
only responsible for one or a few related messages. When pylint runs, it only uses necessary checkers according to activated messages and reports. When all checkers will be splitted, it should improve performance of "pylint --error-only".
So, I can say I'm finally happy with the results of that pylint bugs day! And hopefuly we will be more people for the next edition...
virtualenv, pip and Distribute are tree tools that help developers
and packagers. In this short presentation we will see some virtualenv
capabilities.
Please, keep in mind that all above stuff has been made using :
Debian Lenny, python 2.5 and virtualenv 1.4.5.
virtualenv builds python sandboxes where it is possible to do
whatever you want as a simple user without putting in jeopardy your global
environment.
virtualenv allows you to safety:
- install any python packages
- add debug lines everywhere (not only in your scripts)
- switch between python versions
- try your code as you are a final user
- and so on ...
Prefered way
Just download the virtualenv python script at http://bitbucket.org/ianb/virtualenv/raw/tip/virtualenv.py and call it using python (e.g. python virtualenv.py).
For conveinience, we will refers to this script using virtualenv.
Other ways
For Debian (ubuntu as well) addicts, just do :
$ sudo aptitude install python-virtualenv
Fedora users would do:
$ sudo yum install python-virtualenv
And others can install from PyPI (as superuser):
$ pip install virtualenv
or
$ easy_install pip && pip install virtualenv
You could also get the source here.
To work in a python sandbox, do as follow:
$ virtualenv my_py_env
$ source my_py_env/bin/activate
(my_py_env)$ python
"That's all Folks !"
Once you have finished just do:
(my_py_env)$ deactivate
or quit the tty.
Let's start again ... more slowly. Consider the following environment:
$ pwd
/home/you/some/where
$ ls
Now create a sandbox called my-sandbox:
$ virtualenv my-sandbox
New python executable in "my-sandbox/bin/python"
Installing setuptools............done.
The output said that you have a new python executable and specific
install tools. Your current directory now looks like:
$ ls -Cl
my-sandbox/ README
$ tree -L 3 my-sandbox
my-sandbox/
|-- bin
| |-- activate
| |-- activate_this.py
| |-- easy_install
| |-- easy_install-2.5
| |-- pip
| `-- python
|-- include
| `-- python2.5 -> /usr/include/python2.5
`-- lib
`-- python2.5
|-- ...
|-- orig-prefix.txt
|-- os.py -> /usr/lib/python2.5/os.py
|-- re.py -> /usr/lib/python2.5/re.py
|-- ...
|-- site-packages
| |-- easy-install.pth
| |-- pip-0.6.3-py2.5.egg
| |-- setuptools-0.6c11-py2.5.egg
| `-- setuptools.pth
|-- ...
In addition to the new python executable and the install tools you
have an whole new python environment containing libraries, a
site-packages/ (where your packages will be installed), a bin
directory, ...
- Note:
- virtualenv does not create every file needed to get a whole new python
environment. It uses links to global environment files instead in
order to save disk space end speed up the sandbox creation.
Therefore, there must already have an active python environment
installed on your system.
At this point you have to activate the sandbox in order to use your custom python.
Once activated, python still has access to the global environment but will look at your sandbox first for python's modules:
$ source my-sandbox/bin/activate
(my-sandbox)$ which python
/home/you/some/where/my-sandbox/bin/python
$ echo $PATH
/home/you/some/where/my-sandbox/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
(pyver)$ python -c 'import sys;print sys.prefix;'
/home/you/some/where/my-sandbox
(pyver)$ python -c 'import sys;print "\n".join(sys.path)'
/home/you/some/where/my-sandbox/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg
[...]
/home/you/some/where/my-sandbox
/home/you/personal/PYTHONPATH
/home/you/some/where/my-sandbox/lib/python2.5/
[...]
/usr/lib/python2.5
[...]
/home/you/some/where/my-sandbox/lib/python2.5/site-packages
[...]
/usr/local/lib/python2.5/site-packages
/usr/lib/python2.5/site-packages
[...]
First of all, a (my-sandbox) message is automatically added to your
prompt in order to make it clear that you're using a
python sandbox environment.
Secondly, my-sandbox/bin/ is added to your PATH. So, running
python calls the specific python executable placed in
my-sandbox/bin.
- Note
- It is possible to improve the sandbox isolation by ignoring the
global paths and your PYTHONPATH (see Improve isolation
section).
It is possible to install any packages in the sandbox without any
superuser privilege. For instance, we will install the
pylint development revision
in the sandbox.
Suppose that you have the pylint stable version already installed in
your global environment:
(my-sandbox)$ deactivate
$ python -c 'from pylint.__pkginfo__ import version;print version'
0.18.0
Once your sandbox activated, install the development revision of
pylint as an update:
$ source /home/you/some/where/my-sandbox/bin/activate
(my-sandbox)$ pip install -U hg+http://www.logilab.org/hg/pylint#egg=pylint-0.19
The new package and its dependencies are only installed in the sandbox:
(my-sandbox)$ python -c 'import pylint.__pkginfo__ as p;print p.version, p.__file__'
0.19.0 /home/you/some/where/my-sandbox/lib/python2.6/site-packages/pylint/__pkginfo__.pyc
(my-sandbox)$ deactivate
$ python -c 'import pylint.__pkginfo__ as p;print p.version, p.__file__'
0.18.0 /usr/lib/pymodules/python2.6/pylint/__pkginfo__.pyc
You can safely do any change in the new pylint code or in others
sandboxed packages because your global environment is still
unchanged.
As said before, your sandboxed python sys.path still references the global
system path. You can however hide them by:
- either use the --no-site-packages that do not give
access to the global site-packages directory to the sandbox
- or change your PYTHONPATH in my-sandbox/bin/activate
in the same way as for PATH (see tips)
$ virtualenv --no-site-packages closedPy
$ sed -i '9i PYTHONPATH="$_OLD_PYTHON_PATH"
9i export PYTHONPATH
9i unset _OLD_PYTHON_PATH
40i _OLD_PYTHON_PATH="$PYTHONPATH"
40i PYTHONPATH="."
40i export PYTHONPATH' closedPy/bin/activate
$ source closedPy/bin/activate
(closedPy)$ python -c 'import sys; print "\n".join(sys.path)'
/home/you/some/where/closedPy/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg
/home/you/some/where/closedPy
/home/you/some/where/closedPy/lib/python2.5
/home/you/some/where/closedPy/lib/python2.5/plat-linux2
/home/you/some/where/closedPy/lib/python2.5/lib-tk
/home/you/some/where/closedPy/lib/python2.5/lib-dynload
/usr/lib/python2.5
/usr/lib64/python2.5
/usr/lib/python2.5/lib-tk
/home/you/some/where/closedPy/lib/python2.5/site-packages
$ deactivate
This way, you'll get an even more isolated sandbox, just
as with a brand new python environment.
It is possible to dedicate a sandbox to a
particular version of python by using the --python=PYTHON_EXE
which specifies the interpreter that virtualenv was installed with
(default is /usr/bin/python):
$ virtualenv --python=python2.4 pyver24
$ source pyver24/bin/activate
(pyver24)$ python -V
Python 2.4.6
$ deactivate
$ virtualenv --python=python2.5 pyver25
$ source pyver25/bin/activate
(pyver25)$ python -V
Python 2.5.2
$ deactivate
To distribute your sandbox, you must use the --relocatable option
that makes an existing sandbox relocatable.
This fixes up scripts and makes all .pth files relative
This option should be called just before you distribute the sandbox (each
time you have changed something in your sandbox).
An important point is that the host system should be similar to your own.
Add these scripts to your .bashrc in order to help you using
virtualenv and automate the creation and activation processes.
rel2abs() {
#from http://unix.derkeiler.com/Newsgroups/comp.unix.programmer/2005-01/0206.html
[ "$#" -eq 1 ] || return 1
ls -Ld -- "$1" > /dev/null || return
dir=$(dirname -- "$1" && echo .) || return
dir=$(cd -P -- "${dir%??}" && pwd -P && echo .) || return
dir=${dir%??}
file=$(basename -- "$1" && echo .) || return
file=${file%??}
case $dir in
/) printf '%s\n' "/$file";;
/*) printf '%s\n' "$dir/$file";;
*) return 1;;
esac
return 0
}
function activate(){
if [[ "$1" == "--help" ]]; then
echo -e "usage: activate PATH\n"
echo -e "Activate the sandbox where PATH points inside of.\n"
return
fi
if [[ "$1" == '' ]]; then
local target=$(pwd)
else
local target=$(rel2abs "$1")
fi
until [[ "$target" == '/' ]]; do
if test -e "$target/bin/activate"; then
source "$target/bin/activate"
echo "$target sandbox activated"
return
fi
target=$(dirname "$target")
done
echo 'no sandbox found'
}
function mksandbox(){
if [[ "$1" == "--help" ]]; then
echo -e "usage: mksandbox NAME\n"
echo -e "Create and activate a highly isaolated sandbox named NAME.\n"
return
fi
local name='sandbox'
if [[ "$1" != "" ]]; then
name="$1"
fi
if [[ -e "$1/bin/activate" ]]; then
echo "$1 is already a sandbox"
return
fi
virtualenv --no-site-packages --clear --distribute "$name"
sed -i '9i PYTHONPATH="$_OLD_PYTHON_PATH"
9i export PYTHONPATH
9i unset _OLD_PYTHON_PATH
40i _OLD_PYTHON_PATH="$PYTHONPATH"
40i PYTHONPATH="."
40i export PYTHONPATH' "$name/bin/activate"
activate "$name"
}
- Note:
- The virtualenv-commands and virtualenvwrapper projects add some very
interesting features to virtualenv. So, put on eye on them for more advanced features than the above ones.
I found it to be irreplaceable for testing new configurations or
working on projects with different dependencies. Moreover, I use it to
learn about other python projects, how my project exactly interacts
with its dependencies (during debugging) or to test the final user
experience.
All of this stuff can be done without virtualenv but not in such an easy and secure way.
I will continue the series by introducing other useful projects
to enhance your productivity : pip and Distribute. See you
soon.
We are happy to announce the Astng 0.20.0 and Pylint 0.20.0 releases.
Pylint is a static code checker based on Astng, both depending on logilab-common 0.49.
Astng 0.20.0 is a major refactoring:
instead of parsing and modifying the syntax tree generated from python's _ast or compiler.ast modules,
the syntax tree is rebuilt. Thus the code becomes much clearer, and
all monkey patching will eventually disappear from this module.
Speed improvement is achieved by caching the parsed modules earlier to avoid double parsing,
and avoiding some repeated inferences, all along fixing a lot of important bugs.
Pylint 0.20.0 uses the new Astng, and fixes a lot of bugs too, adding some
new functionality:
- parameters with leading "_" shouldn't count as "local" variables
- warn on assert( a, b )
- warning if return or break inside a finally
- specific message for NotImplemented exception
We would like to thank Chmouel Boudjnah, Johnson Fletcher, Daniel Harding, Jonathan Hartley, Colin Moris, Winfried Plapper, Edward K. Ream and Pierre Rouleau for their contributions, and all other people helping the project to progress.
Hey guys,
we'll hold the next pylint bugs day on april 16th 2010 (friday). If some of you want to come and work with us in our Paris office, you'll be much welcome.
Else you can still join us on jabber / irc:
See you then!
I recently had to (remotely) debug an issue on windows involving
PostgreSQL and PL/Python. Basically, two very similar computers, with
Python2.5 installed via python(x,y), PostgreSQL 8.3.8 installed via
the binary installer. On the first machine create language
plpythonu; worked like a charm, and on the other one, it failed with
C:\\Program Files\\Postgresql\\8.3\\plpython.dll: specified module
could not be found. This is caused by the dynamic linker not finding
some DLL. Using Depends.exe showed that
plpython.dll looks for python25.dll (the one it was built
against in the 8.3.8 installer), but that the DLL was there.
I'll save the various things we tried and jump directly to the
solution. After much head scratching, it turned out that the first
computer had TortoiseHg installed. This caused C:\\Program
Files\\TortoiseHg to be included in the System PATH environment
variable, and that directory contains python25.dll. On the other
hand C:\\Python25 was in the user's PATH environment variable on both
computers. As the database Windows service runs using a dedicated
local account (typically with login postgres), it would not have
C:\\Python25 in its PATH, but if TortoiseHg was there, it would
find the DLL in some other directory. So the solution was to add
C:\\Python25 to the system PATH.
Logilab is proud to announce that the blog entries published on the blogs of http://www.logilab.org and http://www.cubicweb.org are now licensed under a Creative Commons Attribution-Share Alike 2.0 License (check out the footer).
We often use creative commons licensed photographs to illustrate this blog, and felt that being developers of open source software it was quite logical that some of our content should be published under a similar license. Some of the documentation that we release also uses this license, for example the "Building Salome" documentation. This license footer has been integrated to the cubicweb-blog package that is used to publish our sites (as part of cubicweb-forge).
As part of an ongoing customer project, I've been learning about the Condor queue management system (actually it is more than just a batch queue management system, tacking the High-throughput computing problem, but in my current project, we're not using the full possibilities of Condor, and the choice was dictated by other considerations outside the scope of this note). The documentation is excellent, and the features of the product are really amazing (pity the project runs on Windows, and we cannot use 90% of these...).
To launch a job on a computer participating in the Condor farm, you just have to write a job file which looks like this:
Universe=vanilla
Executable=$path_to_executabe
Arguments=$arguments_to_the_executable
InitialDir=$working_directory
Log=$local_logfile_name
Output=$local_file_for_job_stdout
Error=$local_file_for_job_stderr
Queue
and then run condor_submit my_job_file and use condor_q to monitor the status your job (queued, running...)
My program is generating Condor job files and submitting them, and I've spent hours yesterday trying to understand why they were all failing : the stderr file contained a message from Python complaining that it could not import site and exiting.
A point which was not clear in the documentation I read (but I probably overlooked it) is that the executable mentionned in the job file is supposed to be a local file on the submission host which is copied to the computer running the job. In the jobs generated by my code, I was using sys.executable for the Executable field, and a path to the python script I wanted to run in the Arguments field. This resulted in the Python interpreter being copied on the execution host and not being able to run because it was not able to find the standard files it needs at startup.
Once I figured this out, the fix was easy: I made my program write a batch script which launched the Python script and changed the job to run that script.
UPDATE : I'm told there is a Transfer_executable=False line I could have put in the script to achieve the same thing.
(photo by gudi&cris licenced under CC-BY-ND)
This work is a part of the build identification task found in the PEP 385, Migrating from svn to Mercurial:
http://www.python.org/dev/peps/pep-0385/
It was done during the Mercurial sprint hosted at Logilab. If you would like to see the result, just follow
the steps:
hg clone http://hg.xavamedia.nl/cpython/pymigr/
cd pymigr/build-identification
The current Python development branch is first checkout:
svn co http://svn.python.org/projects/python/trunk
A patch will be applied for adding the 'sys.mercurial' attribute
and modifying the build informations:
cp add-hg-build-id.diff trunk/
cd trunk
svn up -r 78019
patch -p0 < add-hg-build-id.diff
The changed made to 'configure.in' need then to be propagated
to the configure script:
autoconf
The configuration is then done by:
./configure --enable-shared --prefix=/dev/null
You should now see changes propagated to the Makefile for finding
the revision, the tag and the branch:
grep MERCURIAL Makefile
Finally, Python can be built:
make
The sys.mercurial attribute should already be present:
LD_LIBRARY_PATH=. ./python
>>> import sys
>>> sys.mercurial
('CPython', '', '')
No tag nor revision have been found as there was no mercurial repository. A test
by the Py_GetBuildInfo() in the C API will also be built:
gcc -o show-build-info -I. -IInclude -L. -lpython2.7 ../show-build-info.c
You can test its result by:
LD_LIBRARY_PATH=. ./show-build-info
-> default, Feb 7 2010, 15:07:46
First a fake mercurial tree is built:
hg init
hg add README
hg ci -m "Initial repo"
hg id
-> 84a6de74e48f tip
Now Python needs to be built with the given mercurial information:
rm Modules/getbuildinfo.o
make
You should then see the current revision number:
LD_LIBRARY_PATH=. ./python
>>> import sys
>>> sys.mercurial
('CPython', 'default', '84a6de74e48f')
and the C API can be tested by:
LD_LIBRARY_PATH=. ./show-build-info
-> default:84a6de74e48f, Feb 7 2010, 15:10:13
The fake mercurial repository can now be cleaned:
rm -rf .hg
Automatic tests for checking the behavior for every cases will now work
build Python and clean afterward. Those tests work only when run from the
trunk svn directory of Python:
python ../test_build_identification.py
The current work is only an attempt to add the mercurial build identification to Python, it
still needs to be checked on production cases. Moreover the build identification on Windows has
not been started yet, it will need to be integrated to the Microsoft Visual Studio building process.
I regularly come across code such as:
output = os.popen('diff -u %s %s' % (appl_file, ref_file), 'r')
Code like this might well work machine but it is buggy and will fail (preferably during the demo or once shipped).
Where is the bug?
It is in the use of %s, which can inject in your command any string you want and also strings you don't want. The problem is that you probably did not check appl_file and ref_file for weird things (spaces, quotes, semi colons...). Putting quotes around the %s in the string will not solve the issue.
So what should you do? The answer is "use the subprocess module": subprocess.Popen takes a list of arguments as first parameter, which are passed as-is to the new process creation system call of your platform, and not interpreted by the shell:
pipe = subprocess.Popen(['diff', '-u', appl_file, ref_file], stdout=subprocess.PIPE)
output = pipe.stdout
By now, you should have guessed that the shell=True parameter of subprocess.Popen should not be used unless you really really need it (and even them, I encourage you to question that need).
|