Blog entries august 2013 [4]

Emacs turned into a IDE with CEDET

2013/08/29 by Anthony Truchet

Abstract

In this post you will find one way, namely thanks to CEDET, of turning your Emacs into an IDE offering features for semantic browsing and refactoring assistance similar to what you can find in major IDE like Visual Studio or Eclipse.

Introduction

Emacs is a tool of choice for the developer: it is very powerful, highly configurable and has a wealth of so called modes to improve many aspects of daily work, especially when editing code.

The point, as you might have realised in case you have already worked with an IDE like Eclipse or Visual Studio, is that Emacs (code) browsing abilities are quite rudimentary... at least out of the box!

In this post I will walk through one way to configure Emacs + CEDET which works for me. This is by far not the only way to get to it but finding this path required several days of wandering between inconsistent resources, distribution pitfall and the like.

I will try to convey relevant parts of what I have learnt on the way, to warn about some pitfalls and also to indicate some interesting direction I haven't followed (be it by choice or necessity) and encourage you to try. Should you try to push this adventure further, your experience will be very much appreciated... and in any case your feedback on this post is also very welcome.

The first part gives some deemed useful background to understand what's going on. If you want to go straight to the how-to please jump directly to the second part.

Sketch map of the jungle

This all started because I needed a development environment to do work remotely on a big, legacy C++ code base from quite a lightweight machine and a weak network connection.

My former habit of using Eclipse CDT and compiling locally was not an option any longer but I couldn't stick to a bare text editor plus remote compilation either because of the complexity of the code base. So I googled emacs IDE code browser and started this journey to set CEDET + ECB up...

I quickly got lost in a jungle of seemingly inconsistent options and I reckon that some background facts are welcome at this point as to why.

Up to this date - sept. 2013 - most of the world is in-between two major releases of Emacs. Whereas Emacs 23.x is still packaged in many stable Linux distribution, the latest release is Emacs 24.3. In this post we will use Emacs 24.x which brings lots of improvements, two of those are really relevant to us:

  • the introduction of a package manager, which is great and (but) changes initialisation
  • the partial integration of some version of CEDET into Emacs since version 23.2

Emacs 24 initialisation

Very basically, Emacs used to read the user's Emacs config (~/.emacs or ~/.emacs.d/init.el) which was responsible for adapting the load-path and issuing the right (require 'stuff) commands and configuring each library in some appropriate sequence.

Emacs 24 introduces ELPA, a new package system and official packages repository. It can be extended by other packages repositories such as Marmalade or MELPA

By default in Emacs 24, the initialisation order is a bit more complex due to packages loading: the user's config is still read but should NOT require the libraries installed through the package system: those are automatically loaded (the former load-path adjustment and (require 'stuff) steps) after the ~/.emacs or ~/.emacs.d/init.el has finished. This makes configuring the loaded libraries much more error-prone, especially for libraries designed to be configured the old way (as of today most libraries, notably CEDET).

Here is a good analysis of the situation and possible options. And for those interested in the details of the new initialisation process, see following sections of the manual:

I first tried to stick to the new-way, setting up hooks in ~/.emacs.d/init.el to be called after loading the various libraries, each library having its own configuration hook, and praying for the interaction between the package manager load order and my hooks to be ok... in vain. So I ended up forcing the initialisation to the old way (see Emacs 24 below).

What is CEDET ?

CEDET is a Collection of Emacs Development Environment Tools. The major word here is collection, do not expect it to be an integrated environment. The main components of (or coupled with) CEDET are:

Semantic
Extract a common semantic from source code in different languages
(e)ctags / GNU global
Traditional (exhuberant) CTags or GNU global can be used as a source of information for Semantic
SemanticDB
SemanticDB provides for caching the outcome of semantic analysis in some database to reduce analysis overhead across several editing sessions
Emacs Code Browser
This component uses information provided by Semantic to offer a browsing GUI with windows for traversing files, classes, dependencies and the like
EDE
This provides a notion of project analogous to most IDE. Even if the features related to building projects are very Emacs/ Linux/ Autotools-centric (and thus not necessarily very helful depending on your project setup), the main point of EDE is providing scoping of source code for Semantic to analyse and include path customisation at the project level.
AutoComplete
This is not part of CEDET but Semantic can be configured as a source of completions for auto-complete to propose to the user.
and more...
Senator, SRecode, Cogre, Speedbar, EIEIO, EAssist are other components of CEDET I've not looked at yet.

To add some more complexity, CEDET itself is also undergoing heavy changes and is in-between major versions. The last standalone release is 1.1 but it has the old source layout and activation method. The current head of development says it is version 2.0, has new layout and activation method, plus some more features but is not released yet.

Integration of CEDET into Emacs

Since Emacs 23.2, CEDET is built into Emacs. More exactly parts of some version of new CEDET are built into Emacs, but of course this built-in version is older than the current head of new CEDET... As for the notable parts not built into Emacs, ECB is the most prominent! But it is packaged into Marmalade in a recent version following head of development closely which, mitigates the inconvenience.

My first choice was using built-in CEDET with ECB installed from the packages repository: the installation was perfectly smooth but I was not able to configure cleanly enough the whole to get proper operation. Although I tried hard, I could not get Semantic to take into account the include paths I configured using my EDE project for example.

I would strongly encourage you to try this way, as it is supposed to require much less effort to set up and less maintenance. Should you succeed I would greatly appreciate some feedback of you experience!

As for me I got down to install the latest version from the source repositories following as closely as possible Alex Ott's advices and using his own fork of ECB to make it compliant with most recent CEDET:

How to set up CEDET + ECB in Emacs 24

Emacs 24

Install Emacs 24 as you wish, I will not cover the various options here but simply summarise the local install from sources I choose.

  1. Get the source archive from http://ftpmirror.gnu.org/emacs/
  2. Extract it somewhere and run the usual (or see the INSTALL file) - configure --prefix=~/local, - make, - make install

Create your emacs personal directory and configuration file ~/.emacs.d/site-lisp/ and ~/.emacs.d/init.el and put this inside the latter:

;; this is intended for manually installed libraries
(add-to-list 'load-path "~/.emacs.d/site-lisp/")

;; load the package system and add some repositories
(require 'package)
(add-to-list 'package-archives
             '("marmalade" . "http://marmalade-repo.org/packages/"))
(add-to-list 'package-archives
             '("melpa" . "http://melpa.milkbox.net/packages/") t)

;; Install a hook running post-init.el *after* initialization took place
(add-hook 'after-init-hook (lambda () (load "post-init.el")))

;; Do here basic initialization, (require) non-ELPA packages, etc.

;; disable automatic loading of packages after init.el is done
(setq package-enable-at-startup nil)
;; and force it to happen now
(package-initialize)
;; NOW you can (require) your ELPA packages and configure them as normal

Useful Emacs packages

Using the emacs commands M-x package-list-packages interactively or M-x package-install <package name>, you can install many packages easily. For example I installed:

Choose your own! I just recommend against installing ECB or other CEDET since we are going to install those from source.

You can also insert or load your usual Emacs configuration here, simply beware of configuring ELPA, Marmalade et al. packages after (package-initialize).

CEDET

  • Get the source and put it under ~/.emacs.d/site-lisp/cedet-bzr. You can either download a snapshot from http://www.randomsample.de/cedet-snapshots/ or check it out of the bazaar repository with:

    ~/.emacs.d/site-lisp$ bzr checkout --lightweight \
    bzr://cedet.bzr.sourceforge.net/bzrroot/cedet/code/trunk cedet-bzr
    
  • Run make (and optionnaly make install-info) in cedet-bzr or see the INSTALL file for more details.

  • Get Alex Ott's minimal CEDET configuration file to ~/.emacs.d/config/cedet.el for example

  • Adapt it to your system by editing the first lines as follows

    (setq cedet-root-path
        (file-name-as-directory (expand-file-name
            "~/.emacs.d/site-lisp/cedet-bzr/")))
    (add-to-list 'Info-directory-list
            "~/projects/cedet-bzr/doc/info")
    
  • Don't forget to load it from your ~/.emacs.d/init.el:

    ;; this is intended for configuration snippets
    (add-to-list 'load-path "~/.emacs.d/")
    ...
    (load "config/cedet.el")
    
  • restart your emacs to check everything is OK; the --debug-init option is of great help for that purpose.

ECB

  • Get Alex Ott ECB fork into ~/.emacs.d/site-lisp/ecb-alexott:

    ~/.emacs.d/site-lisp$ git clone --depth 1  https://github.com/alexott/ecb/
    
  • Run make in ecb-alexott and see the README file for more details.

  • Don't forget to load it from your ~/.emacs.d/init.el:

    (add-to-list 'load-path (expand-file-name
          "~/.emacs.d/site-lisp/ecb-alexott/"))
    (require 'ecb)
    ;(require 'ecb-autoloads)
    

    Note

    You can theoretically use (require 'ecb-autoloads) instead of (require 'ecb) in order to load ECB by need. I encountered various misbehaviours trying this option and finally dropped it, but I encourage you to try it and comment on your experience.

  • restart your emacs to check everything is OK (you probably want to use the --debug-init option).

  • Create a hello.cpp with you CEDET enable Emacs and say M-x ecb-activate to check that ECB is actually installed.

Tune your configuration

Now, it is time to tune your configuration. There is no good recipe from here onward... But I'll try to propose some snippets below. Some of them are adapted from Alex Ott personal configuration

More Semantic options

You can use the following lines just before (semantic-mode 1) to add to the activated features list:

(add-to-list 'semantic-default-submodes 'global-semantic-decoration-mode)
(add-to-list 'semantic-default-submodes 'global-semantic-idle-local-symbol-highlight-mode)
(add-to-list 'semantic-default-submodes 'global-semantic-idle-scheduler-mode)
(add-to-list 'semantic-default-submodes 'global-semantic-idle-completions-mode)

You can also load additional capabilities with those lines after (semantic-mode 1):

(require 'semantic/ia)
(require 'semantic/bovine/gcc) ; or depending on you compiler
; (require 'semantic/bovine/clang)
Auto-completion

If you want to use auto-complete you can tell it to interface with Semantic by configuring it as follows (where AAAAMMDD.rrrr is the date.revision suffix of the version od auti-complete installed by you package manager):

;; Autocomplete
(require 'auto-complete-config)
(add-to-list 'ac-dictionary-directories (expand-file-name
             "~/.emacs.d/elpa/auto-complete-AAAAMMDD.rrrr/dict"))
(setq ac-comphist-file (expand-file-name
             "~/.emacs.d/ac-comphist.dat"))
(ac-config-default)

and activating it in your cedet hook, for example:

...
;; customisation of modes
(defun alexott/cedet-hook ()
...
    (add-to-list 'ac-sources 'ac-source-semantic)
) ; defun alexott/cedet-hook ()
Support for GNU global a/o (e)ctags
;; if you want to enable support for gnu global
(when (cedet-gnu-global-version-check t)
  (semanticdb-enable-gnu-global-databases 'c-mode)
  (semanticdb-enable-gnu-global-databases 'c++-mode))

;; enable ctags for some languages:
;;  Unix Shell, Perl, Pascal, Tcl, Fortran, Asm
(when (cedet-ectag-version-check)
  (semantic-load-enable-primary-exuberent-ctags-support))

Using CEDET for development

Once CEDET + ECB + EDE is up you can start using it for actual development. How to actually use it is beyond the scope of this already too long post. I can only invite you to have a look at:

Conclusion

CEDET provides an impressive set of features both to allow your emacs environment to "understand" your code and to provide powerful interfaces to this "understanding". It is probably one of the very few solution to work with complex C++ code base in case you can't or don't want to use a heavy-weight IDE like Eclipse CDT.

But its being highly configurable also means, at least for now, some lack of integration, or at least a pretty complex configuration. I hope this post will help you to do your first steps with CEDET and find your way to setup and configure it to you own taste.


Pylint 1.0 released!

2013/08/06 by Sylvain Thenault

Hi there,

I'm very pleased to announce, after 10 years of existence, the 1.0 release of Pylint.

This release has a hell long ChangeLog, thanks to many contributions and to the 10th anniversary sprint we hosted during june. More details about changes below.

Chances are high that your Pylint score will go down with this new release that includes a lot of new checks :) Also, there are a lot of improvments on the Python 3 side (notably 3.3 support which was somewhat broken).

You may download and install it from Pypi or from Logilab's debian repositories. Notice Pylint has been updated to use the new Astroid library (formerly known as logilab-astng) and that the logilab-common 0.60 library includes some fixes necessary for using Pylint with Python3 as well as long-awaited support for namespace packages.

For those interested, below is a comprehensive list of what changed:

Command line and output formating

  • A new --msg-template option to control output, deprecating "msvc" and "parseable" output formats as well as killing --include-ids and --symbols options.
  • Fix spelling of max-branchs option, now max-branches.
  • Start promoting usage of symbolic name instead of numerical ids.

New checks

  • "missing-final-newline" (C0304) for files missing the final newline.
  • "invalid-encoded-data" (W0512) for files that contain data that cannot be decoded with the specified or default encoding.
  • "bad-open-mode" (W1501) for calls to open (or file) that specify invalid open modes (Original implementation by Sasha Issayev).
  • "old-style-class" (C1001) for classes that do not have any base class.
  • "trailing-whitespace" (C0303) that warns about trailing whitespace.
  • "unpacking-in-except" (W0712) about unpacking exceptions in handlers, which is unsupported in Python 3.
  • "old-raise-syntax" (W0121) for the deprecated syntax raise Exception, args.
  • "unbalanced-tuple-unpacking" (W0632) for unbalanced unpacking in assignments (bitbucket #37).

Enhanced behaviours

  • Do not emit [fixme] for every line if the config value 'notes' is empty
  • Emit warnings about lines exceeding the column limit when those lines are inside multiline docstrings.
  • Name check enhancement:
    • simplified message,
    • don't double-check parameter names with the regex for parameters and inline variables,
    • don't check names of derived instance class members,
    • methods that are decorated as properties are now treated as attributes,
    • names in global statements are now checked against the regular expression for constants,
    • for toplevel name assignment, the class name regex will be used if pylint can detect that value on the right-hand side is a class (like collections.namedtuple()),
    • add new name type 'class_attribute' for attributes defined in class scope. By default, allow both const and variable names.
  • Add a configuration option for missing-docstring to optionally exempt short functions/methods/classes from the check.
  • Add the type of the offending node to missing-docstring and empty-docstring.
  • Do not warn about redefinitions of variables that match the dummy regex.
  • Do not treat all variables starting with "_" as dummy variables, only "_" itself.
  • Make the line-too-long warning configurable by adding a regex for lines for with the length limit should not be enforced.
  • Do not warn about a long line if a pylint disable option brings it above the length limit.
  • Do not flag names in nested with statements as undefined.
  • Remove string module from the default list of deprecated modules (bitbucket #3).
  • Fix incomplete-protocol false positive for read-only containers like tuple (bitbucket #25).

Other changes

  • Support for pkgutil.extend_path and setuptools pkg_resources (logilab-common #8796).
  • New utility classes for per-checker unittests in testutils.py
  • Added a new base class and interface for checkers that work on the tokens rather than the syntax, and only tokenize the input file once.
  • epylint shouldn't hang anymore when there is a large output on pylint'stderr (bitbucket #15).
  • Put back documentation in source distribution (bitbucket #6).

Astroid

  • New API to make it smarter by allowing transformation functions on any node, providing a register_transform function on the manager instead of the register_transformer to make it more flexible wrt node selection
  • Use this new transformation API to provide support for namedtuple (actually in pylint-brain, logilab-astng #8766)
  • Better description of hashlib
  • Properly recognize methods annotated with abc.abstract{property,method} as abstract.
  • Added the test_utils module for building ASTs and extracting deeply nested nodes for easier testing.

Astroid 1.0 released!

2013/08/02 by Sylvain Thenault

Astroid is the new name of former logilab-astng library. It's an AST library, used as the basis of Pylint and including Python 2.5 -> 3.3 compatible tree representation, statical type inference and other features useful for advanced Python code analysis, such as an API to provide extra information when statistical inference can't overcome Python dynamic nature (see the pylint-brain project for instance).

It has been renamed and hosted to bitbucket to make clear that this is not a Logilab dedicated project but a community project that could benefit to any people manipulating Python code (statistical analysis tools, IDE, browser, etc).

Documentation is a bit rough but should quickly improve. Also a dedicated web-site is now online, visit www.astroid.org (or https://bitbucket.org/logilab/astroid for development).

You may download and install it from Pypi or from Logilab's debian repositories.


Going to DebConf13

2013/08/01 by Julien Cristau

The 14th Debian developers conference (DebConf13) will take place between August 11th and August 18th in Vaumarcus, Switzerland.

Logilab is a DebConf13 sponsor, and I'll attend the conference. There are quite a lot of cloud-related events on the schedule this year, plus the usual impromptu discussions and hallway track. Looking forward to meeting the usual suspects there!

https://www.logilab.org/file/158611/raw/dc13-btn0-going-bg.png