Installation

The procedure of installing your own instance of nodewatcher platform follows.

Note: we are assuming that you are running an UNIX-like operating system.

Warning Regarding Database Backend

nodewatcher assumes working support for transactional savepoints in the database backend. This is only supported in PostgreSQL version 8.0 or higher and therefore this is the only database that is supported by nodewatcher.

The system will still work with MySQL and SQLite but some features regarding error handling and validation may cause unexpected results and even data corruption! Do not use them for production deployment. You have been warned.

Prerequisites

The following (Debian, Ubuntu ...) packages are required:

  • python (>= 2.6, >= 3.0 not supported)
  • python-django (1.4.x)
  • python-rrdtool (with rrdtool >= 1.3.5)
  • python-lxml
  • python-yaml
  • python-memcache
  • fping
  • graphviz
  • memcached
  • python-psycopg2
  • python-django-picklefield
  • python-django-south
  • python-pyparsing
  • python-anyjson
  • python-pymongo
  • python-geoip
  • python-dnspython
  • beanstalkd

The following Python modules are required:

  • celery (2.5.3)
  • django-celery (2.5.5)
  • django-registration (newer than revision 6d2e42a13cb6, > 0.7)
  • django-phonenumber-field
  • python-aprmd5
  • pybeanstalk

The following PostgreSQL extension is required:

Install them all with a single command:

sudo aptitude install python python-django python-rrdtool \
python-lxml python-yaml python-memcache fping graphviz memcached python-psycopg2 \
python-django-picklefield python-django-south python-pyparsing python-anyjson \
python-pymongo beanstalkd python-geoip python-dnspython

Getting Source

Use default branch which contains stable nodewatcher source from the project repository. Get it using this command:

hg clone http://dev.wlan-si.net/hg/nodewatcher

Configuration

There are few preconfigured settings files already provided for you to use as a template/base:

  • settings.py – default settings file for local development
  • settings_wlansi.py – settings file for local development customized for wlan slovenija network
  • settings_production.py – settings file for production instance used in wlan slovenija network

They are all in frontend directory. settings_production.py extends settings_wlansi.py, which in turn extends settings.py. You can start testing nodewatcher immediatelly by using settings.py configuration file. But if you want to modify settings, the recommended way is to make a similar structure, extending and overriding provided settings files. In this way you will not have repository conflicts when provided settings files will get updated. And if you extend then properly, you will get good defaults automatically for new settings.

When starting django, make sure to set the DJANGO_SETTINGS_MODULE environment variable or --settings command line option so that it points to your settings file. Note that your copy of the settings file will not be updated by the source-code repository.

The settings.py settings file contains a lot of commented-out or disabled options (and features). This also means some dependencies are not required when using it. On the other hand settings_production.py uses and enables everything.

Runing Development/Testing Instance

Once you have all prerequisites and nodewatcher itself, you can run it from its frontend directory:

./manage.py runserver

You can then open it in your browser at http://localhost:8000/.

If you want to see how wlan slovenija instance looks like, run instead:

./manage.py runserver --settings=frontend.settings_wlansi

Setting Up Dummy Data

It is often useful to have some real data in your database for local development or to see how the system works with filled data. Currently we provide a cleaned-up (no passwords or other private data) dump of our database for such purposes. The dump is created daily and can be retrieved from our server. As it is made from newest repository version you should probably always update your local version also to newest version prior using this data (otherwise incompatibilities in data models can happen).

In the dump archive there are two separate components. One is a data.json file which is a sanitized database dump of our production setup and the other is the graphs directory that includes some static graphs generated by our setup.

Run in the frontend directory:

./manage.py loadtestdata

This will download dump archive, unpack the graphs directory to the static directory and the data.json to the frontend directory and prepare and populate database with dump data from data.json file.

Setting Up a Production/Clean Environment

You really must use PostgreSQL (see warnings above) so you have to configure it in Django settings file. You should also disable all debugging options. You can simply use setting_production.py as a template/base for your settings file. You will also need to create a file named secrets.py into which you put settings you do not want to have public (and by mistake pushed to the code repository). Here are some suggestions what you can put there:

  • DB_PASSWORD
  • SECRET_KEY
  • GOOGLE_MAPS_API_KEY

Then for clean/empty environment you prepare database with (in the frontend directory):

./manage.py preparedb

It will also ask for initial administrator user data.

IP Pools

In the database you have to define your project and IP pools to be able to register nodes. You can add them for example with following SQL queries:

INSERT INTO nodes_pool
 (family, network, cidr, status, description, ip_subnet, default_prefix_len,
  min_prefix_len, max_prefix_len)
 VALUES(4, '10.88.0.0', 18, 0, 'Test Pool', '10.88.0.0/18', 27, 26, 28);

The pool in this example is 10.88.0.0/18 by default prefixes of length /27 are allocated to nodes, but allocation of sizes /26 through /28 (inclusive) is also allowed.

Values are:

  • family – should be 4 as we do not support IPv6 yet
  • network – network address of your pool
  • cidr – size of your pool (prefix length)
  • status – should be 0 when first creating a toplevel pool
  • description – nice description of the pool
  • ip_subnet – shold be in network/cidr format
  • default_prefix_len – default prefix length allocated to nodes
  • min_prefix_len – min (numerically) prefix length to allow
  • max_prefx_len – max (numerically) prefix length to allow

DNS Zones

In order to setup the DNS zones for the projects you will currently have to manually add the top-level zones into the database and then configure your DNS resolver. The instructions below apply to bind and you should have some experience with setting up DNS servers. First you need to create a zone by executing a command like:

INSERT INTO dns_zone
 (zone, owner_id, active, primary_ns, resp_person, serial, refresh, retry, expire,
  minimum)
 VALUES('xx.wlan', 1, true, 'ns1.xx.wlan.', 'dns@wlan-xx.net.', 1, 10800, 3600,
  604800, 38400);

Values are:

  • zone – should be the zone’s DNS name
  • owner_id – currently unused, should be the administrators uid which is usually 1
  • active – set to true for active zones
  • primary_ns – DNS name of the primary nameserver
  • resp_person – e-mail of DNS admin in hostname notation
  • serial – current serial number, should be set to 1 when creating a zone
  • refresh, retry, expire, minimum – see DNS documentation

After creating a zone you should also create some basic records in order for the zone to work properly:

INSERT INTO dns_record
 (zone_id, name, ttl, type, data, mx_priority)
 VALUES('xx.wlan', '@', 38400, 'SOA', 'xx.wlan.', 0);

INSERT INTO dns_record
 (zone_id, name, ttl, type, data, mx_priority)
 VALUES('xx.wlan', '@', 38400, 'NS', 'ns1.xx.wlan.', 0);

The top-level zone (in our example it is called wlan) must be configured as a zone in your resolver. An example configuration follows:

$TTL 38400
wlan.           IN      SOA     a.root-servers.wlan. dns.wlan-xx.net. (
                        1
                        10800
                        3600
                        604800
                        38400 )

; Root nameservers for this zone
wlan.                   IN      NS      a.root-servers.wlan.
a.root-servers          IN      A       10.x.y.z

; Subdomain delegation
xx                      IN      NS      ns1.xx.wlan.
ns1.xx.wlan.            IN      A       10.x.y.z

; Domain for test DNS checks
dns-test.wlan.          0 IN    A       127.0.0.1

Then you have to configure your DNS resolver to fetch some zones dynamically from the nodewatcher database. This can be done in bind by configuring the DLZ plugin in your named.conf. Sample configuration is as follows:

dlz "wlanXX" {
  database "postgres 1
  {host=localhost dbname=nodewatcher user=nodewatcher password=YOURDBPASSWORD}
  {SELECT zone FROM dns_zone WHERE zone = '$zone$' AND active = true}
  {SELECT ttl, type, case when type = 'TXT' then mx_priority || ' ' || '\"' || data || '\"' when type = 'SOA' then primary_ns || ' ' || resp_person || ' ' || serial || ' ' || refresh || ' ' || retry || ' ' || expire || ' ' || minimum else data end FROM dns_record r, dns_zone z WHERE z.zone = r.zone_id AND zone = '$zone$' AND name = '$record$'}";
};

Note: On some older bind versions keyword parameters to queries should be encased in % and not $ (so you would use %zone% instead of $zone$).

Projects

INSERT INTO nodes_project
 (name, description, pool_id, channel, ssid, ssid_backbone, ssid_mobile,
  zone_id, captive_portal, geo_lat, geo_long, geo_zoom)
 VALUES('ArborMesh', 'Example project on the Moon', 1, 6,
  'open.example.net', 'open.example.net-backbone', 'open.example.net-mobile',
  NULL, true, 46.05, 14.5, 13);

Values are:

  • name – name of the project, for example, city of the network
  • description – nice description of the project
  • pool_id – default IP pool
  • channel – default channel used
  • ssid – SSID used in this project
  • ssid_backbone – SSID used for backbone nodes in this project
  • ssid_mobile – SSID used for mobile nodes in this project
  • zone_id – DNS zone id (NULL if DNS capabilities of ‘’nodewatcher’’ are not used)
  • captive_portal – should the nodes in this project have captive portals?
  • geo_lat – default location of the map when adding a new node (latitude)
  • geo_long – default location of the map when adding a new node (longitude)
  • geo_zoom – default location of the map when adding a new node (zoom)

And then you have to link pool with the project (of course with proper id values):

INSERT INTO nodes_project_pools(project_id, pool_id) VALUES(1, 1);

Running Web Server

For production deployment read Django documentation on the subject. Django’s development web server is not suitable for production use.

Running Data Collection Daemon (Monitor)

Django web interface is just an interface to the database. To populate and update it with real data from the network you have to run also a monitoring daemon.

Run the monitor using command (in monitor directory):

./monitor.py --path=.. --settings=frontend.settings_production

You also need to install olsrd-mod-txtinfo plugin on some node in the network and configure it via OLSR configuration file (also note the node’s firewall configuration). By default monitor expects OLSR txtinfo plugin on localhost. This and other options you can configure in Django settings file.

Checking OLSR txtinfo Plugin

You can check that the txtinfo plugin is working by issuing:

telnet 10.x.y.z 2006
Trying 10.x.y.z...
Connected to 10.x.y.z.
Escape character is '^]'.

Then type GET and press enter. This should output something like:

HTTP/1.0 200 OK
Content-type: text/plain

Table: Links
...lots of data...
Connection closed by foreign host.

This means that the plugin is working properly.

Simulation of Monitor Data

To simulate monitor data you should set MONITOR_ENABLE_SIMULATION to True in your setting file. In this case the whole network is simulated and no node with OLSR providing the data feed is required. This may not be suitable for all test scenarios. Simulation data should be provided in simulator/data directory.

The latest simulation data that can be retrieved from this location. Simply unpack it into simulator/data directory.

Optional Data Archival System

nodewatcher supports an optional data archival system so all graphed data is also stored in a non-RRD database. We currently use MongoDB for this store due to its schemaless document nature and fast operations. In order to use this feature, you need to install and configure a MongoDB instance and then configure nodewatcher via DATA_ARCHIVE_* directives in settings.py. You will also need the pymongo Python driver for MongoDB.

You should familiarize yourself with MongoDB operations, durability limitations and proper deployment modes. Documentation is accessible via the above link.

On-demand Graph Feneration

All graphs are generated on-demand when requested by the web frontend to reduce I/O load on monitor runs. Because this requires additional configuration/setup, default configuration has the on-demand graph generation disabled (and therefore no graphs are displayed). You should configure this after you already have a working monitor setup.

On-demand graph generation requires a working installation of a message broker (for details see Celery documentation). We use MongoDB for this purpuse in production via the mongodb backend. If you already have a working MongoDB installation (it is also used for the optional data archive system) you simply need to set BROKER_HOST and BROKER_PORT to proper values for your MongoDB setup. Check settings_production.py file for an example.

After you have the broker set-up you also need to run celeryd task dispatcher in the background. You can do this simply via manage.py as follows:

./manage.py celeryd -l info -c 4 --maxtasksperchild=50

For production systems you will probably create an init script for starting up the dispatcher. Be sure that the user under which the deamon is executed has privileges to write to GRAPH_DIR. The last thing to do is to set ENABLE_GRAPH_DISPLAY to True in your settings file.

Firmware Image Generator

After you have configured all of the above components you might also want to enable the firmware image generator daemon. As the whole process is based on OpenWrt, you first need to build the imagebuilders for our firmware. The procedure below assumes creation of a new directory, but symlinking or building the imagebuilders on another system (as this is a very CPU and IO intensive process) is also possible.

Setup the needed directories and compile the imagebuilders using the following commands (if you are doing it remotely, you really should run this inside a screen session so that compiling is not interrupted if your session is disconnected):

mkdir build
cd build
hg clone http://dev.wlan-si.net/hg/old/openwrt-nw openwrt-200901
hg clone http://dev.wlan-si.net/hg/nodewatcher
cd nodewatcher/generator
./build_all_generators.sh

This will take a long time and will heavily load your CPU and IO. It is only needed to rebuild the imagebuilders when updating to a new version of the firmware. After the above process is completed without errors you must create the user with an username as configured with IMAGE_GENERATOR_USER in your setttings file. It should be in the same-named group. This will be the user the process will run under. You also need to setup a local instance of the beanstalkd daemon that should run on 127.0.0.1, port 11300 (refer to beanstalkd documentation for details). After that, you may run the image generator using the following commands:

./gennyd.py --path=.. --settings=frontend.settings_production --destination=/srv/www/packages.foonet.org/images

The destination argument should reflect your IMAGES_BINDIST_URL configuration in your settings file. This means that it should point to the physical directory that is backed by the URL. The directory must be writable by the IMAGE_GENERATOR_USER user. After you have configured everything and the generator is running, you should set IMAGE_GENERATOR_ENABLED to True.

Theming

It is possible to configure distributed nodewatcher theme or even develop your own custom theme, see Theming.