1 Open Weather (Working Title)An open weather monitoring and reporting system using a distributed model
The idea behind this project is to provide a system to monitor, log and forecast the weather using a distributed model.
All protocols and interfaces developed by the project will be open and published. All software used in development and produced by the project will be free software, sometimes called open source, and preferably GPL.
At this moment I have no idea how much of this is achievable, certainly many aspects are well outside my sphere of competance. In this document I will outline some ideas for the framework of the project.
The latest version of this document can be found at http://www.catnip.co.uk/wx/
Formal document control will commence when initial draft is stable.
A printable version of this document in OpenDocument Format (ISO/IEC 26300) is available at:
http://www.catnip.co.uk/opendocument/wx.odt
This document may be opened with OpenOffice.org 2 or any other ODF compliant application.
These documents may lag behind the xhtml version when this is under active development.
There are currently three pieces of reference software under development, wxdisp, wxcard and wxlmon, as well as several PHP scripts. Wxcard is software to run on a Microchip PIC 16F877 based weather monitoring card. Wxdisp is an application which takes the data from wxcard, displays it locally and also makes the data available for other applications to use. Wxlmon is a serial interface diagnostic tool. The PHP scripts provide the interface to the internet. Wxcard and wxlmon are Linux based applications that use the GTK+ widget set for the user interface. They should compile on any fairly recent Linux distribution.
Release 0.01. This release is very pre-alpha. It only provides the basic transmission framework plus the modules to transmit wind direction from a 4-bit encoder and temperature from an analogue sensor.
Wxcard01 - 2005-04-15
Release 0.01. A program to display and log packets of data from wxcard. It can also transmit dummy packets (click AUTOSEQ button) and will display and log them if the serial port is looped back.
Wxdisp01 - 2005-04-28
Release 0.01. A program to receive and display packets sent by wxcard. It can also transmit dummy packets and will display them if the serial port is looped back, as shown in the screenshot above. This program should compile on most Linux platforms with GTK installed.
Wxlmon01 - 2005-04-20
The PHP scripts provide an interface to the public internet in conjunction with a Linux - based server running the standard Apache web server software. The first script is rss.php, which presents the local weather data as an RSS feed. The second is wxlocal.php, which presents the local weather data as a web page. Both of these PHP scripts along with their support files and a dummy data file are hosted on this server for demonstration purposes - please see the links below.
Release 0.02. These PHP scripts should work with any recent Linux-Apache-PHP setup. The file getvars.php is an include file which parses the output file processed.txt from wxdisp and places the data in a hash. The file rss.php uses the hash to generate an RSS feed from this weather data, whilst wxlocal.php generates a web page from the same data. The include file locvars.php contains data pertaining to the local site, footer.php generates the web page footer, synopsis.php is a placeholder for a synopsis generator and soon.php is a dummy page for links to features yet to be implemented.
PHP scripts 02 - 2005-05-05
Hopefully hardware details for wxcard will be posted here in the fullness of time. If any of the sensor manufacturers would like to donate hardware this will speed up the process!
The system comprises three main parts:
The local station comprises an array of meteorological sensors which are monitored by a small microprocessor unit. The microprocessor unit converts the analogue signals into digital form and sends them to the local station computer for logging and onward transmission. The local station computer logs the meteorological data and makes this available to the internet in two forms:
Where a Local Station does not have the benefit of an ADSL or other direct internet connection, a facility will be provided to enable another Local Station so equipped to act as a proxy.
The distributed processing subsystem comprises a grid of interconnected machines which look at the RDF data provided by the Local Stations and process this to provide forecast data for different locations.
The User is able to look at the system data in three ways. Firstly the RDF/RSS feed of current data from a given Local Station should be viewable in any basic RSS reader, such as that integrated into the Firefox web browser. Secondly data will be provided by the Local Station web server on a summary page giving the current conditions at the site, and as a series of graphs of recent trends. Thirdly, forecast data will be available from certain nodes of the distributed computing system.
It is hoped that this system will find a use in many parts of the world, and that local nodes will be referenced from distant locations. Thus the design of web page and rss software should allow for easy localisation and have a multilingual capability.
This aspect of the system, at the present time and for the forseeable future, is nothing more than a pipe dream. It would certainly not be viable unless there was a considerable installed base of Local Stations collecting and making available the raw data. In all probability it will never happen, but stranger things have come to pass. I've been wanting to write a reasonable application in Lisp or maybe Haskell for a while - perhaps I'll give it a go!
What I imagine is a grid of processing nodes overlying the grid of physical weather stations - quite possibly residing on the same machines, but of course there is no requirement for the physical layout to match the mapping. So perhaps the processing node nearest each station does a rough forecast based on the rate of change of pressure, the temperature and wind speed. This forecast is then refined by looking at the forecasts for the nearest stations to the one in question, which themselves have been refined by looking... ...and in an iterative fashion the nodes should arrive at a reasonable forecast. The whole network will need to be critically damped to arrive at a forecast in the smallest time possible without becoming unstable. On top of this system there will need to be a layer which can detect meteorological features from the readings - such as cyclonic behaviour, fronts and the like. The output from this layer can be used to further refine the forecasting.
It is not the purpose of this document to mandate the design of the Local Station, but rather to lay a foundation by providing some design criteria, system level protocols and software source code that will enable Local Stations to be built that will work well with the rest of the system. That said, I hope to eventually produce a reference design and perhaps even offer component and circuit board kits in the fullness of time. The intention is to avoid expensive professional weather station equipment and provide instead designs that can be constructed reliably in a home workshop using inexpensive commodity sensors.
The raw data from the Local Station will be made available on the internet in the form of an RSS channel roughly conforming to RSS 1.0.
The first two items in the feed will give the name of the station and the current weather synopsis.
Subsequent items the feed will provide a measurement such as temperature, pressure, etc.
The current reading for each of these subsequent items will be contained in the item <title> field. The form of this field in British English (en-GB) will be:
These <title> fields are the data that will be displayed in a typical RSS reader. It is proposed to develop a supplemental xml namespace 'wx' to:
This is a demonstation of the form the feed might take: RSS
Here is the demonstration RSS feed displayed as a Live Bookmark in Firefox:
The <description> field of each rdf item will provide an exact description of the parameter being measured along with the units of measurement e. g.:
Atmospheric pressure in hectopascals
The local station data may be displayed on a web page. This demonstration is produced from a processed data file by one of the PHP scripts which is contained in the 02 script release. The script generates a page based on xhtml 1.1 and on css2 and may therefore not be suitable for older web browsers. The page has been found to perform well with browsers using the Gecko rendering engine.
The link layer for communications between the data card and the Station Computer will be initially RS485/232, depending on the length of cable run required. USB may be a possibility for future convenience, although it is not suited to long cable runs. The serial protocol will be simple - the data card will send an ad hoc packet of information to the Station Computer each time a measurement is made, and the Station Computer will acknowlege if the checksum is correct.
Full details of the protocol will be given in the appendix.
The local station computer will be a PC compatible type running the Linux operating system. This is because this platform makes it very easy to set up a web server using the Apache software usually included in the Linux distribution. PHP will be used to generate the web pages and RDF files dynamically from the data received from the Data Card. Gnuplot will be used to generate graphs of historical data on demand.
This graph was generated by Gnuplot from a data set provided in a text file.
The Data Card will send readings from the sensors to the station computer on a regular basis - every minute or so. A driver program written in 'C' will listen on the serial port for this information, and store it in a text file. Each parameter will be stored on a separate line along with the time of the reading as a decimal representation of time_t. This text file will be processed at intervals by a Perl script called by cron. This script will perform two functions.
Firstly it will take each reading in turn and append this reading along with the time of the reading to the end of a text file, a file for each reading. These files will comprise the historical record for the station.
Secondly the script will generate a PHP include file which will provide data to generate a synopsis of the current weather conditions at the station. Each reading will be loaded into a separate variable. Where the trend of a reading is required the tail of the appropriate historical record file will be examined and the slope calculated.
The PHP file will be included in the script that generates the RSS channel, and also in the script that generates the about.php page, that will display the current readings. Lock files will be used to avoid contention during updates.
All times will be in UTC.
It is in all probability beyond the scope of this project to generate an accurate visibility figure in metres. However it should be possible to derive a reading of the 'good | fair | poor' variety by looking at a distant object with a webcam and measuring the contrast of the image. The contrast will decrease with visibility.
A small webcam will be employed, it may be possible to use the same camera as that used for visibility above. The camera will need to be mounted to have a clear view of as much of the sky as possible. The signal from this camera will need processing to determine the percentage of blue in the field of view.
The Local Station Data Card will be based on a Microchip 16F877 microcontroller, designed with the gEDA tool set, laid out with PCB and programmed with Gputils. Here is an example of a board produced last year using these techniques.
The primary function of the card is to read the data coming from the meteorological sensors and forward this to the station computer. It may be possible to add other features such as a locally driven display and thermostat functions. However there is always a danger of feature creep compromising the cleanest solution. For example, if a local LCD display is driven from the data card, this could lead to error correction of the (say) humidity sensor being performed on the Data Card, when it may be better to perform this on the floating point processor of the Station Computer. On the other hand, there is no objection to providing the track for these components on the card layout, so the card can be used for more than one purpose.
The digital section of the card is relatively simple to produce, as most of the hard work has been done in the design of the Microchip 16F877 microcontroller. It only remains to add static protection and RFI filtering to the signal lines that go off - board.
The digital sensors used in this project all utilise optical sensors to provide output. They all generate signals with relatively low pulse rates. This means that as long as an optical transducer with a totem-pole type of output is chosen, the sensor can be mounted a fair distance from the Data Card.
Wind Speed will be measured by a conventional rotating cup anemometer arrangement. This device will probably need a fair bit of development, as two problems (at least) will have to be overcome. Firstly a design is required, that whilst easy to make in the average garage workshop, yields a consistent speed of rotation for a given wind speed. It is not desireable to have to calibrate each unit. Secondly, the electrical signal generator will need some thought. Something is required that uses a non - contact technique - e. g. optical - that can handle both very low and very high wind speeds.
Wind direction will be measured by a conventional wind vane connected to a 16 - position Gray Code wheel. This will give a 4 - bit output signal to the Data Card. Fluctuations will be smoothed out by the card software processing to provide the wind direction data.
This sensor will probably be of the common rocking bucket type. This is another area where there is likely to be some tradeoff between sensitivity and heavy load capability.
Every time the bucket tips, a pulse will be sent to the data card, and a 16 - bit counter incremented. This count will be sent to the station computer periodically. When the station computer first receives a count on the precipitation channel, it will merely store the count. The next time it receives a count, it will subtract the stored count from it to obtain the count between the two transmissions. This count between transmissions is added to the cumulative tip count. The count just received will become the new stored count. In this way any packet loss will not affect the accuracy of the cumulative count, but the granularity will be temporarily degraded.
The cumulative tip count is the raw data that is processed to give the cumulative rainfall in millimetres. This figure is reset at local midnight and the final total output as the day's total rainfall.
Sunshine measurement, in principle, seems straightforward enough. What is required is effectively a one - bit analogue to digital converter, returning 1 when there is sunshine, and 0 when there isn't. The devil, as always, is in the details. Sunshine was, and is, recorded by instruments of the Cambell-Stokes variety comprising a glass sphere placed in front of a curved paper chart. When it is sunny, the sunshine is focused onto the chart and burns a trace - measuring the length of the trace for a given day gives the number of hours of sunshine. According to the article "Measuring direct solar radiation" by E. Linacre and B. Geerts "Bright sunshine occurs for a Cambell-Stokes recorder when Rd exceeds 120 Wm-2 onto a surface perpendicular to the beam". However RReDC Glossary of Solar Radiation Resource Terms states that "Bright Sunshine - when the sun casts an obvious shadow or when a Campbell-Stokes sunshine recorder is recording. The lower limit for bright sunshine (based on a Campbell-Stokes recorder) is between 70 W/m2 (very dry air) and 280 W/m2 (very humid air). It seems there are possibly two approaches here. The simplest would be an integrating sphere containing a photo - electric cell set to toggle the output at a given level of incident radiation. This would probably model the Cambell-Stokes recorder reasonably well. The second approach is to measure when the sun casts a shadow. A possibility here would be to form a metal grille into a hemisphere and rotate this over a photo - electric sensor. The width of the bars of the grille would have to be such that they could cast a shadow with a width that exceeds the width of the sensitive area of the detector. When the sky is overcast the light falling on the sensor would be invariant. As the sun comes out it would start to cast shadows that would cross the sensor as the hemisphere rotated. This would give a signal where the rate of change would be a function of the 'sharpness' of the incident light. A large patch of bright sky would cast a large penumbra onto the sensor, so the rate of change would be slow. Differentiating the output and feeding it into a comparator should give the desired result.
This particular sensor will in all probability require the highest sampling rate of any as sunshine can fluctuate rapidly when there are small clouds and a brisk breeze. Thus it may be best to integrate the time on the Data Card by counting the seconds of sunshine in a 32 - bit counter and sending the result periodically to the data card. Another counter could count the transitions of the input, which could be useful in producing a 'sunny | fair | variable | cloudy' type of output.
For the prototype implementation, the following scheme will be used. A photocell in an integrating sphere will feed an output into one of the A to D converter inputs. This allows the sensor switching level and hysterysis to be set in software. The front end will generate true if sunny, otherwise false. This will be integrated in the short term by a 32 - bit counter that will count up if sunny, down if not. At the end of each sampling period, the output will be taken as sunny if the counter is positive. This signal will be sent over the serial interface to the station computer.
When the station computer first receives a signal on the sunshine channel, it will merely store the local time. If the next time it receives a signal on the sunshine channel, the signal is true, the station computer will add the difference between the current time and the stored time to the sunshine hours cumulative total. The current time becomes the new stored time.
The temperature (depending on sensor type), humidity and electrostatic charge sensors will present an analogue signal to the data card. This signal will have to be matched in terms of voltage swing and level to the inputs of the analogue to digital converters integrated into the microprocessor chip. Every effort needs to be made to produce a design that does not need calibration yet is capable of accurate and stable operation over extended periods of time.
The PIC16F877 has a 10 - bit A/D converter and can be configured to use an external voltage reference.
The remainder of the analogue circuitry is that required to provide a stable power supply for the card. It is envisaged that the card will operate from a 12 to 24 V d. c. supply with minimal current requirements, so it can be supplied via the signal cable.
A decision needs to be taken on how to best mount some of the sensors. The ideal from an electrical point of view would be to mount everything on the Data Card. This would mean that the whole Data Card assembly would need to be waterproofed and mounted inside a Stevenson Screen or similar on a clear patch of ground. One serious problem could be heat from the unit affecting the temperature measurement.
Another approach would be to mount the unit inside a building and mount some of the sensors outside. The pressure sensor can be mounted directly on the Data Card. If the reading of this sensor is likely to be disturbed due to forced ventilation of the building, the ported type of sensor can be used with a thin tube leading outside. This leaves the temperature and humidity sensors to be mounted outside. A solution may be to mount the sensors in something like 40cm of white plastic pipe open at each end with a plate suspended over the top to prevent sunlight from entering.
Solar gain appears to be potentially one the greatest source of errors in the system. There are several signs in this area which display the temperature. They only differ with each other by a degree or so if it is overcast, if it is sunny they all disagree by several degrees.
Note that there are several sensors that measure essentially analogue parameters but give a digital output that can be connected directly to a serial input on a microprocessor chip. However, the protocols used are proprietary in nature, some are even subject to patents. As the software running on the microprocessor will be released under the terms of the GNU GPL, this could present a potential problem. For this reason analogue sensors have been used throughout.
Instrumentation amplifiers will be required to adjust the range and level of the incoming analogue signals. The basic form of the design is illustrated in the following diagram prepared with gschem:
Here Vi is the input voltage from the sensor, Vo is the output to the analogue to digital converter, and Vb is the offset or bias voltage.
Vo = (1 + R2 / R1) Vi - (R2 / R1) Vb
This function can be entered into a Perl script, and used to output sets of data points that Gnuplot can use to plot the amplifier transfer functions. It is then easy to visualise the effects of varying the gain and bias points and optimise these so that the outputs have similar range and offset:
The script also outputs the maximum and minimum outputs, and the range:
[pete@emily perl]$ ./tferfunc.pl Vmin = 1.10129999999986 Vmax = 4.75999999999967 Range = 3.65869999999982 Return to exit
These curves were plotted with the following values:
These figures have some implications on on the design of the instrumentation amplifier:
According to the Wikipedia, world temperatures range from 58.0°C, recorded in Al'Aziziyah, Libya, to -89.2°C recorded in Vostok, Antartica. Whilst the upper end of the range is easily measured with bandgap type semiconductor temperature sensors, the lower limit could be more problematical. Presumably anyone setting up this system in a place experiencing temperatures this low will be adept enough at problem - solving to adapt a suitable sensor to the system, and contribute this knowledge back to the project. For the standard offering I will be pragmatic and limit the minimum temperature to that of commonly available sensors, around -40°C
The DS600 analogue temperature sensor from Maxim/Dallas can measure from -40°C to 125°C with an accuracy of 0.5°C over the range -20°C to 100°C. The output voltage change with temperature is +6.45mV/°C with a dc offset of 509mV. Over the desired range -40°C to 60°C this gives a minimum voltage of 0.251V and a maximum of 0.896V, a range of 0.645V.
HIH 3610 0.8 - 4.1V Range 3.3V
According to the Wikipedia, atmospheric pressure can vary from between 869.96mbar (86.996 kPa) and 1085.7mbar (108.57kPa). Presumably these are sea level readings - so the range of the instrument will need to be larger than this to acommodate instruments at high elevations. Assuming that the highest altitude this unit will find itself will be no more than 3000 metres, a sensor range of 550hPa to 1100hPa should be ample.
The MPX4115A pressure sensor from Motorola has a sensitivity of 49.5 mV/kPa and a transfer function Vo = Vs*(0.009*P-0.095). This gives a voltage swing of between 2.0V and 4.475V over the range of normal atmospheric pressure variation plus an allowance for station altitude, assuming a supply voltage of 5.0V.
Electrostatic charge can be measured using something along the lines of the Homebrew Lightning Detector referenced below. There are, however, some practical problems. To start with, I am not sure what meteorological property this thing is measuring, which makes calibration difficult. I would feel it would be irresponsible to advocate a reference design based on the Matchrockets type of device, because of the potential dangers of fabrication, but would not want to preclude people from using this (presumably more responsive) sensor in the system. Another factor is that the sharp object used as a collector will probably not be in an accessible place, and will corrode and blunt over time, causing the calibration to shift. Thus the only solution that suggests itself is to provide the data card with a current sensing input that calibrates itself over time. This will make it impossible to measure differences in absolute levels of charge, but should allow the changes - i. e. the differential - of the signal to be recorded. As this thing has the potential to feed a nasty spike into the system the best implementation is a remote current sensor connected to the data card by a length of optical fibre.
The hyperlinks in the text will go straight to the items of interest rather than coming here, but most linked items will also be listed here for easy reference.
Royal Meteorological Society
The UK Weather Information Site
UK's National Pressure Standards
Altitude & Barometric Pressure
Encyclopedia of the Atmospheric Environment
Measuring direct solar radiation
RReDC Glossary of Solar Radiation Resource Terms
Homebrew Lightning Detector
About Temperature Sensors: A repository about temperature measurements & devices.
Maxim Temperature Sensors
WCARC WX Monitoring - Notes and Links - Sensors
GNU PIC Utilities
GNU Common Lisp
Association of Lisp Users
PHP: Hypertext Preprocessor
The Apache HTTP Server Project
Perl.com: The Source for Perl
The Perl Directory - perl.org
Comprehensive Perl Archive Network
ActiveState - ActivePerl Product Family
OpenOffice.org
Firefox Web Browser
This specification is in the early stages of formulation but can be found at http://www.catnip.co.uk/wx/0.1/
This section intentionally left blank
Data will be transmitted from the Data Card to the Station Computer using a simple serial protocol. The sensor readings will be sent as a set of packets - one for each reading. As each set of packets transmitted will involve no more than 100 bytes, the speed of transmission will be kept low to facilitate transmission over fairly long distances. The physical connection will be via either 3 - wire RS232 for cable runs of a few metres or 5 - wire RS422 / RS485 for longer distances or electrically noisy environments.
Transmission will be 9600 Baud, no parity, one stop bit.
Each normal transmission will comprise:
Each packet will be individually checksummed. As the packets are resent more frequently than the desired measurement period, it is not felt the additional complication of packet acknowledgement and retransmission is justified in the initial scheme. Here are a few packets listed by the diagnostics of the prototype software:
Packet START 02 01 d0 30 0a 00 3f 3e 35 03 Packet WIND DR 02 01 20 20 0a 0d 2f 7e a9 03 Packet TEMP 01 02 01 50 20 0a 00 ff de 0e 03 Packet END 02 01 d1 30 0a 00 00 53 d8 03
The meanings of the various fields are as follows:
Communication between the file logging application (wxdisp in the reference implementation) and the web server is by means of Unix format text files. Before reading or writing to these files, obtain an advisory lock (flock) on the lock file lock.txt.
The file out.text gives the last received raw data from the data card (wxcard in the reference implementation). The file comprises pairs of comma separated data, sender i. d. then value in hexadecimal form. The sender i. d. has the same interpretation as on the serial link above. The order of the pairs and number of pairs is unspecified and should not be relied upon when parsing the line.
d0,0007,10,0005,50,0234,d1,00000000426ac81d
The example shown gives a sequence number of 0x0007, a wind speed of 0x0005, a temperature of 0x0234 and time_t when the measurement was received was 0x00000000426ac81d.
The file processed.text gives the last received data from the data card after processing to derive meaningful values from the raw data. The file comprises pairs of comma separated data, sender i. d. then value in signed floating point decimal form. The sender i. d. has the same interpretation as on the serial link above. The order of the pairs and number of pairs is unspecified and should not be relied upon when parsing the line.
16,5,80,-3.5,208,7,209,1114294301
This line represents a wind speed of 5 km/h, a temperature of -3.5°C, a sequence number of 7 and time_t when the measurement was received was 1114294301.
Some additional sender i. d. numbers appear in this file for derived readings not transmitted over the serial interface:
The log file log.txt is the same format as processed.txt, but with multiple entries.
16,5,80,17,208,1,209,1114294289 16,5,80,17,208,2,209,1114294291 16,5,80,17,208,3,209,1114294293 16,5,80,17,208,4,209,1114294295 16,5,80,17,208,5,209,1114294297 16,5,80,17,208,6,209,1114294299 16,5,80,17,208,7,209,1114294301
The update period of the log file and the size to which this will be allowed to grow before archiving is to be decided.
The UK National Physical Laboratory states that following the 8th Congress of the World Meteorological Organization, from 1 January 1986 the term hectopascal (hPa) is preferred to the numerically identical millibar (mbar) for meteorological purposes. This unit will therefore be used in this project.
Please refer to http://www.gnu.org/copyleft/fdl.html
Copyright © 2005 - 2006 Peter Harlow
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License".
The copyright holders make no representation about the suitability of the specification for any purpose. It is provided "as is" without expressed or implied warranty.
10057
