Linux weather station

Do this:

This is a How-To for the project that I've built since 15-Jan, which you can view at http://wx.ci.net. The main thing that's changed since mid-2000s is that low-power embedded-Linux machines have gotten insanely cheap; alas, the climate-measuring gear hasn't changed at all. Oh, and Weather Underground got bought out by TWC just a couple years ago, alas. Here goes: I'll share the assumptions, parts list, and software setup.

Assumptions[edit | edit source]

You're doing this as a home hobbyist, not a professional.
You have permission/ability to securely mount an anemometer at a high point near your home, workplace, school or friend's place.
You want to bear witness to climate changes over time, in a precise way.
In addition to temperature, you want to measure wind, rainfall, and barometric pressure
You want your data fully accessible online or on your mobile device, and to share it via a website (your own or Weather Underground).
Once set up, you don't want to have to babysit the hardware.
You already have a web server somewhere (local or far-away, Apache or nginx: doesn't matter).
Optionally, you have a local instance of MySQL available somewhere.

Parts List[edit | edit source]

Raspberry Pi with SD Card (type 10, 8GB or bigger) - $30
Davis Instruments station - choose from Vantage Pro 2 wireless with fan-aspirated shield - $695 Vantage Pro 2 wired - $415 Vantage Vue wireless - $310 (Sensors for sunlight UV/visible are bundled at the $850 level)
3 batteries (C size) for Davis console - $8
Davis Instruments logger (required) with WeatherLink - $130
Dahua 3MP outdoor webcam model IPC-HFW4300S - $125
PoE injector for webcam - $20 to $40
10' steel mounting pole (1-1/4" galvanized natural-gas pipe) with end cap U-clamps from Home Depot - $25

Approximate budget: $500 low-end without cam, $1050 high-end with cam

I am not aware of any worthy rival to Davis Instruments; you can get something that might work for less than the low end of my suggested budget, but very likely will make the whole project a huge, unreliable chore. The Davis products are woefully obsolete as UI devices, but I swallowed my criticism and am focused on getting the data online. Dahua does have decent rivals, but at this price-point there really isn't anything worth considering.

Hardware Setup[edit | edit source]

Find the highest point available, put the end cap on your mounting pole
Fasten anemometer to top of pole using Davis-included hardware, and the rest of the instrument array about 3' below the top
Securely attach pole to a wall or railing at the highest structurally-sound part of your property. Make sure it doesn't budge even in high winds. If you got a non-wireless version, route the cable to where you place the console.
Open back of the Davis console, install batteries and the overpriced logger thingie. Find your coffee table, take out the Windows software that came with the logger and put a mug of beer on it. Sip beer slowly at next step.
Plug a USB cable from your Rasberry Pi into the logger, and an Ethernet cable into your LAN (or wifi if you prefer that)
Figure out your latitude/longitude/elevation, enter them into the station console
(Optional) find a good spot for your webcam, run cat5e cable to its location from your Ethernet switch and test with PoE power injector; verify browser access with password admin/admin

Software Setup - WX[edit | edit source]

Get Raspbian onto your SD card; plug a blank card into your Linux or Mac and use the 'dd' command to copy the Debian Wheezy 2015-01-31 image from http://www.raspberrypi.org/downloads/ to the card. Tip for Mac users: use the "rdisk" device, e.g. /dev/rdisk1, otherwise the copy will take hours.
Plug an HDMI monitor and keyboard (temporarily) into the Raspberry Pi and boot it up; go through its dialog, selecting en_US.UTF-8 locale, then log in as 'pi'.
Use fdisk/resize2fs (or the Raspbian installer dialog) to increase size of the root volume (I prefer manual configuration so I can leave some scratch space available to use in a pinch).
Invoke: apt-get install xtide xtide-data python-configobj
sysstat python-mysqldb python-serial python-usb
python-cheetah python-imaging rsync
Invoke apt-get update, then download weewx_3.0.1-1.deb package from http://sourceforge.net/projects/weewx/files, install with 'dpkg -i' command
Verify connectivity with the weather station, and set parameters thus: wee_config_device --set-interval 300 wee_config_device --set-rain-year-start=7 # or whatever month
Set up a vhost on your webserver. My apache2 vhost looks like this:

<VirtualHost *:80> # Server Configuration: ServerName wx.ci.net DocumentRoot "/var/www/htdocs/wx/" ServerAdmin richb@instantlinux.net ServerSignature email DirectoryIndex forecast.html HostNameLookups Off <Directory /var/www/htdocs/wx> Options -Indexes FollowSymLinks AllowOverride All Order allow,deny Allow from all </Directory> </VirtualHost>

Set up a username 'wx' and ssh keypair for uploads to the webserver, on both the Raspberry Pi and the web server: this will be used for rsync. (You can use an existing username on the webserver if you have to.)
If you like my WeeGreen 'skin', download it from https://github.com/instantlinux/weewx-WeeGreen and change the
Download the forecast extension from MIT: http://lancet.mit.edu/mwall/projects/weather/
Invoke: /usr/share/weewx/setup.py install
--extension /var/tmp/weewx-forecast-3.0.2.tgz
The forecast extension comes with its own skin; optionally you can turn that of if you've decided on WeeGreen (or if you're customizing off mine)
Register yourself at Weather Underground, get an API key and a station identifier. My ID is KCASANFR366.
If you prefer 12-hour format on your output, change the formats from %H:%M in /etc/weewx/skins/WeeGreen/skin.conf
Adjust the other preferences in skin.conf
In the top-level /etc/weewx/weewx.conf, there are several settings to update: you can (mostly) ignore the Station section because settings are overridden by what's set in the console, but you'll want to add your XTide location and WU api_key to the Forecast section, and station ID/password in the Wunderground section, rapidfire = True (that part's kinda remarkable), and your web server user/path info to the Rsync section
If you have a MySQL server (strongly recommended), make a blank database and add its credential settings to the Databases section. You actually need two separate databases, I call mine weewx_a and weewx_f, because the forecast and archive engines both write to the same table name. There's a cool script you can download to convert from sqlite3 to mysql if you've already got data collected in sqlite3 (google for 'weewx sqlite3-to-mysql.py').
Add this line to /etc/fstab: tmpfs /var/www/weewx tmpfs nodev,nosuid,noexec,nodiratime,size=4M 0 0
If you have a central rsyslog (also strongly recommended) create the file /etc/rsyslog.d/custom.conf with content like the following:

*.info;*.=notice;*.=warn;auth;user.*;daemon.*;mail.none; \
    cron.!=info;local1.*;local3.none    @@syslog.ci.net:601

Software Setup - Cam[edit | edit source]

Webcam capability is a nice-to-have feature which will drive traffic to your observatory if your camera has a good vantage point. (Mine happens to be aimed at Twin Peaks and the Sutro Tower, with a chunk of western sky where fog and clouds provide dramatic time-lapse opportunities.) Everywhere has unique opportunities, so be creative even if all you think you've got is a view of passing traffic.

The embedded-Linux-based camera I've recommended here, from Dahua, is a hobbyist's dream compared to the likes of much-hyped cloud-centric products like Dropcam: from its brower console, you get a whole ton of tunable settings. It's also going to give you an image 2048x1536 rather than something-by-720, at a fraction of the price.

Log into the camera and confirm you can see image in the "Live" tab
On your local Linux system (could be your Raspberry Pi but I have a better/faster box for pre-processing images), configure a dedicated ftp user/password (sorry, no scp: yeah, I know--Dahua is a "security" company).
In the Dahua's Camera->Video->Snapshot section, select "Best" quality
In the same section, select Customized interval, and choose how often you want your time-lapse images to be uploaded (I picked 150 seconds)
In the Storage->Schedule->Snapshot Schedule, select 24x7 and make sure the "general" (green) band shows up 7 days a week
Although I'm using the Dahua strictly as a webcam, note that while you're installing this, you can build an on-the-cheap burglar alarm out of this thing by enabling motion-detect video and email alerts in the Event->Video Detect->Motion Detect section. Works just like the Dropcam my neighbor showed me after we got hit by burglars. Net effect is the burglar's activities are uploaded to the cloud at least up to the point where he steals your camera.
OK assuming your camera hasn't gotten stolen in the above step (yes, I took my own advice, and hope that you're taking your next sip of beer): go to the Storage->Destination->FTP tab and enter the ftp credentials noted above. I told it to use a directory called "upload". The camera will start sending snapshots to that directory every 150 seconds, in a stupidly-complicated directory hierarchy. But because you know bash, you don't care how silly the pathnames are.
Once you verify that you're getting the uploads to your local ftp way-station directory, you can optionally register your web cam at Wunderground (for inspiration, take a look at any of the Boston-area ones by browsing maps at wunderground.com, or my SF one at http://www.wunderground.com/webcams/RichCastroWX/1/show.html); the process takes only a single "Add Your Webcam" dialog box - pick "ftp" and then tell it where you are and what you want to name the webcam. (You can have more than one.) Shrug off your objections to Internet privacy: we, ISIS and the NSA already know where you are.
Install ImageMagick and ncftp on your local ftp server and set up a crontab to run an uploader script. Notice how my images are a non-standard aspect ratio: I chose to crop the image, and retain as much image-quality as will fit in 150Kb, prior to transferring to wunderground.com. Since you objected to learning bash two steps up, here's my script:

 #! /bin/bash
 # Upload latest image to wunderground

 CAM=$1
 MAX_SIZE=145kb
 MAX_TIME=60
 RETRIES=3

 cd ~/upload/$CAM
 LATEST=`find . -type f -name *.jpg -mmin -5 -print|sort -r |head -1`
 IMG=/tmp/image.jpg
 if [ "$LATEST" != "" ]; then
  convert $LATEST -crop 1920x880+0+0 -define jpeg:extent=$MAX_SIZE $IMG
  while [ $RETRIES -gt 0 ]; do
    START=`date +%s`
    ncftpput -f ~/.ncftp -t $MAX_TIME -V -d /tmp/wxupload-debug.log / $IMG
    RET=$?
    FIN=`date +%s`
    if [ $RET == 0 ]; then
      logger WX upload file=$LATEST bytes=`stat -c %s $IMG` seconds=$((FIN -
START))
      break
    else
      logger -p user.warning WX upload failed file=$LATEST bytes=`stat -c %s
$IMG` seconds=$((FIN - START))
    fi
    RET=$((RET - 1))
    sleep 5
  done
  rm $IMG
 fi

Your wunderground.com ftp-upload credentials (wunderground is not a security company, so they likewise don't do scp) need to be put into the upload role user's ~/.ncftp file in this form:

host webcam.wunderground.com user RichCastroWXCAM1 pass xxxxxx

Wrap-up[edit | edit source]

Early in the above instructions, I mentioned installing sysstat. Next step:

Invoke on the Raspberry Pi: iostat -k -d 60
Wait 10 minutes and take note of the kB_wrtn values. You're looking to make sure the SD card doesn't blow up after a few weeks/months (like mine did last Friday).
If you're seeing any activity on the SD card, double-check your rsyslog, MySQL, and tmpfs mounts. If you've followed these instructions, the SD card should get zero activity and the Raspberry Pi will remain reliable for more years than the external sensors.
Add monitoring of your webcam(s), Raspberry Pi, and web logs to your nagios/zabbix instance
Install the wunderground app on your mobile device
Make sure you've got a one-time backup of the Raspberry Pi's SD card, and daily backups of the video upload directory on your ftp server and your mysql database.
Enjoy a second mug of Sam Adams

That's it, hopefully this HOWTO inspires you to bear witness to global climate change (you don't trust the TV and websites to give you unbiased info, do you?), and saves you lots of time as you juggle the software and hardware bits.

My apologies for the snow in Boston at this time, and for the fact that unless you shell out an extra $kilobuck, the only automated measurement of snowfall you can make is its water content.

-rich

Footnote: the abbreviation "WX" for weather is familiar to me from work in aviation software. But it actually comes from the days of telegraph machines: in Morse code it is .-- -..-