This was an internet of things project with big data storage and analysis.

Monitoring of air quality has become a lot easier in the last few years with the advent of small solid state sensors that are capable of doing a similar job to equipment originally designed for laboratory work which is what is in those large stations that are occasionally seen at traffic intersections.

The objective of this project was to build small monitoring stations, about the size of a large book, that could be easily mounted on buildings the data being sent back wirelessly to servers where it can be collated, displayed and supplied to other interested parties.

Initial system configuration:-

Air Monitoring Stations

Sensors used were developed by Alphasense and mounted on their standard boards which were interfaced to a bespoke processor board doing A to D conversion and generating 5 minute blocks of data which were sent using https: to the data acquisition servers, the boxes being masters.

Data collected was Temperature Deg.C, humidity, acoustic (noise level), NO2, NO, O3, SO2 and CO in PPB and Particulates PM1, PM2.5 and PM10.

Blocks of data were sent  to the acquisition server in a compressed json format over 2G or 3G network using https via the mobile network gateway. The server reply data block could do things like sync the clock to UTC, change the sampling rates of various sensors, ok the data blocks received and, eventually, request the box to upload new software from the distribution server.

Eventually we were going to move away from http, as it is quite a bulky protocol, to CoAP as it is a lot faster and more compact thus saving a lot on mobile network bills, also uses a lot less power.

Data Acquisition and Storage

Deciding on the databases was one of the big decisions at the start of the project. The sensor boxes were obviously going to produce large amounts of data and, depending on the time slice or configuration of the box, different sets of sensor data, both of these challenges made a standard SQL database not the best choice for the job. We chose MongoDB in the end since it could handle the expected amount of data with out slowing appreciably and variable contents for any particular time slice was not a problem since it was a document database.

Once a data block had been received by the server it was checked for validity ie. valid sensor box id, tokens and type, the compressed message block was converted into a more readable json format based on the configuration of the box's sensors and this saved for a month in case we needed to re-generate due to configuration mistakes.

The next step was to convert the voltage produced by each sensor to the equivalent value in say parts per billion for NO2, this involved standard equations for the type of sensor, constants from tests carried out on the sensor after production and the temperature when the sensing was done. Again the block was saved giving the most accurate data for that sensor box.

Another data set produced from the received block was one and ten minute averages. The one hour average was produced once enough unprocessed data had accumulated in the ten second collection.

A future enhancement for the calculations was to be to use machine learning  to find out how the sensors changed as they aged and correct for this.

Configuration API

Configurations for the boxes needed to be created for new boxes and updated at times for old ones. An API was instituted to achieve this which was ip and key locked to another system which had user interfaces for this.

Data Display

Visitors to the site could see a map of the UK with all the public boxes marked. Hovering over the pin showed current status, clicking moved to a data display page for that box.

Users could change the displays as follows:-

• Date and start time
• Number of hours shown
• Sample rate 10s, 1 minute, 10 minute and 1 hour
• Types of sensor data displayed
• Also outliers and raw data

The plots were created using Google graphs and showed the local time and date when the data was collected. This was an interesting challenge as Google graphs assumes the time is in UTC and does an offset to show it in the browsers local time so I had to find out what that was and do some interesting maths to trick the graphs into showing the correct time and date.

User Authentication

User authentication was implemented so that private sensor boxes could only be seen on the map and the data by the users that owned them, the first of those was a company putting sensors to monitor temperature, humidity, carbon monoxide and ambient sound level in council and rental flats.

API's

Two API's were implemented on the front end system. One to allow user creation and box ownership to be defined from another system, the second was to allow the company putting sensor boxes into flats to request the data collected.

Future API was going to be a general data outlet for the public boxes and private boxes if the user has appropriate authentication using OAuth.

Rate Limiting

All the API's and some of the other end points were rate limited to prevent users from hammering them and slowing the system.

Testing

All API's had tests written for them using Codeception a php testing framework, both success and fail were tested for to ensure a solid result. Eventually tests were going to be written for the rest of the software.

Future Development

Queuing was to be implemented on the data capture using Apache Kafka so that any hold up with the database would not stall the input side and further queues used for distributing the data to other clients.

Data capture servers would be put behind a load balancer to ensure no stalling of the collection.

Lots and lots of visitor utilities for consuming and displaying the air quality data.

I was trying to transfer a MySql database from one hosting to another and no matter what I tried, various encodings in phpMyAdmin, using command line dumps and imports, the accented characters were coming out with £ and other corrupt elements both in the database admin and in the resulting website. The database versions were identical, 5.6, and there was no obvious differences in the installed character encodings.

The main result of this was having to leave the site on the old hosting till I found a solution.

The Solution

In the new year I started playing with a single table to see if I could at least import that without problems, no soap until I just tried importing the data into an already created table and It worked! No character corruptions!

Next I tried it with the whole database:-

1. Exported the original database structure without the data and imported that into the new database.
2. Exported the original database data without the structure definition and imported that into the new database.

That worked as well, I just used the utf-8 as the export and import character set.

Why splitting the structure and data import removed the problems but I suspect that the encoding for the database structure clashed with that for the actual data in some way.

So if you are having problems with moving a MySql database give this a go, do not know if similar things happen with MariaDb but perhaps it might be worth a go with similar problems.

The objective of this article is to help people use strong different passwords in all websites with the help of a password vault.

So why is it important to use strong and different passwords?

No sites security is absolutely perfect so there is always a chance that they will loose user information, and if they have not encrypted or only lightly encrypted the passwords then the hackers have can use those passwords in conjunction with other details like emails to hack other sites of which you might be a member.

Even if they cannot crack the passwords if you use the same weak password for all sites there is a chance that they can guess it and use the other information they gleaned to get in anyway.

Hard Copy

One of the better ways to keep passwords secure is to actually write them down on paper. They still have to be easy to type in and you still have to come up with them but its still better than using something memorable.

One of the down sides of this is keeping track of where they are written, post it notes are OK if nobody or only trusted people are liable to use the room where your computer is located, a note book is better and works for a laptop, just don't loose it.

Password Vaults

This is my preferred method of creating and storing passwords, these programmes are available as plug-ins for browsers and apps for mobile phones so they will work both with websites and phone apps.

How are the passwords stored?

One of the usual questions I get asked about password vaults is what if they are hacked? Well the passwords and other information stored in them are heavily encrypted and only the person whose account that is can read the information using their password, the main upshot is hackers might steal this information but they can't read it.

So onto using a password vault, all the examples are made using Bitwarden, my current vault of choice, however all the others work in a similar way.

Installing Bitwarden

Go to the firefox addons menu or settings->appearance->themes->extensions in Google chrome  and search for Bitwarden and click on Bitwarden - Free Password Manager and follow the prompts to install the plugin.

Once the plugin has been installed you will see the Bitwarden shield icon in the browser bar

click on the shield show the menu

Clicking on Create Account will get you here:

Fill it all in, using a nice long password as its its going to be the only one you need from now on. If you are not sure about remembering the password write it down.

You will now be prompted to login

do that using the email you specified and the long password you created and you should see the vault with no entries as we have not created any yet.

Adding entries to Bitwarden

So now you have an account up and running, once you are logged in the shield turns from grey to blue, and if you have saved passwords for the website displayed you will see this .

Adding entries is pretty simple, just go to a site that you want to save the login credentials for and login as normal and bitwarden will display the following at the top of the screen:

simply clicking "Yes, save now" will add this site to your vault.

Now you might want to improve the sites password, to do this go to the change password option of the site, in the new password boxes either right click and select the bitwarden option and select "generate password (copied)" or click on the bitwarden shield in the browser bar and select the password generator at the bottom.

as you can see you can change the way passwords are generated for those sites that need something different but generally a length of 14 and all the options ticked will do the job.

Past the new copied password into the new password fields on the site and save  it, bitwarden will display at the top of the screen:

clicking on "yes, update now" will save the new unguessable password for the site into Bitwarden's vault.

Now logoff from the site. Logging in now consists of going to the sites login page, clicking on the Bitwarden shield and selecting the entry for your login for the site which will fill the correct credentials.

and then you click login to enter the site.

So now as you use sites you have signed up to you can gradually add them to bitwarden's vault and replace your passwords with stronger ones.

Remember you have to login to Bitwarden, usually on starting your browser, to save site login details and auto fill the login form.

Bitwarden app on mobile devices

One of the great things about using a password vault is that you can use them on multiple platforms such as your phone, pad, all computers and browsers such as firefox, chrome and safari.

On you phone simply go to your app store and look for Bitwarden and install, then connect using the same email and password as you used for your browser plugin and low and behold there are all your site credentials on your phone!

The mobile app will work both for your browser app and other applications such as twitter and facebook so effectively crosses the barrier between those and websites. Another nice thing is that once you have authenticated to login to Bitwarden once you can authenticate using your fingerprint if you have that capability on your phone and never have to use the password again.

Occasionally the the app equivalent for a website won't show the login credentials and you may have to search the vault for the correct ones, you should only have to do this once.

More advanced facilities

As you gain confidence using Bitwarden you will find yourself using the more advanced facilities:-

• Multiple logins for a site
• Editing the the description for vault entries
• Saving text notes for critical information
• Changing the url conditions for particular site credentials
• and many other facilities.

Family, team and enterprise accounts have even more facilities

• A shared the vault if needed
• Much larger storage
• File storage
• Vault health report
• Technical support
• Self hosting
• and many other options

Password Managers

There are quite a few password managers out there, don't be shy try one, if it annoys you export your saved sites and import them into another and try that.

Here is quite a good article on some of the better on Cnet

I have spent a large amount of the last couple of years developing this site and have really enjoyed the process.

See the site.

The site was originally built in Drupal 4 and has a data import back end in bash script, also there was a lot of racing news and other content which is not needed on the new site.

See a pdf of a black type report here.

The process to update the code base to modern standards and improve the bloodstock data was as follows:-

1. The old code for report generation was extracted and put into models and helpers, any embeded HTML removed and old syntax updated to stop all the warnings, notices and errors.
2. Controllers and views were developed to display the reports in clean standards compliant HTML.
3. The back end bash import routines that handle new race, sales and stallion fees were updated to work on modern servers and databases, the originals having been developed on spark servers.
4. Report PDFs are generated from the HTML using specialised style sheets and an external service.
5. Payments for the reports are taken using a multi-currency external service which extends Stripe payments.
6. Additional reports have been developed to give clients better information on foal purchases.
7. Semi-manual routines have also been developed to clean up the horse bloodstock data thus giving a much more accurate results.
8. The site was made mobile friendly.

The main technologies used to replace Drupal were:-

• Codeigniter - a light MVC framework which handles buggy old code with resilience.
• jQuery and jQuery UI for the clean JavaScript needed for the functionality.
• MariaDB - Open source database compatible with MySql.
• Bootstrap - implement mobile friendly web pages.
• Composer - easy external PHP library version control.
• SSP - hardened session and auth handler

The process of improving the the site is still ongoing with additional reports, data cleaning routines and an API to share the information coming in the next year.