CANbus data to JSON API

Particle has released the TrackerOne and TrackerSOM, a “a field-ready asset tracking solution” and reference design for a custom tracking solution. The board includes an integrated CAN Bus controller and transceiver able to connect to a car’s OBD-II port.

A couple years ago I built the same thing from parts and an Electron. CAN/OBD-II data comes through the port encoded and the decoding library couldn’t be run on the STM32F205RGT6 120MHz ARM Cortex M3, so I built a service that could be called via a Particle webhook. I’ve been sitting on it for ages, so now I’m making it public:
http://api.greencam.xyz/canbus

The service can be used to translate a single CAN frame or in a batch mode. For single frames, simply include the HEX data in the query parameter:

http://api.greencam.xyz/canbus/single_frame?can_frame=cf00300;d1000014ff0f5e7d

and receive a plaintext result. This method is effective for basic calls from a webhook or script.

To batch process CANbus data, POST the CANbus data as a JSON object, either as a set of frames:

{
    "can_frames": [
    'cf00300;d1000014ff0f5e7d',
    '18fedf00;7da0287d7dfffff0',
    '18feef00;ffffff00067dfffa',
    '18fee000;4624040042240400'
    ]
}

or as a stream separated by \n line endings:

{
    "can_stream": "18fedf00;7da0287d7dfffff0\ncf00300;d1000014ff0f5e7d\n18feef00;ffffff00067dfffa\n18fee000;4624040042240400n",
}

to the URL: http://api.greencam.xyz/canbus/json.

Note: there is a limit of 1000 frames and 200 calls per day or 1 call per second on all of these services.

Feel free to reach out if you’d like to know more, and if you are interested in a tailored tracking solution for more sophisticated applications or original products I am available for consulting or contract.

Flask on Dreamhost Shared Website Hosting

This tutorial will guide you through deploying Flask on a Dreamhost shared plan using Passenger. Dreamhost has first-party technical notes for nearly every step in this process, this walkthrough simply clarifies and connects them into a step-by-step path. Thank you to Matt Carrier‘s excellent tutorial from 2013 for getting me to 90%, this is essentially just an updated version.

Initial Setup

First create a new domain or subdomain on your Dreamhost account – if you reuse a domain, this process will overwrite existing data. (If you don’t own a domain you can register within Dreamhost but a service like Hover.com will make moving easier if/when you outgrow Dreamhost.)

  • Login to your Dreamhost admin panel
  • Go to Domains -> Manage Domains in the left navmenu
  • Click Add Hosting to a Domain / Sub-Domain button on the right.
  • Fill in your Domain to host: with something fun like newproject.mydomain.com.
  • Fill in the next options to your preference.
  • Check the Passenger (Ruby/NodeJS/Python apps only) box. This is important.
  • Click Fully host this domain
  • Wait a few minutes while this domain is created.
  • Refresh the Manage Domains page so you can see the new domain.
  • Under the SECURITY column will be a link HTTPS Not Secure – click this
  • Click the Select this certificate button under the LET’S ENCRYPT SSL CERTIFICATE card. Don’t worry, it is free.

Install Python3

Python 2.x was deprecated January 1, 2020 after a 14 year transition period but Dreamhost still defaults to it for most users. It is also good practice to install a custom, private version to isolate and control your environment from the server’s. The following procedure installs Python3 and changes your user path to default to it. This procedure closely follows Dreamhost’s own support document.

  • SSH into your domain: ssh username@mydomain.com
  • Go to your newly created project’s directory: cd newproject.mydomain.com
  • Execute the following commands in order, one at a time. Some will take several minutes to complete. If a newer version Python3 exists, correct the links and names as appropriate:
mkdir ~/py3_tmp
cd ~/py3_tmp/
wget https://www.python.org/ftp/python/3.8.2/Python-3.8.2.tgz
tar zxvf Python-3.8.2.tgz 
cd Python-3.8.2
./configure --prefix=$HOME/opt/python-3.8.2
make
make install
echo 'export PATH=$HOME/opt/python-3.8.2/bin:$PATH' >> ~/.bash_profile
  • The last command will set the default Python for this user. To activate it, reload your profile: . ~/.bash_profile
  • Now check your work: which python3; double check: python3 --version; and triple-check: pip3 --version. You may delete the py3_tmp directory when you have verified everything.

Install Flask within a virtual environment

Using the private version of Python you’ve just installed, we now create a virtual environment and install Flask. A virtual environment ensures your code, libraries and extensions are isolated from the rest of the applications on the server. Dreamhost also has a KB article walking you through this. In brief:

  • Make sure the latest pip is installed: python3 -m pip install --upgrade pip
  • Install virtualenv: pip3 install virtualenv
  • Check it: which virtualenv
  • Type which python3 and use the output in the next step:
  • Create the venv itself, using the path from the previous step: virtualenv -p /paste/path/from/above/python-3.8.2/bin/python3 venv
  • Activate your venv: source venv/bin/activate
  • and check again: python -V

Your command prompt should now be prepended with the name of the venv, indicating all installations will happen within the venv. Next, Flask itself is easy to install:

  • Make sure pip is updated: pip install --upgrade pip
  • Install Flask: pip install Flask
  • Install any other dependencies you will need, like requests, Flask-Login, Flask-Mail, Flask-SQLAlchemy, etc.

Configure Passenger

Passenger is an open source web and application server and gateway interface. Basically it is the connecting framework for web servers (which Dreamhost provides) to forward requests to web applications or frameworks written in Python (which you will write). Setup is a breeze:

  • create a Passenger configuration file: nano passenger_wsgi.py (feel free to use vi or emacs or your preference. Don’t email me.)
  • Enter the following into passenger_wsgi.py, making sure you replace ‘yourdomain.com’ with your information:
import sys, os
INTERP = os.path.join(os.environ['HOME'], 'yourdomain.com', 'venv', 'bin', 'python3')
if sys.executable != INTERP:
        os.execl(INTERP, INTERP, *sys.argv)
sys.path.append(os.getcwd())

from flask import Flask
application = Flask(__name__)

@application.route('/')
def index():
    return 'Hello from Passenger'
  • Save and close this file.
  • Make the file executable: chmod +x passenger_wsgi.py
  • Create a restart button: mkdir tmp
  • Restart: touch tmp/restart.txt

At this point, everything should be running. Go check it: https://www.<yourdomain>.com will show the text, ‘Hello from Passenger’. If not, double-check everything above.

Create your Flask App

Now we’re going to make a real Flask application. (You will likely want start using your favorite IDE and upload the files to the server when finished rather than editing directly on the server. You will also likely want a local environment and to start a GIT repository. These steps are outside the scope of this tutorial.)

  • Create a directory: mkdir app if you’re still in the server.
  • Create a new file within app/ called __init__.py
  • Enter the following and save:
from flask import Flask

app = Flask(__name__)

from app import routes
  • Create the file routes.py within app/
  • Enter the following and save:
from app import app

@app.route('/')
@app.route('/index')
def index():
    return "Hello from Flask!"
  • Edit passenger_wsgi.py:
import sys, os
INTERP = os.path.join(os.environ['HOME'], 'yourdomain.com', 'venv', 'bin', 'python3')
if sys.executable != INTERP:
        os.execl(INTERP, INTERP, *sys.argv)
sys.path.append(os.getcwd())

sys.path.append('app')
from app import app as application
  • The project structure should look like this:

mydomain.com/
----venv/
----app/
--------init.py
--------routes.py
----passenger_wsgi.py

  • Restart Flask: touch tmp/restart.txt
  • And now go check your application again – you should see ‘Hello from Flask!’

Build

You now have a fully functional, if trivial, Flask app running on Dreamhost Shared webhosting. You can make some amazing software with SQL backends and complex frontends from here – I suggest Miguel Grinberg’s Flask Mega-Tutorial for the basics, or for data science applications, Data Visualization with Python and JavaScript is an excellent resource.

Just One State – immigration lies

My grandmother recently forwarded an email that has been going around the internet. Snopes is my usual source of fact-checking but it, along with about a half dozen other debunking sites, didn’t do this particular chain justice, despite this chain having been around for ~15 years. This was the best of the bunch, which makes sense as the LA Times was, inaccurately, listed as the source of the statistics. So I checked every statement myself. Below is my letter to my grandmother:

Continue reading

Chicago Bus App

Chicago Bus App

Chicago is a Big Data city.

https://data.cityofchicago.org serves 914 distinct, publicly accessible data sets including Beach Water Quality, Rat Infestations, Employee names and salaries. Chicago is “dedicated to promoting access to government data and encouraging the development of creative tools to engage and serve Chicago’s diverse community”. They have a Github site, video tutorials and sample API code. It is an immense data store!

Continue reading

0.6% of my Income

WatchSalaries
Something tickled my pattern matching circuits while reading the countless articles analyzing the pricing of the Apple Watch and Apple’s strategy behind it.

Of the 38ish distinct Apple Watch models there are, to me, only five major style variations (ignoring size): the basic aluminum Sport, at $399, less or more depending on sizes and bands. The $699 steel Watch with classic, leather or modern bands. Upgrade to a steel link band for $999 ($1099 in black). Then there is the gold Edition starting at $10,000 and going all the way to $17,000.

Continue reading

Solar Panel Analysis

I like solar power. I’ve purchased over a dozen consumer solar panels from Brunton to PowerMonkey to Goal Zero in a search of a favorite. Goal Zero’s older Nomad7 (purchased at a nice discount from TheClymb) is the newest leader.

But this isn’t a review of the Goal Zero; others have written excellent solar panel reviews, notably The Wirecutter and OutdoorGearLab. What I want to know is how to get the best performance, how to use it most effectively, to characterize it. What conditions produce best charge? What charge is produced under average conditions?

Continue reading

Kilimanjaro

No moon, no stars, only blackness. Freezing, failing headlamps reveal only a couple of meters of terrain around me. The 60kmph wind howls, cutting off speech and chilling us through to the core; it supports my weight when I lean into it, threatens to topple me if I don’t. The bitter cold begins to affect me: My fingers, my nose are frozen and it creeps into my arms, threatening my core. One foot in front of the other, I trudge onward.Only a few days ago it was sunny and hot; beautiful flowers, trees, birds, monkeys. Then, I could talk to my friends, chat about inconsequential topics, learn Swahili, point out the landscape and features to one another. It was nice in the rain forest. Now there is only rock: black, pitted and dusty lava rock from this sleeping volcano. Now I can hear nothing but the howling wind; we do not attempt to speak for the effort required. Only a few words at the occasional rest stop, mumbled breathlessly through cracked and frozen lips. We must keep our rhythm, must keep moving to stay warm. We must keep moving.

Continue reading

Burden of hope

Ice, extending behind us as far as we could see, crunched under our crampon laden feet. It had been a long day — 12 km across the glacier – but now the edge was in sight, and just beyond that a boat bobbed in the ocean awaiting our return.

We stripped the crampons off, stowed our ice axes and walked across the dirty glacial edge to the pier, then down to the boat. The wind picked up on the water but we couldn’t tear our eyes from the massive 20m cliff of ice to head below decks.

Our guide had disappeared, leaving us to meditate the sight, but soon reappeared bearing a small tray with glasses. Handing one to each of us, he explained: “Argentine Whiskey, the best in the world. That ice – it is 300 years old and taken from the center of the glacier we just crossed.”

As we sipped, Stefanie’s eyes met mine with the same thought: it couldn’t get better than this. The next two weeks would prove us wrong.


Whiskey

More Photos

And more.