Life in Factory Automation
Code->Compile->Run, everyone knows that’s how programmers work. We write stuff, run it through a toolchain, and when satisfied / tested, push to the production system for deployment and restart the app.
It has ever thus been this way. Except where it’s not, and never has been.
There is a place where code runs all the time, and code changes are pushed into the flow as it’s running. Updates take a few milliseconds and suddenly the new code is in use, without any interruption or delay.
Where is this place ?
In manufacturing, specifically in the world of PLCs, or Programmable Logic Controllers — these are the specialized industrial computers that control all the machinery that builds all the world’s stuff.
The PLCs run specialized programming languages, usually Ladder Logic, with a looping flow where all logic is run/analyzed every 5–50ms. For our purposes PLC’s most interesting characteristic is to update their code while they are running — this usually means while the PLC-controlled machinery is running, too. A bit scary.
These PLCs are often controlling critical functions, and their failure or bad code can mean disaster, e.g. damage, explosions, injury, and even loss of life. This is serious business, and probably that last place you’d expect to see code updated on the fly. Especially with code controlling megawatts of power, hundreds of tons of machinery, flammable gas, radiation, and more.
But that’s where we do it, all the time (maybe a bit too much).
Why ?
First, because we can ;) but more importantly because in many instances, we have no opportunity to stop the programs, for this means stopping the machines or the whole factory.
There are numerous industries and processes where this is not safe, nor realistic for a number of reasons (though we do often wait for a ‘pause opportunity’ to make the largest or most dangerous changes).
So, how does it work ?
We usually ‘edit’ our programs in specialized visual editors on laptops that are remotely connected to the running PLCs. We can inspect, test, and even simulate various actions and logic on the PC before we dare try it in the PLC — though in the vast majority of situations, there is just no test environment, nor is one even possible due to the hundreds of inputs and complex logic.
Once our edits or changes are done, we are ready to push to the running PLC. In some PLCs, these changes take effect immediately, but in the best systems, the changes are uploaded to the PLC, but not active until we are ready.
When we think we are ready, the people are clear, and we are prepared, we push the button to make the changes live. The PLC will switch its running code to the new version, and execute it within milliseconds.
If we sense something is wrong, we can push a button to ‘back-out’ the changes and the PLC will switch back to the old code. Once we are satisfied all is okay, we have a button that finally commits the code to the permanent program. It can be quite scary to do this on a big and complex system when there are big changes to be made.
Bad things can and do happen, and it’s a good day when you avoid damaging products, people, or machinery. Overall it’s a pretty robust process and engineers both know their systems well and are usually very careful, though you’d be amazed what we’ll do at 3am to keep things running.
All developers should play with PLCs and Ladder Logic some time in their careers, as a way to expand their horizons across colorful problem domains.
In the end, it’s an exciting technology to work with, and it’s always good for traditional app or web programmers to know how others live and work, that not all coding and computing environments are the same.
It is a bit of life on the edge (not the only way old factory guys lived on the edge), but interesting and does make you be more careful — some day I’ll write about how I forgot a 0 in a formula and a 100 pound motor jumped off the floor and nearly took me out …