Python is a GREAT tool… for prototyping. It's complete lack squiggly brackets, semicolons and dictatorial encouragement of whitespace is probably what launched it into the mainstream as a "the programming language" of the last decade. Combined with the communities RFC like process that enables it to rapidly produce new releases make it almost iOS like.
WHAT DO YOU MEAN YOU'RE NOT USING PYTHON3.6.. err 3.7 YET!?
As programmers though, we get sucked into the hype and tend to model our entire architecture around the hype and hire programers who've drown themselves in the pool of kool-aid. As your application starts to scale, you learn how to do 'multithreading'. More threading becomes more data and with that more customers. As you run faster you start gaining [over] confidence in both your programming ability and your architecture.
Scale
Then there's that one day, you realize the pool you were swimming in is really a kiddie pool. You start googling "python threading sucks". You probably knew at some point you'd need to convert parts of your architecture to something like C, Rust or Go, but that day was far far away. Python had the concept of multithreading and like other pieces of Python it's probably something you can easily learn and bet on. The technical debt starts piling up, just to keep customers happy. Debt that can't easily be transferred to other, more scalable programming languages.
Python is an interpreted language, that's what makes it powerful for prototyping. It's also the thing that makes threading almost IMPOSSIBLE at any sort of scale. That and it's lackluster attempt at concurrency communication (eg: passing messages between threads) simply leaves you with technical debt you can't sustain in the long run. Even if you're one of the 10% who's mastered thread communications in Python, those design patterns rarely translate cleanly to other languages. Use Python for what it's meant for, leave the message passing to the professionals.
As Python3 has matured, so has its multiprocessing framework. This means- that as long as you look at your "multi-threaded" app as a "multi-process" app (.. and use Unix) you're probably going to be OK [in the medium term]. Using this mind-set you avoid all the pitfalls of "livelocks" that can creep up on you while trying to yield your way through a multithreaded application using a single process (interpreter). If you've read this far and are still scratching your head- stick with me here [and bookmark this link for later].
Chances are, you will run into this later in your professional security career. The problem is and your search results will be littered with all the wrong ways to solve this problem. You'll see things like "just write it in Go!" or "just learn how to use the MP framework and pass messages through Queues!". While these solutions work, they're not the right way to think about the problem. You've invested all that time, energy and teamwork in Python, you just need a bridge that gets you to the next step while at the same time, laying the groundwork for the pieces you do need in something like C or go, a design pattern that transcends languages such that you can use the correct language for the job. Most importantly at the correct time.
By using the design patterns in ZeroMQ combined with power of the MP framework- you can accomplish (albeit fairly awkward) multithreading across a larger python application, but in a way that you can re-use that message pattern in other languages. Using ZeroMQ sockets across "threads" (eg: separate processes) puts a few simple things in your favor:
Makes each of these processes independent from the master interpreter.
If a subprocess crashes, it can be re-started.
If a subprocess has a memory leak- it can be killed and restarted, getting your memory back.
Being independent means the master process DOES NOT CARE what language they are written in, just that they communicate cleanly over a ZeroMQ channel.
Gives you some breathing room to scale your application across multiple processes, as well as the time to re-code some of the subprocesses that SHOULD be in another language slowly rather than all at once.
Microservices
Leveraging ZeroMQ from the start of simple things like INPROC or IPC communications (instead of native message queuing), automatically sets your architecture up for cross network scale. Your processes need only switch modes of communication (eg: from an ipc://router.ipc address to a tcp://192.168.1.1:5000 address) as you break pieces of it up across the network. The first step is designing your app around multiprocess first, not threading. At least when it comes to Python.
The neat thing about the PyZMQ framework, it comes with a really neat event-loop and something called a ZMQStream. You can use these patterns to abstract out your own "MyProcess" object that handles both the "multi-process" forks as well as the streaming communication between them. The benefit here- more power, much less code. When you get to that breaking point, where one of the processes needs to be written in C or Go, you can do so without disrupting the entire app architecture. When you're ready to flip the switch you simply tell your app the new address of that process. When things are stable, you can remove the old process code.. when you WANT to, not when you HAVE to.
Need more examples? Check out the latest master of CIFv4…
