Interrupting Python Threads
I'm mulling things over as to how best to interrupt threads with Python using a KeyboardInterrupt and what effectively happens to them whenever you do.
1 The Use Case of that Multithreaded OpenStack Shell I Wrote
In that multithreaded OpenStack shell newstar I wrote, I have a number of threads getting server names from a Queue and running nova show in a subprocess to display information about them. The main thread waits for the Queue to be consumed with Queue.join(). The problem is that Queue.join() cannot be interrupted by a KeyboardInterrupt. Even a SIGINT with a custom handler won't run.
2 Interrupting Sleep Instead
A common suggestion is to have an infinite loop repeatedly wait with time.sleep() and to check if the Queue is empty, as time.sleep() is interruptible. Simply add a Queue.join() call after the loop to make sure it's drained:
try:
while not queue.empty():
time.sleep(10) # Interruptible
except KeyboardInterrupt:
while not queue.empty():
queue.get()
queue.task_done()
queue.join()Note that Thread.join() isn't normally interruptible, but it is at any time when supplied with a timeout argument. It's likely that it simply uses time.sleep() too behind the scenes.
3 What Happens When KeyboardInterrupts Are Issued?
In the beginning I thought that a KeyboardInterrupt would go to the threads currently being joined by the main thread, and any of its running subprocesses, instead of reaching the main thread. I thought I could have some level of control on this by forking and detaching the process from the terminal.
But in fact, a KeyboardInterrupt really hits all the threads – including the main one. The only reason it didn't seem to have any effect on it was simply because the main thread was stuck in that uninterruptible Queue.join().