Identifying producer/consumer scenarios in the wild

I've been working on a data import process the past couple of days, trying to solve some memory issues (OOMEs).  Essential we have a reader (the producer) and a writer (the consumer).  The writer part of this scenario operates much slower than the reader part.  The reader part is implemented as an iterator, so it only produces enough work for the writer to consume.  As this design evolved over time, parallel execution of the writer was added, in an effort to speed up the overall writing process.  The coordination of the parallelization is an ExecutorService implementation.  With this executor service now in place, the iteration of the reader can operate independently of the writer.  Thus, the consumer now starts creating lots of tasks that are then submitted to the executor service, where they queue up.  The executor service is not bounded, so it just keeps accepting tasks.  This wouldn't be a problem if the number of tasks were small and the memory footprint of those tasks was low, but that is not our situation.  Thus, we keep blowing out our Java VM process with OOMEs.  We're in the process of fixing this issue, using a bounded concurrent collection to handle the buffering of items between the reader and the executor service and ultimately the writer.

 

In support of Groovy's dynamic method dispatch

Came across a design issue today where we really could have used Groovy's dynamic dispatch and multi-method support to invoke the proper method based on the interrogation of the runtime type of an object instance passed as a parameter to the method.  This blog posting by MrHaki gives a great description of how Groovy solves this problem without the need of a double dispatch pattern implementation.  We're writing our code in Java, so we don't get the Groovy goodness of dynamic method dispatch and multi-methods.  We resorted to a double dispatch implementation to solve our problem.  We did not use the instanceof operator and a whole bunch of conditionals.

Book review: Clean Coder

Clean Coder, The: A Code of Conduct for Professional Programmers by Robert C. Martin

My rating: 3 of 5 stars


Decent follow-up to Clean Code. Much of the book has been talked about in other avenues. The professional attitude discussion was decent, but mostly common sense. I liked the book, but have seen the content before. Nothing really new here. Definitely meant more for new developers and would wholeheartedly recommend it for that group of people.



View all my reviews

Book review: The Great Derangement

The Great Derangement: A Terrifying True Story of War, Politics, and Religion at the Twilight of the American Empire by Matt Taibbi

My rating: 3 of 5 stars


Not nearly as good at Griftopia, also by Taibbi. Kind of wanders and there doesn't seem to be any real plot to this book. Kind of wonder if Matt was still on drugs when he wrote this thing. The religion parts were semi-interesting, the political insights were boring. Can't recommend this. Some good parts, but the whole doesn't make for a good read.



View all my reviews