Always look for what is missing to work better

DateTime.AfterPost - DateTime.Now = 3 minutes

One of the most common complaints I hear where I work is that the developers don’t have enough coding assignments. 
I can understand that and I too felt sometimes we do a lot of “maintenance” work and less real-time coding.

In 2018 I was in charge of a big project that had very few programming in it. I loved that project because it introduced me to the real power of VMs (You can read about it here).Moreover, that project allowed me to see what was missing from the system we were upgrading.

We didn’t have real-time statistics on what was happening in the system.
We couldn’t say, easily, what was the state of the system, and how much our upgrade improved (or not) the system.
To do that, I had to go to production, copy the relevant info from the DBs, open it in my computer, do some magic in excel and print the charts. Agonizing.

There is a chart in xkcd that I find very useful when thinking if there is a need to automate something or not: https://xkcd.com/1205/.

I quickly realized I will be asked many more times to show updated charts, so I had to automate it. I wrote a simple Stored Procedure in the DB (SQL Server) that will export the data to an XML file:

select @sql = 'bcp "exec <stored procedure>" queryout c:\temp\test.txt -c -t -T -S' + @@servername
Where <stored procedure> is the name of the actual stored procedure that brings the data you need.

After that, I moved the file from production to development and made a simple program that will import it to another database (made only for statistics).
Yay! I had all my data in production so now I can copy it from the DB, paste it in excel and show the updated data each time they needed it!

But that’s not Great! That’s merely OK.

I talked to our DevOps guy and told him I have the data in the DB and ask him what can I do with it? I couldn’t even finish my sentence when he immediately said: use Kibana or Grafana.

Without knowing what that means, I downloaded both and try to play with them. Kibana didn’t run so smoothly so I skipped it and tried Grafana.
The installation was very easy (I’m using windows), and in a few clicks, I connected it to the DB. In less than 5 minutes I saw a beautiful graph with all my data! That’s Great.

This all happened January 2019, nowadays we have more than 5 dashboards with dozens of charts showing statistics of all of our systems.
More than once it helped us realize issues before any alarms went off.
Moreover, we have a TV in our room constantly showing the dashboards.

Soon enough, this small side-project I started became a “real” project and we received full support of our boss to work on it (more than support actually, he REALLY liked the graphs). The added value this graphs gave our systems was huge. We had to write many different small programs to export and import the data from all the different system, each one working different and saving its data differently, and we had to work smart so to not explode the DBs.

Thanks to this, we created an entire project “out of nowhere”, where the team and the managers were the clients. The first thing we now do to any new system we encounter, is to export all the data we can from it to learn as much as we can and have live feed from it.
We are now adding some Anomaly Detection to our data so as to be more intelligent and try to get ahead of any issues.

This project gave me a lot of coding work and I enjoy it a lot up until this day. I started working on it off-hours and when it started to show results I was encouraged to keep working on it.

The main point for this blog, besides telling you a bit about Grafana, which is awesome and I encourage you to try it if you have data laying around (not only SQL, nearly anything can be a source for Grafana) is that there is always something to do and to innovate, especially in dinosaur-aged systems.

You can look at the system and say “meh – it works, no need to touch it” or try to list all the things you can do to make it better.

There is a term in Hebrew – “Big Head”. As opposed to “Small Head” people, “Big Head” people don’t settle on doing only what they are told to do (like “Small Head” people) but try to think what else they can do. What more they can add.

In my opinion, Small head people don’t get too far in life, they get stuck behind. When encountered with an unexpected problem, Big head people will be the one to succeed and life is a journey full of unexpected problems and obstacles.

Comments: 3

  1. […] After that, we changed the whole system to be virtualized (VMWare ESXi as hosts). This simple (it was really simple) upgrade opened a lot of doors and brought this old 90s system to the 21st century.For example, we were able to add real-time statistics monitoring and business logic monitoring to the system. You can read more about that here. […]

  2. […] When I was confronted with the request to see live updates on our systems, I could have developed a system that reads from a database and shows graphs. That would have taken me weeks and it would be nothing compared to Grafana. […]

  3. […] The most fast and easy way I saw was to build a DB containing all the books (so I can show in my Grafana) and if you wanted to take a book, I would change the book’s status. After two days that two […]

Add your comment