Performance monitoring best practices

When you design a servers performance monitoring system there are several things that you will have to consider. Best practices when implementing such systems are:

  • Set up a monitoring configuration
  • Keep monitoring overhead low
  • Centralized place for monitoring
  • Analyze performance results and establish a performance baseline
  • Set alerts
  • Tune performance
  • Plan ahead

When setting up a monitoring system you have to consider what kind of system is “good enough” for you. You will have to decide if you go with an opensource monitoring system or if you go with a commercial system. Since i’m not a fan of commercial closed source system i will focus on opensource solutions:

  1. Nagios – Nagios is a powerful monitoring system that enables organizations to identify and resolve IT infrastructure problems before they affect critical business processes
  2. Cacti – Cacti is a complete network graphing solution designed to harness the power of RRDTool‘s data storage and graphing functionality. Cacti provides a fast poller, advanced graph templating, multiple data acquisition methods, and user management features out of the box.
  3. Munin – Munin the monitoring tool surveys all your computers and remembers what it saw. It presents all the information in graphs through a web interface. Its emphasis is on plug and play capabilities.
  4. You may also want to take a look at http://compari.tech/bandwidthmonitoring for some other useful bandwidth monitoring tools

Nagios has the advantage that it can be set up to send SMS alerts to predefined groups of users in case of alerts. Cacti has the advantage that you can evaluate in time how your systems performs and you can have a good idea of the trends. Munin can monitor certain aspects better than Cacti but is more invasive on the systems you install it on.

Next thing will be to keep the monitoring overhead low. This can be done by :

  1. Don’t query the servers too often.
  2. Monitoring system should be tun on a standalone server that does monitoring and nothing else.
  3. Archive unneeded data.
  4. Use asynchronous requests when possible

On previous point i said that monitoring system should run on a standalone server . This means exactly Centralized place for monitoring .

Ideally, all logs from different areas of monitoring should be stored in a centralized place where one UI can be used to analyze the data. Based on your user scenarios, consider identifying which teams to partner with, so log data can be viewed as a coherent whole. The reasons behind centralization are:

  1. You can easy implement a strict user control / user policy / procedures ( You will need it if you need  Sarbanes-Oxley compliant )
  2. Minimize the admin time. Imagine that you have 20 servers and each one with it’s own monitoring system.
  3. Giving access to some users on relevant graphs / logs is easy
  4. You can get an overview on the whole system

After you implemented the system and data starts to pile up you can do an analysis of performance results . This should be done as often as possible in order to identify trends and also to catch “exceptions”. For example at the end of each month servers that runs accounting will have increased load than on a normal day. If you do not pay attention you might find yourself in pretty delicate position when users requests more capacity or more processing power and according to trend it wasn’t necessary.

After getting a base line for the performance you can Set alerts for moments that systems behave out of the ordinary or for problems with the system. For example if a server uses 15G RAM out of 16G RAM you might want to be notified about that to schedule a downtime to add more RAM or to see what is going on with the applications running on that server.

Performance tunning is a delicate job and take an awful lot of time. Because a system can be optimized according to a scenario. If the data doesn’t fit in that scenario you might need to adjust servers parameters in order to adapt to the scenario. Databases, apache servers, kernel parameters can be tuned to suit your needs.

Also the baseline and graphs of the performance allows you to Plan ahead the evolution of your systems. For example you can predict with good accuracy when or if your will need to purchase new hardware or when you will need to upgrade your existing systems.

Adding Oracle support to PHP

If you want to connect to an Oracle database with PHP you will need the pecl module named oci8

First in order to compile it you will need Oracle Instant Client ( both basic & sdk ) . You can download them from here:

http://www.oracle.com/technology/software/tech/oci/instantclient/htdocs/linuxsoft.html

If you don’t have an Oracle account you will need to create one.

At the time of this post the following files are available: instantclient-basic-linux32-11.2.0.1.zip , instantclient-sdk-linux32-11.2.0.1.zip .

Create a directory in /opt  mkdir /opt/oracle/instantclient and copy those files there . Then unzip them . You will need to create a symbolic link ln -s libclntsh.so.11.1 libclntsh.so

Then you will need to install libaio if you don’t have it already . Don’t forget to add /opt/oracle/instantclient/instantclient_11_2 to /etc/ld.so.conf .

At this point you are ready to install the pecl extension to php.Create a temporary directory ( /tmp/1 ) and cd there.

mkdir /tmp/1

cd /tmp/1

pecl download oci8

tar xf oci8-1.3.5.tar

cd oci8-1.3.5

phpize

./configure –with-oci8=shared,instantclient,/opt/oracle/instantclient/instantclient_11_2

make

make install

edit /etc/php.ini and add

extension=oci8.so

And enjoy oracle extension for php. Note: if you have apache running restart it.


New CMS needed for tfm.ro

tfm.ro needs a new face. in the beginning we started with an inhouse mini CMS. Then we switched to mediawiki and focused on end user documentation and howto. At this point we use wordpress with Structure theme .

What is not ok:

  • Community options are limited
  • User generated content is limited to comments
  • Integration with the development site is done only via an RSS feed
  • Looks old and dusty
  • We focus on development and by doing this the site is most of informations are old and sometime obsolete

So ,  i’m begining to search for an alternative that will have to provide:

  • community options ( registration , posibility to upload pictures , patches , docs etc )
  • bug tracker integrated with the main site
  • online manuals creation / gathering
  • forums

Do you know what options are out there ?

About events

How NOT to take part in the events:

  1. If you want to win, don’t bother.
  2. If you think events are something you should get, because you are kewl, don’t bother.
  3. If you think that prizes in events are for free, you’re wrong and don’t bother.
  4. If you want to take part in event just to bitch at prizes, DO NOT bother.
  5. If you think that the hints should give you precise walkthrough, guess what? – don’t bother!
  6. If you are not used to use the content of you head to the purposes Mother Nature created it – don’t bother!
  7. If you’re not able to read and understand hints – don’t bother!
  8. If you think you’re too 1337 for events, don’t bother!

How NOT to host an event:

  1. If you think that everything will go as you planned, you’re wrong.
  2. If you think that everybody will be happy with prizez, you’re wrong.
  3. If you think contesters will understand your hints and follow them, you’re wrong.
  4. If you hope to hear a ‘thank you’, you’re optimists.
  5. If you are not ready to read complaints, bitching and ‘This should be done this way, not as it was’, contests are not for you.

How to take part in the events:

  1. If you want to take part in the event for fun, not prizes. You’re precious, hope you will do it each time.
  2. If you do use your brain, read hints and do not whine – hope you will take part in all events.
  3. If you’re able to have fun without getting prize – :thumbs up:
  4. If you are able to give sensible critic with good arguments, I bet contest holders will love you.

How to host an event:

  1. Be ready for frustration.
  2. Be ready for bitching.
  3. Be ready for disappointment.
  4. Be ready for nobody understanding your hints.
  5. Be ready for huge costs.
  6. Be ready for everything going wrong.
  7. You know what? SCREW it, go and have a beer with friends. It’s surelly better idea.