Real monitoring, debugging, logging, metrics for real programmers for real production servers

Here are my current thoughts on server monitoring in production for software developers.

So, you plan to publish your server to production (you use docker right?), you have some logging, some monitoring, are you really aware of what you can do with monitoring? what should go to logging? what to audit? jmx? REST? web interface? bewildered by all these options? so are we!

Here is a brief summary of which monitoring tool to  use, prefer to use all of them ofcourse.

When to use
SNMP Alerts
Critical status change, Without false alarms, Prefer simplistic
Email notifications
Medium / High level severity status changes
Hadoop events
All (everything audited to hadoop)
Tweaking system (change log level) internal system info
Main metrics
All (+ability to TRACE)
Admin WEB UI
Global system status, Server Components startup status, have here “support” people in mind.
Java Melody
general app statistics
Gathering statistics + System status
Cross server global system status
Google docs
Procedures & captains log of events
expose your metrics via rest so that QA can gather them, you can create automatic processes to monitoring them
expose a simple /isalive that will do nothing simply check whether your server is alive.  you can utilize it for simple response time, maybe you have some network issues, this will tell you how much time it takes to access your server without any logic done.
Web Flow Tracer
Allow you to see the flow of what happens in your server for a specific request.  Combination of logback + MDC + Filters + in memory appender.