Monitoring and maintenance

Here are Embrace recommendations regarding application monitoring. 

Logs

All application logs to monitor are located in: 

Server

/var/log/embrace/

Worker Linux

/var/log/embrace/

Worker Windows

By default, the logs are located in:

C:\Automate-IT\logs

But in case a shared storage has been mounted, the log destination might be configured in the agent.ini under the LOGS_DIR path.

Applications

Server

All the containers in the docker environment should be monitored.

Worker Linux

All the containers in the docker environment should be monitored.

Worker Windows

The following services should always be running: 

• AIT_Agent_EventsSender
• AIT_Agent_QueueService
• AIT_Agent_WebhookJobs
• AIT_Agent_WorkflowPlayer

In case of an Automate-IT deployment, the following processes should always be running: 

C:\Program Files\Automate-it\ait_agent-2.0\node_app_cmd_processor.bat

Machines

Each machine running a Pulse-IT or an Automate-IT application should be monitored on their: 

• CPU Usage: it should not exceed 80%. High usage may indicate a bottleneck or unoptimized workflows.
• RAM Usage and swap Usage: should not exceed 80%. Memory leaks or misconfigurations can lead to crashes.
• Disk usage and I/O: the disk should remain at least 30% free. Disk full or slow I/O can crash apps or degrade performance.
• Network Bandwidth, packet loss. Latency or congestion may affect app responsiveness.

Database

In case the database has been externalized (RDS, Aurora or self hosted), the auto vacuum should be enabled and vacuum should be regularly monitored with the following query. 

SELECT 
    psu.schemaname AS "schema",
    psu.relname AS table_name,
    pg_size_pretty(sum(pg_total_relation_size(psu.relid))) AS include_ndx_gb,
    pg_size_pretty(sum(pg_table_size(psu.relid))) AS whole_table_gb,
    pg_size_pretty(sum(pg_relation_size(psu.relid))) AS not_toasts_gb,
    pg_size_pretty(sum(pg_table_size(psu.relid) - pg_relation_size(psu.relid))) AS toasts_gb,
    pg_size_pretty(sum(pg_indexes_size(psu.relid))) AS indexes_gb,
    to_char(cl.reltuples, '999,999,990') AS row_count
FROM pg_catalog.pg_statio_user_tables psu
JOIN pg_catalog.pg_class cl ON psu.relid = cl.oid
GROUP BY psu.schemaname, psu.relname, cl.reltuples
ORDER BY sum(pg_total_relation_size(psu.relid)) DESC;

If the database has not been externalized, the postgres container automatically takes care of the database maintenance. 

For vacuum statistics, the following command can be ran: 

docker compose exec postgres psql -U ait ait_factory -c "SELECT relname, last_vacuum, last_autovacuum FROM pg_stat_user_tables;" > /tmp/vaccum.log