ED2 KC- Relation "analytics" does not exist & Null Pointer Exception
Background
This means that Analytics failed. In this case, it was because the Instance was too small and did not have enough memory to complete the index generation step of Analytics, then it crashed.
Check the /system/tasks API to get a better sense of when/why – this is likely where you’ll see the “Null Pointer Exception” Error. Or, you may get an exact error. When checking, I saw only two Analytics Tasks, even though the logs showed more succesful runs. This tells me that Tomcat restarted at some point. The tasks API will hold up 99 task events showing the stages of analytics but is cleared when tomcat restarts.
First, check the DHIS2 GUI to see what sort of parameters / schedule were being used to run analytics, since there was no cron job.
Are they using “regular” analytics or Continuous? That should be apparent in the tasks API as well.
Resolution
In the examples below, the resolution is to size up the server to add more resources to it because it is what caused the error. However, the resolution of the actual issue is to regenerate resource and analytics tables.
Check the Logs (examples)
dhis-analytics-table.log
To quickly see that analytics was completing as expected a lot of the time
nested exception is org.postgresql.util.PSQLException: ERROR: relation "analytics" does not exist
Jul 7 15:26:24 eda91631 server: Position: 97
INFO 2021-07-07T15:10:17,016 Session timeout set to 3600 seconds (DefaultAuthenticationSuccessHandler.java [localhost-startStop-1])
* INFO 2021-07-07T15:10:19,592 No Gateway configuration not found (DefaultGatewayAdministrationService.java [localhost-startStop-1])
* INFO 2021-07-07T15:10:21,228 Executing startup routine [1 of 10, runlevel 2]: defaultAdminUserPopulator (DefaultStartupRoutineExecutor.java [localhost-startStop-1])
* INFO 2021-07-07T15:10:21,244 Executing startup routine [2 of 10, runlevel 3]: PeriodTypePopulator (DefaultStartupRoutineExecutor.java [localhost-startStop-1])
* INFO 2021-07-07T15:10:21,288 Executing startup routine [3 of 10, runlevel 3]: PeriodTypePopulator (DefaultStartupRoutineExecutor.java [localhost-startStop-1])
/var/log/messages.log
To track down the restart after the anticipated tomcat crash, then scroll up through all the other messages in that log to find the precursor out of memory error event shortly before the crash.
Jul 7 15:08:56 eda91631 kernel: Out of memory: Kill process 27594 (java) score 475 or sacrifice child
Jul 7 15:08:56 eda91631 kernel: Killed process 27594 (java) total-vm:4808556kB, anon-rss:1841100kB, file-rss:0kB, shmem-rss:0kB
Jul 7 15:08:56 eda91631 systemd: tomcat.service: main process exited, code=killed, status=9/KILL
Jul 7 15:08:57 eda91631 systemd: Unit tomcat.service entered failed state.
Jul 7 15:00:47 eda91631 server: * INFO 2021-07-07T15:00:47,509 No of analytics table indexes: 623 (DefaultAnalyticsTableService.java [taskScheduler
-23])
Jul 7 15:01:22 eda91631 systemd: Created slice User Slice of root.
Jul 7 15:01:23 eda91631 systemd: Started Session 532 of user root.
Jul 7 15:01:34 eda91631 systemd: Removed slice User Slice of root.
Jul 7 15:08:56 eda91631 kernel: java invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0