aux_clm mizuroute tests with DIFFs from baseline, pass on second attempt

[slevis-lmwg]

Brief summary of bug

DIFFs from baseline appeared in three cprnc.out (clm2.h0a, cpl.hi, mizuroute.h) files:
SMS_D_Ld5.5x5_amazon_rHDMA.I2000Clm60SpMizGs.derecho_gnu.mizuroute-default.GC.0326-172426de_gnu.*.2000-01-06-00000.nc.cprnc.out
BUT the tests passed on the second attempt:

 RMS TWS                              1.3186E+00            NORMALIZED  4.4715E-04
 RMS VOLR                             1.2424E+08            NORMALIZED  2.8238E-01
 RMS VOLRMCH                          1.2424E+08            NORMALIZED  2.8238E-01

 RMS lndExp_Flrr_volr                 1.3186E-03            NORMALIZED  2.8207E-01
 RMS lndExp_Flrr_volrmch              1.3186E-03            NORMALIZED  2.8207E-01
 RMS rofImp_Flrr_volr                 2.7567E+00            NORMALIZED  3.4586E+01
 RMS rofImp_Flrr_volrmch              2.7567E+00            NORMALIZED  3.4586E+01
 RMS rofImp_Forr_rofl                 5.3859E+02            NORMALIZED  2.5794E+00

 RMS basinID                                 NaN            NORMALIZED  0.0000E+00
 RMS reachID                                 NaN            NORMALIZED  0.0000E+00
 RMS time                             5.0000E-01            NORMALIZED  1.1765E-01
 RMS basRunoff                        4.4759E-05            NORMALIZED  8.3166E-01

Same behavior in the intel version of the test.

General bug information

CTSM version you are using: ctsm5.4.028-60-g46199ca79 in PR #3894

Does this bug cause significantly incorrect results in the model's science? No

Configurations affected:

Details of bug

Originally documented in #3894

[samsrabin]

From CTSM SE meeting: Seems like a possible race condition, which means it'll be hard to debug. @ekluzek will talk to Naoki about it. For now, let's just be aware that this might happen during these tests:

• SMS_D_Ld5.5x5_amazon_rHDMA.I2000Clm60SpMizGs.derecho_gnu.mizuroute-default
• SMS_D_Ld5.5x5_amazon_rHDMA.I2000Clm60SpMizGs.derecho_intel.mizuroute-default

A possible clue is that this fails at build time on Izumi nag. Naoki has fixed that so it builds, but @ekluzek needs to test whether that change resolves this issue.

[ekluzek]

I just talked this over with @nmizukami and he hasn't seen this behavior which is good. So we'll just keep this in mind, and see how often we see this again.