Skip to content

Analysis library crash on exit #908

Open
Open
Analysis library crash on exit#908
@sdobbs

Description

@sdobbs
sdobbs
opened on Mar 19, 2025

Rarely, in REST production test runs, there are segfaults as the process is terminating. It looks like this is in the ANALYSIS library.

Example output files are in this directory, and the stack trace is given below:
/volatile/halld/home/gxproj4/jobs/ver13-perl/job_info_120400_191

17:24:16.021 [fatal] Segfault detected!
17:24:16.022 [fatal] Hard exit due to segmentation fault! Backtrace:
17:24:16.022 [fatal] 
17:24:16.022 [fatal]    0: DAnalysis::DSourceComboTimeHandler::Fill_Histograms()
17:24:16.022 [fatal]       hd_root:6076f0
17:24:16.022 [fatal] 
17:24:16.022 [fatal]    1: DAnalysis::DSourceComboer::~DSourceComboer()
17:24:16.022 [fatal]       hd_root:4127a7
17:24:16.022 [fatal] 
17:24:16.022 [fatal]    2: DAnalysisResults_factory::~DAnalysisResults_factory()
17:24:16.022 [fatal]       hd_root:431132
17:24:16.022 [fatal] 
17:24:16.022 [fatal]    3: DAnalysisResults_factory::~DAnalysisResults_factory()
17:24:16.022 [fatal]       hd_root:431799
17:24:16.022 [fatal] 
17:24:16.022 [fatal]    4: JFactorySet::~JFactorySet()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1f9997
17:24:16.022 [fatal] 
17:24:16.022 [fatal]    5: JFactorySet::~JFactorySet()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1f9a82
17:24:16.022 [fatal] 
17:24:16.022 [fatal]    6: JEvent::~JEvent()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1eb6f5
17:24:16.022 [fatal] 
17:24:16.022 [fatal]    7: void __gnu_cxx::new_allocator::destroy(JEvent*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:22d749
17:24:16.022 [fatal] 
17:24:16.022 [fatal]    8: void std::allocator_traits >::destroy(std::allocator&, JEvent*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:22d715
17:24:16.022 [fatal] 
17:24:16.022 [fatal]    9: std::_Sp_counted_ptr_inplace, (__gnu_cxx::_Lock_policy)2>::_M_dispose()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:22d603
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   10: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
17:24:16.022 [fatal]       hd_root:3c811a
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   11: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d7fc9
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   12: std::__shared_ptr::~__shared_ptr()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1f4850
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   13: std::shared_ptr::~shared_ptr()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1f4892
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   14: void std::_Destroy >(std::shared_ptr*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:22b6d5
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   15: void std::_Destroy_aux::__destroy*>(std::shared_ptr*, std::shared_ptr*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:22a4c0
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   16: void std::_Destroy*>(std::shared_ptr*, std::shared_ptr*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:228836
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   17: void std::_Destroy*, std::shared_ptr >(std::shared_ptr*, std::shared_ptr*, std::allocator >&)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:226b4d
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   18: std::vector, std::allocator > >::~vector()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:21fa07
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   19: JEventPool::~JEventPool()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:22d526
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   20: JEventPool::~JEventPool()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:22d54e
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   21: JTopologyBuilder::~JTopologyBuilder()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:218519
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   22: void __gnu_cxx::new_allocator::destroy(JTopologyBuilder*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1eace1
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   23: void std::allocator_traits >::destroy(std::allocator&, JTopologyBuilder*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e9861
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   24: std::_Sp_counted_ptr_inplace, (__gnu_cxx::_Lock_policy)2>::_M_dispose()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e883d
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   25: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
17:24:16.022 [fatal]       hd_root:3c811a
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   26: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d7fc9
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   27: std::__shared_ptr::~__shared_ptr()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d580c
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   28: std::shared_ptr::~shared_ptr()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d5828
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   29: jana::components::JComponent::Service::~Service()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d585a
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   30: JExecutionEngine::~JExecutionEngine()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:20bde3
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   31: JExecutionEngine::~JExecutionEngine()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:20be0a
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   32: std::default_delete::operator()(JExecutionEngine*) const
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1dd23e
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   33: std::_Sp_counted_deleter, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e8f76
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   34: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
17:24:16.022 [fatal]       hd_root:3c811a
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   35: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d7fc9
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   36: std::__shared_ptr::~__shared_ptr()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d9e90
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   37: std::shared_ptr::~shared_ptr()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d9ed2
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   38: std::pair >::~pair()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e6818
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   39: void __gnu_cxx::new_allocator > > >::destroy > >(std::pair >*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e6838
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   40: void std::allocator_traits > > > >::destroy > >(std::allocator > > >&, std::pair >*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e5459
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   41: std::_Rb_tree >, std::_Select1st > >, std::less, std::allocator > > >::_M_destroy_node(std::_Rb_tree_node > >*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e3609
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   42: std::_Rb_tree >, std::_Select1st > >, std::less, std::allocator > > >::_M_drop_node(std::_Rb_tree_node > >*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e18c9
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   43: std::_Rb_tree >, std::_Select1st > >, std::less, std::allocator > > >::_M_erase(std::_Rb_tree_node > >*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1dfca5
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   44: std::_Rb_tree >, std::_Select1st > >, std::less, std::allocator > > >::~_Rb_tree()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1dd068
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   45: std::map, std::less, std::allocator > > >::~map()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d9098
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   46: JServiceLocator::~JServiceLocator()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1dcc7a
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   47: std::default_delete::operator()(JServiceLocator*) const
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1dcca0
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   48: std::unique_ptr >::~unique_ptr()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d8dc4
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   49: JApplication::~JApplication()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d2524
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   50: main
17:24:16.022 [fatal]       hd_root:3a4505
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   51: ???
17:24:16.022 [fatal]       /lib64/libc.so.6:295d0
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   52: __libc_start_main
17:24:16.022 [fatal]       /lib64/libc.so.6:29680
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   53: _start
17:24:16.022 [fatal]       hd_root:3b1c85
17:24:16.022 [fatal]

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions