Re: invalid free in XrdProofdProofServMgr::Recover()

    Dear Thiemo,

    I am not really able to reproduce the problem with your files. Can you please switch full debugging with the directive

    xpd.trace all

    and send me the logfile?

    Thanks,

    Gerri

Thiemo Nagel wrote:
> Hello,
>
> I stumbled upon a reproducible invalid free in
> XrdProofdProofServMgr::Recover(). A valgrind transcript of the xrootd
> session is below. It seems that something was corrupted in my
> /tmp/.x* directories because moving them out of the way resolved the
> problem. (I have attached a tarball of these, in case you're
> interested.)
>
> Kind regards,
>
> Thiemo
>
>
>
> Copr. 2007 Stanford University, xrootd version 2.9.0 build 20081122-0000
> ++++++ xrootd protocol initialization started.
> Config warning: 'xrootd.seclib' not specified; strong authentication
> disabled!
> Config warning: 'xrootd.fslib' not specified; using native file system.
> Copr. 2007 Stanford University/SLAC sfs (Standard File System) v 9.0n
> Config warning: 'xrootd.prepare logdir' not specified; prepare
> tracking disabled.
> Config warning: only '/tmp' will be exported.
> ------ xrootd protocol initialization completed.
> 090122 16:14:28 001 xpd-I: Manager::Config: configuring
> 090122 16:14:28 001 xpd-I: Manager::Config: listening on port 1093
> 090122 16:14:28 001 xpd-I: Manager::Config: using temp dir: /tmp
> 090122 16:14:28 001 xpd-I: Manager::Config: role set to: any
> 090122 16:14:28 001 xpd-I: Manager::Config: admin path set to:
> /tmp/.xproofd.1093
> 090122 16:14:28 001 xpd-I: Manager::Config: PROOF pool:
> root://nailgun.e18.physik.tu-muenchen.de
> 090122 16:14:28 001 xpd-I: Manager::Config: PROOF pool namespace:
> /proofpool
> 090122 16:14:28 001 xpd-I: Manager::LoadScheduler: scheduler loaded:
> type: default
> 090122 16:14:28 001 xpd-I: Manager::Config: user config files are
> disabled
> 090122 16:14:28 001 xpd-I: Manager::Config: list of superusers: tnagel
> 090122 16:14:28 001 xpd-I: Manager::Config: bare lib path for
> proofserv: /usr/lib/debug:/usr/X11R6/lib/debug
> 090122 16:14:28 001 xpd-I: Group::Print: +++ Group: default
> 090122 16:14:28 001 xpd-I: Group::Print: +++ Priority: -1, fraction: -1
> 090122 16:14:28 001 xpd-I: Group::Print: +++ End of Group: default
> 090122 16:14:29 001 xpd-I: NetMgr::Config: configuring
> 090122 16:14:29 001 xpd-I: NetMgr::Config: PROOF config file: none
> 090122 16:14:29 001 xpd-I: NetMgr::Config: 4 worker nodes defined
> 090122 16:14:29 001 xpd-I: PriorityMgr::Config: configuring
> 090122 16:14:29 001 xpd-I: PriorityMgr::Config: no priority changes
> requested
> 090122 16:14:29 001 xpd-I: PriorityMgr::Config: poller thread started
> 090122 16:14:29 001 xpd-I: ROOTMgr::Config: configuring
> 090122 16:14:29 001 xpd-I: ROOTMgr::Config: ROOT dist: '5.22/00
> 5.22/00 /afs/e18/sw/ROOT/root_v5.22.00.gcc412' validated
> 090122 16:14:29 001 xpd-I: ClientMgr::Config: configuring
> 090122 16:14:29 001 xpd-I: ClientMgr::Config: clients admin path set
> to: /tmp/.xproofd.1093/clients
> 090122 16:14:29 001 xpd-I: ClientMgr::Config: XRD seclib not
> specified; strong authentication disabled
> 090122 16:14:29 001 xpd-I: ClientMgr::Config: cron thread started
> 090122 16:14:29 001 xpd-I: ProofServMgr::Config: configuring
> 090122 16:14:29 001 xpd-I: ProofServMgr::Config: setting internal
> timeout to 10 secs
> 090122 16:14:29 001 xpd-I: ProofServMgr::Config: client sessions kept
> idle for 0 secs after disconnection
> 090122 16:14:29 001 xpd-I: ProofServMgr::Config: active sessions admin
> path set to: /tmp/.xproofd.1093/activesessions
> 090122 16:14:29 001 xpd-I: ProofServMgr::Config: terminated sessions
> admin path set to /tmp/.xproofd.1093/terminatedsessions
> 090122 16:14:29 001 xpd-I: ProofServMgr::PrepareSessionRecovering:
> session recover thread started
> 090122 16:14:30 29867 xpd-I: ProofServCron: next full sessions check
> in 30 secs
> 090122 16:14:30 001 xpd-I: ProofServMgr::Config: cron thread started
> 090122 16:14:30 001 xpd-I: Manager::Config: manager cron thread started
> 090122 16:14:30 001 xpd-I: Protocol::Configure: global manager created
> 090122 16:14:30 001 xpd-I: Protocol::Configure: xproofd protocol
> version 0.5 build 20081122-0000 successfully loaded
> ------ xrootd anon_at_nailgun.e18.physik.tu-muenchen.de:1094
> initialization completed.
> 090122 16:14:30 29867 xpd-I: tnagel.29659:30_at_localhost:
> ClientMgr::MapClient: user tnagel logged-in (privileged); type: Internal
> 090122 16:14:30 29867 xpd-I: tnagel.29657:36_at_localhost:
> ClientMgr::MapClient: user tnagel logged-in (privileged); type: Internal
> 090122 16:14:30 29867 xpd-I: tnagel.29655:37_at_localhost:
> ClientMgr::MapClient: user tnagel logged-in (privileged); type: Internal
> ==29867==
> ==29867== Thread 13:
> ==29867== Invalid read of size 4
> ==29867== at 0x499D890:
> XrdProofdProofServMgr::Recover(XpdClientSessions*) (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/lib/libXrdProofd.so.5.22)
> ==29867== by 0x499DFE3:
> XrdProofdProofServMgr::RecoverActiveSessions() (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/lib/libXrdProofd.so.5.22)
> ==29867== by 0x499E409: XrdProofdProofServRecover(void*) (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/lib/libXrdProofd.so.5.22)
> ==29867== by 0x808A2DC: XrdSysThread_Xeq (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/bin/xrootd)
> ==29867== by 0x405923F: start_thread (in
> /lib/tls/i686/cmov/libpthread-2.3.6.so)
> ==29867== by 0x424949D: clone (in /lib/tls/i686/cmov/libc-2.3.6.so)
> ==29867== Address 0x5A4C514 is 0 bytes after a block of size 36 alloc'd
> ==29867== at 0x401DB31: operator new(unsigned)
> (vg_replace_malloc.c:163)
> ==29867== by 0x499EB7C: XrdProofdProofServMgr::ResolveSession(char
> const*) (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/lib/libXrdProofd.so.5.22)
> ==29867== by 0x49A0B90:
> XrdProofdProofServMgr::PrepareSessionRecovering() (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/lib/libXrdProofd.so.5.22)
> ==29867== by 0x49A257D: XrdProofdProofServMgr::Config(bool) (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/lib/libXrdProofd.so.5.22)
> ==29867== by 0x498515A: XrdProofdManager::Config(bool) (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/lib/libXrdProofd.so.5.22)
> ==29867== by 0x49B58C4: XrdProofdProtocol::Configure(char*,
> XrdProtocol_Config*) (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/lib/libXrdProofd.so.5.22)
> ==29867== by 0x49B641B: XrdgetProtocol (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/lib/libXrdProofd.so.5.22)
> ==29867== by 0x8078D2C: XrdProtLoad::getProtocol(char const*, char
> const*, char*, XrdProtocol_Config*) (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/bin/xrootd)
> ==29867== by 0x8078DD8: XrdProtLoad::Load(char const*, char const*,
> char*, XrdProtocol_Config*) (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/bin/xrootd)
> ==29867== by 0x807DA0C: XrdConfig::Setup(char*) (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/bin/xrootd)
> ==29867== by 0x807E8EA: XrdConfig::Configure(int, char**) (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/bin/xrootd)
> ==29867== by 0x8076E24: main (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/bin/xrootd)
> ==29867==
> ==29867== Invalid free() / delete / delete[]
> ==29867== at 0x401CCBC: operator delete(void*)
> (vg_replace_malloc.c:244)
> ==29867== by 0x499D8A5:
> XrdProofdProofServMgr::Recover(XpdClientSessions*) (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/lib/libXrdProofd.so.5.22)
> ==29867== by 0x499DFE3:
> XrdProofdProofServMgr::RecoverActiveSessions() (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/lib/libXrdProofd.so.5.22)
> ==29867== by 0x499E409: XrdProofdProofServRecover(void*) (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/lib/libXrdProofd.so.5.22)
> ==29867== by 0x808A2DC: XrdSysThread_Xeq (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/bin/xrootd)
> ==29867== by 0x405923F: start_thread (in
> /lib/tls/i686/cmov/libpthread-2.3.6.so)
> ==29867== by 0x424949D: clone (in /lib/tls/i686/cmov/libc-2.3.6.so)
> ==29867== Address 0x5A4C50C is 28 bytes inside a block of size 36
> alloc'd
> ==29867== at 0x401DB31: operator new(unsigned)
> (vg_replace_malloc.c:163)
> ==29867== by 0x499EB7C: XrdProofdProofServMgr::ResolveSession(char
> const*) (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/lib/libXrdProofd.so.5.22)
> ==29867== by 0x49A0B90:
> XrdProofdProofServMgr::PrepareSessionRecovering() (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/lib/libXrdProofd.so.5.22)
> ==29867== by 0x49A257D: XrdProofdProofServMgr::Config(bool) (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/lib/libXrdProofd.so.5.22)
> ==29867== by 0x498515A: XrdProofdManager::Config(bool) (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/lib/libXrdProofd.so.5.22)
> ==29867== by 0x49B58C4: XrdProofdProtocol::Configure(char*,
> XrdProtocol_Config*) (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/lib/libXrdProofd.so.5.22)
> ==29867== by 0x49B641B: XrdgetProtocol (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/lib/libXrdProofd.so.5.22)
> ==29867== by 0x8078D2C: XrdProtLoad::getProtocol(char const*, char
> const*, char*, XrdProtocol_Config*) (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/bin/xrootd)
> ==29867== by 0x8078DD8: XrdProtLoad::Load(char const*, char const*,
> char*, XrdProtocol_Config*) (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/bin/xrootd)
> ==29867== by 0x807DA0C: XrdConfig::Setup(char*) (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/bin/xrootd)
> ==29867== by 0x807E8EA: XrdConfig::Configure(int, char**) (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/bin/xrootd)
> ==29867== by 0x8076E24: main (in
> /afs/e18.ph.tum.de/sw/ROOT/root_v5.22.00.gcc412/bin/xrootd)
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Accept: session pointer
> undefined or socket invalid: 0
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Recover: problems accepting
> callback:
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Accept: session pointer
> undefined or socket invalid: 0
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Recover: problems accepting
> callback:
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Accept: session pointer
> undefined or socket invalid: 0
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Recover: problems accepting
> callback:
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Accept: session pointer
> undefined or socket invalid: 0
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Recover: problems accepting
> callback:
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Accept: session pointer
> undefined or socket invalid: 0
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Recover: problems accepting
> callback:
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Accept: session pointer
> undefined or socket invalid: 0
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Recover: problems accepting
> callback:
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Accept: session pointer
> undefined or socket invalid: 0
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Recover: problems accepting
> callback:
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Accept: session pointer
> undefined or socket invalid: 0
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Recover: problems accepting
> callback:
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Accept: session pointer
> undefined or socket invalid: 0
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Recover: problems accepting
> callback:
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Accept: session pointer
> undefined or socket invalid: 0
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Recover: problems accepting
> callback:
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Accept: session pointer
> undefined or socket invalid: 0
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Recover: problems accepting
> callback:
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Accept: session pointer
> undefined or socket invalid: 0
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Recover: problems accepting
> callback:
> 090122 16:14:30 29867 xpd-E: ProofServMgr::Accept: session pointer
> undefined or socket invalid: 0

-- 
+--------------------------------------------------------------------------+
  Gerardo GANIS    PH Department, CERN
        address    CERN, CH 1211 Geneve 23  
                   room: 32-RC-017, tel / fax: +412276 76439 / 69133
         e-mail    gerardo.ganis_at_cern.ch
+--------------------------------------------------------------------------+